Vengala Vinay

Having 12+ Years of Experience in Software Development

Home » Disaster Recovery 101: Architecting Auto-Failovers for Elasticsearch and Ruby Deployments on Google Cloud

Disaster Recovery 101: Architecting Auto-Failovers for Elasticsearch and Ruby Deployments on Google Cloud

Designing for Resilience: Elasticsearch Auto-Failover on Google Cloud

Achieving high availability for Elasticsearch clusters, especially when coupled with critical application tiers like Ruby on Rails, demands a robust disaster recovery strategy. This isn’t about manual intervention; it’s about architecting for automated failover. On Google Cloud Platform (GCP), this translates to leveraging managed services and intelligent orchestration.

Elasticsearch Cluster Architecture for HA

A production-ready Elasticsearch cluster for HA should comprise multiple nodes distributed across different availability zones within a GCP region. We’ll focus on a setup with dedicated master, data, and ingest nodes. For automated failover, the key is a quorum-based system for master election and resilient data storage.

Master Node Resilience

Elasticsearch’s master election process relies on a quorum. A minimum of three master-eligible nodes is the baseline for HA. If a master node fails, the remaining master-eligible nodes can elect a new master. To ensure this process is robust, we’ll deploy these nodes across distinct availability zones.

Data Node Redundancy and Sharding

Data nodes store the actual indices. For data durability and availability, we configure indices with sufficient replicas. A common strategy is to have at least one replica shard for every primary shard, ideally distributed across different availability zones. This ensures that if a node or an entire availability zone becomes unavailable, data is still accessible from its replicas.

Leveraging Google Cloud Services

While Elasticsearch itself provides replication, GCP offers foundational services that enhance our failover capabilities:

  • Google Kubernetes Engine (GKE): For orchestrating Elasticsearch pods and managing their lifecycle.
  • Persistent Disks (Regional PDs): For stateful Elasticsearch data, ensuring data availability across zones.
  • Cloud Load Balancing: To distribute traffic to healthy Elasticsearch nodes and application instances.
  • Cloud Monitoring & Cloud Logging: For observing cluster health and triggering automated actions.

Implementing Auto-Failover with GKE and StatefulSets

Deploying Elasticsearch on GKE using StatefulSets is the idiomatic approach for stateful applications. StatefulSets provide stable network identifiers, persistent storage, and ordered deployment/scaling. This is crucial for Elasticsearch’s internal discovery and data management.

GKE Cluster Setup

Ensure your GKE cluster is configured with nodes spread across multiple zones within a single region. This is a prerequisite for cross-zone resilience.

Elasticsearch Helm Chart Configuration

We’ll use the official Elasticsearch Helm chart, customizing it for HA and GCP integration. Key parameters to focus on include:

Master Nodes Configuration

Ensure `node.roles` includes `master` and `ingest` (or `data_warm`/`data_cold` depending on your workload). Set `cluster.initial_master_nodes` to include the names of your master-eligible nodes. For HA, a minimum of 3 master nodes is recommended.

Data Nodes Configuration

Configure `node.roles` to `data`. Crucially, use `volumeClaimTemplates` to provision Persistent Disks. For cross-zone resilience, consider using **Regional Persistent Disks** if your Elasticsearch version and GKE version support it. If not, you’ll need a strategy for data replication that accounts for zone failures.

Persistent Storage with Regional Persistent Disks

Regional Persistent Disks automatically replicate data between two zones within a region. If one zone fails, the disk can be attached to a VM in the other zone. This is a powerful feature for stateful workloads like Elasticsearch.

Example `values.yaml` Snippet for Elasticsearch Helm Chart

clusterName: "my-es-cluster"
nodeGroup: "master" # Example for master nodes
replicas: 3 # Number of master nodes

# For master nodes, ensure they are master-eligible
esJavaOpts: "-Xms1g -Xmx1g" # Adjust JVM heap size as needed

# Example for data nodes
# nodeGroup: "data"
# replicas: 3 # Number of data nodes

# Persistent storage configuration
volumeClaimTemplate:
  accessModes: ["ReadWriteOnce"]
  storageClassName: "your-regional-pd-storage-class" # e.g., "premium-rwo" for Regional SSD
  resources:
    requests:
      storage: 100Gi # Adjust size as needed

# Network configuration for internal discovery
service:
  type: "ClusterIP"
  port: 9200

# Pod anti-affinity to ensure nodes are spread across availability zones
antiAffinity: "soft" # Or "hard" for stricter placement

# Elasticsearch configuration overrides
extraEnvs:
  - name: "discovery.seed_hosts"
    value: "my-es-cluster-master.default.svc.cluster.local" # Adjust service name and namespace
  - name: "cluster.routing.allocation.disk.threshold_enabled"
    value: "true"
  - name: "cluster.routing.allocation.disk.watermark.low"
    value: "85%"
  - name: "cluster.routing.allocation.disk.watermark.high"
    value: "90%"
  - name: "cluster.routing.allocation.disk.watermark.flood_stage"
    value: "95%"
  - name: "cluster.routing.allocation.enable"
    value: "all" # Ensure shards are allocated across nodes and zones

# Master nodes specific configuration
master:
  replicas: 3
  persistence:
    enabled: true
    storageClassName: "your-regional-pd-storage-class"
    size: 50Gi
  nodeAttributes:
    node.roles: "master,ingest" # Or other roles as needed

# Data nodes specific configuration (if separate)
data:
  replicas: 3
  persistence:
    enabled: true
    storageClassName: "your-regional-pd-storage-class"
    size: 200Gi
  nodeAttributes:
    node.roles: "data"

Deploying Elasticsearch on GKE

Once your `values.yaml` is configured, deploy the chart:

helm repo add elastic https://helm.elastic.co
helm repo update
helm install my-es elastic/elasticsearch -f values.yaml --namespace elasticsearch --create-namespace

Ruby Application Deployment and Failover

Your Ruby application (e.g., Rails) needs to be aware of the Elasticsearch cluster’s health and be able to switch to a healthy endpoint if the primary one becomes unavailable. This involves:

Service Discovery and Load Balancing

Expose your Elasticsearch cluster via a Kubernetes Service. This service will provide a stable DNS name for your application to connect to. For true auto-failover, you’ll want to use a load balancer that can health-check Elasticsearch nodes.

Kubernetes Service for Elasticsearch

apiVersion: v1
kind: Service
metadata:
  name: elasticsearch-service
  namespace: elasticsearch # Match your Elasticsearch deployment namespace
spec:
  selector:
    app: elasticsearch # This should match the labels of your Elasticsearch pods
  ports:
    - protocol: TCP
      port: 9200
      targetPort: 9200
  type: ClusterIP # Or LoadBalancer if exposing externally

Application-Level Resilience

Your Ruby application needs to implement retry logic and potentially circuit breaker patterns when interacting with Elasticsearch. The `elasticsearch-ruby` client offers some built-in capabilities, but you might need to extend them.

Configuring the Elasticsearch Ruby Client

When initializing the client, provide multiple hosts. The client will attempt to connect to them in order and will automatically retry on failure. For GKE, you can point to the Kubernetes Service name.

require 'elasticsearch'

# Assuming your Elasticsearch cluster is exposed via a Kubernetes Service named 'elasticsearch-service'
# in the 'elasticsearch' namespace.
# The client will automatically discover and connect to healthy nodes behind this service.
# For more advanced failover, you might need a dedicated load balancer or a custom discovery mechanism.

client = Elasticsearch::Client.new(
  hosts: ['http://elasticsearch-service.elasticsearch.svc.cluster.local:9200'],
  retry_on_failure: 5, # Number of retries
  transport_options: {
    request: { timeout: 200 } # Request timeout in seconds
  }
)

# Example of a simple health check within your application
begin
  if client.ping
    puts "Connected to Elasticsearch!"
  else
    puts "Could not connect to Elasticsearch."
    # Implement fallback logic here, e.g., serve stale data, show an error page
  end
rescue Elasticsearch::Transport::Transport::Errors::ServiceUnavailable => e
  puts "Elasticsearch service unavailable: #{e.message}"
  # Implement fallback logic
rescue StandardError => e
  puts "An unexpected error occurred: #{e.message}"
  # Implement fallback logic
end

GKE Ingress and Load Balancing for Application Tier

Deploy your Ruby application on GKE using Deployments. Use a GKE Ingress controller (like GCE Ingress or Nginx Ingress) to expose your application. Configure health checks on the Ingress to ensure traffic is only sent to healthy application pods. This is crucial for the application tier’s availability.

Monitoring and Automated Recovery

Automated failover is only as good as the monitoring that triggers it. GCP’s Cloud Monitoring and Logging are essential here.

Elasticsearch Cluster Health Monitoring

Use the Elasticsearch APIs to monitor cluster health, node status, and disk usage. Integrate these metrics with Cloud Monitoring. Key metrics include:

  • Cluster status (green, yellow, red)
  • Number of nodes
  • Shard allocation status
  • Disk usage per node

GKE Pod and Node Health

GKE automatically restarts unhealthy pods. For node failures, GKE can be configured to automatically provision new nodes to replace failed ones, especially if using cluster autoscaling.

Custom Alerting and Remediation

Set up Cloud Monitoring alerts for critical Elasticsearch metrics (e.g., cluster status red, high disk usage, nodes becoming unresponsive). These alerts can trigger:

  • Notifications to your operations team.
  • Cloud Functions or Cloud Run jobs to perform automated remediation steps (e.g., restarting specific pods, rebalancing shards if manual intervention is required).

Example Cloud Monitoring Alert Configuration (Conceptual)

You would configure a metric-based alert in Cloud Monitoring. For instance, an alert for `elasticsearch.cluster.status` with a value of `red` for more than 5 minutes.

Application Health Checks

Ensure your Ruby application pods expose a `/health` endpoint that checks its connection to Elasticsearch and its own internal state. GKE Ingress and Kubernetes Liveness/Readiness probes will use this to manage traffic to healthy application instances.

Ruby Application Health Check Endpoint (Rails Example)

# config/routes.rb
get '/health', to: 'health#show'

# app/controllers/health_controller.rb
class HealthController < ApplicationController
  def show
    # Check Elasticsearch connection
    begin
      # Assuming 'client' is your configured Elasticsearch::Client instance
      if client.ping
        render json: { status: 'ok', elasticsearch: 'connected' }, status: :ok
      else
        render json: { status: 'error', elasticsearch: 'disconnected' }, status: :service_unavailable
      end
    rescue Elasticsearch::Transport::Transport::Errors::ServiceUnavailable
      render json: { status: 'error', elasticsearch: 'unavailable' }, status: :service_unavailable
    rescue StandardError
      render json: { status: 'error', elasticsearch: 'error' }, status: :internal_server_error
    end
  end

  private

  # Lazy initialize Elasticsearch client
  def client
    @client ||= Elasticsearch::Client.new(
      hosts: ['http://elasticsearch-service.elasticsearch.svc.cluster.local:9200'],
      retry_on_failure: 0, # Let Kubernetes/Ingress handle retries for now
      transport_options: {
        request: { timeout: 10 } # Shorter timeout for health check
      }
    )
  end
end

Advanced Considerations: Multi-Region Failover

For true disaster recovery across regions, the architecture becomes significantly more complex. It typically involves:

  • Cross-Region Replication (CRR) for Elasticsearch: Using tools like Logstash or custom solutions to replicate data to an Elasticsearch cluster in a different region.
  • Global Load Balancing: Services like Cloud Load Balancing with global forwarding rules and backend services that can direct traffic to the closest or healthiest region.
  • Application-Level Routing: The application must be able to discover and connect to the active Elasticsearch cluster in the failover region.
  • Data Synchronization Challenges: Ensuring data consistency between regions during normal operation and during failover is a major hurdle.

For most use cases, a well-architected single-region, multi-zone deployment with automated failover provides a very high level of availability and is a more achievable goal for robust disaster recovery.

Recent Posts

Top Categories

Our Products

Our Services