Vengala Vinay

Having 9+ Years of Experience in Software Development

Home » Disaster Recovery 101: Architecting Auto-Failovers for Elasticsearch and Laravel Deployments on AWS

Disaster Recovery 101: Architecting Auto-Failovers for Elasticsearch and Laravel Deployments on AWS

Elasticsearch Cluster Architecture for High Availability

Achieving robust disaster recovery for Elasticsearch hinges on a well-architected cluster that inherently supports high availability. This means moving beyond a single-node or simple master/data node setup to a distributed system with redundancy at every critical layer. For production deployments on AWS, we leverage multiple Availability Zones (AZs) to mitigate single datacenter failures.

A typical HA Elasticsearch cluster comprises dedicated master nodes, data nodes, and ingest/coordinating nodes. For disaster recovery, we ensure that data nodes are replicated across at least two, preferably three, AZs. This is managed through Elasticsearch’s shard allocation awareness settings.

Configuring Elasticsearch Shard Allocation Awareness

Shard allocation awareness is crucial for distributing shards and replicas across different physical locations (AZs in AWS). This prevents a single AZ failure from rendering your data inaccessible. We configure this in the elasticsearch.yml file on each node.

First, ensure your EC2 instances are tagged with an attribute that identifies their Availability Zone. AWS provides the `topology.kubernetes.io/zone` or a custom tag like `aws:availability-zone` which can be leveraged. Elasticsearch can then be configured to use this tag.

Example elasticsearch.yml Configuration

On each Elasticsearch node, modify /etc/elasticsearch/elasticsearch.yml:

cluster.routing.allocation.awareness.attributes: aws:availability-zone
cluster.routing.allocation.awareness.force.aws:availability-zone.values: us-east-1a,us-east-1b,us-east-1c

In this configuration:

  • cluster.routing.allocation.awareness.attributes: Specifies the node attribute that Elasticsearch should use for awareness. We’re using a custom tag `aws:availability-zone`.
  • cluster.routing.allocation.awareness.force.aws:availability-zone.values: Lists the specific values for the awareness attribute that nodes should be aware of. This ensures shards are distributed across these defined AZs.

With this setting, Elasticsearch will attempt to place shards and their replicas on nodes residing in different AZs. For a production setup with 3 AZs, you’d typically configure 3 master-eligible nodes (one in each AZ) and a sufficient number of data nodes distributed across these AZs to hold your data and its replicas.

Automated Elasticsearch Failover with AWS Services

True automated failover for Elasticsearch requires more than just replication. It involves detecting failures and orchestrating recovery. AWS services like Route 53, Elastic Load Balancing (ELB), and custom Lambda functions can be combined to achieve this.

Route 53 Health Checks and Failover Routing

We use Route 53 to manage DNS resolution for our Elasticsearch endpoint. By configuring health checks against our Elasticsearch cluster, we can automatically reroute traffic away from unhealthy endpoints.

A common strategy is to have a primary Elasticsearch cluster in one region and a secondary, read-only (or warm) cluster in a different region for DR. Route 53 can manage failover between these two endpoints.

Route 53 Health Check Configuration (Conceptual)

Create a Route 53 health check that probes a specific endpoint on your Elasticsearch cluster (e.g., a coordinating node or an ELB in front of it). This health check should be configured to fail if it doesn’t receive a successful response within a defined threshold.

Then, create a DNS record (e.g., es.yourdomain.com) with failover routing policy. The primary record points to your active cluster’s ELB/IP, and the secondary record points to your DR cluster’s ELB/IP. The health check is associated with the primary record.

Elasticsearch Cluster State and Node Monitoring

Directly monitoring individual Elasticsearch nodes for health is essential. Elasticsearch exposes its cluster health via the `_cluster/health` API. We can use this to determine if the cluster is in a red or yellow state, indicating data unavailability or unassigned shards.

A Lambda function can periodically poll the `_cluster/health` API. If the cluster state is unhealthy (e.g., `status` is not `green` for an extended period), it can trigger a failover process.

Lambda Function for Elasticsearch Health Monitoring (Python Example)

import json
import boto3
import requests
import os

ES_ENDPOINT = os.environ['ES_ENDPOINT'] # e.g., "search-my-cluster-xxxx.us-east-1.es.amazonaws.com"
ROUTE53_RECORD_SET_NAME = os.environ['ROUTE53_RECORD_SET_NAME'] # e.g., "es.yourdomain.com."
ROUTE53_ZONE_ID = os.environ['ROUTE53_ZONE_ID'] # e.g., "Z1ABCDEFGHIJKLMN"
HEALTH_CHECK_THRESHOLD = 3 # Number of consecutive failures before triggering failover

def lambda_handler(event, context):
    unhealthy_count = event.get('unhealthy_count', 0)

    try:
        response = requests.get(f"https://{ES_ENDPOINT}/_cluster/health", timeout=10)
        response.raise_for_status() # Raise an exception for bad status codes
        health_data = response.json()

        if health_data['status'] != 'green':
            print(f"Elasticsearch cluster is not green. Status: {health_data['status']}")
            unhealthy_count += 1
        else:
            unhealthy_count = 0 # Reset count if healthy

    except requests.exceptions.RequestException as e:
        print(f"Error checking Elasticsearch health: {e}")
        unhealthy_count += 1

    if unhealthy_count >= HEALTH_CHECK_THRESHOLD:
        print(f"Elasticsearch cluster unhealthy for {unhealthy_count} checks. Initiating failover.")
        trigger_route53_failover()
        return {'status': 'failover_initiated', 'unhealthy_count': 0} # Reset count after failover
    else:
        print(f"Elasticsearch health check passed. Unhealthy count: {unhealthy_count}")
        return {'status': 'healthy', 'unhealthy_count': unhealthy_count}

def trigger_route53_failover():
    # This is a simplified example. In a real-world scenario, you'd need to:
    # 1. Identify the current primary and secondary records.
    # 2. Update the primary record to point to the secondary endpoint.
    # 3. Potentially disable the health check for the old primary.
    # This requires careful management of Route 53 Change Sets.

    # Placeholder for actual Route 53 API call to switch records
    print("Simulating Route 53 failover record update...")
    # Example: Using boto3 to update a record set (requires IAM permissions)
    # r53 = boto3.client('route53')
    # change_batch = {
    #     'Comment': 'Failover to secondary Elasticsearch cluster',
    #     'Changes': [
    #         {
    #             'Action': 'UPSERT',
    #             'ResourceRecordSet': {
    #                 'Name': ROUTE53_RECORD_SET_NAME,
    #                 'Type': 'A', # or CNAME, depending on your setup
    #                 'TTL': 60,
    #                 'AliasTarget': { # If using Alias records
    #                     'HostedZoneId': 'SECONDARY_ELB_HOSTED_ZONE_ID',
    #                     'DNSName': 'secondary.es.yourdomain.com',
    #                     'EvaluateTargetHealth': False
    #                 }
    #             }
    #         }
    #     ]
    # }
    # r53.change_resource_record_sets(
    #     HostedZoneId=ROUTE53_ZONE_ID,
    #     ChangeBatch=change_batch
    # )
    pass

This Lambda function would be triggered periodically by CloudWatch Events (e.g., every minute). The unhealthy_count is passed in the event payload to maintain state across invocations, preventing immediate failover on a single transient network glitch.

Laravel Application Integration and Failover Strategy

Your Laravel application needs to be aware of the Elasticsearch endpoint. The primary mechanism for this is through environment variables and Laravel’s configuration system.

Environment Configuration for Elasticsearch Endpoint

In your Laravel application’s .env file, define the Elasticsearch host. This should ideally be the Route 53 DNS name that resolves to your active cluster.

ELASTICSEARCH_HOST=es.yourdomain.com
ELASTICSEARCH_PORT=9200
ELASTICSEARCH_USERNAME=elastic
ELASTICSEARCH_PASSWORD=your_password

Your Laravel service provider or configuration files will then use these variables. For example, using the popular Elasticsearch PHP client:

Example Laravel Service Provider Snippet

use Elasticsearch\ClientBuilder;
use Illuminate\Support\ServiceProvider;

class ElasticsearchServiceProvider extends ServiceProvider
{
    public function register()
    {
        $this->app->singleton(Client::class, function ($app) {
            $config = [
                'hosts' => [
                    config('services.elasticsearch.host') . ':' . config('services.elasticsearch.port'),
                ],
                'basicAuthentication' => [
                    config('services.elasticsearch.username'),
                    config('services.elasticsearch.password'),
                ],
                // Add other client options as needed
            ];

            // If using AWS Elasticsearch Service with IAM authentication,
            // you'd configure the AwsConnection plugin here.
            // Example:
            // $awsConfig = [
            //     'region' => env('AWS_REGION', 'us-east-1'),
            //     'credentials' => [
            //         'key'    => env('AWS_ACCESS_KEY_ID'),
            //         'secret' => env('AWS_SECRET_ACCESS_KEY'),
            //     ],
            // ];
            // $config['connectionSelector'] = new AwsConnection($awsConfig);


            return ClientBuilder::create()->setHosts($config['hosts'])->setBasicAuthentication($config['basicAuthentication'][0], $config['basicAuthentication'][1])->build();
        });
    }
}

When Route 53 performs a failover, it updates the DNS record for es.yourdomain.com to point to the DR cluster. Laravel’s application, by resolving this DNS name, will automatically start connecting to the new endpoint without application code changes.

Orchestrating Cross-Region Elasticsearch Failover

For true disaster recovery, a cross-region strategy is paramount. This involves setting up a secondary Elasticsearch cluster in a different AWS region. This secondary cluster can be:

  • Hot Standby: A fully functional cluster, potentially scaled down, that can take over immediately. Requires data replication.
  • Warm Standby: A cluster with recent data, but possibly with slower indexing or search capabilities, requiring more time to become fully operational.
  • Cold Standby: Primarily for backups, requiring significant time to provision and restore data.

For automated failover, a hot or warm standby is necessary. Data replication can be achieved using Elasticsearch’s cross-cluster replication (CCR) feature or by setting up a dedicated replication mechanism.

Cross-Cluster Replication (CCR) Setup

CCR allows you to replicate indices from a primary cluster to a secondary cluster. This is configured on the leader (primary) cluster.

Configuring CCR (Conceptual)

1. Configure Remote Cluster: On the follower (secondary) cluster, define the leader cluster as a remote cluster.

# On the follower cluster's elasticsearch.yml
cluster.remote.leader_cluster_alias:
  seeds: "leader-es-node1:9300,leader-es-node2:9300" # Or use the public endpoint if accessible
  # Add security settings if applicable (e.g., transport layer TLS)

2. Create Replication Policy: On the leader cluster, define a replication policy for the indices you want to replicate.

PUT _ccr/ાલ/my-replication-policy
{
  "leader_alias": "leader_cluster_alias",
  "index_patterns": ["my-app-logs-*"],
  "settings": {
    "index.number_of_replicas": 1,
    "index.refresh_interval": "5s"
  }
}

3. Start Replication: On the follower cluster, start the replication for specific indices.

POST my-app-logs-000001/_ccr/ાલ/start
{
  "remote_cluster": "leader_cluster_alias",
  "index_name": "my-app-logs-000001"
}

When a failover is triggered, Route 53 will point your application to the secondary region. The secondary Elasticsearch cluster, being a hot standby with replicated data, can then serve read requests immediately. For write operations, you might need to reconfigure the replication direction or promote the follower cluster to be the new leader.

Automated Failover Orchestration with AWS Lambda and Step Functions

A comprehensive automated failover process often involves more than just DNS changes. AWS Step Functions can orchestrate complex workflows, including:

  • Detecting Elasticsearch cluster failure (via CloudWatch alarms and Lambda).
  • Initiating Route 53 DNS failover.
  • If using a warm/cold standby, provisioning additional resources in the DR region.
  • Reconfiguring cross-cluster replication to point from the DR region back to the primary (once it’s restored).
  • Notifying operations teams.

Example Step Functions State Machine (Conceptual)

{
  "Comment": "Elasticsearch Failover Workflow",
  "StartAt": "CheckElasticsearchHealth",
  "States": {
    "CheckElasticsearchHealth": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:123456789012:function:ElasticsearchHealthChecker",
      "Next": "HandleHealthCheckResult"
    },
    "HandleHealthCheckResult": {
      "Type": "Choice",
      "Choices": [
        {
          "Variable": "$.status",
          "StringEquals": "failover_initiated",
          "Next": "InitiateFailover"
        }
      ],
      "Default": "Healthy"
    },
    "InitiateFailover": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:123456789012:function:Route53FailoverOrchestrator",
      "Next": "NotifyFailoverComplete"
    },
    "Healthy": {
      "Type": "Pass",
      "End": true
    },
    "NotifyFailoverComplete": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:123456789012:function:NotificationService",
      "Parameters": {
        "Message": "Elasticsearch failover initiated to DR region."
      },
      "End": true
    }
  }
}

The ElasticsearchHealthChecker Lambda would be the same as described earlier, but instead of directly calling Route 53, it would update the Step Functions state with the result. The Route53FailoverOrchestrator Lambda would then perform the actual Route 53 record updates and potentially other DR region activation steps.

Considerations for Write Operations During Failover

When failing over to a DR cluster, especially if it’s a warm standby or if replication was one-way, handling write operations requires careful planning:

  • Application-Level Retries: Ensure your Laravel application has robust retry mechanisms for Elasticsearch writes.
  • Re-establishing Writes: Once the primary cluster is restored, you’ll need to re-establish write capabilities. This might involve:
    • Promoting the DR cluster to be the primary.
    • Reconfiguring CCR to replicate from the new primary back to the old primary (now a standby).
    • If the primary cluster was only down temporarily, simply resuming writes to it after it’s healthy.
  • Data Consistency: Understand the potential for data loss during failover if replication is not synchronous. CCR offers synchronous replication for specific use cases, but it has performance implications.

Implementing automated failover for Elasticsearch and integrated applications like Laravel on AWS is a multi-faceted endeavor. It requires a deep understanding of Elasticsearch’s HA features, AWS networking and DNS services, and robust application-level resilience patterns. By combining these elements, you can build a system that automatically recovers from failures, minimizing downtime and ensuring business continuity.