Disaster Recovery 101: Architecting Auto-Failovers for Elasticsearch and Laravel Deployments on DigitalOcean

Elasticsearch Cluster Health and Failover Strategies

Achieving high availability for Elasticsearch is paramount for any mission-critical application. A single-node Elasticsearch cluster is a single point of failure. For robust disaster recovery, we must architect for redundancy and automated failover. This involves understanding Elasticsearch’s internal mechanisms for node discovery, shard allocation, and cluster state management.

A typical production Elasticsearch deployment utilizes a master-eligible node, data nodes, and ingest nodes. For high availability, we need multiple master-eligible nodes to ensure quorum. A minimum of three master-eligible nodes is recommended to prevent split-brain scenarios. Data nodes should also be replicated across availability zones or even regions if the cloud provider supports it. DigitalOcean’s Droplets and Kubernetes offerings allow for deployment across different regions, which is crucial for true disaster recovery.

Configuring Elasticsearch for High Availability on DigitalOcean

Let’s outline the configuration for a highly available Elasticsearch cluster deployed on DigitalOcean Droplets. We’ll assume three Droplets designated as master-eligible nodes and at least two Droplets for data nodes. For simplicity, we’ll use a single region initially, but the principles extend to multi-region deployments.

On each Elasticsearch node, the elasticsearch.yml configuration file needs to be adjusted. Key parameters include:

• cluster.name: A unique name for your Elasticsearch cluster.
• node.master: Set to true for master-eligible nodes.
• node.data: Set to true for data nodes.
• discovery.seed_hosts: A list of IP addresses or hostnames of other master-eligible nodes.
• cluster.initial_master_nodes: A list of node names that are eligible to become the initial master. This is crucial for bootstrapping the cluster.
• network.host: Bind to a specific IP address or 0.0.0.0 to listen on all interfaces.
• http.port and transport.port: Standard Elasticsearch ports.

Here’s an example snippet for a master-eligible node:

cluster.name: my-production-cluster
node.name: es-master-01
node.master: true
node.data: false
network.host: 0.0.0.0
http.port: 9200
transport.port: 9300

discovery.seed_hosts:
  - "192.168.1.101:9300" # es-master-01
  - "192.168.1.102:9300" # es-master-02
  - "192.168.1.103:9300" # es-master-03

cluster.initial_master_nodes:
  - "es-master-01"
  - "es-master-02"
  - "es-master-03"

xpack.security.enabled: true # Recommended for production
xpack.security.http.ssl.enabled: true
xpack.security.transport.ssl.enabled: true

And for a data node:

cluster.name: my-production-cluster
node.name: es-data-01
node.master: false
node.data: true
network.host: 0.0.0.0
http.port: 9200
transport.port: 9300

discovery.seed_hosts:
  - "192.168.1.101:9300" # es-master-01
  - "192.168.1.102:9300" # es-master-02
  - "192.168.1.103:9300" # es-master-03

xpack.security.enabled: true # Recommended for production
xpack.security.http.ssl.enabled: true
xpack.security.transport.ssl.enabled: true

Ensure that the `discovery.seed_hosts` list on all nodes contains the IP addresses of all master-eligible nodes. The `cluster.initial_master_nodes` should be set to the names of the master-eligible nodes during the initial cluster formation. Once the cluster has formed, this setting can be removed or commented out for subsequent restarts.

Automating Elasticsearch Failover with Keepalived

While Elasticsearch itself handles node failures and shard rebalancing, applications need a stable endpoint to connect to. This is where a virtual IP (VIP) managed by a High Availability (HA) solution like Keepalived comes into play. Keepalived uses the Virtual Router Redundancy Protocol (VRRP) to manage a floating IP address across multiple servers.

We’ll set up two Droplets as Keepalived servers, each running an Elasticsearch client or proxy (like Nginx or HAProxy) that points to the Elasticsearch cluster. The VIP will float between these two servers.

On each Keepalived server (e.g., keepalived-01 and keepalived-02), install Keepalived:

sudo apt update
sudo apt install keepalived -y

Configure /etc/keepalived/keepalived.conf. The key is to define a VRRP instance with a unique `vrrp_instance` name, a `state` (MASTER for one, BACKUP for the other), a `virtual_router_id`, and the `virtual_ipaddress`.

On keepalived-01 (MASTER):

vrrp_script chk_es_proxy {
    script "/usr/local/bin/check_es_proxy.sh"
    interval 2
    weight 2
    fall 2
    rise 2
}

vrrp_instance VI_1 {
    state MASTER
    interface eth0 # Replace with your primary network interface
    virtual_router_id 51
    priority 101 # Higher priority for MASTER
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass your_secret_password
    }
    virtual_ipaddress {
        192.168.1.200/24 dev eth0 # Your Virtual IP
    }
    track_script {
        chk_es_proxy
    }
}

On keepalived-02 (BACKUP):

vrrp_script chk_es_proxy {
    script "/usr/local/bin/check_es_proxy.sh"
    interval 2
    weight 2
    fall 2
    rise 2
}

vrrp_instance VI_1 {
    state BACKUP
    interface eth0 # Replace with your primary network interface
    virtual_router_id 51
    priority 100 # Lower priority for BACKUP
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass your_secret_password
    }
    virtual_ipaddress {
        192.168.1.200/24 dev eth0 # Your Virtual IP
    }
    track_script {
        chk_es_proxy
    }
}

The `track_script` is crucial. It monitors the health of the Elasticsearch proxy running on the Keepalived server. If the proxy fails, Keepalived will transition the VIP to the other node. Create the health check script /usr/local/bin/check_es_proxy.sh:

#!/bin/bash

# Check if the Elasticsearch proxy (e.g., Nginx) is running and responding
# This script assumes Nginx is configured to proxy to Elasticsearch on localhost:8080
if curl -s -o /dev/null -w "%{http_code}" http://localhost:8080/ _cluster/health | grep -q "200"; then
    exit 0 # Healthy
else
    exit 1 # Unhealthy
fi

Make the script executable:

sudo chmod +x /usr/local/bin/check_es_proxy.sh

Restart Keepalived on both nodes:

sudo systemctl restart keepalived

Now, your applications will connect to 192.168.1.200, and Keepalived will ensure this IP is always reachable via one of the two proxy servers. If one proxy server fails or its Elasticsearch proxy becomes unresponsive, Keepalived will automatically move the VIP to the other server.

Laravel Application Integration and Failover

Your Laravel application needs to be configured to use the Virtual IP (VIP) for its Elasticsearch connection. This is typically done within your Laravel application’s configuration files, specifically config/elasticsearch.php if you’re using a popular package like the official Elasticsearch PHP client or a Laravel-specific wrapper.

Assuming you are using the official Elasticsearch PHP client and have a configuration like this:

// config/elasticsearch.php (example)
return [
    'default' => env('ELASTICSEARCH_HOSTS', 'http://127.0.0.1:9200'),

    'connections' => [
        'default' => [
            'hosts' => [
                env('ELASTICSEARCH_HOSTS', 'http://127.0.0.1:9200'),
            ],
            // ... other client options
        ],
    ],
];

You would set your environment variable ELASTICSEARCH_HOSTS to point to the VIP:

# .env file
ELASTICSEARCH_HOSTS=http://192.168.1.200:9200

When the primary Keepalived server fails, and the VIP moves to the backup server, your Laravel application, still pointing to the same VIP, will automatically connect to the new active proxy server. The proxy server will then forward requests to the healthy Elasticsearch cluster.

Advanced Considerations: Multi-Region DR and Application-Level Retries

For true disaster recovery, deploying Elasticsearch and Keepalived across multiple DigitalOcean regions is essential. This involves:

• Multi-Region Elasticsearch Cluster: Configure Elasticsearch nodes in different regions. This requires careful network configuration and potentially using a cross-region replication strategy if synchronous replication is not feasible due to latency.
• Regional VIPs: Each region would have its own Keepalived setup and VIP.
• Global Load Balancer: A global load balancer (e.g., DigitalOcean’s Load Balancers, or a third-party solution like Cloudflare or AWS Route 53 with health checks) would direct traffic to the active region’s VIP. The global load balancer must have health checks configured to detect regional failures and reroute traffic.
• Application-Level Retries: Even with automated failover, transient network issues or brief cluster unavailability can occur. Implement robust retry mechanisms within your Laravel application for Elasticsearch operations. Libraries like Guzzle (used by many HTTP clients) have built-in retry capabilities.

Example of adding retries to an Elasticsearch client in Laravel:

use Elasticsearch\ClientBuilder;
use GuzzleHttp\HandlerStack;
use GuzzleHttp\Middleware;
use GuzzleHttp\Psr7\Request;
use GuzzleHttp\Psr7\Response;
use Psr\Http\Message\RequestInterface;
use Psr\Http\Message\ResponseInterface;

// ... inside a service provider or configuration file

$stack = HandlerStack::create();

// Add retry middleware
$stack->push(Middleware::retry(function (
    RequestInterface $request,
    ResponseInterface $response = null,
    $exception = null
) {
    // Limit retries to 3
    if (func_num_args() > 2) {
        return false;
    }

    // Retry on connection exceptions or 5xx server errors
    if ($exception instanceof \GuzzleHttp\Exception\ConnectException ||
        ($response && $response->getStatusCode() >= 500)) {
        // Log the retry attempt
        \Log::warning('Retrying Elasticsearch request', [
            'request' => $request->getUri(),
            'response_status' => $response ? $response->getStatusCode() : 'N/A',
            'exception' => $exception ? $exception->getMessage() : 'N/A',
        ]);
        return true;
    }

    return false;
}, 3000)); // Retry every 3 seconds

$client = ClientBuilder::create()
    ->setHosts([env('ELASTICSEARCH_HOSTS')])
    ->setHandler($stack)
    // ... other client configurations
    ->build();

// Now use this $client instance in your application
// $client->index([...]);

Implementing these strategies ensures that your Elasticsearch data remains accessible and your Laravel application can gracefully handle infrastructure failures, providing a resilient and highly available service.