Vengala Vinay

Having 9+ Years of Experience in Software Development

Home » Disaster Recovery 101: Architecting Auto-Failovers for Elasticsearch and Magento 2 Deployments on Linode

Disaster Recovery 101: Architecting Auto-Failovers for Elasticsearch and Magento 2 Deployments on Linode

Elasticsearch Cluster Architecture for High Availability

Achieving true disaster recovery for Elasticsearch hinges on a robust, multi-region architecture. For a Magento 2 deployment, this means ensuring search functionality remains available even if an entire Linode region becomes inaccessible. We’ll focus on a primary/secondary active-passive setup, leveraging Elasticsearch’s built-in replication and shard management capabilities, augmented by external monitoring and failover orchestration.

A typical Elasticsearch cluster for Magento 2 will consist of multiple nodes. For high availability, we need at least three master-eligible nodes and a sufficient number of data nodes to handle the load and replication factor. A common pattern is to deploy these across different availability zones within a region for resilience against single datacenter failures. For cross-region DR, we’ll replicate data to a separate cluster in a different Linode region.

Cross-Cluster Replication (CCR) Configuration

Elasticsearch’s Cross-Cluster Replication (CCR) is the cornerstone of our data synchronization strategy. This feature allows us to replicate indices from a primary cluster to a secondary cluster. For Magento 2, this typically involves replicating the product catalog, layered navigation, and search-related indices.

First, configure the remote cluster connection on the secondary cluster. This involves defining the primary cluster as a remote connection. Ensure network connectivity between the Linode regions, potentially using Linode’s private networking or VPNs if direct public access is not desired or feasible.

Remote Cluster Configuration (on Secondary Cluster)

Edit the elasticsearch.yml file on each node of your secondary Elasticsearch cluster. Add the following configuration:

cluster:
  remote:
    primary_cluster:
      seeds:
        - primary-es-node-1.example.com:9300
        - primary-es-node-2.example.com:9300
        - primary-es-node-3.example.com:9300
      skip_unavailable: false

Replace primary-es-node-X.example.com:9300 with the actual hostnames and transport ports of your primary Elasticsearch cluster nodes. Restart the Elasticsearch service on the secondary cluster nodes for these changes to take effect.

Configuring Replication Policies

Once the remote cluster is configured, you can define replication policies. This is typically done via the Elasticsearch API. We’ll use the _ccr/follow API to establish replication from specific indices in the primary cluster to the secondary cluster. It’s crucial to replicate only the necessary indices to minimize overhead and potential conflicts.

Example: Replicating Magento’s product catalog index.

curl -X POST "https://secondary-es-node-1.example.com:9200/_ccr/follow/magento_products" -H 'Content-Type: application/json' -d'
{
  "remote_cluster": "primary_cluster",
  "leader_index": "magento_products",
  "follow_index": "magento_products_replica"
}'

This command initiates replication for the magento_products index from the primary_cluster to a new index named magento_products_replica on the secondary cluster. You would repeat this for all critical Magento search indices. The follow_index can be the same name as the leader index if you intend to switch over directly, or a distinct name for a staging/DR environment.

Automated Failover Orchestration for Elasticsearch

Manual failover is not an option for true disaster recovery. We need an automated system to detect primary cluster failure and switch traffic to the secondary cluster. This involves several components:

  • Health Monitoring: Regularly check the health of the primary Elasticsearch cluster.
  • Failover Trigger: A mechanism to initiate the failover process when the primary is deemed unhealthy.
  • Traffic Redirection: Update application configurations (e.g., Magento’s app/etc/env.php) or DNS records to point to the secondary cluster.
  • Failback Mechanism: A controlled process to return operations to the primary cluster once it’s restored.

Health Monitoring Script (Python Example)

A simple Python script can periodically check the health of the primary Elasticsearch cluster. This script would run on a dedicated monitoring server or a control node.

import requests
import time
import json
import os

PRIMARY_ES_URL = "https://primary-es-node-1.example.com:9200"
HEALTH_CHECK_INTERVAL = 60  # seconds
FAILOVER_TRIGGER_THRESHOLD = 3  # consecutive failures

def check_es_health(url):
    try:
        response = requests.get(f"{url}/_cluster/health", auth=('elastic', 'changeme'), verify=False) # Use proper cert verification in production
        response.raise_for_status()
        health_data = response.json()
        return health_data.get("status") in ["green", "yellow"]
    except requests.exceptions.RequestException as e:
        print(f"Error checking Elasticsearch health: {e}")
        return False

def trigger_failover():
    print("Primary Elasticsearch cluster is unhealthy. Initiating failover...")
    # Implement failover logic here:
    # 1. Update Magento env.php or DNS
    # 2. Notify operations team
    # 3. Potentially pause CCR on secondary if it's now primary
    pass

if __name__ == "__main__":
    consecutive_failures = 0
    while True:
        if check_es_health(PRIMARY_ES_URL):
            print("Primary Elasticsearch cluster is healthy.")
            consecutive_failures = 0
        else:
            consecutive_failures += 1
            print(f"Primary Elasticsearch cluster unhealthy. Failure count: {consecutive_failures}/{FAILOVER_TRIGGER_THRESHOLD}")
            if consecutive_failures >= FAILOVER_TRIGGER_THRESHOLD:
                trigger_failover()
                # In a real scenario, you might want to break or enter a different state
                # after triggering failover to avoid repeated triggers.
                break
        time.sleep(HEALTH_CHECK_INTERVAL)

Note: In a production environment, replace verify=False with actual SSL certificate verification. Also, manage Elasticsearch credentials securely (e.g., using environment variables or a secrets manager). The trigger_failover function needs to be expanded to handle the actual redirection logic.

Magento 2 Application Configuration Update

The most direct way to redirect Magento 2 traffic to the secondary Elasticsearch cluster is by updating its configuration file, app/etc/env.php. This requires a mechanism to programmatically modify this file and then clear the Magento cache.

A shell script can be used to perform this update. This script would be invoked by the failover orchestration system.

#!/bin/bash

# Configuration for the secondary Elasticsearch cluster
SECONDARY_ES_HOST="secondary-es-node-1.example.com"
SECONDARY_ES_PORT="9200"
MAGENTO_ROOT="/var/www/html/magento2" # Adjust path as needed

# Backup the current env.php
cp "$MAGENTO_ROOT/app/etc/env.php" "$MAGENTO_ROOT/app/etc/env.php.bak_$(date +%Y%m%d_%H%M%S)"

# Update env.php with secondary Elasticsearch details
# This uses 'sed' to find and replace the Elasticsearch connection details.
# Be very careful with the exact patterns to avoid unintended replacements.
sed -i "s/'host' => '[^']*'/'host' => '$SECONDARY_ES_HOST'/" "$MAGENTO_ROOT/app/etc/env.php"
sed -i "s/'port' => '[^']*'/'port' => '$SECONDARY_ES_PORT'/" "$MAGENTO_ROOT/app/etc/env.php"

# Ensure correct Elasticsearch scheme (http/https) and authentication are set if needed.
# Example for HTTPS:
# sed -i "s/'scheme' => '[^']*'/ 'scheme' => 'https'/" "$MAGENTO_ROOT/app/etc/env.php"
# sed -i "s/'username' => '[^']*'/ 'username' => 'elastic'/" "$MAGENTO_ROOT/app/app/etc/env.php"
# sed -i "s/'password' => '[^']*'/ 'password' => 'changeme'/" "$MAGENTO_ROOT/app/etc/env.php"


# Clear Magento cache
cd "$MAGENTO_ROOT"
php bin/magento setup:di:compile
php bin/magento cache:clean
php bin/magento cache:flush

echo "Magento 2 env.php updated to use secondary Elasticsearch cluster."
echo "Magento cache cleared."

Important Considerations:

  • The sed commands are highly dependent on the exact structure of your env.php. Test thoroughly.
  • Ensure the user running this script has write permissions to app/etc/env.php and can execute PHP CLI commands.
  • Consider using a configuration management tool (Ansible, Chef, Puppet) for more robust and repeatable deployments.
  • If using DNS for redirection, ensure TTL values are set appropriately to minimize propagation delays.

Magento 2 Application-Level Failover

While Elasticsearch CCR and external orchestration handle the data and infrastructure failover, the Magento 2 application itself needs to be aware of and resilient to the Elasticsearch cluster status. This involves configuring Magento’s search engine adapter.

Configuring Elasticsearch Engine in Magento 2

Magento 2 uses the Elasticsearch search engine adapter. The configuration for this adapter is primarily defined in app/etc/env.php. When setting up the secondary cluster, you would configure it here. During a failover, the script mentioned previously modifies these settings.

// Example snippet from app/etc/env.php
'indexer' => [
    'design' => [
        'design_expression_engine' => 'Magento\\Framework\\DB\\Adapter\\Pdo\\Mysql',
        'design_db_connection' => 'default_setup'
    ],
    'update_schedule' => [
        'cron_run_schedule' => '*/5 * * * *',
        'history_limit' => 100
    ]
],
'search' => [
    'engine' => 'elasticsearch7', // or 'elasticsearch6'
    'elasticsearch7' => [
        'index_prefix' => 'magento2',
        'hosts' => [
            [
                'host' => 'primary-es-node-1.example.com', // This will be updated on failover
                'port' => '9200',
                'scheme' => 'https', // or 'http'
                'username' => 'elastic',
                'password' => 'changeme',
                'timeout' => 15
            ]
        ],
        'options' => [
            'timeout' => 15,
            'verify_ssl' => true, // Set to false if using self-signed certs (not recommended)
            'ssl_cert_path' => '/path/to/your/cert.pem', // If needed
            'ssl_key_path' => '/path/to/your/key.pem' // If needed
        ]
    ]
],

The failover script directly modifies the hosts array and potentially other connection parameters within the 'elasticsearch7' (or 'elasticsearch6') section of env.php. After the update, Magento’s dependency injection and cache need to be refreshed.

Handling Search Index Rebuilds Post-Failover

When a failover occurs, the secondary Elasticsearch cluster might not have the absolute latest data if there was a brief period of network partition or if replication lag existed. Magento’s search indices need to be rebuilt or reindexed to ensure consistency.

The failover orchestration should ideally trigger a reindex after the application has been successfully redirected to the secondary cluster. This can be done via Magento’s CLI commands.

# After env.php update and cache flush, execute reindex
cd /var/www/html/magento2 # Adjust path as needed
php bin/magento indexer:reindex catalogsearch_fulltext
php bin/magento indexer:reindex catalogsearch_document
# Reindex other relevant Magento search indices if applicable

This process can be time-consuming for large catalogs. Consider running it during off-peak hours or implementing a phased reindexing strategy if possible. The catalogsearch_fulltext index is the primary one for product searches.

Disaster Recovery for Other Critical Components

Elasticsearch is only one piece of the puzzle. A comprehensive DR strategy for Magento 2 on Linode must also address other critical components:

Database (MySQL/MariaDB) Replication

Magento’s primary database is critical. For DR, implement robust database replication. Linode offers managed MySQL/MariaDB databases which often have built-in replication features. For self-managed instances, consider:

  • Master-Replica Replication: Set up a replica database in a separate Linode region.
  • Asynchronous vs. Synchronous Replication: Asynchronous is common for cross-region DR to minimize latency impact, but carries a small risk of data loss. Synchronous replication offers zero data loss but significantly increases write latency.
  • Automated Failover: Tools like Orchestrator, ProxySQL, or custom scripts can manage database failover. This involves promoting the replica to master and updating application connection strings.

The process for updating Magento’s env.php for database credentials is similar to the Elasticsearch update, requiring careful scripting and cache clearing.

Object Storage (e.g., Redis, Varnish)

Redis: For session storage and caching, Redis instances should also be replicated across regions. Consider using Redis Sentinel for high availability within a region and a separate replicated Redis instance in the DR region. Magento’s env.php will need to point to the DR Redis instance during failover.

Varnish: Varnish is a caching HTTP reverse proxy. For DR, you would typically deploy a separate Varnish instance in the DR region. Cache invalidation and synchronization between Varnish instances across regions can be complex. Often, the strategy is to let the DR Varnish instance rebuild its cache from the origin (Magento application) once traffic is directed to it.

CDN and DNS Management

Your Content Delivery Network (CDN) and DNS provider play a crucial role in directing traffic to the DR site. Ensure your DNS records have low TTLs (Time To Live) to facilitate quick propagation of changes. Many CDNs offer features for health checks and automatic failover between origin servers or regions.

Testing and Validation

A disaster recovery plan is only as good as its last successful test. Regularly schedule and execute DR drills. These tests should simulate various failure scenarios (e.g., entire region outage, specific service failure) and validate:

  • The automated failover process.
  • Data consistency across primary and secondary clusters/databases.
  • Application functionality on the DR site.
  • Recovery Time Objective (RTO) and Recovery Point Objective (RPO) adherence.

Document the entire failover and failback procedure meticulously. Automate as much of the testing process as possible to ensure consistency and reduce human error.