Vengala Vinay

Having 9+ Years of Experience in Software Development

Home » Disaster Recovery 101: Architecting Auto-Failovers for Elasticsearch and C++ Deployments on OVH

Disaster Recovery 101: Architecting Auto-Failovers for Elasticsearch and C++ Deployments on OVH

Elasticsearch Cluster Architecture for High Availability on OVH

Achieving true disaster recovery for Elasticsearch hinges on a robust, multi-region or multi-zone architecture. For deployments on OVH, this typically involves leveraging their dedicated servers or VPS instances across different geographical locations. The core principle is to maintain a quorum of master-eligible nodes and data nodes that can withstand the failure of an entire datacenter or availability zone. We’ll focus on a setup using two distinct OVH regions, each hosting a full Elasticsearch cluster, with cross-region replication or a federated approach for critical data.

A common pattern is to deploy a primary cluster in one region and a secondary, read-only or standby cluster in another. For automatic failover, we need a mechanism to detect primary cluster failure and promote the secondary. This is where external orchestration and health checking become paramount.

Configuring Elasticsearch for Resilience

Each Elasticsearch node requires careful configuration to support high availability. Key parameters include:

  • discovery.seed_hosts: Essential for nodes to find each other. This should list the IP addresses or hostnames of potential master-eligible nodes in both regions.
  • cluster.initial_master_nodes: Specifies the nodes that are eligible to become the initial master.
  • node.master, node.data, node.ingest: Define the role of each node. For DR, ensure both regions have master-eligible nodes.
  • xpack.security.enabled: Crucial for securing your cluster, especially when nodes communicate across regions.
  • cluster.routing.allocation.enable: Controls shard allocation. For failover, you might temporarily disable allocation to a failed region.

Consider the following elasticsearch.yml snippet for a master-eligible node in Region A:

cluster.name: "my-es-cluster"
node.name: "es-node-a1"
network.host: "0.0.0.0"
http.port: 9200
transport.port: 9300

discovery.seed_hosts:
  - "192.168.1.10:9300"  # Node 1 in Region A
  - "192.168.1.11:9300"  # Node 2 in Region A
  - "192.168.2.10:9300"  # Node 1 in Region B (DR)
  - "192.168.2.11:9300"  # Node 2 in Region B (DR)

cluster.initial_master_nodes:
  - "es-node-a1"
  - "es-node-a2"
  - "es-node-b1"
  - "es-node-b2"

node.master: true
node.data: true
node.ingest: true

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.http.ssl.enabled: true

# Adjust shard allocation settings as needed for your DR strategy
# cluster.routing.allocation.enable: all

The configuration for Region B nodes would be similar, with appropriate `node.name` and network interface settings.

Implementing Cross-Region Data Synchronization

For automatic failover, data must be consistent or near-consistent between regions. Elasticsearch’s built-in cross-cluster replication (CCR) is the preferred method for this. CCR allows you to replicate indices from a leader cluster to a follower cluster.

First, configure the remote cluster connection on the follower cluster (Region B). This requires setting up an API key or certificate-based authentication.

# On Region B's elasticsearch.yml
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.http.ssl.enabled: true

# Define the remote cluster (Region A)
remote.clusters:
  - "region_a_cluster"

# Configure connection details for Region A
# This can be done via API or by adding to elasticsearch.yml
# Example using API (requires appropriate privileges):
# PUT _cluster/settings
# {
#   "persistent": {
#     "cluster": {
#       "remote": {
#         "region_a_cluster": {
#           "seeds": ["192.168.1.10:9300", "192.168.1.11:9300"],
#           "skip_unavailable": false
#         }
#       }
#     }
#   }
# }

Then, on the follower cluster (Region B), create a follower index that replicates from the leader index in Region A:

# On Region B
PUT /my-replicated-index/_ccr/follow
{
  "remote_cluster": "region_a_cluster",
  "leader_index": "my-original-index"
}

Ensure your network configuration on OVH allows secure communication (e.g., VPN, private network, or TLS-encrypted public access) between the Elasticsearch nodes in different regions.

Automating Failover with External Orchestration

Elasticsearch’s built-in quorum and replication are foundational, but true automatic failover requires an external system to monitor the primary cluster’s health and initiate the switch. A common approach involves a dedicated monitoring service or a Kubernetes operator.

We can use a simple Python script running on a separate, highly available instance (e.g., a small VPS or a managed service) to perform health checks and trigger failover actions.

The script will periodically check the health of the primary Elasticsearch cluster (Region A). If it becomes unresponsive or enters a degraded state (e.g., insufficient master nodes), the script will:

  • Attempt to stop indexing to the primary cluster.
  • Promote the follower cluster (Region B) to become the new primary.
  • Update DNS records or load balancer configurations to point traffic to Region B.
  • Notify relevant teams.
import requests
import time
import json
import os

PRIMARY_ES_URL = "http://primary-es-endpoint:9200"
SECONDARY_ES_URL = "http://secondary-es-endpoint:9200"
HEALTH_CHECK_INTERVAL = 30  # seconds
FAILOVER_THRESHOLD = 3  # consecutive failures to trigger failover

primary_failures = 0
is_primary_active = True

def check_primary_health():
    try:
        response = requests.get(f"{PRIMARY_ES_URL}/_cluster/health", timeout=5)
        response.raise_for_status()
        health = response.json()
        # More sophisticated checks can be added here (e.g., number of nodes, status)
        return health.get("status") in ["green", "yellow"]
    except requests.exceptions.RequestException:
        return False

def promote_secondary():
    print("Attempting to promote secondary cluster...")
    try:
        # 1. Stop CCR on secondary (if it's still following)
        # This is a simplified example; actual stop might involve API calls to disable follower indices
        # For demonstration, we assume it's already configured to be promoted.

        # 2. Update DNS/Load Balancer (example using a hypothetical API)
        # update_dns("my-app.example.com", "secondary-es-endpoint-ip")

        # 3. Reconfigure primary cluster to point to secondary (if needed for writes)
        # This might involve re-enabling writes on follower indices or reconfiguring applications.
        # For simplicity, we assume applications will be reconfigured to point to the new primary.

        print("Secondary cluster promoted successfully. Traffic should be redirected.")
        return True
    except Exception as e:
        print(f"Error promoting secondary cluster: {e}")
        return False

def update_dns(domain, ip_address):
    # This is a placeholder. In a real-world scenario, you'd integrate with your DNS provider's API
    # (e.g., OVH's API, Cloudflare API, AWS Route 53 API).
    print(f"Updating DNS for {domain} to point to {ip_address} (simulated).")
    # Example: requests.put(f"https://api.dns-provider.com/records/{domain}", json={"ip": ip_address})
    pass

def main():
    global primary_failures, is_primary_active

    while True:
        if check_primary_health():
            primary_failures = 0
            if not is_primary_active:
                print("Primary cluster is back online. Failback logic can be implemented here.")
                is_primary_active = True
        else:
            primary_failures += 1
            print(f"Primary cluster health check failed. Consecutive failures: {primary_failures}")

            if primary_failures >= FAILOVER_THRESHOLD and is_primary_active:
                print("Primary cluster is down. Initiating failover...")
                if promote_secondary():
                    is_primary_active = False
                    # Once failover is complete, the monitoring script might stop or reconfigure itself
                    # to monitor the new primary. For this example, we'll keep it running to monitor the new primary.
                    # In a real scenario, you might want to re-point this script too.
                else:
                    print("Failover process failed. Manual intervention required.")
                    # Potentially alert more aggressively or enter a safe mode.

        time.sleep(HEALTH_CHECK_INTERVAL)

if __name__ == "__main__":
    main()

This script needs to be deployed on a highly available monitoring instance. For OVH, consider using a small VPS with a managed OS and ensuring its own network connectivity is robust. The `update_dns` function is a critical placeholder; you’ll need to integrate with your DNS provider’s API (e.g., OVH’s API for managing DNS zones) to automate the IP address change.

C++ Application Deployment and Failover Strategy

For C++ applications, especially those interacting with Elasticsearch, the failover strategy must be application-aware. This involves:

  • Connection Pooling and Retries: Implement robust retry mechanisms with exponential backoff for Elasticsearch client connections.
  • Configuration Management: Dynamically update application configurations to point to the new primary Elasticsearch endpoint after a failover.
  • Service Discovery: Utilize a service discovery mechanism (e.g., Consul, etcd, or even DNS) that can be updated during failover.

Let’s consider a simplified C++ client using a hypothetical Elasticsearch client library. The key is to abstract the Elasticsearch endpoint and allow it to be reconfigured.

#include <iostream>
#include <string>
#include <vector>
#include <chrono>
#include <thread>
#include <stdexcept>

// Hypothetical Elasticsearch client library
// In reality, you'd use a library like elasticsearch-cpp or a custom HTTP client
class ElasticsearchClient {
public:
    ElasticsearchClient(const std::string& host, int port) : host_(host), port_(port) {
        connect();
    }

    void connect() {
        std::cout << "Attempting to connect to Elasticsearch at " << host_ << ":" << port_ << std::endl;
        // Simulate connection attempt
        std::this_thread::sleep_for(std::chrono::seconds(1));
        if (host_.find("unreachable") != std::string::npos) {
            throw std::runtime_error("Connection failed: Host is unreachable.");
        }
        std::cout << "Successfully connected to Elasticsearch." << std::endl;
        is_connected_ = true;
    }

    void disconnect() {
        std::cout << "Disconnecting from Elasticsearch." << std::endl;
        is_connected_ = false;
    }

    bool is_connected() const {
        return is_connected_;
    }

    void index_document(const std::string& index, const std::string& id, const std::string& document) {
        if (!is_connected_) {
            throw std::runtime_error("Not connected to Elasticsearch.");
        }
        std::cout << "Indexing document " << id << " to index " << index << std::endl;
        // Simulate indexing
        std::this_thread::sleep_for(std::chrono::milliseconds(500));
    }

    void update_endpoint(const std::string& host, int port) {
        if (is_connected_) {
            disconnect();
        }
        host_ = host;
        port_ = port;
        try {
            connect();
        } catch (const std::runtime_error& e) {
            std::cerr << "Failed to reconnect after endpoint update: " << e.what() << std::endl;
            is_connected_ = false; // Ensure state is correct
        }
    }

private:
    std::string host_;
    int port_;
    bool is_connected_ = false;
};

// Main application logic
int main() {
    std::string current_es_host = "primary-es-host.example.com"; // This would be dynamically updated
    int current_es_port = 9200;
    ElasticsearchClient es_client(current_es_host, current_es_port);

    int retry_count = 0;
    const int MAX_RETRIES = 5;
    const std::chrono::seconds RETRY_DELAY(5);

    while (true) {
        try {
            if (!es_client.is_connected()) {
                std::cerr << "Elasticsearch client is not connected. Attempting to reconnect..." << std::endl;
                es_client.connect();
                retry_count = 0; // Reset retry count on successful connection
                continue;
            }

            // Simulate application work that requires indexing
            es_client.index_document("my-app-logs", "log-123", "{ \"message\": \"Application event\" }");
            retry_count = 0; // Reset retry count on successful operation

        } catch (const std::runtime_error& e) {
            std::cerr << "Operation failed: " << e.what() << std::endl;
            retry_count++;

            if (retry_count >= MAX_RETRIES) {
                std::cerr << "Max retries reached. Initiating failover procedure..." << std::endl;
                // In a real system, this would trigger a request to a service discovery or configuration manager
                // to get the new primary endpoint.
                std::string new_es_host = "secondary-es-host.example.com"; // This would come from service discovery
                int new_es_port = 9200;

                if (new_es_host != current_es_host) {
                    std::cout << "Failover detected. Updating Elasticsearch endpoint to " << new_es_host << std::endl;
                    es_client.update_endpoint(new_es_host, new_es_port);
                    current_es_host = new_es_host;
                    current_es_port = new_es_port;
                    retry_count = 0; // Reset after successful endpoint update
                } else {
                    std::cerr << "Failover attempted, but endpoint did not change. Manual intervention may be needed." << std::endl;
                    // Consider a more aggressive alert or shutdown if failover is critical
                }
            } else {
                std::cout << "Retrying in " << RETRY_DELAY.count() << " seconds..." << std::endl;
                std::this_thread::sleep_for(RETRY_DELAY);
            }
        }
        // Simulate some work between indexing operations
        std::this_thread::sleep_for(std::chrono::seconds(5));
    }

    return 0;
}

The C++ application needs a mechanism to be notified of the endpoint change. This could be achieved by:

  • Polling a Configuration Service: The application periodically queries a central configuration store (e.g., etcd, Consul) for the current Elasticsearch endpoint.
  • Receiving Updates via a Message Queue: A failover orchestrator publishes an event to a message queue (e.g., Kafka, RabbitMQ) that the C++ application subscribes to.
  • DNS Updates: If using DNS for service discovery, the application might rely on DNS TTLs and re-resolving the hostname. However, this is less immediate and can be problematic with aggressive caching.

For OVH deployments, consider using their managed Kubernetes service (if applicable) or deploying a dedicated service discovery tool like Consul on highly available instances across your chosen regions. The C++ application’s build process should ideally inject or fetch the initial endpoint configuration, and a runtime mechanism must handle updates.

Orchestration and Monitoring on OVH

To manage this complex setup on OVH, a layered approach to orchestration and monitoring is essential:

  • OVH Infrastructure Management: Use OVH’s control panel or API to provision and manage dedicated servers or VPS instances in different regions. Ensure network connectivity (e.g., VPN tunnels, private network configurations) is established between these instances for secure Elasticsearch communication.
  • Container Orchestration (Optional but Recommended): If using Kubernetes, deploy Elasticsearch using the Elastic Cloud on Kubernetes (ECK) operator. ECK simplifies Elasticsearch cluster management, including multi-cluster setups and upgrades. Your C++ applications can also be deployed as containers, managed by Kubernetes, and configured via ConfigMaps or Secrets that are updated during failover.
  • Health Checking and Alerting: Implement comprehensive health checks for both Elasticsearch clusters and your C++ applications. Tools like Prometheus with Alertmanager are excellent choices. Configure Prometheus to scrape metrics from Elasticsearch (via its metrics endpoint) and your C++ applications. Alertmanager can then trigger notifications (e.g., PagerDuty, Slack, email) based on predefined alert rules.
  • Failover Triggering: The Python script (or a more sophisticated orchestrator) acts as the trigger. It should be deployed on a highly available setup itself, perhaps a small, resilient VPS or even a managed Kubernetes cluster with high availability configured.

For DNS-based failover, OVH provides DNS management tools. You can automate DNS record updates using their API. For example, to update an A record:

# Example using OVH API (requires authentication and specific API calls)
# This is a conceptual example; actual implementation requires using an OVH API client library or direct HTTP requests.

API_ENDPOINT="https://api.ovh.com/1.0/domain/zone/yourdomain.com/record"
RECORD_ID="your_record_id" # ID of the A record to update
NEW_IP="1.2.3.4" # IP of the secondary Elasticsearch cluster's load balancer

# Authenticate and make a PUT request to update the record
# curl -X PUT "$API_ENDPOINT/$RECORD_ID" -H "X-Auth-Token: YOUR_TOKEN" -H "Content-Type: application/json" -d "{\"subDomain\": \"es\", \"fieldType\": \"A\", \"target\": \"$NEW_IP\", \"ttl\": 300}"
echo "Simulating OVH DNS update for es.yourdomain.com to $NEW_IP"

The key is to ensure that the failover process is idempotent and can be re-run safely if an intermediate step fails. Thorough testing of the failover and failback procedures in a staging environment is non-negotiable.