Vengala Vinay

Having 9+ Years of Experience in Software Development

Home » Disaster Recovery 101: Architecting Auto-Failovers for Redis and C++ Deployments on AWS

Disaster Recovery 101: Architecting Auto-Failovers for Redis and C++ Deployments on AWS

Designing for Resilience: Redis Sentinel and C++ Client Auto-Failover on AWS

Achieving true high availability for critical services necessitates robust disaster recovery strategies. For applications leveraging Redis as a high-performance data store and a C++ client for low-latency access, implementing automated failover is paramount. This post details the architectural patterns and specific configurations required to build an auto-failover system for Redis, managed by Redis Sentinel, and how C++ clients can seamlessly adapt to these changes within an AWS environment.

Redis Sentinel: The Heart of Auto-Failover

Redis Sentinel is Redis’s high-availability solution. It provides monitoring, notification, and automatic failover for Redis master instances. A Sentinel system consists of multiple Sentinel processes, each monitoring a Redis master and its replicas. If a master is deemed unavailable by a quorum of Sentinels, they initiate a failover process, promoting a replica to become the new master.

Sentinel Configuration for High Availability

Deploying Sentinel requires careful configuration. For production environments, running at least three Sentinel instances across different Availability Zones (AZs) is a best practice to ensure quorum even if one AZ becomes unavailable. Each Sentinel instance needs to know about the master it’s monitoring and the quorum required for a failover.

# sentinel.conf
port 26379
dir /tmp

# Monitor the Redis master instance named 'mymaster'
# It's running on 127.0.0.1:6379
# A quorum of 2 Sentinels is required to trigger a failover
# If the master is down for 15 seconds, it's considered down
# After 30 seconds, Sentinels will try to failover
sentinel monitor mymaster 127.0.0.1 6379 2
sentinel down-after-milliseconds mymaster 15000
sentinel failover-timeout mymaster 30000

# Specify the replicas to monitor. Sentinel will automatically discover them.
# sentinel parallel-syncs mymaster 1 # Number of replicas that can sync with the new master simultaneously

# Optional: Configure Sentinel to notify an external script on events
# sentinel notification-script mymaster /path/to/your/notification_script.sh

In an AWS context, these Sentinel instances would typically run on separate EC2 instances, ideally distributed across different AZs. The Redis master and its replicas would also be deployed with similar AZ distribution for maximum resilience. For example, a master in `us-east-1a`, a replica in `us-east-1b`, and another replica in `us-east-1c`, with Sentinels also spread across these AZs.

C++ Client Integration with Redis Sentinel

The C++ client needs to be aware of the Sentinel cluster to dynamically discover the current Redis master. Modern Redis client libraries for C++ often have built-in support for Sentinel. If not, a custom solution can be implemented by querying Sentinel for the master address and then connecting to it. The key is to handle connection errors gracefully and re-query Sentinel upon failure.

Using a Sentinel-Aware C++ Redis Client

Libraries like hiredis, when used with its Sentinel extensions, can abstract away the complexity of failover. The client connects to one or more Sentinel instances, asks for the current master of a given master name (e.g., ‘mymaster’), and then connects to that master. If a connection fails or a timeout occurs, the client can be instructed to re-query Sentinel for the new master address.

#include <iostream>
#include <hiredis/hiredis.h>
#include <hiredis/adapters/libevent.h> // Or your preferred adapter

int main() {
    // Sentinel connection details
    const char* sentinel_host = "your_sentinel_ip_1"; // Or multiple IPs for redundancy
    int sentinel_port = 26379;
    const char* master_name = "mymaster";

    // Initialize hiredis context for Sentinel
    redisContext* sentinel_ctx = redisConnect(sentinel_host, sentinel_port);
    if (sentinel_ctx && !sentinel_ctx->err) {
        std::cout << "Connected to Sentinel." << std::endl;
    } else {
        std::cerr << "Failed to connect to Sentinel: " << (sentinel_ctx ? sentinel_ctx->errstr : "unknown error") << std::endl;
        if (sentinel_ctx) redisFree(sentinel_ctx);
        return 1;
    }

    // Get the current master from Sentinel
    redisReply* reply = (redisReply*)redisCommand(sentinel_ctx, "SENTINEL master %s", master_name);
    redisReply* master_info = nullptr;
    redisContext* redis_ctx = nullptr;

    if (reply && reply->type == REDIS_REPLY_ARRAY && reply->elements >= 1) {
        // The first element is the master name, second is IP, third is port, etc.
        // We need the IP and port.
        if (reply->element[3]->type == REDIS_REPLY_STRING && reply->element[4]->type == REDIS_REPLY_STRING) {
            const char* master_ip = reply->element[3]->str;
            int master_port = std::stoi(reply->element[4]->str);

            std::cout << "Discovered master: " << master_ip << ":" << master_port << std::endl;

            // Connect to the actual Redis master
            redis_ctx = redisConnect(master_ip, master_port);
            if (redis_ctx && !redis_ctx->err) {
                std::cout << "Connected to Redis master." << std::endl;
                // Now you can perform Redis operations
                redisReply* pong = (redisReply*)redisCommand(redis_ctx, "PING");
                if (pong) {
                    std::cout << "PING response: " << pong->str << std::endl;
                    freeReplyObject(pong);
                }
            } else {
                std::cerr << "Failed to connect to Redis master: " << (redis_ctx ? redis_ctx->errstr : "unknown error") << std::endl;
            }
        }
        freeReplyObject(reply);
    } else {
        std::cerr << "Failed to get master info from Sentinel." << std::endl;
        if (reply) freeReplyObject(reply);
    }

    // Clean up
    if (redis_ctx) redisFree(redis_ctx);
    if (sentinel_ctx) redisFree(sentinel_ctx);

    return 0;
}

For robust applications, the client should implement a retry mechanism. If a command to the Redis master fails (e.g., with a connection error or a “MOVED” error indicating the master has changed), the client should disconnect, re-query Sentinel for the current master, and attempt to reconnect and re-issue the command. This logic can be encapsulated within a connection manager class.

AWS Deployment Considerations

Deploying Redis Sentinel and clients on AWS involves leveraging its networking and compute services effectively. Using Elastic Network Interfaces (ENIs) with static private IP addresses for Redis instances and Sentinels can simplify configuration and DNS management. AWS Route 53 can be used to manage DNS records for the Redis master, updating them via API calls during a failover, although Sentinel’s client-side discovery is often preferred for lower latency.

Automating Failover Detection and Response

While Redis Sentinel handles the Redis failover, integrating this with broader AWS infrastructure monitoring and C++ client behavior is key. AWS CloudWatch can monitor Sentinel and Redis instance health. Custom Lambda functions or EC2-based agents can be triggered by CloudWatch alarms to perform actions like updating DNS, notifying teams, or even orchestrating client-side re-initialization if the C++ client doesn’t have sophisticated Sentinel discovery built-in.

Example: Sentinel Notification Script

A simple shell script can be configured in sentinel.conf to be executed upon specific Sentinel events, such as a failover. This script can then trigger further automation.

#!/bin/bash

# sentinel.conf: sentinel notification-script /etc/redis/sentinel_notify.sh

# Arguments passed by Sentinel:
# $1: Sentinel current state (s_down_master, master_down_by_sentinel, master_failover, etc.)
# $2: Master name
# $3: Master IP
# $4: Master Port

EVENT=$1
MASTER_NAME=$2
MASTER_IP=$3
MASTER_PORT=$4

echo "$(date): Sentinel event '$EVENT' for master '$MASTER_NAME' ($MASTER_IP:$MASTER_PORT)" >> /var/log/sentinel_events.log

case "$EVENT" in
    "master-failover")
        NEW_MASTER_IP=$5 # Sentinel passes new master IP as 5th argument
        NEW_MASTER_PORT=$6 # Sentinel passes new master port as 6th argument
        echo "$(date): Failover complete. New master is $NEW_MASTER_IP:$NEW_MASTER_PORT" >> /var/log/sentinel_events.log

        # Example: Trigger a Lambda function to update DNS or notify systems
        # aws lambda invoke --function-name YourFailoverNotificationFunction --payload "{\"master_name\": \"$MASTER_NAME\", \"new_master_ip\": \"$NEW_MASTER_IP\"}" output.json
        ;;
    "s_down_master")
        echo "$(date): Master $MASTER_IP:$MASTER_PORT is S_DOWN." >> /var/log/sentinel_events.log
        ;;
    "master_down_by_sentinel")
        echo "$(date): Master $MASTER_IP:$MASTER_PORT is confirmed DOWN by Sentinel." >> /var/log/sentinel_events.log
        ;;
    *)
        echo "$(date): Unhandled event '$EVENT'." >> /var/log/sentinel_events.log
        ;;
esac

exit 0

This script logs events and can be extended to call AWS APIs (e.g., using the AWS CLI) to update DNS records in Route 53, trigger an SNS notification, or initiate other automated recovery procedures. For C++ clients that do not have built-in Sentinel support, this script could also trigger a mechanism to signal them to re-evaluate their Redis master connection.

Conclusion

Architecting for auto-failover with Redis Sentinel and C++ clients on AWS involves a multi-layered approach. By correctly configuring Redis Sentinel for quorum and resilience, ensuring C++ clients can dynamically discover master changes, and leveraging AWS services for monitoring and automation, you can build a highly available Redis deployment that minimizes downtime during failures.