Disaster Recovery 101: Architecting Auto-Failovers for Redis and Python Deployments on DigitalOcean

Establishing a High-Availability Redis Cluster on DigitalOcean

Achieving true high availability for Redis, especially in a cloud environment like DigitalOcean, necessitates a robust failover strategy. We’ll architect a solution using Redis Sentinel, a critical component for monitoring Redis instances and orchestrating automatic failovers. This setup will involve at least three Sentinel instances for quorum and redundancy, alongside a primary Redis master and at least one replica.

Sentinel Configuration for Automatic Failover

Each Sentinel instance requires a configuration file (e.g., sentinel.conf). The key directives ensure proper monitoring and failover behavior. We’ll deploy these on separate Droplets for maximum isolation.

Sentinel Configuration File (`sentinel.conf`)

Create a sentinel.conf file on each Sentinel Droplet. The following configuration is a baseline; adjust ports and IP addresses as per your deployment.

# Sentinel Configuration Example

port 26379
daemonize yes
pidfile /var/run/redis_sentinel.pid
logfile /var/log/redis/sentinel.log

# Monitor the Redis master. 'mymaster' is the name we give to this Redis setup.
# 192.168.1.10:6379 is the IP and port of the primary Redis master.
# 2 is the quorum: the minimum number of Sentinels that must agree that the master is down.
# 1 is the failover timeout: how long (in milliseconds) Sentinel waits before starting a failover.
# Adjust these values based on your network latency and tolerance for false positives.
sentinel monitor mymaster 192.168.1.10 6379 2

# The name of the master is 'mymaster'.
# The down-after-milliseconds is the time in milliseconds the master must be unreachable
# for it to be considered in 'down' state by a Sentinel.
sentinel down-after-milliseconds mymaster 5000

# The failover-timeout is the maximum time in milliseconds for a failover to complete.
sentinel failover-timeout mymaster 60000

# The parallel-syncs is the number of replicas that can be reconfigured to sync
# with the new master in parallel.
sentinel parallel-syncs mymaster 1

# Optional: Define a password for Sentinel to connect to Redis instances
# sentinel auth-pass mymaster YourRedisPassword

# Optional: Specify the Redis data directory for replicas if they need to be created
# sentinel data-dir /var/lib/redis/sentinel

Starting Redis and Sentinel Services

Ensure your Redis master and replicas are running with appropriate configurations, and then start the Sentinel services.

Redis Master Configuration (`redis.conf`)

# redis.conf for Master
port 6379
daemonize yes
pidfile /var/run/redis_6379.pid
logfile /var/log/redis/redis-server.log
dir /var/lib/redis
# If using Sentinel authentication
# requirepass YourRedisPassword

Redis Replica Configuration (`redis.conf`)

# redis.conf for Replica
port 6379
daemonize yes
pidfile /var/run/redis_6379.pid
logfile /var/log/redis/redis-server.log
dir /var/lib/redis
replicaof 192.168.1.10 6379 # Point to your Redis master
# If using Sentinel authentication
# requirepass YourRedisPassword

Starting Services (Example on Ubuntu/Debian)

# On Redis Master Droplet
sudo systemctl start redis-server

# On Redis Replica Droplet(s)
sudo systemctl start redis-server

# On each Sentinel Droplet
sudo systemctl start redis-sentinel

Integrating Python Applications with Redis Sentinel

Your Python application needs to be aware of the Redis cluster’s state and be able to connect to the current master, even after a failover. The redis-py library, with Sentinel support, simplifies this significantly.

Python Client Configuration using `redis-py`

Instead of directly connecting to a single Redis instance, you’ll configure your client to use Sentinel. This allows the client to query Sentinel for the current master’s address.

import redis

# List of Sentinel host:port tuples
SENTINEL_HOSTS = [('192.168.1.20', 26379), ('192.168.1.21', 26379), ('192.168.1.22', 26379)]
MASTER_NAME = 'mymaster' # Must match the 'sentinel monitor' name

try:
    # Create a Redis Sentinel client
    sentinel = redis.Sentinel(SENTINEL_HOSTS, socket_timeout=0.5)

    # Get the current master connection
    # If password is set in sentinel.conf and redis.conf
    # master = sentinel.master_for(MASTER_NAME, socket_timeout=0.5, password='YourRedisPassword')
    master = sentinel.master_for(MASTER_NAME, socket_timeout=0.5)

    # Test the connection and perform an operation
    master.set('mykey', 'myvalue')
    value = master.get('mykey')
    print(f"Successfully connected to Redis master. Value for 'mykey': {value.decode('utf-8')}")

    # You can also get a replica connection if needed
    # replica = sentinel.slave_for(MASTER_NAME, socket_timeout=0.5)
    # print(f"Connected to a replica: {replica.client_list()}")

except redis.exceptions.ConnectionError as e:
    print(f"Could not connect to Redis Sentinel or master: {e}")
except Exception as e:
    print(f"An unexpected error occurred: {e}")

Handling Failover in Application Logic

The `redis-py` Sentinel client automatically handles reconnections to the new master after a failover. However, your application might need to gracefully handle temporary unavailability during the failover process. Implementing retry mechanisms with exponential backoff is a good practice.

import redis
import time
import logging

logging.basicConfig(level=logging.INFO)

SENTINEL_HOSTS = [('192.168.1.20', 26379), ('192.168.1.21', 26379), ('192.168.1.22', 26379)]
MASTER_NAME = 'mymaster'
MAX_RETRIES = 5
INITIAL_BACKOFF = 1 # seconds

def get_redis_master():
    """
    Attempts to get a Redis master connection with retry logic.
    """
    sentinel = redis.Sentinel(SENTINEL_HOSTS, socket_timeout=0.5)
    retries = 0
    backoff_time = INITIAL_BACKOFF

    while retries < MAX_RETRIES:
        try:
            # If password is set:
            # master = sentinel.master_for(MASTER_NAME, socket_timeout=0.5, password='YourRedisPassword')
            master = sentinel.master_for(MASTER_NAME, socket_timeout=0.5)
            # Perform a quick check to ensure connection is live
            master.ping()
            logging.info("Successfully connected to Redis master.")
            return master
        except redis.exceptions.ConnectionError as e:
            logging.warning(f"Connection attempt {retries + 1}/{MAX_RETRIES} failed: {e}. Retrying in {backoff_time} seconds...")
            time.sleep(backoff_time)
            retries += 1
            backoff_time = min(backoff_time * 2, 30) # Exponential backoff, capped at 30s
        except Exception as e:
            logging.error(f"An unexpected error occurred during connection: {e}")
            # Depending on the error, you might want to retry or raise immediately
            time.sleep(backoff_time)
            retries += 1
            backoff_time = min(backoff_time * 2, 30)

    logging.error(f"Failed to connect to Redis master after {MAX_RETRIES} retries.")
    return None

# Example usage:
if __name__ == "__main__":
    redis_client = get_redis_master()

    if redis_client:
        try:
            redis_client.set('app_status', 'operational')
            status = redis_client.get('app_status')
            print(f"App status from Redis: {status.decode('utf-8')}")
        except redis.exceptions.ConnectionError as e:
            logging.error(f"Error performing Redis operation after connection: {e}. Application might need to re-establish connection.")
        except Exception as e:
            logging.error(f"An unexpected error occurred during Redis operation: {e}")
    else:
        logging.error("Application cannot proceed without Redis connection.")
        # Implement application-level fallback or error handling here

Automated Failover Testing and Monitoring

Regularly testing your failover mechanism is crucial. You can simulate a master failure by stopping the Redis master process or by manually commanding Sentinel to failover.

Simulating a Master Failure

To test the failover, you can stop the Redis master process on its Droplet. Sentinel should detect the failure and promote a replica.

# On the Redis Master Droplet
sudo systemctl stop redis-server
# Or, to simulate a network partition, you could use iptables to block traffic
# sudo iptables -A INPUT -p tcp --dport 6379 -j DROP

After stopping the master, monitor the Sentinel logs on your Sentinel Droplets. You should see messages indicating that the master is down and a failover is being initiated.

# On a Sentinel Droplet (tailing the log file)
sudo tail -f /var/log/redis/sentinel.log

Once the failover is complete, verify that a new master has been elected and that your Python application can connect to it. You can also use `redis-cli` to check the status:

# On any machine with redis-cli installed, pointing to a Sentinel
redis-cli -h 192.168.1.20 -p 26379 SENTINEL master mymaster
redis-cli -h 192.168.1.20 -p 26379 SENTINEL replicas mymaster

Monitoring Sentinel Health

Beyond Redis itself, monitoring the health of your Sentinel instances is paramount. Use tools like Prometheus with a Redis Exporter and a Sentinel Exporter, or DigitalOcean’s built-in monitoring, to track Sentinel availability, leader election status, and failover events.

Sentinel Uptime: Ensure all Sentinel processes are running.
Quorum Status: Verify that a sufficient number of Sentinels are active and communicating.
Master Status: Monitor the health of the current Redis master as reported by Sentinel.
Failover Events: Log and alert on any failover occurrences, as they indicate a problem that needs investigation.

Alerting on Sentinel failures or prolonged master unavailability is critical for proactive issue resolution.