Vengala Vinay

Having 9+ Years of Experience in Software Development

Home » Disaster Recovery 101: Architecting Auto-Failovers for Redis and Perl Deployments on OVH

Disaster Recovery 101: Architecting Auto-Failovers for Redis and Perl Deployments on OVH

Redis Sentinel for High Availability

Achieving automated failover for Redis deployments hinges on leveraging Redis Sentinel. Sentinel is a distributed system that provides high availability for Redis. It monitors Redis instances, detects failures, and can initiate automatic failover by promoting a replica to a master. For this architecture, we’ll assume a primary Redis instance and at least two replicas, all managed by a quorum of Sentinel processes.

A typical Sentinel configuration file (sentinel.conf) would look something like this:

port 26379
sentinel monitor mymaster 192.168.1.100 6379 2
sentinel down-after-milliseconds mymaster 5000
sentinel failover-timeout mymaster 10000
sentinel parallel-syncs mymaster 1
sentinel auth-pass mymaster YourRedisPassword

Key directives:

  • port 26379: The port Sentinel listens on.
  • sentinel monitor mymaster <ip> <port> <quorum>: This is the core directive. It tells Sentinel to monitor a Redis master named ‘mymaster’ at the specified IP and port. The <quorum> value (e.g., 2) is the number of Sentinels that must agree a master is down before initiating a failover. This prevents split-brain scenarios.
  • sentinel down-after-milliseconds mymaster 5000: The time in milliseconds after which a Redis instance is considered down if it doesn’t respond to PING.
  • sentinel failover-timeout mymaster 10000: The maximum time in milliseconds to complete a failover.
  • sentinel parallel-syncs mymaster 1: The number of replicas that can be reconfigured to sync with the new master in parallel after a failover.
  • sentinel auth-pass mymaster <password>: If your Redis master requires authentication, this directive provides the password.

Ensure you have at least three Sentinel instances running on separate hosts for robust high availability. These Sentinels will communicate with each other to maintain consensus on the state of the Redis cluster.

Perl Application Integration with Redis Sentinel

Your Perl applications need to be aware of the current Redis master. Instead of hardcoding the master’s IP address, they should query Sentinel for the current master’s address. The Redis::Sentinel Perl module is ideal for this.

First, ensure you have the module installed:

cpan Redis::Sentinel

Here’s a sample Perl script demonstrating how to connect to Redis via Sentinel:

use strict;
use warnings;
use Redis::Sentinel;
use Redis;

my $sentinel_hosts = [
    { host => '192.168.1.101', port => 26379 },
    { host => '192.168.1.102', port => 26379 },
    { host => '192.168.1.103', port => 26379 },
];
my $service_name = 'mymaster';
my $redis_password = 'YourRedisPassword'; # If applicable

my $sentinel = Redis::Sentinel->new(
    sentinels => $sentinel_hosts,
    service_name => $service_name,
    redis_args => [ password => $redis_password ] # Pass password to Redis client
);

eval {
    my $redis_client = $sentinel->discover_master;

    # Use the Redis client as usual
    $redis_client->set('mykey', 'myvalue');
    my $value = $redis_client->get('mykey');
    print "Value: $value\n";

    # Example of handling potential connection errors during operation
    # In a real application, you'd wrap critical Redis operations in try/catch
    # and potentially re-discover the master if a connection error occurs.
};
if ($@) {
    warn "Error connecting to Redis or performing operation: $@\n";
    # In a production system, you might trigger alerts or attempt to re-discover
    # the master after a short delay.
}

The Redis::Sentinel->new() constructor takes an array of Sentinel host/port pairs and the name of the Redis master service. The discover_master method queries the Sentinels to find the current master. If a failover occurs, subsequent calls to discover_master will return the new master’s address. It’s crucial to implement retry logic and error handling in your application to gracefully manage temporary network issues or the brief unavailability during a failover event.

OVH Infrastructure Setup and Configuration

On OVH, you’ll typically deploy your Redis instances and Sentinel processes on separate virtual machines (e.g., Public Cloud Instances). For network isolation and security, it’s recommended to use private networks within your OVH project.

1. Network Configuration:

  • Create a private network (e.g., using OVH’s SDN capabilities) that spans the availability zone(s) where your Redis and Sentinel instances will reside.
  • Assign static private IP addresses to each Redis master, replica, and Sentinel instance within this private network. This ensures consistent internal addressing, which is vital for Sentinel’s monitoring and for your applications to reliably discover services.

2. Firewall Rules:

  • Redis Instances: Allow TCP traffic on port 6379 (or your configured Redis port) from your application servers and from your Sentinel instances. Restrict access from all other sources.
  • Sentinel Instances: Allow TCP traffic on port 26379 (or your configured Sentinel port) between Sentinel instances themselves. Allow TCP traffic on port 26379 from your application servers (if they need to query Sentinel directly for discovery, though the Perl module abstracts this). Crucially, allow TCP traffic on port 6379 from Sentinels to the Redis master and replicas they are monitoring.

You can configure these firewall rules using OVH’s security group or network firewall features within the Public Cloud interface.

3. Deployment Steps:

  • Provision your OVH instances.
  • Install Redis server on your master and replica instances. Configure them for replication (replicaof directive in redis.conf) and ensure they are accessible via their private IPs.
  • Install Redis Sentinel on at least three separate instances. Configure sentinel.conf as described earlier, pointing to the correct private IPs of your Redis master.
  • Start the Redis and Sentinel services.
  • Verify connectivity: From an application server, attempt to connect to Redis via Sentinel using the Perl script. From a Sentinel host, ensure it can ping the Redis master and replicas.

Automated Failover Testing and Monitoring

Regularly testing your failover mechanism is non-negotiable. A common method is to simulate a master failure.

Testing Procedure:

  • Identify the current Redis master.
  • Gracefully shut down the Redis master instance (e.g., using redis-cli shutdown).
  • Monitor the Sentinel logs on your Sentinel hosts. You should see events indicating the master is down and a failover is being initiated.
  • Observe the Sentinel logs and your application’s behavior. Your Perl application, upon encountering a connection error or after a short retry interval, should be able to discover the new master promoted by Sentinel.
  • Verify data integrity by checking a few key-value pairs on the new master.
  • Bring the old master back online. It should be automatically configured as a replica of the new master (if replica-serve-stale-data yes is not set and failover is successful).

Monitoring:

  • Sentinel Health: Monitor the Sentinel processes themselves. Ensure they are running and can communicate with each other. Tools like Prometheus with the redis_exporter can scrape Sentinel metrics.
  • Redis Metrics: Monitor key Redis metrics such as memory usage, connected clients, latency, and replication lag.
  • Application Performance: Track application error rates, particularly Redis connection errors, and overall response times.

Implement robust alerting based on these metrics. For instance, alert if a Sentinel instance becomes unreachable, if the number of connected clients to the master drops unexpectedly, or if replication lag exceeds a defined threshold. This proactive monitoring ensures you are aware of potential issues before they impact your users.