Disaster Recovery 101: Architecting Auto-Failovers for Redis and PHP Deployments on Linode

Establishing a Redis Sentinel Cluster for High Availability

For critical applications relying on Redis, a single instance is a single point of failure. Redis Sentinel provides high availability by monitoring Redis instances, performing automatic failovers, and providing configuration updates to clients. We’ll architect a robust Sentinel setup on Linode, ensuring minimal downtime.

A typical Sentinel deployment involves at least three Sentinel instances to ensure quorum for failover decisions. This prevents split-brain scenarios where a minority of Sentinels might incorrectly initiate a failover. We’ll assume three Linode instances, each running a Redis master, a Redis replica, and a Redis Sentinel process.

Redis Configuration (Master & Replica)

On each Linode instance designated for Redis, ensure your redis.conf is configured for replication and persistence. For simplicity, we’ll use RDB persistence. For production, consider AOF or a combination.

On the Master Node (e.g., 192.168.1.101):

# redis.conf (Master)
port 6379
daemonize yes
pidfile /var/run/redis_6379.pid
logfile /var/log/redis/redis-server.log
dir /var/lib/redis

# Persistence (RDB)
save 900 1
save 300 10
save 60 10000
dbfilename dump.rdb

# Binding to a specific IP for Sentinel access
bind 192.168.1.101 127.0.0.1
protected-mode no

# Replication (This instance is the master, so no replicaof directive)
# masterauth is only needed if you have replication authentication enabled
# requirepass your_redis_password

On the Replica Node (e.g., 192.168.1.102):

# redis.conf (Replica)
port 6379
daemonize yes
pidfile /var/run/redis_6379.pid
logfile /var/log/redis/redis-server.log
dir /var/lib/redis

# Persistence (RDB) - Replicas can also persist, but it's less critical for HA
save 900 1
save 300 10
save 60 10000
dbfilename dump.rdb

# Binding to a specific IP for Sentinel access
bind 192.168.1.102 127.0.0.1
protected-mode no

# Replication
replicaof 192.168.1.101 6379
# masterauth your_redis_password # Uncomment and set if masterauth is used

Ensure the Redis data directory (/var/lib/redis) and log directory (/var/log/redis) exist and have appropriate permissions for the Redis user. Start Redis on both instances:

sudo systemctl start redis-server
sudo systemctl enable redis-server

Redis Sentinel Configuration

On each of the three Linode instances (let’s say 192.168.1.101, 192.168.1.102, and 192.168.1.103), configure sentinel.conf. We’ll run Sentinel on the default port 26379.

On each Sentinel Node (e.g., 192.168.1.101, 192.168.1.102, 192.168.1.103):

# sentinel.conf
port 26379
daemonize yes
pidfile /var/run/redis/redis-sentinel.pid
logfile /var/log/redis/redis-sentinel.log
dir /var/lib/redis # Sentinel also needs a directory to store state

# Monitor the master Redis instance.
# 'mymaster' is the name of the master Redis instance.
# 192.168.1.101 6379 is the IP and port of the master.
# 2 is the quorum: the minimum number of Sentinels that must agree
# that the master is down for a failover to be initiated.
# 15000 is the failover timeout in milliseconds.
sentinel monitor mymaster 192.168.1.101 6379 2

# If the master is unreachable for more than 15 seconds,
# Sentinels will start to consider it failed.
sentinel down-after-milliseconds mymaster 15000

# Number of replicas that can be reconfigured at the same time during a failover.
sentinel parallel-syncs mymaster 1

# The failover timeout. If a failover is not completed within this time,
# other Sentinels will try to take over.
sentinel failover-timeout mymaster 60000

# If you are using Redis authentication, specify the password here.
# sentinel auth-pass mymaster your_redis_password

# Bind Sentinel to specific IPs for external access if needed
# bind 192.168.1.101 127.0.0.1
# protected-mode no

Start Sentinel on all three nodes:

sudo systemctl start redis-sentinel
sudo systemctl enable redis-sentinel

After starting, check the logs (/var/log/redis/redis-sentinel.log) on each Sentinel instance. They should discover each other and the master/replica setup. You can also use redis-cli -p 26379 sentinel masters to verify the state.

PHP Application Integration with Redis Sentinel

Your PHP application needs to be aware of the Sentinel cluster to connect to the current Redis master. The most robust way to achieve this is by using a Redis client library that supports Sentinel. The phpredis extension is a good choice, but it requires manual configuration for Sentinel. A more common and easier-to-configure approach is using a library like Predis.

Using Predis for Sentinel Integration

First, install Predis via Composer:

composer require predis/predis

Then, configure your PHP application to connect using the Sentinel configuration. Instead of connecting directly to a single Redis IP, you provide a list of Sentinel servers and the name of the master you are monitoring.

require 'vendor/autoload.php';

// Define your Sentinel servers and the master name
$sentinels = [
    'tcp://192.168.1.101:26379',
    'tcp://192.168.1.102:26379',
    'tcp://192.168.1.103:26379',
];
$masterName = 'mymaster';

// Optional: Redis connection parameters (if any, e.g., password)
$redisOptions = [
    'parameters' => [
        // 'password' => 'your_redis_password',
        'sentinel_master_name' => $masterName,
    ],
];

try {
    // Create a Predis client that connects via Sentinel
    $client = new Predis\Client($sentinels, $redisOptions);

    // You can now use the client as usual. Predis will automatically
    // discover the current master and connect to it.
    $client->set('mykey', 'myvalue');
    $value = $client->get('mykey');

    echo "Successfully connected to Redis. Value for 'mykey': " . $value . "\n";

    // To verify which server you are connected to:
    $info = $client->info();
    echo "Connected to: " . $info['Server']['redis_version'] . " on " . $info['Server']['host'] . ":" . $info['Server']['port'] . "\n";

} catch (Predis\Connection\ConnectionException $e) {
    // Handle connection errors. This might happen if all Sentinels are down
    // or if the master is unreachable and no failover has completed yet.
    echo "Could not connect to Redis: " . $e->getMessage() . "\n";
    // Implement your application's fallback or retry logic here.
} catch (Exception $e) {
    echo "An unexpected error occurred: " . $e->getMessage() . "\n";
}

When the Redis master fails, Sentinel will detect it, elect a new master from the replicas, and reconfigure the remaining replicas. Predis, when it next attempts to communicate with Redis (or on initial connection), will query the Sentinels for the new master’s address and transparently switch its connection.

Simulating a Failover for Testing

To ensure your setup is robust, you must simulate failover scenarios. The easiest way is to stop the Redis master process.

Steps:

• Identify the current master using redis-cli -p 26379 sentinel master mymaster on any Sentinel node.
• Stop the Redis master process on its Linode instance: sudo systemctl stop redis-server.
• Monitor the Sentinel logs (/var/log/redis/redis-sentinel.log) on all Sentinel nodes. You should see messages indicating the master is down, Sentinels are agreeing, and a failover is occurring.
• Once the failover is complete (this typically takes seconds), check the Sentinel status again: redis-cli -p 26379 sentinel master mymaster. It should now report a different IP address as the master.
• Restart the old master: sudo systemctl start redis-server. It should automatically connect as a replica to the new master.
• Test your PHP application. It should seamlessly reconnect to the new master.

For more advanced testing, you can also simulate network partitions between Sentinels or Redis instances to verify quorum and failover logic under adverse conditions.

Architectural Considerations for Production

While this setup provides basic HA, consider these points for production:

• Network Segmentation: Ensure your Redis and Sentinel instances are on a private network for security and performance. Linode’s VPC or private IP addressing is crucial here.
• Resource Allocation: Dedicate sufficient CPU, RAM, and disk I/O to your Redis instances. Monitor resource usage closely.
• Persistence Strategy: For critical data, RDB alone might not be enough. Consider enabling AOF (Append Only File) with appendfsync everysec or always for better durability, though it impacts performance.
• Monitoring & Alerting: Integrate Sentinel’s status with your broader monitoring system (e.g., Prometheus, Datadog). Set up alerts for Sentinel failures, failovers, and Redis performance issues.
• Configuration Management: Use tools like Ansible, Chef, or Puppet to automate the deployment and configuration of Redis and Sentinel across your Linode instances.
• Client-Side Timeout Handling: Ensure your PHP application has appropriate connection and read timeouts configured in Predis (or your chosen client) and robust error handling for transient network issues or during failovers.
• Security: Always use requirepass and masterauth for Redis authentication, and ensure Sentinel is configured with sentinel auth-pass if authentication is enabled. Restrict network access to Redis and Sentinel ports.

By implementing Redis Sentinel and integrating it correctly with your PHP application using a capable client library like Predis, you establish a highly available Redis layer that significantly enhances your application’s resilience against single-node failures on Linode.