Disaster Recovery 101: Architecting Auto-Failovers for Redis and Laravel Deployments on Linode
Establishing a Redis Sentinel Cluster for High Availability
For robust Redis deployments, especially those backing critical Laravel applications, a single Redis instance is a single point of failure. Redis Sentinel provides high availability by monitoring Redis instances, performing automatic failovers, and providing configuration updates to clients. We’ll architect a three-node Sentinel cluster for redundancy, ensuring that even if one Sentinel node fails, the cluster remains operational.
First, let’s configure the primary Redis instance. Ensure Redis is installed on your Linode instances. We’ll assume three nodes: `redis-master` (your primary), `redis-slave-1`, and `redis-slave-2` (for replication and potential failover targets).
Redis Master Configuration (`redis-master`)
Edit your `redis.conf` file (typically located at `/etc/redis/redis.conf`).
# redis-master/redis.conf port 6379 daemonize yes pidfile /var/run/redis_6379.pid logfile /var/log/redis/redis-server.log dir /var/lib/redis # Enable RDB persistence (adjust as needed) save 900 1 save 300 10 save 60 10000 # Enable AOF persistence for better durability appendonly yes appendfilename "appendonly.aof" # Bind to a specific IP address for Sentinel access (replace with your private IP) bind 192.168.1.100 127.0.0.1 # Require a password for security requirepass YOUR_STRONG_REDIS_PASSWORD
Redis Slave Configuration (`redis-slave-1`, `redis-slave-2`)
On each slave node, configure replication. The `replicaof` directive (or `slaveof` in older versions) points to the master. Ensure the master password is provided.
# redis-slave-X/redis.conf port 6379 daemonize yes pidfile /var/run/redis_6379.pid logfile /var/log/redis/redis-server.log dir /var/lib/redis # Enable RDB persistence (optional for slaves, but good practice) save 900 1 save 300 10 save 60 10000 # Enable AOF persistence (optional for slaves) appendonly yes appendfilename "appendonly.aof" # Bind to a specific IP address bind 192.168.1.101 127.0.0.1 # For redis-slave-1 # bind 192.168.1.102 127.0.0.1 # For redis-slave-2 # Replication settings replicaof 192.168.1.100 6379 # Point to your redis-master IP masterauth YOUR_STRONG_REDIS_PASSWORD
After configuring and restarting Redis on all nodes, verify replication status from the master:
redis-cli -h 192.168.1.100 -a YOUR_STRONG_REDIS_PASSWORD REPLICA OF
You should see output listing your slaves. Now, let’s set up the Sentinels.
Redis Sentinel Configuration (`redis-sentinel-1`, `redis-sentinel-2`, `redis-sentinel-3`)
Sentinel configuration is typically in a separate file, `sentinel.conf`. We’ll deploy Sentinels on dedicated nodes or co-locate them with slaves. For this example, let’s assume three dedicated Sentinel nodes: `redis-sentinel-1`, `redis-sentinel-2`, and `redis-sentinel-3`.
# redis-sentinel-X/sentinel.conf port 26379 daemonize yes pidfile /var/run/redis-sentinel.pid logfile /var/log/redis/sentinel.log dir /var/lib/redis # Monitor the master Redis instance # The 'quorum' is the number of Sentinels that must agree a master is down # before initiating a failover. For 3 Sentinels, a quorum of 2 is typical. # The 'down-after-milliseconds' is how long a master must be unreachable # before it's considered down. # The 'failover-timeout' is the maximum time for a failover to complete. # The 'parallel-syncs' is the number of replicas that can sync with the # new master simultaneously after a failover. sentinel monitor mymaster 192.168.1.100 6379 2 sentinel down-after-milliseconds mymaster 5000 sentinel failover-timeout mymaster 60000 sentinel parallel-syncs mymaster 1 # If you have a password for Redis, specify it here sentinel auth-pass mymaster YOUR_STRONG_REDIS_PASSWORD # Specify the IP addresses Sentinels should bind to bind 192.168.1.110 127.0.0.1 # For redis-sentinel-1 # bind 192.168.1.111 127.0.0.1 # For redis-sentinel-2 # bind 192.168.1.112 127.0.0.1 # For redis-sentinel-3
Start the Sentinel processes on each of your Sentinel nodes. You can verify their status and cluster configuration using `redis-cli` connected to any Sentinel:
redis-cli -p 26379 -h 192.168.1.110 SENTINEL masters
This command will show details about the `mymaster` configuration, including the current master, its slaves, and the number of Sentinels monitoring it. To test failover, stop the Redis master process:
# On redis-master sudo systemctl stop redis-server
Monitor the Sentinel logs. Within a minute or two, you should see Sentinel logs indicating a master failure and a failover process. The Sentinels will elect a new master from the available slaves. After the failover, query the Sentinels again to see the updated master:
redis-cli -p 26379 -h 192.168.1.110 SENTINEL master mymaster
The output will now show the IP address of the newly elected master.
Integrating Laravel with Redis Sentinel
Laravel’s cache and session drivers can be configured to use Redis. For Sentinel integration, you need to specify the Sentinel hosts and the master name in your `.env` file. The `predis` client library, which Laravel uses by default for Redis, supports Sentinel out of the box.
Environment Configuration (`.env`)
Update your Laravel application’s `.env` file with the following settings. Replace the IP addresses with your actual Sentinel node IPs and ensure the `REDIS_PASSWORD` matches your Redis configuration.
# .env REDIS_HOST=127.0.0.1 # This is ignored when using Sentinels, but required by some Laravel versions REDIS_PORT=6379 # This is ignored when using Sentinels # Sentinel configuration REDIS_SENTINEL_HOSTS="192.168.1.110:26379,192.168.1.111:26379,192.168.1.112:26379" REDIS_SENTINEL_SERVICE_NAME=mymaster REDIS_PASSWORD=YOUR_STRONG_REDIS_PASSWORD REDIS_CLIENT=predis # Ensure predis is installed: composer require predis/predis
Laravel’s default Redis configuration in `config/database.php` is designed to pick up these Sentinel settings automatically if `REDIS_CLIENT` is set to `predis` and the Sentinel variables are present.
Laravel Redis Configuration (`config/database.php`)
The relevant section in `config/database.php` looks like this. No modifications are typically needed if you’re using `predis` and have set the `.env` variables correctly.
<?php
// ... other configurations
'redis' => [
'client' => env('REDIS_CLIENT', 'phpredis'), // Defaults to phpredis, change to 'predis' in .env
'options' => [
'cluster' => env('REDIS_CLUSTER', 'redis'),
'parameters' => [
'password' => env('REDIS_PASSWORD'),
'sentinel' => [
'hosts' => explode(',', env('REDIS_SENTINEL_HOSTS', '')),
'service_name' => env('REDIS_SENTINEL_SERVICE_NAME', 'mymaster'),
],
],
],
// ... other redis configurations
],
// ... other database configurations
?>
When your Laravel application attempts to connect to Redis, the `predis` client will use the provided Sentinel hosts to discover the current master for `mymaster`. If a failover occurs, the Sentinels will update their internal state, and subsequent connections from your Laravel application will automatically be directed to the new master after a brief period.
Automating Database Backups with Linode Object Storage
Regular, automated backups are non-negotiable for disaster recovery. We’ll leverage Linode’s Object Storage for cost-effective, off-site storage of Redis data. This involves using `redis-cli` to save the RDB file and then uploading it to Object Storage.
Setting up Linode Object Storage
1. **Create a Bucket:** In your Linode Cloud Manager, navigate to “Object Storage” and create a new bucket. Choose a region close to your Linode instances for lower latency.
2. **Generate Access Keys:** Under “Access Keys,” generate a new key pair. You’ll need the **Access Key ID** and the **Secret Access Key**. Store these securely.
3. **Install `s3cmd`:** This command-line utility simplifies interaction with S3-compatible storage, including Linode Object Storage.
sudo apt update sudo apt install s3cmd
4. **Configure `s3cmd`:** Run the configuration wizard, providing your Linode Object Storage credentials and endpoint. The endpoint URL will be in the format `https://[region].linodeobjects.com` (e.g., `https://us-east-1.linodeobjects.com`).
s3cmd --configure
Follow the prompts. For “Encryption password,” you can leave it blank unless you require client-side encryption. Test the configuration:
s3cmd ls
Automated Backup Script
Create a shell script (e.g., `/usr/local/bin/redis_backup.sh`) on your Redis master node to perform the backup. This script will save the RDB file, compress it, and upload it to your Object Storage bucket.
#!/bin/bash
# --- Configuration ---
REDIS_HOST="192.168.1.100"
REDIS_PORT="6379"
REDIS_PASSWORD="YOUR_STRONG_REDIS_PASSWORD"
BACKUP_DIR="/var/backups/redis"
BUCKET_NAME="your-linode-bucket-name" # Replace with your bucket name
S3_PATH="redis-backups" # Optional: a prefix for your backups in the bucket
TIMESTAMP=$(date +"%Y%m%d_%H%M%S")
FILENAME="redis_master_${TIMESTAMP}.rdb.gz"
LOG_FILE="/var/log/redis_backup.log"
# --- Ensure backup directory exists ---
mkdir -p ${BACKUP_DIR}
# --- Log function ---
log_message() {
echo "$(date +"%Y-%m-%d %H:%M:%S") - $1" | tee -a ${LOG_FILE}
}
log_message "Starting Redis backup..."
# --- Save Redis data to RDB file ---
# Use --dbfilename to specify the output file name directly, avoiding default redis.rdb
# Use --dir to specify the directory where the dump will be saved
if redis-cli -h ${REDIS_HOST} -p ${REDIS_PORT} -a ${REDIS_PASSWORD} SAVE; then
log_message "Redis SAVE command executed successfully."
# Find the latest RDB file. Assumes redis.conf has 'dbfilename dump.rdb' or similar.
# If you specified a different dbfilename in redis.conf, adjust accordingly.
# A more robust approach would be to read dbfilename from redis.conf.
RDB_FILE=$(redis-cli -h ${REDIS_HOST} -p ${REDIS_PORT} -a ${REDIS_PASSWORD} CONFIG GET dbfilename | grep -v dbfilename | tr -d ' ')
if [ -z "$RDB_FILE" ]; then
RDB_FILE="dump.rdb" # Default if not found
fi
RDB_PATH="${BACKUP_DIR}/${RDB_FILE}"
if [ -f "${RDB_PATH}" ]; then
log_message "Found RDB file at ${RDB_PATH}."
# --- Compress the RDB file ---
if gzip -c "${RDB_PATH}" > "${BACKUP_DIR}/${FILENAME}"; then
log_message "RDB file compressed to ${BACKUP_DIR}/${FILENAME}."
# --- Upload to Linode Object Storage ---
if s3cmd put "${BACKUP_DIR}/${FILENAME}" s3://${BUCKET_NAME}/${S3_PATH}/${FILENAME}; then
log_message "Backup uploaded successfully to s3://${BUCKET_NAME}/${S3_PATH}/${FILENAME}."
# --- Clean up old local backups (optional) ---
find ${BACKUP_DIR} -type f -name "redis_master_*.rdb.gz" -mtime +7 -delete
log_message "Old local backups (older than 7 days) removed."
# --- Clean up old remote backups (optional) ---
# This requires more advanced s3cmd scripting or lifecycle policies on Linode
# For simplicity, we'll omit remote cleanup here.
log_message "Backup process completed."
exit 0
else
log_message "ERROR: Failed to upload backup to Linode Object Storage."
exit 1
fi
else
log_message "ERROR: Failed to compress RDB file."
exit 1
fi
else
log_message "ERROR: RDB file not found at ${RDB_PATH} after SAVE command."
exit 1
fi
else
log_message "ERROR: Redis SAVE command failed."
exit 1
fi
Make the script executable:
sudo chmod +x /usr/local/bin/redis_backup.sh
Scheduling Backups with Cron
Use cron to schedule the backup script. For example, to run the backup daily at 3:00 AM:
sudo crontab -e
Add the following line:
0 3 * * * /usr/local/bin/redis_backup.sh
This setup ensures that your Redis data is regularly backed up to a separate, durable storage location, significantly improving your disaster recovery posture.
Automating Laravel Application Failover
While Redis Sentinel handles the data layer failover, your Laravel application needs to be deployed in a way that can also failover. This typically involves a load balancer and multiple application servers. Linode’s NodeBalancers are ideal for this.
Load Balancer Configuration (Linode NodeBalancer)
1. **Create a NodeBalancer:** In your Linode Cloud Manager, navigate to “NodeBalancers” and create a new one. Choose a region close to your application servers.
2. **Add Nodes:** Add your Laravel application servers as “Nodes.” Configure health checks to ensure the NodeBalancer only sends traffic to healthy instances. A common health check is an HTTP GET request to a specific endpoint (e.g., `/health`) on your application servers.
3. **Configure Frontend:** Set up the frontend configuration (e.g., port 80 for HTTP, 443 for HTTPS) and associate it with your nodes.
Application Health Check Endpoint
Create a simple route in your Laravel application to act as a health check. This endpoint should return a 200 OK status if the application is healthy, including its connection to Redis.
// routes/web.php or routes/api.php
use Illuminate\Support\Facades\Route;
use Illuminate\Support\Facades\Cache;
use Illuminate\Support\Facades\Redis;
Route::get('/health', function () {
try {
// Check Redis connection
Redis::connection()->client()->ping();
// Check a simple cache entry
Cache::put('health_check', true, 1);
if (Cache::get('health_check')) {
return response()->json(['status' => 'ok', 'message' => 'Application is healthy and Redis is accessible.']);
} else {
return response()->json(['status' => 'error', 'message' => 'Application is healthy, but cache is not working correctly.'], 500);
}
} catch (\Exception $e) {
return response()->json(['status' => 'error', 'message' => 'Failed to connect to Redis: ' . $e->getMessage()], 500);
}
});
Ensure this endpoint is accessible and returns a 200 OK status when your application is functioning correctly. The NodeBalancer will use this to determine if an application server is available to receive traffic.
Deployment Strategy for Failover
Deploy your Laravel application to at least two separate Linode instances. These instances should be configured identically, including their connection strings to the Redis Sentinel cluster. When a failover occurs:
- If the primary Redis master fails, Redis Sentinel will promote a slave to become the new master.
- Your Laravel application instances, configured to use Sentinel, will automatically connect to the new Redis master.
- If an application server becomes unhealthy (e.g., due to application errors, network issues, or Redis unavailability), the NodeBalancer’s health checks will detect this.
- The NodeBalancer will stop sending traffic to the unhealthy application server and redirect all incoming requests to the remaining healthy application server(s).
This multi-layered approach ensures that both your data store and your application layer are resilient to failures, providing a robust disaster recovery solution.