Disaster Recovery 101: Architecting Auto-Failovers for Redis and Laravel Deployments on DigitalOcean
Automated Redis Failover with Sentinel
For high availability of Redis, Sentinel is the de facto standard. It provides monitoring, notification, and automatic failover. We’ll configure a basic Sentinel setup for a primary and replica Redis instance, and then discuss how to scale this to multiple replicas and Sentinels for robustness.
First, ensure you have Redis installed on your DigitalOcean Droplets. We’ll assume two Droplets: one for the primary Redis (e.g., 10.10.10.1) and one for the replica (e.g., 10.10.10.2). A third Droplet (e.g., 10.10.10.3) will host our Sentinel process.
Configuring Redis Primary and Replica
On the primary Redis Droplet (10.10.10.1), edit your redis.conf file. Ensure it’s configured to bind to your private IP and has a password set for security.
# redis.conf on 10.10.10.1 bind 10.10.10.1 port 6379 requirepass your_redis_password replica-serve-stale-data yes appendonly yes
On the replica Redis Droplet (10.10.10.2), configure it to replicate from the primary. Again, bind to its private IP and set a password.
# redis.conf on 10.10.10.2 bind 10.10.10.2 port 6379 requirepass your_redis_password replicaof 10.10.10.1 6379 masterauth your_redis_password replica-serve-stale-data yes appendonly yes
Restart both Redis instances:
sudo systemctl restart redis-server
Setting Up Redis Sentinel
On the Sentinel Droplet (10.10.10.3), create a sentinel.conf file. This configuration tells Sentinel how to monitor your Redis master and what quorum is needed for a failover.
# sentinel.conf on 10.10.10.3 port 26379 daemonize yes pidfile /var/run/redis/redis-sentinel.pid logfile /var/log/redis/redis-sentinel.log # Monitor your master Redis instance # Format: sentinel monitorsentinel monitor mymaster 10.10.10.1 6379 2 # Specify the password for connecting to Redis sentinel auth-pass mymaster your_redis_password # How long Redis master is considered unavailable (milliseconds) sentinel down-after-milliseconds mymaster 5000 # How long Sentinel waits before starting a failover (milliseconds) sentinel failover-timeout mymaster 10000 # Number of replicas that will be reconfigured when a failover occurs sentinel parallel-syncs mymaster 1
Start the Sentinel process:
redis-sentinel /etc/redis/sentinel.conf
To achieve higher availability for Sentinel itself, deploy at least three Sentinel instances across different Droplets or availability zones. The quorum value in the sentinel.conf should be set to (N/2) + 1, where N is the total number of Sentinels. For example, with 3 Sentinels, a quorum of 2 is appropriate.
Integrating Laravel with Redis Sentinel
Laravel’s cache and session drivers can be configured to use Redis Sentinel. Update your config/database.php file. You’ll need to define a new Redis connection that points to your Sentinel cluster.
<?php
return [
// ... other configurations
'redis' => [
'client' => env('REDIS_CLIENT', 'phpredis'),
'default' => [
'password' => env('REDIS_PASSWORD', null),
'host' => env('REDIS_HOST', '127.0.0.1'),
'port' => env('REDIS_PORT', 6379),
'database' => env('REDIS_DB', 0),
],
'sentinel' => [
'password' => env('REDIS_PASSWORD', null),
'scheme' => 'tcp',
'host' => env('REDIS_SENTINEL_HOST', '10.10.10.3'), // Your Sentinel IP
'port' => env('REDIS_SENTINEL_PORT', 26379),
'sentinels' => [ // List of all Sentinel instances
['host' => env('REDIS_SENTINEL_HOST_1', '10.10.10.3'), 'port' => env('REDIS_SENTINEL_PORT', 26379)],
// Add more sentinels if you have them configured
// ['host' => '10.10.10.4', 'port' => 26379],
// ['host' => '10.10.10.5', 'port' => 26379],
],
'service' => env('REDIS_SENTINEL_SERVICE', 'mymaster'), // The master name defined in sentinel.conf
'database' => env('REDIS_SENTINEL_DB', 0),
],
],
// ... other configurations
];
Update your .env file to use the Sentinel configuration:
REDIS_CLIENT=phpredis REDIS_HOST=127.0.0.1 # This is ignored when using sentinel config REDIS_PORT=6379 # This is ignored when using sentinel config REDIS_PASSWORD=your_redis_password REDIS_DB=0 REDIS_SENTINEL_HOST=10.10.10.3 REDIS_SENTINEL_PORT=26379 REDIS_SENTINEL_SERVICE=mymaster REDIS_SENTINEL_DB=0
Then, in your config/cache.php and config/session.php, set the driver to use the Sentinel connection:
// config/cache.php
'default' => env('CACHE_DRIVER', 'file'),
'stores' => [
// ...
'redis' => [
'driver' => 'redis',
'connection' => 'sentinel', // Use the sentinel connection defined in database.php
],
// ...
],
// config/session.php
'driver' => env('SESSION_DRIVER', 'file'),
'stores' => [
// ...
'redis' => [
'driver' => 'redis',
'connection' => 'sentinel', // Use the sentinel connection defined in database.php
],
// ...
],
Testing the Failover
To test the failover, manually stop the primary Redis process on 10.10.10.1:
sudo systemctl stop redis-server
Monitor the Sentinel logs on 10.10.10.3. You should see messages indicating that the master is down and a failover is being initiated. After a short period, Sentinel will promote one of the replicas to be the new master. Your Laravel application, configured to use the Sentinel connection, should automatically connect to the new master without manual intervention.
You can verify the new master by connecting to Sentinel and running:
redis-cli -p 26379 SENTINEL master mymaster
Automated Database Failover with DigitalOcean Managed Databases
For relational databases like PostgreSQL or MySQL, DigitalOcean’s Managed Databases offer a robust, built-in solution for high availability and automated failover. This abstracts away much of the complexity of setting up replication, monitoring, and failover mechanisms yourself.
Provisioning a Highly Available Database Cluster
When creating a new database cluster in the DigitalOcean control panel, select the “Highly Available” option. This provisions a primary node and at least one replica node. DigitalOcean handles the replication setup and ensures that in the event of a primary node failure, a replica is automatically promoted to become the new primary. The connection string provided by DigitalOcean will point to a virtual IP or DNS name that resolves to the current primary node, ensuring your application seamlessly connects to the active database.
For example, when creating a PostgreSQL cluster, you’d select the HA option. DigitalOcean will provide you with a connection string like:
postgresql://user:[email protected]:25060/database?sslmode=require
The your-db-hostname.db.ondigitalocean.com endpoint is managed by DigitalOcean and will always resolve to the current primary node, even after a failover.
Configuring Laravel for DigitalOcean Managed Databases
To integrate your Laravel application with a DigitalOcean Managed Database, you simply update your .env file with the credentials and hostname provided by DigitalOcean. Ensure your config/database.php is set up to use these environment variables.
// config/database.php (relevant section)
'connections' => [
// ...
'pgsql' => [
'driver' => 'pgsql',
'url' => env('DATABASE_URL'), // Or configure host, port, database, username, password individually
'host' => env('DB_HOST', '127.0.0.1'),
'port' => env('DB_PORT', '5432'),
'database' => env('DB_DATABASE', 'forge'),
'username' => env('DB_USERNAME', 'forge'),
'password' => env('DB_PASSWORD', ''),
'charset' => 'utf8',
'prefix' => '',
'prefix_indexes' => true,
'search_path' => 'public',
'sslmode' => 'prefer',
],
// ...
],
And your .env file:
DB_CONNECTION=pgsql DB_HOST=your-db-hostname.db.ondigitalocean.com DB_PORT=25060 DB_DATABASE=your_database_name DB_USERNAME=your_db_user DB_PASSWORD=your_db_password
Alternatively, if you use the DATABASE_URL format:
DATABASE_URL=postgresql://your_db_user:[email protected]:25060/your_database_name?sslmode=require
Simulating Database Failover
DigitalOcean’s Managed Databases handle failover automatically. While you cannot directly trigger a failover through the API or control panel in a production scenario without risk, you can observe the behavior during planned maintenance events or by contacting DigitalOcean support for a controlled test. The key is that your application, using the provided managed endpoint, will continue to function as the underlying infrastructure seamlessly switches to a replica.
The primary benefit here is the reduction in operational overhead. Instead of managing complex replication and failover scripts for your database, you rely on DigitalOcean’s expertise and infrastructure to ensure high availability.
Orchestrating Application Failover with Load Balancers
For your Laravel application servers, implementing automated failover typically involves a load balancer. DigitalOcean’s Load Balancers are a managed service that can distribute traffic across multiple Droplets running your application. They also provide health checks to automatically remove unhealthy Droplets from the pool of active servers.
Configuring a DigitalOcean Load Balancer
1. Create a Load Balancer: In the DigitalOcean control panel, navigate to “Networking” and select “Load Balancers.” Click “Create Load Balancer.”
2. Add Droplets: Select the Droplets that are running your Laravel application. Ensure these Droplets are configured identically and are part of a stateless deployment for seamless failover.
3. Configure Load Balancing:
Protocol: HTTP or HTTPS (recommended) Frontend Port: 80 or 443 Backend Port: 80 (or whatever port your Laravel app listens on, typically via Nginx/Apache) Algorithm: Round Robin (common choice)
4. Configure Health Checks: This is crucial for automated failover. The load balancer will periodically send requests to a specified path on your backend Droplets to check their health.
Protocol: HTTP Path: /health (You'll need to create a simple route in Laravel for this) Port: 80 Check Interval: 10 seconds (adjust as needed) Response Timeout: 5 seconds (adjust as needed) Healthy Threshold: 2 (number of consecutive successful checks) Unhealthy Threshold: 3 (number of consecutive failed checks)
Implementing a Laravel Health Check Endpoint
Create a simple route in your Laravel application to respond to the health check requests. This route should perform basic checks, such as verifying database connectivity.
// routes/web.php
use Illuminate\Support\Facades\Route;
use Illuminate\Support\Facades\DB;
use Illuminate\Support\Facades\Cache;
Route::get('/health', function () {
try {
// Check database connection
DB::connection()->getPdo();
// Check Redis connection (if applicable)
// Ensure your .env is configured for the correct Redis connection
$redis = Cache::store('redis')->connection();
$redis->ping();
// Add any other critical service checks here
return response('OK', 200);
} catch (\Exception $e) {
// Log the error for debugging
\Log::error("Health check failed: " . $e->getMessage());
return response('Service Unavailable', 503);
}
});
Ensure your web server (Nginx/Apache) is configured to serve this route and that the load balancer can reach it.
Testing Application Failover
To test the load balancer’s failover capability, manually stop your Laravel application’s web server (e.g., Nginx) on one of the Droplets:
sudo systemctl stop nginx
Observe the DigitalOcean Load Balancer dashboard. After the configured unhealthy threshold is met, the Droplet will be marked as unhealthy, and traffic will no longer be routed to it. Subsequent requests will be directed to the remaining healthy Droplets. Once you restart the web server on the affected Droplet, the health checks should pass again, and it will be added back into the rotation.
This layered approach—Sentinel for Redis, Managed Databases for relational data, and Load Balancers for application servers—provides a robust and largely automated disaster recovery strategy for your Laravel applications on DigitalOcean.