Vengala Vinay

Having 9+ Years of Experience in Software Development

Home » Disaster Recovery 101: Architecting Auto-Failovers for Redis and WordPress Deployments on DigitalOcean

Disaster Recovery 101: Architecting Auto-Failovers for Redis and WordPress Deployments on DigitalOcean

Automating Redis Failover with Sentinel

For Redis deployments, high availability is paramount. Manual failover is a non-starter in production environments. Redis Sentinel is the built-in solution for monitoring Redis instances, handling automatic failover, and providing configuration to clients. This section details the setup of a robust Redis Sentinel cluster on DigitalOcean.

We’ll deploy a master Redis instance and at least two replica instances. For Sentinel, we need a minimum of three instances to achieve quorum for leader election and failover decisions. This ensures that a single point of failure doesn’t compromise the Sentinel cluster itself.

Sentinel Configuration

Each Sentinel instance requires a configuration file. Let’s assume we have three Droplets for Redis (redis-master, redis-replica-1, redis-replica-2) and three Droplets for Sentinel (sentinel-1, sentinel-2, sentinel-3). The following configuration should be applied to each Sentinel Droplet, adjusting the IP addresses accordingly.

On sentinel-1, the configuration might look like this:

# sentinel-1.conf
port 26379
dir /tmp

# Monitor the master Redis instance
# The quorum is the number of Sentinels that must agree that the master is down
# before initiating a failover. A quorum of 2 is sufficient for 3 Sentinels.
sentinel monitor mymaster 10.10.0.1 6379 2

# The down-after-milliseconds is the time in milliseconds the master must be
# unreachable for it to be considered "down".
sentinel down-after-milliseconds mymaster 5000

# The parallel-syncs is the number of replicas that can be reconfigured in parallel
# to point to the new master after a failover.
sentinel parallel-syncs mymaster 1

# The failover-timeout is the maximum time in milliseconds to complete a failover.
sentinel failover-timeout mymaster 60000

# Optional: Specify the IP address of the Sentinel itself if it's not obvious
# sentinel resolve-host mymaster 10.10.0.10

# Optional: If you have password-protected Redis instances
# sentinel auth-pass mymaster YourRedisPassword

Repeat this configuration on sentinel-2 and sentinel-3, ensuring the sentinel monitor directive points to the IP address of your primary Redis master (e.g., 10.10.0.1) and that the Sentinel IP addresses are correctly listed in the `sentinel resolve-host` directive if used. The quorum value should be set to a majority of your Sentinel instances (e.g., 2 for 3 Sentinels, 3 for 5 Sentinels).

Starting Redis and Sentinel Services

First, ensure your Redis instances are running. On redis-master:

# On redis-master Droplet
sudo systemctl start redis-server
sudo systemctl enable redis-server

# Configure replicas to connect to the master
# On redis-replica-1 and redis-replica-2 Droplets:
sudo sed -i 's/# replicaof <masterip> <masterport>/replicaof 10.10.0.1 6379/' /etc/redis/redis.conf
sudo systemctl restart redis-server
sudo systemctl enable redis-server

Next, start the Sentinel services on their respective Droplets:

# On sentinel-1, sentinel-2, sentinel-3 Droplets
sudo systemctl start redis-sentinel
sudo systemctl enable redis-sentinel

After starting, you can check the status of the Sentinel cluster. On any Sentinel node, you can use redis-cli -p 26379 to interact with Sentinel. For example:

# On any Sentinel Droplet
redis-cli -p 26379
127.0.0.1:26379> sentinel master mymaster
127.0.0.1:26379> sentinel replicas mymaster
127.0.0.1:26379> sentinel sentinels mymaster

The sentinel master mymaster command should show the current master, its IP, port, and the number of replicas. The sentinel replicas mymaster command will list the current replicas. The sentinel sentinels mymaster command will list all known Sentinel instances for that master.

Testing Failover

To test the failover, you can manually stop the master Redis process. On the redis-master Droplet:

# On redis-master Droplet
sudo systemctl stop redis-server

Within a short period (depending on your down-after-milliseconds and network latency), the Sentinel cluster should detect the master as down, elect a new master from the replicas, and reconfigure the remaining replicas. You can monitor the Sentinel logs (typically in /var/log/redis/sentinel.log or via journalctl -u redis-sentinel) to observe the failover process. After the failover, run redis-cli -p 26379 sentinel master mymaster on a Sentinel node to see the new master.

WordPress High Availability with Load Balancers and Database Replication

Achieving high availability for WordPress involves ensuring both the web application layer and the database layer are resilient to failures. On DigitalOcean, this typically means using multiple Droplets for web servers behind a load balancer and employing a managed database solution with replication or setting up your own replicated MySQL cluster.

Load Balancer Configuration (DigitalOcean Load Balancer)

DigitalOcean’s managed Load Balancer is a straightforward way to distribute traffic across multiple WordPress web server Droplets. We’ll configure it to forward HTTP and HTTPS traffic.

Assuming you have at least two WordPress Droplets (wp-web-1 and wp-web-2) with their private IPs (e.g., 10.10.0.2 and 10.10.0.3), you would create a Load Balancer in the DigitalOcean control panel with the following settings:

  • Frontend Configuration:
  • Protocol: HTTP, Port: 80
  • Protocol: HTTPS, Port: 443 (if using SSL)
  • Backend Pool:
  • Add your WordPress Droplets (wp-web-1, wp-web-2) with their private IP addresses.
  • Port: 80 (or 443 if your WordPress instances are configured for SSL termination at the web server level).
  • Health Check:
  • Protocol: HTTP
  • Port: 80
  • Path: /wp-cron.php or a custom health check endpoint (e.g., /healthz) that returns a 200 OK. This is crucial for detecting unhealthy web servers.
  • Interval: 10 seconds
  • Timeout: 5 seconds
  • Unhealthy Threshold: 3
  • Healthy Threshold: 2
  • Sticky Sessions:
  • Enable if your WordPress site relies on session data that isn’t stored externally (e.g., in Redis or a shared file system). This ensures a user’s requests are consistently sent to the same web server.

Once configured, point your domain’s A record to the Load Balancer’s public IP address.

Database High Availability (Managed Databases for PostgreSQL/MySQL)

For WordPress, the database is often the single point of failure. DigitalOcean’s Managed Databases offer built-in high availability with automatic failover. We’ll focus on MySQL as it’s more common for WordPress.

When creating a Managed Database for MySQL cluster, select the “High Availability” option. This provisions a primary node and one or more read replicas that are automatically synchronized. In case of a primary node failure, DigitalOcean automatically promotes a replica to become the new primary.

You will receive connection details for the primary node. Your WordPress wp-config.php file needs to be updated to use these credentials. Crucially, for read-heavy workloads, you can also configure WordPress to use read replicas for non-critical queries to offload the primary. This requires a plugin like “HyperDB” or custom code.

Configuring WordPress for Managed Database HA

Update your wp-config.php file on each WordPress web server Droplet with the connection details provided by DigitalOcean for your Managed Database cluster. Ensure the database user has appropriate permissions.

/**
 * Database settings for DigitalOcean Managed Databases.
 */
define( 'DB_NAME', 'your_database_name' );
define( 'DB_USER', 'your_database_user' );
define( 'DB_PASSWORD', 'your_database_password' );
define( 'DB_HOST', 'your_managed_database_host.do.digitalocean.com:25060' ); // Example: includes port
define( 'DB_CHARSET', 'utf8mb8' );
define( 'DB_COLLATE', '' );

// If using read replicas with a plugin like HyperDB, you'd define additional constants or configurations here.
// For example, HyperDB might use a DB_PRIMARY and DB_SECONDARY array.
// For simplicity, this example focuses on connecting to the primary.

If you are not using a plugin to manage read replicas, ensure your DB_HOST points to the primary endpoint provided by DigitalOcean. The Managed Database service handles the failover transparently at the database level. Your WordPress application will attempt to reconnect to the new primary if the old one becomes unavailable.

Shared File System for WordPress Uploads

A critical component often overlooked in HA WordPress setups is the shared file system for uploads (wp-content/uploads). If each web server Droplet has its own local uploads directory, content uploaded on one server won’t be available on the other. Solutions include:

  • DigitalOcean Spaces with a CDN: Mount Spaces as a volume using tools like s3fs or use a plugin that directly uploads to Spaces. This is a highly scalable and distributed approach.
  • NFS (Network File System): Set up an NFS server (e.g., on a dedicated Droplet or a NAS) and mount the wp-content/uploads directory on all WordPress web servers. This is simpler but can become a bottleneck or single point of failure if not managed carefully.
  • Managed Object Storage (e.g., AWS S3, Google Cloud Storage): Similar to DigitalOcean Spaces, these can be integrated via plugins or tools like s3fs.

For this architecture, we’ll assume DigitalOcean Spaces is used. You would configure your WordPress site (typically via a plugin like “WP Offload Media Lite” or “S3 Media Maestro”) to upload media directly to a Spaces bucket. This eliminates the need for a shared file system on the web server layer for uploads.

Integrating Redis with WordPress for Caching

To further enhance performance and offload the database, Redis can be used as an object cache for WordPress. This requires a WordPress plugin that supports Redis object caching.

WordPress Redis Object Cache Plugin

The most popular plugin for this is “Redis Object Cache” by Till Krüss. After installing and activating it, you’ll need to configure it to connect to your Redis Sentinel cluster. The plugin typically uses the wp-config.php file for configuration.

/**
 * Redis Object Cache settings.
 *
 * Ensure your Redis Sentinel cluster is accessible from your WordPress web servers.
 * The plugin will automatically connect to the master managed by Sentinel.
 */
define( 'WP_REDIS_CLIENT', 'phpredis' ); // Or 'pecl_redis' if you installed the PECL extension
define( 'WP_REDIS_SCHEME', 'redis' ); // Use 'rediss' for SSL
define( 'WP_REDIS_HOST', '10.10.0.1' ); // IP of one of your Sentinel nodes
define( 'WP_REDIS_PORT', 26379 ); // Sentinel port
define( 'WP_REDIS_DATABASE', 0 ); // Redis database index (0-15)
define( 'WP_REDIS_SENTINEL', true ); // Enable Sentinel mode
define( 'WP_REDIS_SENTINEL_MASTER', 'mymaster' ); // The name of your Redis master as defined in Sentinel config

// If your Redis instances are password protected:
// define( 'WP_REDIS_PASSWORD', 'YourRedisPassword' );

// If your Redis instances are configured with TLS/SSL:
// define( 'WP_REDIS_SCHEME', 'rediss' );
// define( 'WP_REDIS_TLS_VERIFY_PEER', true ); // Or false, depending on your setup
// define( 'WP_REDIS_TLS_PEER_NAME', 'your-redis-host.example.com' );

Important Considerations:

  • The WP_REDIS_HOST and WP_REDIS_PORT should point to one of your Sentinel nodes. The plugin, when WP_REDIS_SENTINEL is true, will communicate with Sentinel to discover the current Redis master.
  • Ensure your WordPress Droplets can reach your Redis and Sentinel Droplets over their private network interfaces. Adjust DigitalOcean VPC firewall rules if necessary.
  • If you are using Redis for session management (e.g., with a plugin like “Session Manager”), ensure that the WP_REDIS_HOST and WP_REDIS_PORT are accessible and configured correctly for session persistence.

Putting It All Together: Architecture Overview

This architecture provides a robust, highly available WordPress deployment on DigitalOcean:

  • User Traffic: Enters through the DigitalOcean Load Balancer.
  • Load Balancer: Distributes traffic to healthy WordPress web server Droplets. It also performs health checks to remove unresponsive web servers from rotation.
  • WordPress Web Servers: Multiple Droplets running Nginx/Apache and PHP-FPM. They serve the WordPress application.
  • Shared Media: Uploaded media is stored directly in DigitalOcean Spaces, accessible globally and served via CDN.
  • Object Caching: Redis Sentinel cluster provides a highly available object cache for WordPress, managed by the “Redis Object Cache” plugin. Sentinel handles automatic failover of the Redis master.
  • Database: DigitalOcean Managed Database for MySQL (High Availability configuration) provides a resilient database layer with automatic failover.

This layered approach ensures that individual component failures are handled gracefully, minimizing downtime and maintaining application availability.