Vengala Vinay

Having 9+ Years of Experience in Software Development

Home » Disaster Recovery 101: Architecting Auto-Failovers for Redis and WordPress Deployments on OVH

Disaster Recovery 101: Architecting Auto-Failovers for Redis and WordPress Deployments on OVH

Establishing a High-Availability Redis Cluster for WordPress

For WordPress deployments demanding resilience, a robust Redis setup is paramount. We’ll architect an auto-failover solution using Redis Sentinel, ensuring minimal downtime during node failures. This guide assumes you have basic familiarity with Linux administration and networking, and access to OVH cloud infrastructure.

Our strategy involves a primary Redis instance, a replica, and at least two Sentinel nodes for quorum and monitoring. This setup provides automatic detection of primary failure and promotion of the replica to primary. We’ll deploy these on separate OVH instances for true fault tolerance.

Redis Installation and Configuration

Begin by installing Redis on each of your chosen instances. For Debian/Ubuntu-based systems:

sudo apt update
sudo apt install redis-server -y

Next, we need to configure Redis for replication and Sentinel monitoring. Edit the Redis configuration file, typically located at /etc/redis/redis.conf. Ensure the following directives are set:

# For the primary Redis instance
port 6379
bind 0.0.0.0
replica-serve-stale-data yes
replica-read-only no
appendonly yes
appendfilename "appendonly.aof"
protected-mode no # Be cautious, ensure firewall rules are in place

# For the replica Redis instance
port 6379
bind 0.0.0.0
replica-serve-stale-data yes
replica-read-only yes
appendonly yes
appendfilename "appendonly.aof"
protected-mode no # Be cautious, ensure firewall rules are in place
replicaof <primary_ip_address> 6379 # Replace <primary_ip_address>

After modifying redis.conf, restart the Redis service on all nodes:

sudo systemctl restart redis-server

Verify replication status from the replica node:

redis-cli -h <replica_ip_address> replicaof no one
redis-cli -h <replica_ip_address> INFO replication

You should see output indicating the replica is connected to the primary and the replication status is ‘connected’.

Configuring Redis Sentinel for Auto-Failover

Redis Sentinel is responsible for monitoring Redis instances, detecting failures, and initiating failover. Install Sentinel on at least two separate instances (ideally three or more for robust quorum).

sudo apt update
sudo apt install redis-sentinel -y

Edit the Sentinel configuration file, typically /etc/redis/sentinel.conf. Configure it as follows:

port 26379
bind 0.0.0.0
daemonize yes
pidfile /var/run/redis/redis-sentinel.pid
logfile /var/log/redis/redis-sentinel.log

# Monitor the primary Redis master
# The quorum is the number of Sentinels that must agree that the master is down
# before initiating a failover. For a 3-node Sentinel setup, a quorum of 2 is typical.
# For a 2-node Sentinel setup, a quorum of 2 is also recommended to avoid split-brain.
sentinel monitor mymaster <primary_ip_address> 6379 2

# The down-after-milliseconds is the time in milliseconds the master must be unreachable
# for a Sentinel to consider it down.
sentinel down-after-milliseconds mymaster 5000

# The failover-timeout is the time in milliseconds to wait for a failover to complete.
sentinel failover-timeout mymaster 60000

# The parallel-syncs is the number of replicas that can be reconfigured in parallel
# during a failover.
sentinel parallel-syncs mymaster 1

# Optional: Specify the password if your Redis instances are password-protected
# sentinel auth-pass mymaster <your_redis_password>

Restart the Sentinel service on each Sentinel node:

sudo systemctl restart redis-sentinel

Verify Sentinel status from any Sentinel node:

redis-cli -p 26379 -h <sentinel_ip_address> sentinel master mymaster

This command will show details about the monitored master, including the number of replicas and Sentinels. Ensure your primary Redis instance is listed as the master and the replica is listed as a replica.

WordPress Configuration for Redis Failover

Your WordPress application needs to be aware of the Redis cluster and be able to connect to the current primary. The most robust way to achieve this is by configuring your WordPress application to connect to the Sentinel port and let Sentinel direct it to the active master.

Using a WordPress plugin like “Redis Object Cache” or a custom integration, you’ll typically configure the connection details. For Sentinel support, you’ll need to specify the Sentinel host(s) and port, along with the master name (e.g., mymaster).

Here’s a conceptual example of how you might configure this in your wp-config.php or via a plugin’s settings. The exact implementation depends on the Redis client library and the WordPress plugin used.

define('WP_REDIS_CLIENT', 'phpredis'); // Or your preferred client
define('WP_REDIS_SCHEME', 'tcp');
define('WP_REDIS_HOST', '<sentinel_ip_address_1>,<sentinel_ip_address_2>'); // Comma-separated list of Sentinel IPs
define('WP_REDIS_PORT', '26379');
define('WP_REDIS_SENTINEL_MASTER', 'mymaster'); // The name defined in sentinel.conf
// define('WP_REDIS_PASSWORD', '<your_redis_password>'); // If using password protection

When WordPress attempts to connect, the Redis client library (configured to use Sentinel) will query the Sentinels for the current master of mymaster and connect to it. If a failover occurs, subsequent connection attempts will be directed to the newly promoted primary.

Testing the Auto-Failover Mechanism

To validate the setup, simulate a primary Redis node failure. This can be done by stopping the Redis service on the primary instance:

# On the primary Redis server
sudo systemctl stop redis-server

Monitor the Redis Sentinel logs (/var/log/redis/redis-sentinel.log) and the Sentinel CLI output:

redis-cli -p 26379 -h <sentinel_ip_address> sentinel master mymaster

You should observe Sentinels detecting the failure, agreeing on a quorum, and initiating a failover. The replica should be promoted to master, and WordPress should automatically reconnect to the new primary. Check your WordPress site for any errors during this period. A brief period of unavailability is expected, but it should recover automatically.

To restore the original primary, start the Redis service again on the old primary node. It will likely be configured as a replica of the new primary automatically by Sentinel. If not, you may need to manually reconfigure it.

Architecting WordPress High-Availability with OVH Load Balancers and Database Replication

Beyond Redis, ensuring WordPress itself is highly available requires a multi-pronged approach involving load balancing and database replication. OVH’s load balancing services, combined with MySQL replication, form the backbone of a resilient WordPress deployment.

OVH Load Balancer Configuration

OVH offers various load balancing solutions. For a typical WordPress setup, an L7 (HTTP/HTTPS) load balancer is most suitable. This allows for intelligent traffic distribution based on HTTP headers, cookies, and other application-level data.

Navigate to your OVH Control Panel, select “Load Balancers” and create a new instance. Configure the following:

  • Frontend Configuration: Define the public IP address and ports (e.g., 80, 443) that your load balancer will listen on.
  • Backend Pool: Add your WordPress web servers (instances running Apache or Nginx) to a backend pool.
  • Load Balancing Algorithm: Choose an algorithm like Round Robin, Least Connections, or Source IP Hash. Round Robin is a good starting point.
  • Health Checks: Crucially, configure health checks for your backend servers. This typically involves checking a specific URL (e.g., /healthz.php or a custom endpoint) on your WordPress instances. The load balancer will stop sending traffic to unhealthy servers.
  • SSL Termination: For HTTPS, you can terminate SSL at the load balancer, simplifying certificate management on your web servers.

Ensure your WordPress web servers are configured to accept traffic from the load balancer’s IP range and that their firewall rules allow incoming connections from the load balancer.

MySQL Replication for WordPress Database

A single MySQL instance is a single point of failure. Implementing replication provides redundancy. We’ll set up asynchronous replication with a primary (master) and at least one replica (slave).

Primary MySQL Server Configuration (my.cnf):

[mysqld]
server-id = 1
log_bin = /var/log/mysql/mysql-bin.log
binlog_format = ROW
relay_log = /var/log/mysql/mysql-relay-bin.log
read_only = 0 # Set to 1 for replicas

Restart MySQL and create a replication user:

-- On the primary MySQL server
CREATE USER 'repl_user'@'%' IDENTIFIED BY 'your_replication_password';
GRANT REPLICATION SLAVE ON *.* TO 'repl_user'@'%';
FLUSH PRIVILEGES;

-- Get the current binary log file and position
SHOW MASTER STATUS;

Note down the File and Position from the SHOW MASTER STATUS output. This is critical for setting up the replica.

Replica MySQL Server Configuration (my.cnf):

[mysqld]
server-id = 2 # Must be unique for each server
log_bin = /var/log/mysql/mysql-bin.log
binlog_format = ROW
relay_log = /var/log/mysql/mysql-relay-bin.log
read_only = 1 # Replicas should generally be read-only

Restart MySQL on the replica. Then, configure replication:

-- On the replica MySQL server
CHANGE MASTER TO
  MASTER_HOST='<primary_mysql_ip_address>',
  MASTER_USER='repl_user',
  MASTER_PASSWORD='your_replication_password',
  MASTER_LOG_FILE='<master_log_file_from_show_master_status>',
  MASTER_LOG_POS=<master_log_pos_from_show_master_status>;
START SLAVE;

Verify replication status:

SHOW SLAVE STATUS\G;

Ensure Slave_IO_Running and Slave_SQL_Running are both Yes, and Seconds_Behind_Master is low (ideally 0).

WordPress Application Configuration for Database Failover

Your WordPress wp-config.php file needs to point to the primary database. For automatic failover of the database itself, you’ll need a more advanced solution. Options include:

  • Proxy/Connection Manager: Tools like ProxySQL or MaxScale can sit in front of your MySQL servers, manage connections, and handle failover.
  • Application-Level Logic: Custom PHP code or a sophisticated WordPress plugin that monitors database health and updates wp-config.php or uses a connection pool that can switch masters.
  • Managed Database Services: OVH’s managed database offerings might provide built-in high-availability features.

For a simpler, albeit less automated, approach, you would manually update wp-config.php after a database failover is initiated. A more robust solution involves a database proxy.

Example wp-config.php for a single primary:

define( 'DB_NAME', 'your_database_name' );
define( 'DB_USER', 'your_db_user' );
define( 'DB_PASSWORD', 'your_db_password' );
define( 'DB_HOST', '<primary_mysql_ip_address>' ); // This would need to be updated on failover
define( 'DB_CHARSET', 'utf8mb4' );
define( 'DB_COLLATE', '' );

Orchestrating Failover with External Tools

Achieving true auto-failover for the database often requires external orchestration. Tools like:

  • MHA (Master High Availability Manager and tools for MySQL): A popular open-source solution for MySQL master failover.
  • Orchestrator: A modern, actively maintained tool for MySQL replication topology management and failover.
  • Keepalived/Pacemaker: For managing virtual IP addresses and ensuring a service is always available on one node.

These tools monitor the primary database and, upon failure, promote a replica and update the necessary configurations or manage a virtual IP that WordPress connects to. Integrating these requires careful planning and testing.

By combining OVH’s load balancing, Redis Sentinel for caching resilience, and a robust MySQL replication strategy with an appropriate failover mechanism, you can architect a highly available WordPress deployment capable of withstanding significant infrastructure failures.