Disaster Recovery 101: Architecting Auto-Failovers for Redis and Magento 2 Deployments on DigitalOcean
Redis Sentinel for High Availability
For Magento 2 deployments, Redis is a critical component for caching sessions, configuration, and full-page caches. Ensuring its high availability is paramount. Redis Sentinel provides a robust solution for automatic failover. We’ll architect a multi-node Redis Sentinel setup on DigitalOcean droplets.
A typical setup involves at least three Redis nodes (master, replica) and at least three Sentinel nodes. This redundancy ensures that even if one Redis node or one Sentinel node fails, the system can detect the failure and promote a replica to master without manual intervention.
Sentinel Configuration
On each Sentinel node, the redis-sentinel.conf file needs to be configured. The key directives are:
port 26379: The default Sentinel port.sentinel monitor mymaster 10.10.0.1 6379 2: This tells Sentinel to monitor a master Redis instance namedmymaster, located at10.10.0.1:6379. The2indicates that at least two Sentinels must agree that the master is down before initiating a failover.sentinel down-after-milliseconds mymaster 5000: The master is considered down if it’s unreachable for 5000 milliseconds (5 seconds).sentinel failover-timeout mymaster 60000: The timeout for the failover process.sentinel parallel-syncs mymaster 1: The number of replicas that can be reconfigured to sync with the new master in parallel.sentinel auth-pass mymaster YourRedisPassword: If your Redis master requires authentication, specify the password here.
Ensure that the IP addresses (e.g., 10.10.0.1) are the private IP addresses of your Redis nodes within your DigitalOcean VPC. The Sentinel nodes should be distributed across different availability zones or even regions for maximum resilience, though for simplicity, we’ll assume they are within the same VPC.
Here’s an example redis-sentinel.conf for one of your Sentinel nodes:
port 26379 sentinel monitor mymaster 10.10.0.1 6379 2 sentinel down-after-milliseconds mymaster 5000 sentinel failover-timeout mymaster 60000 sentinel parallel-syncs mymaster 1 sentinel auth-pass mymaster MySuperSecretRedisPassword
Redis Server Configuration
Each Redis server (master and replicas) needs to be configured to allow replication and, if necessary, authentication.
On the master Redis node (e.g., 10.10.0.1):
port 6379 daemonize yes pidfile /var/run/redis_6379.pid logfile /var/log/redis/redis-server.log dbfilename dump.rdb dir /var/lib/redis # For Sentinel monitoring and replication bind 0.0.0.0 protected-mode no # If using authentication requirepass MySuperSecretRedisPassword
On the replica Redis nodes (e.g., 10.10.0.2, 10.10.0.3):
port 6379 daemonize yes pidfile /var/run/redis_6379.pid logfile /var/log/redis/redis-server.log dbfilename dump.rdb dir /var/lib/redis # For Sentinel monitoring and replication bind 0.0.0.0 protected-mode no # Replication configuration replicaof 10.10.0.1 6379 # If using authentication requirepass MySuperSecretRedisPassword
After configuring and starting Redis and Sentinel services on all nodes, you can verify the setup using redis-cli. Connect to any Sentinel instance:
redis-cli -p 26379 127.0.0.1:26379> SENTINEL masters
This command should list your mymaster, its current master IP, number of replicas, and number of Sentinels monitoring it. You can also check the status of replicas and Sentinels.
Magento 2 Configuration for Redis Failover
Magento 2’s configuration for Redis is typically managed in app/etc/env.php. When using Redis Sentinel, you need to point Magento to the Sentinel instances and specify the master name. Magento’s Redis client library will then query Sentinel to discover the current master’s IP address.
The env.php configuration should look like this:
<?php
return [
'backend' => [
'front' => [
'Mage_Adminhtml' => [
'default' => [
'Mage_Core_Model_Session' => [
'save_handler' => 'redis',
'save_path' => 'tcp://127.0.0.1:6379?persistent=1&database=2',
'redis_host' => '127.0.0.1',
'redis_port' => '6379',
'redis_password' => 'MySuperSecretRedisPassword',
'redis_timeout' => '2.5',
'redis_read_timeout' => '1',
'redis_persistent' => '1',
'redis_database' => '2',
'redis_compress_data' => '1'
]
]
]
]
],
'cache' => [
'frontend' => [
'default' => [
'backend' => 'Magento\\Framework\\Cache\\Backend\\Redis',
'backend_options' => [
'server' => '127.0.0.1',
'port' => '6379',
'database' => '0',
'password' => 'MySuperSecretRedisPassword',
'compress_data' => '1',
'compression_library' => 'gzip'
]
],
'page_cache' => [
'backend' => 'Magento\\Framework\\Cache\\Backend\\Redis',
'backend_options' => [
'server' => '127.0.0.1',
'port' => '6379',
'database' => '1',
'password' => 'MySuperSecretRedisPassword',
'compress_data' => '1',
'compression_library' => 'gzip'
]
]
]
],
'session' => [
'save' => 'redis',
'redis' => [
'host' => '127.0.0.1',
'port' => '6379',
'password' => 'MySuperSecretRedisPassword',
'timeout' => '2.5',
'read_timeout' => '1',
'persistent' => '1',
'database' => '2',
'compression_library' => 'gzip'
]
]
];
?>
Important Note: The above configuration points to 127.0.0.1:6379. This is a common pattern where the Magento application servers have a local Redis client that is configured to use Sentinel. The actual Redis master/replica instances are on separate droplets. To achieve this, you would typically run a local Redis instance on each Magento application server that is configured as a Sentinel client, or more directly, configure Magento’s Redis adapter to use Sentinel directly.
The more robust and recommended approach for Sentinel integration within Magento is to modify the env.php to point directly to the Sentinel cluster. This requires a custom Redis adapter or a Magento extension that supports Sentinel discovery. However, if you are using a Redis client library that natively supports Sentinel (like Predis), you can configure it to connect to Sentinel.
For a direct Sentinel integration without a local Redis instance, the env.php would need to be adapted. Magento’s default Redis adapter doesn’t directly support Sentinel connection strings. You’d typically use a library like Predis, which does. If you’re using Predis, your configuration might look conceptually like this (this requires modifying Magento’s Redis client implementation or using an extension):
<?php
// Conceptual example for env.php using Predis with Sentinel
return [
// ... other configurations
'cache' => [
'frontend' => [
'default' => [
'backend' => 'Magento\\Framework\\Cache\\Backend\\Redis',
'backend_options' => [
'redis_client_class' => 'Predis\\Client', // Specify Predis client
'redis_client_parameters' => [
'sentinels' => [
['host' => '10.10.0.4', 'port' => 26379], // Sentinel 1 IP
['host' => '10.10.0.5', 'port' => 26379], // Sentinel 2 IP
['host' => '10.10.0.6', 'port' => 26379] // Sentinel 3 IP
],
'service' => 'mymaster', // The master name defined in Sentinel config
'password' => 'MySuperSecretRedisPassword',
'database' => '0',
// ... other Predis options
],
'compress_data' => '1',
'compression_library' => 'gzip'
]
],
// ... page_cache configuration
]
],
// ... session configuration
];
?>
This requires ensuring that the predis/predis library is installed in your Magento project (e.g., via Composer) and that Magento’s Redis backend is configured to use it. You might need to create a custom module to override Magento’s default Redis client factory to properly inject Predis with Sentinel configuration.
Simulating a Failover
To test the failover mechanism, you can manually stop the master Redis process. On the current master node:
sudo systemctl stop redis-server
Observe the logs on the Sentinel nodes. Within a few seconds (based on the down-after-milliseconds setting), the Sentinels should detect the master as down, elect a new master from the available replicas, and reconfigure the remaining replicas to follow the new master.
You can verify the new master by connecting to any Sentinel:
redis-cli -p 26379 127.0.0.1:26379> SENTINEL master mymaster
This will show the details of the current master, including its new IP address. Magento applications configured to use Sentinel should automatically reconnect to the new master without downtime, provided the network connectivity is stable and the configuration is correct.
DigitalOcean Load Balancer Integration
While Redis Sentinel handles the failover of the Redis cluster itself, you’ll want to ensure your Magento application servers can always reach the *current* Redis master. A DigitalOcean Load Balancer can be configured to point to your Redis nodes. However, a standard load balancer doesn’t understand Sentinel’s dynamic master promotion.
A common strategy is to use a load balancer with a health check that targets the Redis port (6379). The health check should be configured to expect a specific response or simply check for a TCP connection. When a failover occurs, the new master will become available, and the load balancer will start directing traffic to it.
However, this approach has a critical limitation: the load balancer doesn’t *know* which node is the master. It might send traffic to a replica that is not yet promoted or is in the process of being promoted. This can lead to intermittent errors.
A more robust solution involves a dedicated proxy layer that understands Redis Sentinel, such as:
- Envoy Proxy: Can be configured with Redis Sentinel discovery.
- HAProxy: With appropriate configuration and potentially custom health checks, can be made to work, though it’s less direct than Envoy for Sentinel.
- Redis Enterprise’s built-in proxy: If you were using Redis Enterprise.
For a DigitalOcean setup without a dedicated proxy, the most practical approach is often to configure your Magento application servers to use Sentinel directly (as shown in the conceptual env.php example using Predis) and ensure your application can tolerate brief connection interruptions during failover. The application servers themselves would connect to the Sentinel IPs, not a load balancer.
If you must use a load balancer for Redis, ensure its health checks are aggressive enough to quickly remove a failed master but also consider that it might direct traffic to a replica before Sentinel has fully completed the promotion. This is where direct Sentinel client integration in the application is superior.
Automated Deployment and Management
To truly achieve automated failover, the deployment and management of Redis and Sentinel nodes should be automated. Tools like Ansible, Terraform, or Kubernetes (if applicable) can be used to:
- Provision DigitalOcean droplets.
- Install and configure Redis and Redis Sentinel.
- Set up replication and Sentinel monitoring.
- Manage firewall rules to allow inter-node communication.
- Update Magento’s
env.phpor deploy custom modules for Sentinel integration.
For instance, an Ansible playbook could ensure that Redis and Sentinel are installed, configured with the correct IPs, started, and enabled to run on boot across your fleet of droplets. This ensures consistency and reduces manual errors.
# Example Ansible task for configuring Redis Sentinel
- name: Configure Redis Sentinel
template:
src: redis-sentinel.conf.j2
dest: /etc/redis/redis-sentinel.conf
notify: Restart redis-sentinel
# Example Jinja2 template for redis-sentinel.conf.j2
port 26379
{% for master_ip in redis_masters %}
sentinel monitor mymaster {{ master_ip }} 6379 2
sentinel down-after-milliseconds mymaster 5000
sentinel failover-timeout mymaster 60000
sentinel parallel-syncs mymaster 1
sentinel auth-pass mymaster {{ redis_password }}
{% endfor %}
By combining Redis Sentinel for high availability with automated deployment and configuration management, you create a resilient Redis infrastructure for your Magento 2 deployment that can withstand node failures with minimal to no manual intervention.