Vengala Vinay

Having 12+ Years of Experience in Software Development

Home » Disaster Recovery 101: Architecting Auto-Failovers for Redis and Magento 2 Deployments on Linode

Disaster Recovery 101: Architecting Auto-Failovers for Redis and Magento 2 Deployments on Linode

Redis Sentinel for High Availability

For Magento 2 deployments, Redis is a critical component for caching sessions, configuration, and page caches. Ensuring its high availability is paramount. Redis Sentinel provides a robust solution for monitoring Redis instances and automating failover in case of master node failure. This section details the setup of a Redis Sentinel cluster on Linode.

We’ll deploy three Redis nodes (one master, two replicas) and three Sentinel nodes. This configuration ensures that even if one Redis node and one Sentinel node fail, the cluster remains operational.

Redis Installation and Configuration

On each of your chosen Linode instances (let’s assume three for Redis and three for Sentinel, though they can be co-located for smaller deployments), install Redis:

  • Ubuntu/Debian:
    sudo apt update && sudo apt install redis-server -y
  • CentOS/RHEL:
    sudo yum update && sudo yum install redis -y

Next, configure the Redis master instance. Edit the Redis configuration file (typically /etc/redis/redis.conf):

  • Set a strong `requirepass` for security.
  • Enable replication by setting `replica-serve-stale-data yes` and `replica-read-only yes` (if you want replicas to only serve stale data and be read-only).
  • Ensure `bind` is set to the private IP address of the Linode instance to allow communication within your private network.

Example snippet for redis.conf on the master node:

# /etc/redis/redis.conf (Master Node)
port 6379
daemonize yes
pidfile /var/run/redis_6379.pid
logfile /var/log/redis/redis-server.log
bind 192.168.1.10  # Replace with your Linode's private IP
requirepass your_very_strong_redis_password
masterauth your_very_strong_redis_password
replica-serve-stale-data yes
replica-read-only yes

On the replica nodes, configure them to replicate from the master. The configuration is similar, but with these key differences:

# /etc/redis/redis.conf (Replica Node)
port 6379
daemonize yes
pidfile /var/run/redis_6379.pid
logfile /var/log/redis/redis-server.log
bind 192.168.1.11  # Replace with this replica's private IP
requirepass your_very_strong_redis_password
replicaof 192.168.1.10 6379  # Master IP and Port
masterauth your_very_strong_redis_password
replica-serve-stale-data yes
replica-read-only yes

Restart Redis on all nodes after applying changes:

sudo systemctl restart redis-server

Verify replication status using redis-cli on a replica:

redis-cli -a your_very_strong_redis_password
127.0.0.1:6379> REPLICAINFO
# Replication
master_host:192.168.1.10
master_port:6379
master_link_status:up
# ... other details

Redis Sentinel Configuration

Install Redis Sentinel on your designated Sentinel nodes. Often, it’s installed alongside Redis:

# On Sentinel nodes
sudo apt install redis-sentinel -y  # Debian/Ubuntu
sudo yum install redis -y          # CentOS/RHEL (sentinel is part of redis package)

Create a Sentinel configuration file (e.g., /etc/redis/sentinel.conf). Ensure Sentinel is configured to run as a daemon and logs its activity.

# /etc/redis/sentinel.conf (On each Sentinel node)
port 26379
daemonize yes
pidfile /var/run/redis-sentinel.pid
logfile /var/log/redis/redis-sentinel.log
bind 192.168.1.20  # Replace with this Sentinel's private IP

# Define the master to monitor
# Format: sentinel monitor <master-name> <ip> <port> <quorum>
# Quorum is the number of Sentinels that must agree a master is down.
# For 3 Sentinels, a quorum of 2 is recommended.
sentinel monitor mymaster 192.168.1.10 6379 2

# Authentication for master and replicas
sentinel auth-pass mymaster your_very_strong_redis_password

# Failover timeout (milliseconds)
sentinel down-after-milliseconds mymaster 5000

# Parallel syncs allowed during failover
sentinel parallel-syncs mymaster 1

# Notification script (optional, for custom actions on failover)
# sentinel notification-script mymaster /path/to/your/notification_script.sh

Start the Sentinel service on each Sentinel node:

sudo systemctl start redis-sentinel
sudo systemctl enable redis-sentinel

Verify Sentinel status. You can connect to a Sentinel instance using redis-cli -p 26379 and run SENTINEL MASTERS and SENTINEL SENTINELS.

Magento 2 Configuration for Redis Sentinel

Update your Magento 2 app/etc/env.php file to point to the Sentinel cluster. Magento will automatically discover the current master through Sentinel.

return [
    'backend' => [
        'frontName' => 'admin_secret_path'
    ],
    'crypt' => [
        'key' => 'your_magento_encryption_key'
    ],
    'db' => [
        'table_prefix' => ''
    ],
    'cache' => [
        'frontend' => [
            'default' => [
                'backend' => 'Magento\\Framework\\Cache\\Backend\\Redis',
                'options' => [
                    'report_buffer_size' => '10485760',
                    'password' => 'your_very_strong_redis_password',
                    'database' => '0',
                    'compression_library' => 'gzip',
                    'compress_data' => '1',
                    'cache_dir' => '/var/www/html/magento2/var/cache/',
                    'persistent' => '',
                    'redis_host' => '192.168.1.10', // Master IP (or any Sentinel IP)
                    'redis_port' => '6379',
                    'redis_password' => 'your_very_strong_redis_password',
                    'sentinel_master_name' => 'mymaster', // Must match sentinel.conf
                    'sentinel_hosts' => '192.168.1.20:26379,192.168.1.21:26379,192.168.1.22:26379' // Comma-separated list of Sentinel IPs and ports
                ]
            ],
            'page_cache' => [
                'backend' => 'Magento\\Framework\\Cache\\Backend\\Redis',
                'options' => [
                    'report_buffer_size' => '10485760',
                    'password' => 'your_very_strong_redis_password',
                    'database' => '1', // Use a different DB for page cache if desired
                    'compression_library' => 'gzip',
                    'compress_data' => '1',
                    'cache_dir' => '/var/www/html/magento2/var/cache/',
                    'persistent' => '',
                    'redis_host' => '192.168.1.10', // Master IP (or any Sentinel IP)
                    'redis_port' => '6379',
                    'redis_password' => 'your_very_strong_redis_password',
                    'sentinel_master_name' => 'mymaster', // Must match sentinel.conf
                    'sentinel_hosts' => '192.168.1.20:26379,192.168.1.21:26379,192.168.1.22:26379' // Comma-separated list of Sentinel IPs and ports
                ]
            ]
        ]
    ]
];

Note that redis_host and redis_port in env.php can point to any of the Sentinel nodes or even the current master. Magento’s Redis client will use the sentinel_hosts and sentinel_master_name to discover the actual master. It’s good practice to list all Sentinel nodes in sentinel_hosts for redundancy.

Automated Failover Testing

To test the failover mechanism, you can manually stop the Redis master process:

# On the current Redis master node
sudo systemctl stop redis-server

Observe the Sentinel logs (/var/log/redis/redis-sentinel.log) on the Sentinel nodes. You should see messages indicating that a master is down, a leader Sentinel has been elected, and a replica has been promoted to master. After a short period, Magento should automatically reconnect to the new master.

You can verify the new master by connecting to any Sentinel and running SENTINEL MASTERS. The IP address listed for the master should now be one of the former replicas.

Nginx Configuration for Load Balancing and Failover

While Redis Sentinel handles the database layer’s high availability, Nginx can be configured to provide load balancing and failover for your Magento 2 web servers. This ensures that if one web server becomes unresponsive, traffic is automatically routed to healthy instances.

We’ll set up Nginx in a primary/secondary or active/passive configuration, or more commonly, an active/active setup with health checks.

Nginx Installation and Basic Setup

Install Nginx on your load balancer Linode instance and your Magento web server Linode instances.

# On Load Balancer and Web Servers
sudo apt update && sudo apt install nginx -y # Debian/Ubuntu
sudo yum update && sudo yum install nginx -y # CentOS/RHEL

Ensure Nginx is running and enabled:

sudo systemctl start nginx
sudo systemctl enable nginx

Nginx Load Balancer Configuration

On the Nginx load balancer Linode, create or edit an Nginx configuration file (e.g., /etc/nginx/sites-available/magento_lb). We’ll use the upstream module for defining backend servers and health checks.

# /etc/nginx/sites-available/magento_lb
# Load balancer for Magento 2

# Define the upstream group of Magento web servers
# The 'max_fails' and 'fail_timeout' parameters configure health checks.
# Nginx will consider a server down after 'max_fails' consecutive failed checks
# within 'fail_timeout' seconds.
upstream magento_servers {
    server 192.168.2.10:80 max_fails=3 fail_timeout=30s; # Primary Magento Web Server 1
    server 192.168.2.11:80 max_fails=3 fail_timeout=30s; # Primary Magento Web Server 2
    # Add more web servers as needed
}

server {
    listen 80;
    server_name yourdomain.com www.yourdomain.com;

    # Health check endpoint for Nginx to monitor server health.
    # This endpoint should return a 200 OK status code if the server is healthy.
    # You might need to create a simple PHP script at this location.
    location /nginx_health_check {
        access_log off;
        return 200 'OK';
        add_header Content-Type text/plain;
    }

    location / {
        proxy_pass http://magento_servers;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # Optional: For WebSocket support (e.g., for admin notifications)
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";

        # Timeout settings
        proxy_connect_timeout 60s;
        proxy_send_timeout 60s;
        proxy_read_timeout 60s;
    }

    # Optional: Redirect HTTP to HTTPS
    # if ($scheme != "https") {
    #     return 301 https://$host$request_uri;
    # }
}

# Optional: HTTPS server block (requires SSL certificate configuration)
# server {
#     listen 443 ssl http2;
#     server_name yourdomain.com www.yourdomain.com;
#
#     ssl_certificate /etc/letsencrypt/live/yourdomain.com/fullchain.pem;
#     ssl_certificate_key /etc/letsencrypt/live/yourdomain.com/privkey.pem;
#     include /etc/letsencrypt/options-ssl-nginx.conf;
#     ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
#
#     location / {
#         proxy_pass http://magento_servers;
#         proxy_set_header Host $host;
#         proxy_set_header X-Real-IP $remote_addr;
#         proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
#         proxy_set_header X-Forwarded-Proto $scheme;
#
#         proxy_http_version 1.1;
#         proxy_set_header Upgrade $http_upgrade;
#         proxy_set_header Connection "upgrade";
#
#         proxy_connect_timeout 60s;
#         proxy_send_timeout 60s;
#         proxy_read_timeout 60s;
#     }
# }

Create a simple health check script on each Magento web server. For example, create /var/www/html/nginx_health_check.php:

<?php
header('Content-Type: text/plain');
echo 'OK';
http_response_code(200);
?>

Make sure this script is accessible via HTTP and returns a 200 OK status. You might need to adjust your Magento 2 nginx.conf or Apache .htaccess to allow access to this specific file without going through Magento’s routing.

Enable the Nginx site configuration and test the configuration:

sudo ln -s /etc/nginx/sites-available/magento_lb /etc/nginx/sites-enabled/
sudo nginx -t
sudo systemctl reload nginx

Magento 2 Web Server Configuration

On each Magento web server, ensure Nginx (or Apache) is configured to listen on the private IP address and is accessible by the load balancer. The Magento 2 nginx.conf (or Apache virtual host) should be set up to serve the application. Crucially, ensure the server_name directive matches what the load balancer is sending, or use a wildcard if appropriate.

For Nginx on web servers, ensure the server_name directive is set correctly. If you have multiple web servers, they should all be configured to respond to the same server_name or IP address that the load balancer is configured to proxy to.

# /etc/nginx/sites-available/magento2 (on each web server)
server {
    listen 80;
    server_name 192.168.2.10; # Or your Linode's private IP
    root /var/www/html/magento2/public_html; # Adjust path as needed
    index index.php index.html index.htm;

    # ... other Magento Nginx configurations ...

    # Health check endpoint configuration
    location /nginx_health_check {
        try_files /nginx_health_check.php =404;
    }

    location ~ \.php$ {
        # ... PHP-FPM configuration ...
        include snippets/fastcgi-php.conf;
        fastcgi_pass unix:/var/run/php/php8.1-fpm.sock; # Adjust PHP version
    }

    # ... other Magento Nginx configurations ...
}

Automated Failover Testing for Nginx

To test the Nginx load balancer’s failover capabilities, stop the Nginx service on one of the Magento web servers:

# On one of the Magento web servers
sudo systemctl stop nginx

Within the `fail_timeout` period (30 seconds in our example), Nginx will detect the failure. Subsequent requests to yourdomain.com will be routed to the remaining healthy web server. If you stop Nginx on both web servers, users will receive an Nginx error page (e.g., 502 Bad Gateway).

You can monitor Nginx error logs on the load balancer for messages indicating server failures.

Advanced Considerations and Best Practices

Database Backups: Implement a robust, automated backup strategy for your Magento database (MySQL, etc.). This is crucial for any disaster recovery plan, as failover only addresses availability, not data loss.

Configuration Management: Use tools like Ansible, Chef, or Puppet to automate the deployment and configuration of Redis, Sentinel, Nginx, and Magento. This ensures consistency and reduces manual errors.

Monitoring and Alerting: Integrate comprehensive monitoring with tools like Prometheus, Grafana, or Datadog. Set up alerts for Redis Sentinel status changes, Nginx health check failures, and resource utilization on all servers.

DNS Failover: For true disaster recovery, consider a multi-region or multi-datacenter deployment. In such scenarios, DNS-based failover (e.g., using Linode’s DNS Manager with health checks or a third-party service like Cloudflare) can direct traffic to an entirely different region if the primary region becomes unavailable.

Magento Cache Management: Ensure your Magento cache is properly configured to use the highly available Redis cluster. Regularly clear and flush caches as part of your deployment or maintenance routines.

Security: Always use strong passwords for Redis and Sentinel. Restrict access to Redis and Sentinel ports using Linode’s firewall or by binding them to private IP addresses only. Encrypt traffic between your load balancer and web servers using SSL/TLS.

Testing: Regularly test your failover procedures. Schedule downtime during off-peak hours to simulate failures and verify that your automated systems kick in as expected. Document your DR plan and test it at least quarterly.

Recent Posts

Top Categories

Our Products

Our Services