Vengala Vinay

Having 12+ Years of Experience in Software Development

Home » Disaster Recovery 101: Architecting Auto-Failovers for Redis and Laravel Deployments on OVH

Disaster Recovery 101: Architecting Auto-Failovers for Redis and Laravel Deployments on OVH

Automated Redis Failover with Sentinel and OVH Load Balancers

For applications heavily reliant on Redis, particularly those built with frameworks like Laravel, ensuring high availability through automated failover is paramount. This section details the architecture for achieving this using Redis Sentinel and OVH’s robust load balancing infrastructure.

Redis Sentinel is the de facto standard for Redis high availability. It provides monitoring, notification, and automatic failover. We’ll deploy Sentinel in a quorum-based configuration across multiple availability zones within OVH to prevent split-brain scenarios.

Sentinel Deployment Strategy

A minimum of three Sentinel instances are recommended for a quorum. Deploying them on separate, geographically distinct (within OVH’s available regions/zones) virtual machines or containers ensures resilience against single-point failures. Each Sentinel instance will monitor the primary Redis instance and its replicas.

Sentinel Configuration (`sentinel.conf`)

Here’s a sample `sentinel.conf` file for a Sentinel instance. Adjust `sentinel.conf` on each Sentinel node accordingly. Ensure the `master-name`, `master-ip`, and `master-port` reflect your primary Redis instance.

# sentinel.conf

port 26379
daemonize yes
pidfile /var/run/redis_sentinel.pid
logfile /var/log/redis/sentinel.log

# The name of the master Redis server to monitor.
# This name is arbitrary but must be consistent across all Sentinels.
# The IP and port are the address of the master Redis instance.
# The last argument is the quorum: the number of Sentinels that must agree
# that the master is down for a failover to be initiated.
sentinel monitor mymaster 10.0.0.10 6379 2

# If the master is not available, Sentinels will wait for this amount of time
# before starting to look for a new master.
sentinel down-after-milliseconds mymaster 5000

# Number of Sentinels that must agree that the master is unreachable.
# This is the quorum. A value of 2 means that at least 2 Sentinels must agree.
# For 3 Sentinels, a quorum of 2 is sufficient.
sentinel failover-timeout mymaster 60000

# The number of replicas that can be promoted to master at the same time.
# This is useful for limiting the impact of failover on your application.
sentinel parallel-syncs mymaster 1

# Optional: Specify the password for Redis authentication if applicable.
# sentinel auth-pass mymaster YourRedisPassword

# Optional: Specify the password for Sentinel authentication if applicable.
# sentinel sentinel-auth-pass mymaster YourSentinelPassword

# Optional: Specify the Redis configuration file to use for replicas.
# sentinel replica-config-file mymaster /etc/redis/replica.conf

Redis Deployment and Replication

Deploy your primary Redis instance and at least one replica. For high availability, deploy replicas in different OVH availability zones. Sentinel will automatically detect and manage these replicas.

OVH Load Balancer Integration

The key to seamless application failover lies in abstracting the Redis endpoint. OVH’s Load Balancer (Network Load Balancer or Application Load Balancer) can be configured to point to your Redis instances. Crucially, the load balancer needs to be aware of the *current* primary Redis instance.

Dynamic Backend Configuration with OVH API

Sentinel can be configured to trigger scripts upon failover events. This is where automation truly shines. We can leverage the OVH API to dynamically update the load balancer’s backend pool when a failover occurs.

Sentinel `post-failover` Script Example (Bash)

Create a script that Sentinel will execute after a successful failover. This script will identify the new primary Redis instance and update the OVH Load Balancer configuration.

#!/bin/bash

# Script to update OVH Load Balancer after Redis Sentinel failover

# --- Configuration ---
SENTINEL_MASTER_NAME="mymaster"
OVH_APPLICATION_KEY="YOUR_OVH_APPLICATION_KEY"
OVH_APPLICATION_SECRET="YOUR_OVH_APPLICATION_SECRET"
OVH_CONSUMER_KEY="YOUR_OVH_CONSUMER_KEY"
OVH_LOAD_BALANCER_ID="YOUR_LOAD_BALANCER_ID" # e.g., "lb-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
OVH_FRONTEND_ID="YOUR_FRONTEND_ID" # e.g., "fe-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
OVH_BACKEND_POOL_ID="YOUR_BACKEND_POOL_ID" # e.g., "bp-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
REDIS_PORT="6379"
NEW_PRIMARY_IP="$2" # Sentinel passes the new primary IP as the second argument

# --- Authentication Token Generation ---
# This is a simplified example. In production, manage tokens securely and refresh them.
# For a robust solution, consider a dedicated service account or a more sophisticated token management.
AUTH_URL="https://auth.cloud.ovh.net/v1/auth/tokens"
AUTH_PAYLOAD=$(cat <&2
    exit 1
fi

# --- Update Load Balancer Backend ---
# The OVH API for load balancers can be complex. This example assumes a Network Load Balancer
# and updates the IP address of a backend server. Adjust for your specific LB type and configuration.

# First, we need to find the existing backend server entry to update it.
# This might involve listing backend servers and filtering by IP or a known identifier.
# For simplicity, let's assume we know the backend server ID or can find it.
# In a real-world scenario, you'd likely have a fixed backend server entry for the primary.

# Example: Assuming we are replacing an existing backend server entry.
# You might need to DELETE the old entry and CREATE a new one, or UPDATE an existing one.
# The exact API calls depend on the OVH Load Balancer service and its version.

# Let's assume we need to update an existing backend server configuration.
# We'll need to know the specific backend server's ID within the backend pool.
# This is a placeholder and requires detailed knowledge of your OVH LB setup.

# Example: Update an existing backend server's IP address.
# This is a hypothetical API call. Consult OVH API documentation for exact endpoints and payloads.
# You might need to GET the current backend pool configuration, modify it, and then PUT it back.

# For a Network Load Balancer, you might be updating the 'ip' field of a backend server.
# Let's assume we have a backend server with ID 'bs-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'
# that we want to update.

# --- Placeholder for actual OVH API call to update backend ---
# This part is highly dependent on your specific OVH Load Balancer configuration (e.g., Network LB vs. Application LB)
# and the specific backend server you are targeting.
# You would typically:
# 1. GET the current backend pool configuration.
# 2. Identify the backend server entry corresponding to the old primary.
# 3. Update its IP address to NEW_PRIMARY_IP.
# 4. PUT the modified backend pool configuration back.

# Example structure for updating a backend server (hypothetical):
# BACKEND_SERVER_ID="bs-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
# UPDATE_URL="${OVH_ENDPOINT}/cloud/project/YOUR_PROJECT_ID/network/loadBalancers/${OVH_LOAD_BALANCER_ID}/pools/${OVH_BACKEND_POOL_ID}/servers/${BACKEND_SERVER_ID}"
# UPDATE_PAYLOAD=$(cat <> /var/log/redis/sentinel_failover.log

exit 0

Important Considerations for the Script:

  • OVH API Credentials: Securely manage your OVH API credentials (Application Key, Secret, Consumer Key). Consider using environment variables or a secrets management system.
  • Authentication: The script includes a basic token acquisition mechanism. For production, implement robust token refresh and error handling.
  • Load Balancer Type: The example assumes a Network Load Balancer. If you are using an Application Load Balancer, the API calls and payload structure will differ significantly. Consult the OVH API documentation for your specific load balancer service.
  • Backend Server Identification: The script needs a reliable way to identify the backend server entry in the load balancer that corresponds to the primary Redis instance. This might involve querying the current backend configuration and updating a specific entry, or even deleting and recreating backend entries.
  • Error Handling: Implement comprehensive error handling for API calls and script execution.
  • Testing: Thoroughly test this script in a staging environment before deploying to production. Simulate failovers and verify that the load balancer is updated correctly.

Registering the Script with Sentinel

Edit your `sentinel.conf` file on each Sentinel instance to include the `post-failover` directive:

# sentinel.conf (add this line)
sentinel post-failover-script /path/to/your/ovh_lb_update_script.sh

Ensure the script is executable (`chmod +x /path/to/your/ovh_lb_update_script.sh`) and that the Sentinel process has the necessary permissions to execute it and access the network.

Laravel Application Configuration for Redis Failover

Your Laravel application needs to be configured to connect to the Redis endpoint managed by the OVH Load Balancer. This abstracts away the underlying Redis instance, allowing the application to remain unaware of failovers.

Environment Configuration (`.env`)

In your Laravel application’s `.env` file, configure the Redis connection to point to the IP address and port of your OVH Load Balancer.

REDIS_HOST=your_ovh_load_balancer_ip
REDIS_PASSWORD=null
REDIS_PORT=6379

If you are using Redis authentication, ensure the `REDIS_PASSWORD` is set correctly. The load balancer itself might not require authentication, but it will forward traffic to Redis, which likely does.

Redis Service Configuration (`config/database.php`)

The `config/database.php` file in your Laravel application should be set up to use the environment variables for Redis connection details. The default configuration usually handles this correctly.

<?php

return [

    // ... other configurations

    'redis' => [

        'client' => env('REDIS_CLIENT', 'phpredis'),

        'default' => [
            'host' => env('REDIS_HOST', '127.0.0.1'),
            'password' => env('REDIS_PASSWORD', null),
            'port' => env('REDIS_PORT', 6379),
            'database' => env('REDIS_DB', 0),
        ],

        'cache' => [
            'host' => env('REDIS_HOST', '127.0.0.1'),
            'password' => env('REDIS_PASSWORD', null),
            'port' => env('REDIS_PORT', 6379),
            'database' => env('REDIS_CACHE_DB', 1),
        ],

    ],

];

When a Redis failover occurs, Sentinel will detect it, promote a replica, and trigger the script to update the OVH Load Balancer. The load balancer will then start directing traffic to the new primary Redis instance. Your Laravel application, still pointing to the load balancer’s IP, will seamlessly continue to operate without manual intervention.

Monitoring and Alerting for Redis and Sentinel

A robust disaster recovery strategy is incomplete without comprehensive monitoring and alerting. We need to ensure that both Redis instances and Sentinel are functioning correctly and that failovers are happening as expected.

Key Metrics to Monitor

  • Redis Latency: Track PING response times and command execution latencies.
  • Redis Memory Usage: Monitor `used_memory` and `used_memory_rss` to prevent OOM errors.
  • Redis Connections: Keep an eye on `connected_clients` and `blocked_clients`.
  • Replication Lag: For replicas, monitor `master_repl_offset` and `slave_repl_offset` to ensure they are in sync.
  • Sentinel Status: Monitor the health of each Sentinel instance. Are they reachable? Are they reporting master/replica status correctly?
  • Failover Events: Log and alert on Sentinel’s failover events.
  • OVH Load Balancer Health Checks: Configure health checks on the OVH Load Balancer to verify the availability of the current Redis primary.

Tools and Integrations

Several tools can be integrated for comprehensive monitoring:

  • Prometheus & Grafana: Use `redis_exporter` for Prometheus to scrape Redis metrics. Grafana can then visualize these metrics and trigger alerts based on defined thresholds.
  • Datadog, New Relic, etc.: Commercial APM tools often have Redis integrations that can provide deep insights and alerting capabilities.
  • OVH Control Panel: Utilize OVH’s built-in monitoring and alerting features for your load balancers and other infrastructure components.
  • Custom Scripting: For Sentinel-specific events, leverage Sentinel’s `notify-script` directive to send custom alerts via email, Slack, or other notification channels.

Sentinel `notify-script` Example

This script can be triggered by Sentinel for various events, including failovers. It receives event details as arguments.

#!/bin/bash

# sentinel_notify.sh
# Script to handle notifications from Redis Sentinel

# Arguments passed by Sentinel:
# $1: Event type (e.g., +failover-end, +sdown, +srepl)
# $2: Master name
# $3: Master IP
# $4: Master Port
# $5: Replica IP (if applicable)
# $6: Replica Port (if applicable)
# $7: Number of Sentinels reporting the event

EVENT_TYPE="$1"
MASTER_NAME="$2"
MASTER_IP="$3"
MASTER_PORT="$4"
REPLICA_IP="$5"
REPLICA_PORT="$6"
SENTINEL_COUNT="$7"

LOG_FILE="/var/log/redis/sentinel_notify.log"

echo "$(date) - Event: ${EVENT_TYPE}, Master: ${MASTER_NAME} (${MASTER_IP}:${MASTER_PORT}), Replica: ${REPLICA_IP}:${REPLICA_PORT}, Sentinels: ${SENTINEL_COUNT}" >> "$LOG_FILE"

# Example: Send an alert for a successful failover
if [ "$EVENT_TYPE" == "+failover-end" ]; then
    NEW_PRIMARY_IP="$MASTER_IP" # After failover, MASTER_IP becomes the new primary
    MESSAGE="Redis failover completed for ${MASTER_NAME}. New primary is ${NEW_PRIMARY_IP}:${MASTER_PORT}."
    echo "$MESSAGE" >> "$LOG_FILE"
    # Add your notification logic here (e.g., send email, Slack message)
    # curl -X POST -H 'Content-type: application/json' --data '{"text":"'"$MESSAGE"'"}' YOUR_SLACK_WEBHOOK_URL
fi

# Example: Alert for a master being down
if [ "$EVENT_TYPE" == "+sdown" ]; then
    MESSAGE="Redis master ${MASTER_NAME} (${MASTER_IP}:${MASTER_PORT}) is marked as Subjectively Down."
    echo "$MESSAGE" >> "$LOG_FILE"
fi

# Example: Alert for a master being down (Objectively Down)
if [ "$EVENT_TYPE" == "+odown" ]; then
    MESSAGE="Redis master ${MASTER_NAME} (${MASTER_IP}:${MASTER_PORT}) is marked as Objectively Down. Failover may be imminent."
    echo "$MESSAGE" >> "$LOG_FILE"
fi

exit 0

Add the `notify-script` directive to your `sentinel.conf`:

# sentinel.conf (add this line)
sentinel notify-script /path/to/your/sentinel_notify.sh

By combining automated failover mechanisms with proactive monitoring and alerting, you can build a highly resilient Redis deployment that minimizes downtime and ensures the stability of your Laravel applications on OVH.

Recent Posts

Top Categories

Our Products

Our Services