Vengala Vinay

Having 9+ Years of Experience in Software Development

Home » Disaster Recovery 101: Architecting Auto-Failovers for Redis and Shopify Deployments on OVH

Disaster Recovery 101: Architecting Auto-Failovers for Redis and Shopify Deployments on OVH

Automating Redis Failover with Sentinel and OVH Load Balancers

For applications relying on Redis for caching or session management, high availability is paramount. Implementing an automated failover strategy for Redis, especially within an OVH cloud environment, requires careful consideration of both Redis’s native high-availability features and external load balancing mechanisms. This section details the setup of Redis Sentinel for automatic master-replica failover and how to integrate it with OVH’s Load Balancer service for seamless client redirection.

Redis Sentinel Configuration for High Availability

Redis Sentinel is the official high-availability solution for Redis. It provides monitoring, notification, and automatic failover. A typical Sentinel setup involves at least three Sentinel instances to ensure quorum and avoid split-brain scenarios. Each Sentinel instance needs to be configured to monitor the Redis master and its replicas.

Here’s a sample configuration for a Sentinel instance. This configuration should be replicated across all your Sentinel nodes, adjusting the IP addresses and ports as necessary.

`sentinel.conf` Example

# sentinel.conf

port 26379
daemonize yes
pidfile /var/run/redis_sentinel.pid
logfile /var/log/redis/sentinel.log

# Specify the master to monitor.
# The quorum is the number of Sentinels that must agree that the master is down
# for a failover to be initiated. A quorum of 2 is the minimum for 3 Sentinels.
# The down-after-milliseconds is the time in milliseconds the master must be
# unreachable before it's considered down.
# The failover-timeout is the maximum time in milliseconds to complete a failover.
# The parallel-syncs is the number of replicas that can sync with the new master
# at the same time during a failover.
sentinel monitor mymaster 192.168.1.100 6379 2
sentinel down-after-milliseconds mymaster 5000
sentinel failover-timeout mymaster 60000
sentinel parallel-syncs mymaster 1

# Optional: Configure Sentinel to notify an external script on failover events.
# sentinel notification-script mymaster /path/to/your/notification-script.sh

# Optional: Configure Sentinel to run a script to reconfigure clients after failover.
# This is crucial for automated client redirection.
sentinel client-reconfiguration-script mymaster /path/to/your/reconfigure_clients.sh

Ensure that the Redis master and replica instances are configured to allow connections from Sentinel instances. The `protected-mode no` directive in `redis.conf` might be necessary if Redis is not bound to specific IPs and is accessible from the Sentinel network. Also, ensure the `bind` directive in `redis.conf` allows connections from Sentinel and other Redis instances.

OVH Load Balancer Integration for Client Redirection

While Redis Sentinel handles the internal failover of Redis instances, clients need a stable endpoint to connect to. OVH’s Load Balancer service can act as this stable endpoint. The strategy here is to point the Load Balancer to the current Redis master. When a failover occurs, Sentinel can trigger a script that updates the Load Balancer’s backend pool to point to the new master.

OVH Load Balancer Configuration

You’ll need to create a Load Balancer instance in your OVH control panel. Configure it with a frontend listening on a specific port (e.g., 6379) and a backend pool. Initially, the backend pool should contain only the IP address of your primary Redis master.

Frontend Configuration:

  • Protocol: TCP
  • Port: 6379
  • Load Balancing Method: Round Robin (or Least Connections, though for a single active master, this is less critical)

Backend Pool Configuration:

  • Add a backend server with the IP address of your Redis master (e.g., 192.168.1.100) and port 6379.
  • Configure health checks to ensure the Load Balancer knows if the backend server is responsive. A simple TCP check on port 6379 is usually sufficient for Redis.

Automating Load Balancer Updates with Sentinel’s `client-reconfiguration-script`

The `sentinel client-reconfiguration-script` directive in `sentinel.conf` is key. This script is executed by Sentinel when a failover is completed. It receives arguments detailing the old master and the new master. We can leverage this to interact with the OVH API to update the Load Balancer’s backend pool.

First, you’ll need to obtain API credentials for OVH and install the OVH API client library (e.g., `ovh-python`).

OVH API Credentials Setup

Navigate to your OVH control panel, go to “API Consumers” under “User Settings,” and create a new consumer. Grant it the necessary permissions, particularly for managing Load Balancers.

`reconfigure_clients.sh` Script Example (using Python and OVH API)

This script will be executed by Sentinel. It needs to identify the new master’s IP and update the OVH Load Balancer. We’ll use a Python script for this, triggered by the shell script.

#!/bin/bash

# reconfigure_clients.sh
# Arguments passed by Sentinel:
# $1: Sentinel name (e.g., mymaster)
# $2: Old master IP address
# $3: Old master IP port
# $4: New master IP address
# $5: New master IP port

OLD_MASTER_IP=$2
NEW_MASTER_IP=$4
OVH_LOAD_BALANCER_ID="your_lb_id" # Replace with your OVH Load Balancer ID
OVH_BACKEND_SERVER_ID="your_backend_server_id" # Replace with the ID of the backend server for Redis

# Call the Python script to update the OVH Load Balancer
python /opt/redis/reconfigure_ovh_lb.py \
    --lb-id "$OVH_LOAD_BALANCER_ID" \
    --backend-id "$OVH_BACKEND_SERVER_ID" \
    --new-ip "$NEW_MASTER_IP" \
    --port 6379

exit 0

# /opt/redis/reconfigure_ovh_lb.py
import ovh
import argparse
import sys

def update_ovh_lb_backend(lb_id, backend_id, new_ip, port):
    """
    Updates an OVH Load Balancer backend server IP.
    """
    try:
        client = ovh.Client() # Assumes OVH credentials are set in environment variables or ~/.ovh.conf
    except Exception as e:
        print(f"Error initializing OVH client: {e}", file=sys.stderr)
        sys.exit(1)

    try:
        # Get current backend server details to update
        backend_server_info = client.get(f'/cloud/loadBalancer/{lb_id}/backend/{backend_id}')
        if not backend_server_info:
            print(f"Error: Backend server {backend_id} not found for LB {lb_id}", file=sys.stderr)
            sys.exit(1)

        # Update the backend server with the new IP and port
        # Note: The OVH API might require a PUT request to update specific fields.
        # The exact structure for updating a backend server's IP might vary.
        # This is a conceptual example. You might need to fetch the entire backend
        # configuration, modify the IP, and then PUT it back.
        # For simplicity, let's assume a direct update method exists or we can
        # replace the server. A more robust approach might involve deleting and re-adding.

        # A more realistic approach: Fetch the backend configuration, modify, and PUT.
        # Let's assume we need to update the 'address' field.
        # The API might look like this:
        update_payload = {
            "address": f"{new_ip}:{port}",
            # Other fields might need to be preserved or re-specified
            # "status": backend_server_info.get("status", "active"),
            # "weight": backend_server_info.get("weight", 1),
            # "sslDiscovery": backend_server_info.get("sslDiscovery", False)
        }
        # The actual API endpoint for updating a specific backend server might differ.
        # Consult OVH API documentation for the precise endpoint and payload.
        # Example: client.put(f'/cloud/loadBalancer/{lb_id}/backend/{backend_id}', update_payload)
        # For demonstration, let's simulate the update.
        print(f"Simulating update for LB {lb_id}, Backend {backend_id}: Setting IP to {new_ip}:{port}")

        # A more common pattern for cloud APIs is to update the specific resource.
        # If the API doesn't support direct IP update, you might need to:
        # 1. Get the backend configuration.
        # 2. Remove the old server entry.
        # 3. Add a new server entry with the new IP.
        # 4. Update the backend configuration.

        # Let's assume a PUT operation on the backend resource itself,
        # or a specific endpoint for server updates.
        # Example using a hypothetical PUT to update the server's address:
        # response = client.put(f'/cloud/loadBalancer/{lb_id}/backend/{backend_id}', {"address": f"{new_ip}:{port}"})
        # print(f"OVH API Response: {response}")

        # If the API requires replacing the server within the backend:
        # 1. Get the backend's servers list.
        # 2. Find and remove the old server.
        # 3. Add the new server.
        # 4. Update the backend.
        # This is more complex and depends heavily on the specific API structure.

        # For a practical implementation, you'd consult the OVH API docs:
        # https://docs.ovh.com/gb/en/cloud/api/
        # Search for "cloud loadbalancer" and "backend" operations.

        # As a placeholder, let's just print the action.
        print(f"Successfully updated OVH Load Balancer {lb_id} backend {backend_id} to use {new_ip}:{port}")
        return True

    except Exception as e:
        print(f"Error updating OVH Load Balancer: {e}", file=sys.stderr)
        return False

if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="Update OVH Load Balancer backend with new Redis master IP.")
    parser.add_argument("--lb-id", required=True, help="OVH Load Balancer ID.")
    parser.add_argument("--backend-id", required=True, help="OVH Load Balancer backend server ID.")
    parser.add_argument("--new-ip", required=True, help="New Redis master IP address.")
    parser.add_argument("--port", type=int, required=True, help="Redis port.")

    args = parser.parse_args()

    if update_ovh_lb_backend(args.lb_id, args.backend_id, args.new_ip, args.port):
        sys.exit(0)
    else:
        sys.exit(1)

Important Considerations for the Script:

  • OVH API Credentials: Ensure your OVH API credentials (consumer key, secret key, application key, application secret) are securely configured. The `ovh-python` library typically reads these from environment variables (`OVH_ENDPOINT`, `OVH_APPLICATION_KEY`, `OVH_APPLICATION_SECRET`, `OVH_CONSUMER_KEY`, `OVH_CONSUMER_SECRET`) or from the `~/.ovh.conf` file.
  • OVH API Documentation: The exact API calls for updating a backend server’s IP can be intricate. You must consult the official OVH API documentation for the precise endpoints and payload structures. The example above is illustrative. You might need to fetch the backend configuration, modify the `address` field, and then use a `PUT` request to update it.
  • Error Handling: The script includes basic error handling. In production, you’ll want more robust logging and potentially retry mechanisms.
  • Permissions: The API consumer must have sufficient permissions to modify Load Balancer configurations.
  • Backend Server ID: You need to identify the specific backend server ID within your Load Balancer that corresponds to your Redis master. This can be found in the OVH control panel or via the API.
  • Idempotency: The script should ideally be idempotent, meaning running it multiple times with the same new IP has no adverse effects.

Shopify Deployment Considerations: API-Driven Configuration and Webhooks

Shopify deployments, particularly those involving custom applications or integrations, often rely on external services for configuration, data storage, or event processing. When these external services (like Redis, as discussed) have failover mechanisms, your Shopify application needs to be aware of and adapt to these changes. For Shopify, this typically means leveraging its robust API and webhook system.

Dynamic Configuration Management

If your Shopify app stores connection details for external services (e.g., Redis connection strings), these details must be dynamically updated upon service failover. This can be achieved by:

  • Centralized Configuration Service: Maintain a separate configuration service (e.g., using Consul, etcd, or a simple database) that your Shopify app queries. When Redis fails over, the `client-reconfiguration-script` (or a similar mechanism) updates this central configuration service. Your Shopify app then fetches the updated configuration.
  • Shopify App Settings API: If your application uses Shopify’s App Settings API to store configuration, the failover script can directly update these settings via the Shopify Admin API. This makes the configuration accessible within the Shopify admin interface for your app.

Leveraging Shopify Webhooks for State Changes

While Shopify webhooks primarily deal with store events (orders, products, etc.), you can use them indirectly. If your external service failover impacts functionality that users interact with on Shopify (e.g., a custom product configurator that uses Redis for state), you might need to inform users of potential temporary disruptions or guide them to a fallback mechanism. However, for direct service failover, the primary mechanism is usually the Shopify app itself querying an updated configuration source.

Example: Updating Shopify App Settings

Let’s extend the `reconfigure_clients.sh` script to also update Shopify app settings if your app stores Redis connection details there.

#!/bin/bash

# reconfigure_clients.sh (Extended for Shopify API)

OLD_MASTER_IP=$2
NEW_MASTER_IP=$4
OVH_LOAD_BALANCER_ID="your_lb_id"
OVH_BACKEND_SERVER_ID="your_backend_server_id"

# Update OVH Load Balancer
python /opt/redis/reconfigure_ovh_lb.py \
    --lb-id "$OVH_LOAD_BALANCER_ID" \
    --backend-id "$OVH_BACKEND_SERVER_ID" \
    --new-ip "$NEW_MASTER_IP" \
    --port 6379

# Update Shopify App Settings (if applicable)
# Assumes you have a script or tool to interact with Shopify Admin API
# This might involve using a Shopify API client library (e.g., in Python or Ruby)
# and updating a specific app setting key (e.g., 'redis_connection_string').

# Example using a hypothetical 'update_shopify_setting.py' script:
# This script would need Shopify API credentials (API key, password, shop name).
# It would fetch the current settings, modify the Redis connection string,
# and push the update back.
#
# python /opt/shopify/update_shopify_setting.py \
#     --shop "your-shop-name.myshopify.com" \
#     --api-key "your_shopify_api_key" \
#     --api-password "your_shopify_api_password" \
#     --setting-key "redis_connection_string" \
#     --new-value "redis://${NEW_MASTER_IP}:6379"

echo "Failover process completed for Redis master."

exit 0

The `update_shopify_setting.py` script would use the Shopify Admin API to modify application-specific settings. This ensures that any part of your Shopify application that reads these settings will automatically pick up the new Redis master address after a failover.

Testing and Monitoring the Failover Process

Thorough testing is critical. Simulate Redis master failures by stopping the Redis master process. Monitor:

  • Redis Sentinel logs for failover initiation and completion.
  • The `client-reconfiguration-script` execution and its output (check logs for the Python script).
  • OVH Load Balancer health checks and backend status.
  • Your Shopify application’s ability to connect to the new Redis master.
  • Application functionality that relies on Redis.

Implement comprehensive monitoring for all components: Redis instances, Sentinel instances, and the OVH Load Balancer. Set up alerts for Sentinel failures, Load Balancer health check failures, and application-level Redis connection errors.