Vengala Vinay

Having 9+ Years of Experience in Software Development

Home » Disaster Recovery 101: Architecting Auto-Failovers for DynamoDB and Shopify Deployments on DigitalOcean

Disaster Recovery 101: Architecting Auto-Failovers for DynamoDB and Shopify Deployments on DigitalOcean

Establishing a Multi-Region DynamoDB Strategy

For critical applications leveraging Amazon DynamoDB, a robust disaster recovery (DR) strategy is paramount. This involves architecting for automatic failover to a secondary region in the event of a primary region outage. While DynamoDB Global Tables offer a managed solution for multi-region replication, understanding the underlying mechanisms and how to orchestrate failover at the application layer is crucial for scenarios where fine-grained control or custom failover logic is required.

The core of a DynamoDB DR strategy revolves around maintaining consistent data across regions and directing application traffic to the healthy region. For this, we’ll focus on a scenario where we have two primary regions: `us-east-1` (primary) and `us-west-2` (secondary). We’ll assume a custom replication mechanism or leverage DynamoDB Streams with a Lambda function to propagate changes. The critical component is the application’s ability to detect an unhealthy region and switch its endpoint.

Application-Level Health Checks and Endpoint Switching

The application layer must be responsible for monitoring the health of the DynamoDB endpoint in each region. This can be achieved through periodic, low-latency health check queries against a dedicated “heartbeat” item in a specific DynamoDB table. If these checks consistently fail for a given region, the application should initiate a failover.

Consider a Python application using the AWS SDK (Boto3). We can implement a simple health check function:

import boto3
from botocore.exceptions import ClientError
import logging
import os

# Configure logging
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')

# Environment variables for region and table names
PRIMARY_REGION = os.environ.get("PRIMARY_REGION", "us-east-1")
SECONDARY_REGION = os.environ.get("SECONDARY_REGION", "us-west-2")
HEALTH_CHECK_TABLE = os.environ.get("HEALTH_CHECK_TABLE", "app-health-status")
HEALTH_CHECK_KEY = os.environ.get("HEALTH_CHECK_KEY", "region_status")
HEALTH_CHECK_VALUE = os.environ.get("HEALTH_CHECK_VALUE", "operational")
FAILOVER_THRESHOLD = int(os.environ.get("FAILOVER_THRESHOLD", 3)) # Number of consecutive failures to trigger failover

# Initialize DynamoDB clients for both regions
try:
    primary_dynamodb = boto3.resource('dynamodb', region_name=PRIMARY_REGION)
    secondary_dynamodb = boto3.resource('dynamodb', region_name=SECONDARY_REGION)
    primary_table = primary_dynamodb.Table(HEALTH_CHECK_TABLE)
    secondary_table = secondary_dynamodb.Table(HEALTH_CHECK_TABLE)
    logging.info(f"DynamoDB clients initialized for {PRIMARY_REGION} and {SECONDARY_REGION}")
except Exception as e:
    logging.error(f"Failed to initialize DynamoDB clients: {e}")
    exit(1)

# Global state for tracking failures
region_failures = {
    PRIMARY_REGION: 0,
    SECONDARY_REGION: 0
}

def check_region_health(region_name: str) -> bool:
    """
    Performs a health check against a specific DynamoDB region.
    Returns True if healthy, False otherwise.
    """
    table = primary_table if region_name == PRIMARY_REGION else secondary_table
    try:
        response = table.get_item(
            Key={HEALTH_CHECK_KEY: region_name}
        )
        item = response.get('Item')
        if item and item.get('status') == HEALTH_CHECK_VALUE:
            logging.info(f"Health check successful for {region_name}")
            return True
        else:
            logging.warning(f"Health check failed for {region_name}: Item not found or status incorrect.")
            return False
    except ClientError as e:
        logging.error(f"DynamoDB ClientError during health check for {region_name}: {e}")
        return False
    except Exception as e:
        logging.error(f"Unexpected error during health check for {region_name}: {e}")
        return False

def update_region_status(region_name: str, status: str):
    """
    Updates the status of a region in the health check table.
    """
    table = primary_table if region_name == PRIMARY_REGION else secondary_table
    try:
        table.put_item(
            Item={
                HEALTH_CHECK_KEY: region_name,
                'status': status,
                'last_updated': boto3.utils.datetime.datetime.utcnow().isoformat()
            }
        )
        logging.info(f"Updated status for {region_name} to '{status}'")
    except ClientError as e:
        logging.error(f"DynamoDB ClientError updating status for {region_name}: {e}")
    except Exception as e:
        logging.error(f"Unexpected error updating status for {region_name}: {e}")

def monitor_regions():
    """
    Monitors health of both regions and triggers failover if necessary.
    """
    global region_failures

    regions = [PRIMARY_REGION, SECONDARY_REGION]
    current_active_region = get_current_active_region() # Assume this function determines the currently active region

    for region in regions:
        if check_region_health(region):
            region_failures[region] = 0 # Reset failure count if healthy
            if region != current_active_region:
                logging.info(f"Region {region} is healthy. Considering failback or failover.")
                # Logic to potentially failback if the primary is healthy again
                if region == PRIMARY_REGION and current_active_region == SECONDARY_REGION:
                    logging.info(f"Primary region {PRIMARY_REGION} is healthy. Initiating failback.")
                    set_active_region(PRIMARY_REGION)
                    current_active_region = PRIMARY_REGION
        else:
            region_failures[region] += 1
            logging.warning(f"Health check failed for {region}. Consecutive failures: {region_failures[region]}")
            if region_failures[region] >= FAILOVER_THRESHOLD:
                logging.error(f"Region {region} has exceeded failure threshold. Initiating failover.")
                if region == PRIMARY_REGION:
                    logging.info(f"Primary region {PRIMARY_REGION} is unhealthy. Failing over to {SECONDARY_REGION}.")
                    set_active_region(SECONDARY_REGION)
                    current_active_region = SECONDARY_REGION
                elif region == SECONDARY_REGION and current_active_region == PRIMARY_REGION:
                    logging.warning(f"Secondary region {SECONDARY_REGION} is unhealthy. Cannot failover from {PRIMARY_REGION}.")
                    # In a real-world scenario, you might have a tertiary region or alert aggressively.
                elif region == SECONDARY_REGION and current_active_region == SECONDARY_REGION:
                    logging.error(f"Both regions are unhealthy or unreachable. Critical failure.")
                    # Trigger critical alerts.

def get_current_active_region() -> str:
    """
    Determines the currently active region. This could be stored in a separate
    configuration table, environment variable, or derived from DNS.
    For simplicity, we'll assume it's stored in the health check table.
    """
    try:
        response = primary_table.get_item(Key={HEALTH_CHECK_KEY: 'active_region'})
        item = response.get('Item')
        if item and 'region' in item:
            return item['region']
        else:
            logging.warning("Active region not found. Defaulting to primary.")
            return PRIMARY_REGION
    except ClientError as e:
        logging.error(f"Error retrieving active region: {e}. Defaulting to primary.")
        return PRIMARY_REGION
    except Exception as e:
        logging.error(f"Unexpected error retrieving active region: {e}. Defaulting to primary.")
        return PRIMARY_REGION

def set_active_region(region_name: str):
    """
    Sets the active region. This is a critical operation that should be
    idempotent and carefully managed.
    """
    try:
        primary_table.put_item(
            Item={
                HEALTH_CHECK_KEY: 'active_region',
                'region': region_name,
                'last_updated': boto3.utils.datetime.datetime.utcnow().isoformat()
            }
        )
        logging.info(f"Successfully set active region to: {region_name}")
        # In a real system, this would also trigger DNS updates or load balancer reconfigurations.
    except ClientError as e:
        logging.error(f"Error setting active region to {region_name}: {e}")
    except Exception as e:
        logging.error(f"Unexpected error setting active region to {region_name}: {e}")

if __name__ == "__main__":
    # This would typically run as a background service or scheduled task.
    # For demonstration, we'll run it once.
    logging.info("Starting region monitoring...")
    # Initialize the health check table if it doesn't exist
    try:
        primary_table.load()
    except ClientError as e:
        if e.response['Error']['Code'] == 'ResourceNotFoundException':
            logging.info(f"Health check table '{HEALTH_CHECK_TABLE}' not found. Creating it in {PRIMARY_REGION}.")
            primary_dynamodb.create_table(
                TableName=HEALTH_CHECK_TABLE,
                KeySchema=[{'AttributeName': HEALTH_CHECK_KEY, 'KeyType': 'HASH'}],
                AttributeDefinitions=[{'AttributeName': HEALTH_CHECK_KEY, 'AttributeType': 'S'}],
                ProvisionedThroughput={'ReadCapacityUnits': 5, 'WriteCapacityUnits': 5}
            )
            # Also create in secondary region
            logging.info(f"Creating health check table '{HEALTH_CHECK_TABLE}' in {SECONDARY_REGION}.")
            secondary_dynamodb.create_table(
                TableName=HEALTH_CHECK_TABLE,
                KeySchema=[{'AttributeName': HEALTH_CHECK_KEY, 'KeyType': 'HASH'}],
                AttributeDefinitions=[{'AttributeName': HEALTH_CHECK_KEY, 'AttributeType': 'S'}],
                ProvisionedThroughput={'ReadCapacityUnits': 5, 'WriteCapacityUnits': 5}
            )
            # Wait for tables to be active (simplified for example)
            import time
            time.sleep(10)
            # Initialize statuses
            update_region_status(PRIMARY_REGION, "operational")
            update_region_status(SECONDARY_REGION, "operational")
            set_active_region(PRIMARY_REGION)
        else:
            raise
    
    monitor_regions()
    logging.info("Region monitoring complete.")

This Python script, when run periodically (e.g., via a cron job or a dedicated microservice), will:

  • Initialize Boto3 clients for both regions.
  • Define a health check table (e.g., app-health-status) with a key like region_status and a value indicating operational status.
  • Periodically query a specific item (e.g., {'region_status': 'us-east-1'}) in each region’s health check table.
  • Track consecutive failures for each region.
  • If a region exceeds a predefined failure threshold (FAILOVER_THRESHOLD), it triggers a failover.
  • The set_active_region function is crucial. In a production system, this would not only update a configuration item but also trigger DNS changes (e.g., updating an AWS Route 53 record) or reconfigure load balancers to point to the healthy region’s application instances.

Orchestrating Failover with DNS and Load Balancers

The application-level health check is only one part of the puzzle. The actual traffic redirection needs to happen at the network edge. This is typically achieved using a combination of DNS and load balancers.

We can use AWS Route 53 with health checks and failover routing policies. Each region would have its own set of application instances behind a regional load balancer (e.g., AWS Application Load Balancer – ALB). Route 53 would then point to these ALBs.

Route 53 Health Checks and Failover Routing

For each region, configure a Route 53 health check that monitors a specific endpoint on your application instances (e.g., /health endpoint). This endpoint should, in turn, query DynamoDB to confirm its operational status.

Then, create a Route 53 record set with a “Failover” routing policy:

Record Name: app.yourdomain.com
Type: A
Routing Policy: Failover
Primary Record:
  Alias Target: ALB for us-east-1 (e.g., dualstack.alb-us-east-1.amazonaws.com)
  Health Check ID: [ID of health check for us-east-1]
Secondary Record:
  Alias Target: ALB for us-west-2 (e.g., dualstack.alb-us-west-2.amazonaws.com)
  Health Check ID: [ID of health check for us-west-2]

When the primary health check fails, Route 53 automatically starts returning the IP addresses for the secondary record. This DNS propagation time is a critical factor in your Recovery Time Objective (RTO).

Application Deployment on DigitalOcean

On DigitalOcean, the principles remain similar, but the specific services change. You’d likely use Droplets, potentially managed by Kubernetes (DOKS), and DigitalOcean Load Balancers.

The application instances in each region would be deployed independently. For example, you might have a Kubernetes cluster in a DigitalOcean region in `NYC3` and another in `SFO3`.

Kubernetes Health Checks and Service Discovery

Within Kubernetes, you’d define livenessProbe and readinessProbe for your application pods. These probes would also need to check DynamoDB connectivity.

apiVersion: v1
kind: Pod
metadata:
  name: my-app
spec:
  containers:
  - name: app-container
    image: your-docker-image
    ports:
    - containerPort: 8080
    livenessProbe:
      httpGet:
        path: /health
        port: 8080
      initialDelaySeconds: 15
      periodSeconds: 20
    readinessProbe:
      httpGet:
        path: /health
        port: 8080
      initialDelaySeconds: 5
      periodSeconds: 10
    env:
    - name: PRIMARY_REGION
      value: "nyc3" # DigitalOcean region identifier
    - name: SECONDARY_REGION
      value: "sfo3"
    - name: HEALTH_CHECK_TABLE
      value: "app-health-status"
    - name: HEALTH_CHECK_KEY
      value: "region_status"
    - name: HEALTH_CHECK_VALUE
      value: "operational"
    - name: FAILOVER_THRESHOLD
      value: "3"
    # AWS credentials would be configured via IAM roles or secrets

The /health endpoint on your application would perform the DynamoDB health check as described in the Python example.

DigitalOcean Load Balancers and DNS

You would provision a DigitalOcean Load Balancer in each region, pointing to the application pods within that region’s Kubernetes cluster. The load balancer would have a public IP address.

For DNS, you would use a third-party DNS provider or DigitalOcean’s DNS. Similar to Route 53, you’d configure A records with failover logic. This might involve:

  • Setting up a primary A record pointing to the IP of the `NYC3` load balancer.
  • Setting up a secondary A record pointing to the IP of the `SFO3` load balancer.
  • Configuring external health checks for these DNS records. If your DNS provider doesn’t offer advanced failover routing, you might need an intermediary service or a custom DNS solution.

Alternatively, and often more robustly, you can use a service like Cloudflare or Akamai’s DNS with their advanced health checking and failover capabilities. These services can monitor your load balancer endpoints and automatically update DNS records.

Data Replication and Consistency

While the failover mechanism handles traffic redirection, ensuring data consistency across regions is critical. For DynamoDB, this typically involves:

DynamoDB Global Tables

The most straightforward and managed approach is to use DynamoDB Global Tables. This feature automatically replicates data changes across multiple regions. When you create a Global Table, DynamoDB handles the replication for you. Failover then becomes primarily an application and DNS routing concern.

If you are using Global Tables, your application’s write operations should target the DynamoDB endpoint in the currently active region. DynamoDB will then replicate the data to other regions. Reads can be served from the closest region for lower latency.

Custom Replication with DynamoDB Streams and Lambda

If Global Tables are not suitable (e.g., due to cost, specific control requirements, or using a non-AWS DynamoDB-compatible database), you can implement custom replication. A common pattern involves:

  • Enabling DynamoDB Streams on your primary table.
  • Creating a Lambda function that is triggered by these streams.
  • The Lambda function processes the stream records (INSERT, MODIFY, REMOVE) and writes them to the corresponding table in the secondary region.
  • Crucially, you need to handle potential conflicts and ensure idempotency.
import json
import boto3
import os

SECONDARY_REGION = os.environ['SECONDARY_REGION']
SECONDARY_TABLE_NAME = os.environ['SECONDARY_TABLE_NAME']
HEALTH_CHECK_TABLE = os.environ.get("HEALTH_CHECK_TABLE", "app-health-status")
HEALTH_CHECK_KEY = os.environ.get("HEALTH_CHECK_KEY", "region_status")
HEALTH_CHECK_VALUE = os.environ.get("HEALTH_CHECK_VALUE", "operational")

dynamodb = boto3.resource('dynamodb')
secondary_table = dynamodb.Table(SECONDARY_TABLE_NAME, region_name=SECONDARY_REGION)

def lambda_handler(event, context):
    # Check if the primary region is healthy before replicating
    # This is a simplified check; a more robust solution would involve
    # a dedicated health check mechanism for the replication process itself.
    try:
        primary_dynamodb_client = boto3.client('dynamodb', region_name=os.environ['AWS_REGION']) # Lambda runs in a region
        health_table_primary = primary_dynamodb_client.Table(HEALTH_CHECK_TABLE)
        response = health_table_primary.get_item(Key={HEALTH_CHECK_KEY: os.environ['AWS_REGION']})
        item = response.get('Item')
        if not item or item.get('status') != HEALTH_CHECK_VALUE:
            print(f"Primary region {os.environ['AWS_REGION']} is not operational. Skipping replication.")
            return {'statusCode': 503, 'body': 'Primary region not operational'}
    except Exception as e:
        print(f"Error checking primary region health: {e}. Proceeding with replication cautiously.")
        # Decide whether to proceed or halt based on your risk tolerance.

    for record in event['Records']:
        if record['eventName'] == 'INSERT' or record['eventName'] == 'MODIFY':
            new_image = record['dynamodb']['NewImage']
            # Transform new_image to match secondary table schema if necessary
            # For simplicity, assuming schema is identical
            try:
                secondary_table.put_item(Item=new_image)
                print(f"Successfully replicated item: {new_image.get('id')}") # Assuming 'id' is a primary key
            except Exception as e:
                print(f"Error replicating item {new_image.get('id')}: {e}")
                # Implement retry logic or dead-letter queue for failed writes
        elif record['eventName'] == 'REMOVE':
            old_image = record['dynamodb']['OldImage']
            # Assuming 'id' is the primary key
            try:
                secondary_table.delete_item(Key={'id': old_image['id']})
                print(f"Successfully deleted item: {old_image.get('id')}")
            except Exception as e:
                print(f"Error deleting item {old_image.get('id')}: {e}")
                # Implement retry logic or dead-letter queue for failed deletes

    return {'statusCode': 200, 'body': 'Replication complete'}

This Lambda function needs to be configured with appropriate IAM permissions to read from DynamoDB Streams and write to the secondary DynamoDB table. It also needs environment variables for the secondary region and table name. The health check within the Lambda is a basic example; a more robust solution would involve a separate mechanism to ensure the replication process itself is healthy.

Considerations for Shopify Deployments

Shopify deployments, especially those involving custom applications or extensive integrations, also require a DR strategy. While Shopify manages its core platform, your custom code, themes, and integrations need protection.

Shopify App Backups

For Shopify apps that store data outside of Shopify’s managed services (e.g., in your own database like DynamoDB, or a separate SQL database), the DR strategies discussed above apply. Regularly back up your app’s configuration, code, and any associated databases.

If your app relies on external services, ensure those services also have DR plans in place. For instance, if your app uses a third-party API, understand its uptime guarantees and potential failure modes.

Theme and Asset Management

Shopify themes and assets are managed by Shopify. However, it’s good practice to:

  • Maintain local copies of your theme code.
  • Use version control (e.g., Git) for all theme modifications.
  • Regularly back up any custom code or scripts deployed via Shopify’s theme editor or custom app extensions.

Integration Points

If your Shopify store integrates with external systems (ERPs, CRMs, fulfillment services), these integration points are potential single points of failure. Ensure:

  • The external systems have their own DR and high availability strategies.
  • Your integration logic is resilient to temporary outages of these external systems (e.g., using queues, retry mechanisms).
  • You have a plan to re-establish integrations if a DR event occurs.

Testing and Validation

A DR plan is only effective if it’s regularly tested. Schedule periodic DR drills to simulate region failures. This should involve:

  • Manually triggering a failover scenario.
  • Verifying that traffic is successfully redirected to the secondary region.
  • Confirming that the application functions correctly and data is accessible.
  • Measuring the time taken for failover (RTO) and data loss (RPO).
  • Documenting the results and identifying areas for improvement.

Automated testing of the failover mechanism itself is also highly recommended. This could involve a separate script that attempts to break the primary region’s health checks and verifies that the DNS or load balancer configuration updates as expected.