Vengala Vinay

Having 9+ Years of Experience in Software Development

Home » Disaster Recovery 101: Architecting Auto-Failovers for DynamoDB and Perl Deployments on OVH

Disaster Recovery 101: Architecting Auto-Failovers for DynamoDB and Perl Deployments on OVH

Establishing a Multi-Region DynamoDB Strategy

For critical applications, a single-region DynamoDB deployment is a single point of failure. Architecting for disaster recovery necessitates a multi-region strategy. This involves replicating your DynamoDB tables across geographically distinct AWS regions. DynamoDB Global Tables provide an active-active multi-region solution, automatically handling data replication and conflict resolution. However, for CTOs and VPs of Engineering focused on cost-efficiency and granular control, a more manual, yet robust, approach using DynamoDB Streams and Lambda functions can be implemented. This allows for custom replication logic and failover triggers.

The core components of this custom multi-region setup are:

  • Primary Region DynamoDB Table: The main operational data store.
  • Secondary Region DynamoDB Table: A replica for failover.
  • DynamoDB Streams: Captures item-level changes in the primary table.
  • AWS Lambda Function: Triggered by DynamoDB Streams, responsible for replicating changes to the secondary region.
  • Application-Level Failover Logic: Code within your application or a separate service to detect primary region unavailability and redirect traffic to the secondary region.

Implementing DynamoDB Replication with Lambda and Streams

Let’s outline the setup for replicating data from a primary region (e.g., `us-east-1`) to a secondary region (e.g., `eu-west-1`).

First, ensure your DynamoDB table in the primary region has DynamoDB Streams enabled. We’ll use the NEW_AND_OLD_IMAGES view type to capture all changes.

Enabling DynamoDB Streams

This can be done via the AWS Management Console, AWS CLI, or SDKs. Using the AWS CLI:

aws dynamodb update-table --table-name YourTableName --stream-specification StreamEnabled=true,StreamViewType=NEW_AND_OLD_IMAGES --region us-east-1

Creating the Replication Lambda Function

This Lambda function will be triggered by the stream. It needs to process the stream records and write them to the secondary region’s DynamoDB table. We’ll use Python for this example due to its excellent AWS SDK (Boto3) support.

import json
import boto3
import os

# Initialize DynamoDB clients for both regions
primary_region = os.environ.get('PRIMARY_REGION', 'us-east-1')
secondary_region = os.environ.get('SECONDARY_REGION', 'eu-west-1')
table_name = os.environ.get('TABLE_NAME', 'YourTableName')

dynamodb_primary = boto3.resource('dynamodb', region_name=primary_region)
dynamodb_secondary = boto3.resource('dynamodb', region_name=secondary_region)

primary_table = dynamodb_primary.Table(table_name)
secondary_table = dynamodb_secondary.Table(table_name)

def lambda_handler(event, context):
    for record in event['Records']:
        event_name = record['eventName'] # INSERT, MODIFY, REMOVE

        # We are interested in INSERT and MODIFY for replication
        if event_name in ['INSERT', 'MODIFY']:
            new_image = record['dynamodb']['NewImage']
            # DynamoDB stream images are in a specific format, need to convert
            # For simplicity, assuming direct mapping. In production, handle complex types.
            item_to_replicate = boto3.dynamodb.types.TypeDeserializer().deserialize(new_image)

            try:
                secondary_table.put_item(Item=item_to_replicate)
                print(f"Successfully replicated {event_name} for item: {item_to_replicate.get('id')}") # Assuming 'id' is your partition key
            except Exception as e:
                print(f"Error replicating item: {item_to_replicate.get('id')}. Error: {e}")
                # Implement dead-letter queue or retry mechanism here

        elif event_name == 'REMOVE':
            keys = record['dynamodb']['Keys']
            # Keys are also in stream format, deserialize
            key_to_delete = boto3.dynamodb.types.TypeDeserializer().deserialize(keys)

            try:
                secondary_table.delete_item(Key=key_to_delete)
                print(f"Successfully deleted item with key: {key_to_delete}")
            except Exception as e:
                print(f"Error deleting item with key: {key_to_delete}. Error: {e}")
                # Implement dead-letter queue or retry mechanism here

    return {
        'statusCode': 200,
        'body': json.dumps('Replication process completed.')
    }

Deploy this Lambda function and configure it to be triggered by the DynamoDB Stream of your primary table. Ensure the Lambda function’s IAM role has permissions to read from the DynamoDB stream and write to the DynamoDB table in the secondary region.

Orchestrating Application Failover with Perl

The application layer is where the actual failover decision is made. For a Perl-based deployment, this often involves modifying connection strings or using a load balancer/proxy that can be reconfigured. A common pattern is to have a health check mechanism that monitors the primary DynamoDB endpoint. If the health check fails repeatedly, the application logic (or a supporting script) initiates the failover.

Health Check Mechanism

A simple health check can be a Perl script that attempts a read operation on a known item in the primary DynamoDB table. If this operation times out or returns an error, it’s flagged as unhealthy.

use strict;
use warnings;
use AWS::DynamoDB::V2; # Assuming you're using a Perl AWS SDK

my $primary_region = 'us-east-1';
my $table_name = 'YourTableName';
my $health_check_key = 'health_check_id'; # A specific item for health checks

my $dynamodb = AWS::DynamoDB::V2->new(
    region => $primary_region,
    # Add credentials or IAM role configuration here
);

sub check_primary_health {
    my $start_time = time();
    my $timeout = 5; # seconds

    eval {
        my $result = $dynamodb->get_item({
            TableName => $table_name,
            Key => {
                'id' => { 'S' => $health_check_key } # Adjust key schema as per your table
            },
            ConsistentRead => 1, # Ensure we read the latest
        });

        if ($result && $result->{Item}) {
            print "Primary DynamoDB is healthy.\n";
            return 1;
        } else {
            print "Primary DynamoDB returned no item for health check.\n";
            return 0;
        }
    };
    if ($@) {
        my $error = $@;
        print "Error checking primary DynamoDB health: $error\n";
        return 0;
    }
}

# Example usage:
# if (!check_primary_health()) {
#     print "Primary DynamoDB is unhealthy. Initiating failover...\n";
#     # Call failover procedure
# }

Failover Orchestration Script

This script would be responsible for reconfiguring the application to point to the secondary region. This could involve:

  • Updating a configuration file that the Perl application reads.
  • Triggering an API call to a load balancer (e.g., AWS ELB, HAProxy) to change its backend targets.
  • Sending a signal to a service discovery mechanism.

For a direct application configuration change, a Perl script might look like this:

use strict;
use warnings;
use File::Slurp; # For easy file reading/writing

my $config_file = '/etc/myapp/config.ini'; # Path to your application's config
my $secondary_region = 'eu-west-1';
my $new_db_endpoint = "dynamodb.$secondary_region.amazonaws.com"; # Or your custom endpoint if applicable

sub perform_failover {
    print "Performing failover to secondary region: $secondary_region\n";

    # 1. Update application configuration
    my $config_content = read_file($config_file);
    if ($config_content =~ s/region\s*=\s*us-east-1/$secondary_region/g) {
        write_file($config_file, $config_content);
        print "Updated config file $config_file with region $secondary_region.\n";
    } else {
        print "Could not find or update region in config file.\n";
        # Log error, potentially alert
        return 0;
    }

    # 2. Signal application to reload configuration (if necessary)
    # This depends on your application's architecture.
    # Example: send a SIGHUP to the main application process.
    # my $pid = get_app_pid(); # Function to get your app's PID
    # if ($pid) {
    #     kill(SIGHUP, $pid) or print "Failed to send SIGHUP to PID $pid: $!\n";
    # }

    # 3. Update DNS or Load Balancer (if applicable)
    # This would typically involve calling external APIs (e.g., AWS Route 53, HAProxy API)
    # For simplicity, this example focuses on config file update.

    print "Failover initiated. Application should now use $secondary_region.\n";
    return 1;
}

# Example usage:
# if (!check_primary_health()) {
#     if (perform_failover()) {
#         print "Failover successful.\n";
#     } else {
#         print "Failover failed.\n";
#         # Implement escalation procedures
#     }
# }

OVH Specific Considerations and HAProxy Integration

When deploying on OVH, you might be using their Public Cloud instances and potentially their managed HAProxy service or self-hosted HAProxy. The failover strategy needs to account for this infrastructure.

HAProxy as a Failover Layer

HAProxy can act as a crucial layer for abstracting the database endpoint from your application. Instead of the application directly connecting to DynamoDB, it connects to HAProxy, which then forwards requests. This makes failover much cleaner.

You would configure HAProxy with two backend server groups: one pointing to the primary region’s DynamoDB endpoint (or an intermediary service in that region) and another to the secondary. Health checks are configured on HAProxy to monitor the availability of the primary backend.

HAProxy Configuration Example

This is a simplified example. In a real-world scenario, you’d likely have a more complex setup, possibly involving multiple HAProxy instances for high availability of the proxy layer itself.

frontend http_app
    bind *:80
    mode http
    default_backend app_backends

backend app_backends
    mode http
    balance roundrobin
    option httpchk GET /health # Assuming your app has a /health endpoint
    server app_primary 192.168.1.10:80 check # Your primary application server
    server app_secondary 192.168.1.11:80 check # Your secondary application server

# This is where the DynamoDB endpoint would be managed.
# If your Perl app connects directly to DynamoDB, HAProxy might not be involved at the DB layer.
# If you have a custom API layer that abstracts DynamoDB, HAProxy would point to that API layer.

# For direct DynamoDB failover, the application logic (Perl script) would reconfigure
# the application's internal AWS SDK settings or environment variables.
# If you were using a service like RDS, HAProxy could manage RDS endpoint failover.
# For DynamoDB, the application itself must be aware of the region change.

The key here is that the Perl application’s configuration for the AWS SDK (region, endpoint if custom) needs to be updated. The health check script would run periodically, and if it detects an issue, it would trigger the Perl failover script to update the application’s configuration file and signal it to reload.

OVH Network and Security Groups

Ensure that your OVH security groups and network configurations allow traffic from your application servers to the DynamoDB endpoints in both regions. When failing over, the application will need to reach the DynamoDB endpoint in the secondary region. This typically involves allowing outbound traffic to the AWS region’s DynamoDB service endpoint (e.g., `dynamodb.eu-west-1.amazonaws.com`).

Automated Failover Workflow Summary

1. Continuous Monitoring: A background Perl script (or a cron job executing a Perl script) periodically runs the check_primary_health function against the primary DynamoDB region.

2. Health Check Failure: If check_primary_health returns false for a configured number of consecutive checks (to avoid flapping), an alert is triggered, and the failover process begins.

3. Failover Execution: The perform_failover Perl script is invoked. It:

  • Updates the application’s configuration file to point to the secondary AWS region.
  • Signals the main Perl application process to reload its configuration (e.g., via SIGHUP).
  • Optionally, updates DNS records or load balancer configurations if applicable.

4. Application Reconfiguration: The Perl application re-initializes its AWS SDK clients using the new region configuration.

5. Post-Failover Monitoring: A new health check is initiated, now targeting the secondary region’s DynamoDB endpoint. Once confirmed healthy, the failover is considered complete.

6. Replication Continuity: The Lambda function continues to replicate data from the primary region to the secondary region. If the primary region becomes available again, a separate “failback” procedure would be needed, which involves reversing the configuration changes and ensuring data consistency.

Considerations for Failback

Automating failback is as critical as automating failover. This typically involves:

  • Ensuring the primary region is fully operational and healthy.
  • Re-establishing bidirectional replication if your strategy involves active-active or if the secondary region has diverged significantly.
  • Updating application configuration to point back to the primary region.
  • Signaling the application to reload configuration.
  • Verifying data consistency after failback.

For a custom replication setup, failback can be complex. If the primary region was down for an extended period, the secondary region might have diverged. You’d need a strategy to reconcile these differences, potentially involving a period where writes are only allowed in the primary region while the secondary catches up, or a more complex merge strategy.