Vengala Vinay

Having 12+ Years of Experience in Software Development

Home » Disaster Recovery 101: Architecting Auto-Failovers for MongoDB and Perl Deployments on AWS

Disaster Recovery 101: Architecting Auto-Failovers for MongoDB and Perl Deployments on AWS

Establishing a Multi-AZ MongoDB Replica Set with AWS EC2 and EBS

Achieving automated failover for a MongoDB deployment on AWS necessitates a robust replica set architecture spanning multiple Availability Zones (AZs). This ensures high availability and resilience against single-point failures. We’ll leverage AWS EC2 instances for our MongoDB nodes and EBS volumes for persistent storage, configured as a replica set.

The core components are: a primary node, one or more secondary nodes, and potentially an arbiter. For simplicity and demonstration, we’ll outline a three-node replica set (one primary, one secondary, one arbiter) distributed across two AZs. In a production environment, a minimum of three data-bearing nodes (primary + two secondaries) is strongly recommended for quorum and data redundancy.

EC2 Instance Configuration

We’ll provision EC2 instances with sufficient CPU, RAM, and network throughput for MongoDB. For storage, we’ll attach EBS volumes. The type of EBS volume (e.g., gp3, io2) should be chosen based on I/O performance requirements. Ensure these instances are launched within a VPC with appropriate security group rules allowing MongoDB traffic (default port 27017) between nodes and from application servers.

Example EC2 launch parameters (conceptual):

  • Instance Type: e.g., m5.large or r5.large (depending on memory needs)
  • AMI: Latest Amazon Linux 2 or Ubuntu LTS
  • Storage: Root EBS volume (e.g., 50 GiB gp3) + Data EBS volume (e.g., 200 GiB io2 Block Express for high IOPS)
  • Networking: VPC, Subnet in AZ-a, Security Group allowing port 27017 ingress from internal IPs.
  • Tagging: `Name: mongo-node-1`, `Role: mongodb-replica-set`

EBS Volume Attachment and Mounting

After launching the EC2 instances, attach the data EBS volumes. On each instance, format the volume and mount it to a designated directory (e.g., /data/db).

# On each EC2 instance (after attaching EBS volume)
sudo mkfs -t xfs /dev/xvdf  # Replace /dev/xvdf with your EBS device name
sudo mkdir -p /data/db
sudo mount /dev/xvdf /data/db
echo '/dev/xvdf /data/db xfs defaults,nofail 0 2' | sudo tee -a /etc/fstab # Ensure mount on reboot
sudo chown -R mongodb:mongodb /data/db # Assuming 'mongodb' user/group exists
sudo chmod -R 755 /data/db

MongoDB Installation and Configuration

Install MongoDB Community Edition on each instance. The configuration file (mongod.conf) is crucial for enabling replica set functionality.

Example installation steps (Ubuntu/Debian):

# On each EC2 instance
wget -qO - https://www.mongodb.org/static/pgp/server-5.0.asc | sudo apt-key add -
echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu focal/mongodb-org/5.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-5.0.list
sudo apt update
sudo apt install -y mongodb-org

# Ensure mongod service is stopped before configuration
sudo systemctl stop mongod

Now, configure /etc/mongod.conf on each node. The key parameters are replication.replSetName, storage.dbPath, and net.bindIp. For production, net.bindIp should be set to the instance’s private IP or 0.0.0.0 if secured by security groups.

# /etc/mongod.conf on mongo-node-1 (Primary candidate)
storage:
  dbPath: /data/db
  journal:
    enabled: true
systemLog:
  destination: file
  path: /var/log/mongodb/mongod.log
  logAppend: true
net:
  port: 27017
  bindIp: 127.0.0.1,10.0.1.10 # Replace with instance's private IP and localhost
processManagement:
  fork: true
  pidFilePath: /var/run/mongodb/mongod.pid
  # Example for systemd:
  # systemLog:
  #   path: /var/log/mongodb/mongod.log
  #   logAppend: true
  #   logRotate:
  #     sizeThresholdMB: 256
  #     timeLimitSeconds: 60
  # pidFilePath: /var/run/mongodb/mongod.pid
replication:
  replSetName: myReplicaSet # This name must be consistent across all nodes
sharding:
  clusterRole: configsvr # Only if this node is part of a sharded cluster, otherwise omit
security:
  authorization: enabled # Recommended for production

Repeat this configuration on all nodes, adjusting net.bindIp to each instance’s respective private IP. For the arbiter node, omit the storage section and set replication.arbiterOnly: true.

Initializing the Replica Set

Start the mongod service on all nodes. Then, connect to one of the nodes (preferably the one intended to be the initial primary) using the mongo shell and initiate the replica set configuration.

# On mongo-node-1 (or any node)
sudo systemctl start mongod
sudo systemctl enable mongod # To start on boot

# Connect to mongo shell
mongo --host 10.0.1.10 --port 27017 # Use the private IP of the node

# Inside the mongo shell:
rs.initiate(
  {
    _id : "myReplicaSet",
    members: [
      { _id: 0, host: "10.0.1.10:27017" }, # Primary candidate
      { _id: 1, host: "10.0.1.11:27017" }, # Secondary
      { _id: 2, host: "10.0.1.12:27017", arbiterOnly: true } # Arbiter
    ]
  }
)

Verify the replica set status:

# Inside the mongo shell:
rs.status()

You should see output indicating the primary, secondaries, and arbiter, along with their states (PRIMARY, SECONDARY, ARBITER).

Automating Failover with AWS CloudWatch and Lambda

Manual failover is not an option for true disaster recovery. We need an automated mechanism. AWS CloudWatch Alarms, triggered by specific metrics, can invoke AWS Lambda functions to perform failover actions.

Monitoring MongoDB Health

CloudWatch can monitor various metrics, but for MongoDB, we need to go deeper. We can use the mongostat or mongotop tools, or query MongoDB’s internal status. A common approach is to check if a node is reachable and if it’s currently the primary. If the current primary becomes unreachable or is no longer primary, a failover should be initiated.

We can set up a CloudWatch Agent on each MongoDB instance to collect custom metrics or send logs that can be monitored. Alternatively, a separate monitoring instance can periodically query the replica set status.

A simpler approach for basic health checks is to use CloudWatch’s built-in EC2 metrics (e.g., CPU Utilization, Network In/Out) and combine them with a custom health check endpoint exposed by a small application running alongside MongoDB.

Custom Health Check Endpoint (Perl Example)

A lightweight Perl script using Plack and Starlet can expose an HTTP endpoint that checks MongoDB’s status. This endpoint can be polled by CloudWatch.

#!/usr/bin/perl
use strict;
use warnings;
use Plack::Builder;
use MongoDB;
use MongoDB::Connection;

my $mongo_host = '127.0.0.1'; # Or the instance's private IP
my $mongo_port = 27017;
my $repl_set_name = 'myReplicaSet';

my $app = sub {
    my $env = shift;
    my $response_code = 200;
    my $response_body = "OK";

    eval {
        my $conn = MongoDB::Connection->new(
            host => $mongo_host,
            port => $mongo_port,
            connect_timeout => 5, # seconds
            read_timeout => 5,
        );

        # Check replica set status
        my $admin_db = $conn->get_database('admin');
        my $repl_status = $admin_db->command({ replSetGetStatus => 1 });

        if (exists $repl_status->{ok} && $repl_status->{ok}) {
            my $primary_member = undef;
            foreach my $member (@{$repl_status->{members}}) {
                if ($member->{stateStr} eq 'PRIMARY') {
                    $primary_member = $member;
                    last;
                }
            }

            if (!defined $primary_member) {
                $response_code = 503; # Service Unavailable
                $response_body = "MongoDB: No primary found in replica set '$repl_set_name'";
            }
        } else {
            $response_code = 503;
            $response_body = "MongoDB: Replica set status command failed";
        }
    };

    if ($@) {
        $response_code = 503;
        $response_body = "MongoDB: Connection error - " . $@;
    }

    return [
        $response_code,
        [ 'Content-Type' => 'text/plain' ],
        [ $response_body ]
    ];
};

builder {
    mount '/' => $app;
};

To run this, you’ll need Perl, Plack, and the MongoDB Perl driver installed. Then, run it using a PSGI server like Starlet:

# On each MongoDB instance
sudo cpanm Plack MongoDB::Connection Starlet # Install dependencies

# Save the script as e.g., /opt/mongo_health_check.pl
sudo chmod +x /opt/mongo_health_check.pl

# Run in background (e.g., using systemd or supervisor)
starlet --host 127.0.0.1 --port 8080 /opt/mongo_health_check.pl

CloudWatch Alarm Configuration

Create a CloudWatch Alarm that monitors the HTTP status code of the health check endpoint on each MongoDB instance. If the endpoint returns a 5xx status code (indicating a problem), the alarm should trigger.

Steps:

  • Navigate to CloudWatch -> Alarms -> Create alarm.
  • Select metric: Choose “Custom Namespaces” and find your health check metric (e.g., “HTTP 5xx Errors” if you’ve configured a custom metric or use a synthetic monitor). A more direct way is to use a CloudWatch Synthetics Canaries to ping the HTTP endpoint.
  • Configure the alarm: Set the threshold for “5xx errors” to be >= 1 for a duration of, say, 5 minutes.
  • Actions: Configure the alarm to publish to an SNS topic.

AWS Lambda Function for Failover

The SNS topic will trigger an AWS Lambda function. This function will be responsible for orchestrating the failover process. It needs to identify the current primary, determine a suitable replacement secondary, and promote it.

The Lambda function will need IAM permissions to:

  • Describe EC2 instances (to find the current primary and potential secondaries).
  • Modify EC2 instance attributes (e.g., stop/start if necessary, though not typical for MongoDB failover).
  • Interact with the MongoDB replica set (requires network access to MongoDB nodes, potentially via VPC peering or NAT Gateway/Instance).
import boto3
import pymongo
import os

# Environment variables for configuration
MONGO_REPLICA_SET_NAME = os.environ.get('MONGO_REPLICA_SET_NAME', 'myReplicaSet')
# List of potential MongoDB node private IPs (or DNS names)
# This should be dynamically discovered or managed.
MONGO_NODE_IPS = os.environ.get('MONGO_NODE_IPS', '10.0.1.10,10.0.1.11,10.0.1.12').split(',')
MONGO_PORT = int(os.environ.get('MONGO_PORT', 27017))
MONGO_USERNAME = os.environ.get('MONGO_USERNAME') # If auth is enabled
MONGO_PASSWORD = os.environ.get('MONGO_PASSWORD') # If auth is enabled

ec2 = boto3.client('ec2')

def get_current_primary_ip(nodes):
    """
    Connects to MongoDB nodes to find the current primary.
    Returns the IP of the primary, or None if not found or error.
    """
    for ip in nodes:
        try:
            client = pymongo.MongoClient(
                ip,
                MONGO_PORT,
                serverSelectionTimeoutMS=5000, # 5 seconds timeout
                username=MONGO_USERNAME,
                password=MONGO_PASSWORD,
                authSource='admin' # Or your auth database
            )
            # The ismaster command is cheap and does not require auth.
            client.admin.command('ismaster')
            # Check replica set status to confirm primary role
            repl_status = client.admin.command('replSetGetStatus')
            for member in repl_status.get('members', []):
                if member.get('stateStr') == 'PRIMARY':
                    return ip
        except pymongo.errors.ConnectionFailure as e:
            print(f"Could not connect to {ip}: {e}")
            continue
        except Exception as e:
            print(f"Error checking {ip}: {e}")
            continue
    return None

def promote_secondary(primary_ip, nodes):
    """
    Attempts to promote a secondary node if the primary is down.
    This is a simplified example. In reality, you might need to:
    1. Identify a healthy secondary.
    2. Ensure it has caught up.
    3. Use rs.stepDown() on the old primary if it comes back online.
    """
    if not primary_ip:
        print("No current primary IP provided for stepDown.")
        return False

    print(f"Attempting to step down the old primary: {primary_ip}")
    try:
        client = pymongo.MongoClient(
            primary_ip,
            MONGO_PORT,
            serverSelectionTimeoutMS=5000,
            username=MONGO_USERNAME,
            password=MONGO_PASSWORD,
            authSource='admin'
        )
        client.admin.command({'replSetStepDown': 1})
        print(f"Successfully initiated stepDown on {primary_ip}.")
        return True
    except Exception as e:
        print(f"Failed to step down {primary_ip}: {e}")
        return False

def lambda_handler(event, context):
    print(f"Received event: {event}")

    # Extract information from SNS message
    # This part depends on the exact SNS message format
    message = event['Records'][0]['Sns']['Message']
    message_data = json.loads(message) # Assuming message is JSON

    # Example: Alarm is for a specific instance that went unhealthy
    # You might need to parse the alarm name or dimensions to get the instance ID
    # For simplicity, we'll assume we know which instance is unhealthy or we query all.

    # Find the current primary
    current_primary = get_current_primary_ip(MONGO_NODE_IPS)

    if not current_primary:
        print("Could not determine current primary. Cannot proceed with failover.")
        return {'statusCode': 500, 'body': 'Failed to find primary'}

    print(f"Current primary identified as: {current_primary}")

    # If the alarm triggered on the current primary, attempt to step it down
    # This logic needs refinement: the alarm might trigger because the primary is DOWN,
    # not because it's reachable but unhealthy.
    # A more robust approach:
    # 1. Check if the instance that triggered the alarm is the current primary.
    # 2. If it is, and it's unreachable, then we need to promote a secondary.
    # 3. If it's reachable but unhealthy (e.g., health check endpoint returns 503),
    #    we might still want to step it down.

    # For this example, let's assume the alarm means the primary is DOWN.
    # We need to find a healthy secondary and promote it.
    # MongoDB replica sets automatically elect a new primary if the current one is unavailable.
    # The primary role of this Lambda might be to *ensure* the old primary is stepped down
    # if it comes back online, to prevent split-brain scenarios.

    # Simplified logic: If the alarm triggered, and the current primary is still the same,
    # try to step it down. This is a bit of a race condition.
    # A better approach: The replica set *should* elect a new primary automatically.
    # This Lambda's job is more about notification and potentially cleanup.

    # Let's refine: The alarm indicates a problem. If the current primary is the one
    # that's problematic (e.g., the instance associated with the alarm is the primary),
    # we might want to force a stepDown if it's still reachable.
    # However, if it's unreachable, the replica set will elect a new primary.

    # A more practical Lambda:
    # 1. Get replica set status.
    # 2. Identify the current primary.
    # 3. If the instance that triggered the alarm IS the primary, and it's UNREACHABLE:
    #    - Log this. The replica set should handle election.
    # 4. If the instance that triggered the alarm IS the primary, and it's REACHABLE but UNHEALTHY:
    #    - Attempt rs.stepDown() on it.
    # 5. If the instance that triggered the alarm is a SECONDARY and is UNHEALTHY:
    #    - Log this. It might affect quorum or future elections.

    # For this example, we'll focus on the automatic election.
    # The primary role of this Lambda is to *notify* and potentially *reconfigure*
    # applications if their connection string needs updating (though ideally, they use
    # a replica set-aware driver that handles this).

    print("Replica set should automatically elect a new primary if the old one is unavailable.")
    print("Consider adding logic here to update application connection strings if necessary.")
    print("Or, if the old primary comes back, ensure it steps down gracefully.")

    # Example: If the old primary comes back online and is still the primary, step it down.
    # This requires checking if the instance that triggered the alarm is the *current* primary.
    # This requires parsing the event to get the instance ID.

    # For now, we'll just log and potentially send notifications.
    sns = boto3.client('sns')
    sns.publish(
        TopicArn=os.environ.get('NOTIFICATION_SNS_TOPIC_ARN'),
        Message=f"MongoDB failover event detected. Current primary: {current_primary}. Check logs for details.",
        Subject="MongoDB Failover Alert"
    )

    return {
        'statusCode': 200,
        'body': 'Failover process initiated (or logged).'
    }

Important Considerations for the Lambda Function:

  • Authentication: If MongoDB authentication is enabled, the Lambda function needs credentials. Store these securely in AWS Secrets Manager or Parameter Store and grant the Lambda function IAM permissions to access them.
  • Network Access: The Lambda function must be able to reach your MongoDB instances. This typically means running the Lambda function within your VPC, with appropriate security group rules and potentially a NAT Gateway or VPC Endpoint for access.
  • Replica Set Election: MongoDB’s replica set protocol is designed to automatically elect a new primary if the current primary becomes unavailable. The primary role of your Lambda function might be to *monitor* this process, *notify* administrators, and *ensure* that if the old primary comes back online, it gracefully steps down to avoid split-brain scenarios. The rs.stepDown() command is crucial here.
  • Application Reconnection: Ensure your application clients are configured to use replica set connection strings and are resilient to primary changes. Drivers typically handle this automatically.
  • Error Handling: Implement robust error handling and logging within the Lambda function.

Integrating Perl Applications with MongoDB Replica Sets

Perl applications interacting with a MongoDB replica set should leverage the MongoDB driver and use a replica set connection string. This allows the driver to be aware of the replica set topology and automatically discover the current primary.

Example connection string:

mongodb://mongo-node-1.example.com:27017,mongo-node-2.example.com:27017,mongo-node-3.example.com:27017/?replicaSet=myReplicaSet&authSource=admin

And the Perl code to connect:

#!/usr/bin/perl
use strict;
use warnings;
use MongoDB;
use MongoDB::Connection;

# Use a replica set connection string
my $mongo_uri = "mongodb://mongo-node-1.example.com:27017,mongo-node-2.example.com:27017,mongo-node-3.example.com:27017/?replicaSet=myReplicaSet&authSource=admin";

# If authentication is enabled, add username and password
# my $mongo_uri = "mongodb://myuser:[email protected]:27017,...";

my $conn;
eval {
    $conn = MongoDB::Connection->new(
        uri => $mongo_uri,
        connect_timeout => 10, # seconds
        read_timeout => 10,
    );
};

if ($@) {
    die "Failed to connect to MongoDB: $@\n";
}

my $db = $conn->get_database('my_application_db');

# Perform operations
my $collection = $db->get_collection('users');
my $result = $collection->insert({ name => 'Test User', email => '[email protected]' });

print "Inserted document with ID: ", $result->{'inserted_id'}, "\n";

# The driver will automatically handle failover and reconnect to the new primary
# when operations are performed after a primary change.

# Example of checking the primary (for informational purposes)
my $admin_db = $conn->get_database('admin');
my $repl_status = $admin_db->command({ replSetGetStatus => 1 });

if (exists $repl_status->{ok} && $repl_status->{ok}) {
    foreach my $member (@{$repl_status->{members}}) {
        if ($member->{stateStr} eq 'PRIMARY') {
            print "Current primary is: ", $member->{name}, "\n";
            last;
        }
    }
}

By using the replica set connection string and a replica set-aware driver, your Perl application will automatically discover the new primary after a failover event, minimizing downtime and manual intervention.

Recent Posts

Top Categories

Our Products

Our Services