Disaster Recovery 101: Architecting Auto-Failovers for MongoDB and Perl Deployments on Linode

Establishing a MongoDB Replica Set for High Availability

A robust disaster recovery strategy for MongoDB hinges on a well-configured replica set. This ensures data redundancy and automatic failover in case of node failure. We’ll focus on a three-node setup for quorum and resilience, deployed on Linode instances. Each node should have dedicated storage, ideally SSDs for performance.

First, ensure MongoDB is installed on each Linode instance. For this example, we’ll assume Ubuntu 22.04 LTS. The configuration file, typically located at /etc/mongod.conf, needs to be adjusted on each node.

Node 1 (Primary Candidate) Configuration

On the first node, modify /etc/mongod.conf as follows:

# /etc/mongod.conf on node1
storage:
  dbPath: /var/lib/mongodb
  journal:
    enabled: true
systemLog:
  destination: file
  path: /var/log/mongodb/mongod.log
  logAppend: true
net:
  bindIp: 0.0.0.0
  port: 27017
security:
  keyFile: /etc/mongo-keyfile
  authorization: enabled
replication:
  replSetName: rs0
sharding:
  clusterRole: configsvr
  configDB: rs0/node1.yourdomain.com:27019,node2.yourdomain.com:27019,node3.yourdomain.com:27019
# For production, consider disabling this for security
# processManagement:
#   fork: true
#   pidFilePath: /var/run/mongodb/mongod.pid

Crucially, we enable replication with replSetName: rs0. The bindIp: 0.0.0.0 allows connections from other nodes. For security, we’ll set up a keyfile for internal authentication.

Node 2 & 3 (Secondary Candidates) Configuration

Nodes 2 and 3 will have a similar configuration, but without the sharding section. They will act as secondaries in the replica set.

# /etc/mongod.conf on node2 and node3
storage:
  dbPath: /var/lib/mongodb
  journal:
    enabled: true
systemLog:
  destination: file
  path: /var/log/mongodb/mongod.log
  logAppend: true
net:
  bindIp: 0.0.0.0
  port: 27017
security:
  keyFile: /etc/mongo-keyfile
  authorization: enabled
replication:
  replSetName: rs0

Keyfile Generation and Distribution

Generate a keyfile on one node and distribute it securely to all other nodes. Ensure strict file permissions.

# On node1:
openssl rand -base64 756 > /etc/mongo-keyfile
chmod 400 /etc/mongo-keyfile
chown mongodb:mongodb /etc/mongo-keyfile

# Securely copy /etc/mongo-keyfile to node2 and node3
# On node2 and node3:
chmod 400 /etc/mongo-keyfile
chown mongodb:mongodb /etc/mongo-keyfile

Initializing the Replica Set

After configuring and restarting MongoDB on all nodes, connect to the primary candidate (node1) and initiate the replica set.

# Connect to node1's mongo shell
mongo --host node1.yourdomain.com --port 27017

# Inside the mongo shell:
rs.initiate(
  {
    _id: "rs0",
    members: [
      { _id: 0, host: "node1.yourdomain.com:27017" },
      { _id: 1, host: "node2.yourdomain.com:27017" },
      { _id: 2, host: "node3.yourdomain.com:27017" }
    ]
  }
)

# Verify the status
rs.status()

The rs.status() command will show the state of each member. It might take a few moments for all members to sync and establish their roles (PRIMARY, SECONDARY).

Perl Application Integration and Failover Handling

Your Perl application needs to be aware of the MongoDB replica set and handle potential connection issues gracefully. We’ll use the MongoDB Perl driver.

Perl MongoDB Driver Configuration

When connecting, specify the replica set name and provide a comma-separated list of hosts. The driver will automatically discover the current primary.

use MongoDB;
use strict;
use warnings;

my $dsn = "mongodb://node1.yourdomain.com:27017,node2.yourdomain.com:27017,node3.yourdomain.com:27017/?replicaSet=rs0";
my $client;
my $db;

eval {
    $client = MongoDB::MongoClient->new(
        host => $dsn,
        timeout => 5000, # Connection timeout in milliseconds
        pool_size => 10,
        connect_timeout => 5000,
    );
    $db = $client->get_database('your_database');
};
if ($@) {
    die "Failed to connect to MongoDB: $@\n";
}

# Example: Insert a document
my $collection = $db->get_collection('your_collection');
my $result = $collection->insert({ name => "Test Document", timestamp => time });

print "Document inserted successfully.\n";

Implementing Connection Retries and Error Handling

Automatic failover means the driver will attempt to reconnect to a new primary if the current one becomes unavailable. However, your application should implement retry logic for transient network issues or during the brief failover window.

use MongoDB;
use strict;
use warnings;

my $dsn = "mongodb://node1.yourdomain.com:27017,node2.yourdomain.com:27017,node3.yourdomain.com:27017/?replicaSet=rs0";
my $client;
my $db;
my $max_retries = 5;
my $retry_delay = 2; # seconds

for (1..$max_retries) {
    eval {
        $client = MongoDB::MongoClient->new(
            host => $dsn,
            timeout => 5000,
            pool_size => 10,
            connect_timeout => 5000,
        );
        $db = $client->get_database('your_database');
        # Perform a simple operation to confirm connection
        $db->command({ ping => 1 });
        print "Successfully connected to MongoDB.\n";
        last; # Exit loop on success
    };
    if ($@) {
        warn "Attempt $_ failed: $@\n";
        if ($ _ == $max_retries) {
            die "Failed to connect to MongoDB after $max_retries retries.\n";
        }
        sleep $retry_delay;
    }
}

# Proceed with database operations if connection was successful
my $collection = $db->get_collection('your_collection');
# ... your application logic ...

This loop attempts to establish a connection, performing a simple ping command to verify. If it fails, it waits and retries. This pattern is essential for resilience during failover events.

Automated Failover Testing and Monitoring

Regularly testing your failover mechanism is critical. This involves simulating node failures and observing how the replica set and your application react.

Simulating Node Failure

You can simulate a failure by stopping the mongod service on a secondary node, or even the primary node.

# On the node you want to simulate failure for:
sudo systemctl stop mongod

Observe the rs.status() output on one of the remaining nodes. You should see the failed node marked as `DOWN` or `UNREACHABLE`. Within a short period (typically seconds, depending on election timeouts), a new primary should be elected among the remaining healthy nodes.

Monitor your Perl application logs for any connection errors during this period. The retry logic should handle these gracefully, and the application should resume normal operation once a new primary is available.

Monitoring Tools

Leverage Linode’s monitoring capabilities for CPU, memory, and disk I/O. For MongoDB-specific metrics, consider:

MongoDB Atlas Monitoring (if applicable, though we’re on Linode)
Prometheus with MongoDB Exporter: Deploy a Prometheus server and configure the mongodb_exporter to scrape metrics from your replica set.
Nagios/Zabbix: Set up checks for MongoDB service availability and replica set health.

Key metrics to monitor include:

Replica Set Status (Primary, Secondary, Down)
Network Latency between nodes
Disk I/O and Disk Space
CPU and Memory Utilization
Oplog Lag

Securing MongoDB and Network Access

Production deployments require robust security. Ensure:

Authentication Enabled: As configured with the keyfile and authorization: enabled.
Network Isolation: Use Linode’s firewall to restrict access to MongoDB ports (default 27017) only from your application servers and other MongoDB nodes.
TLS/SSL Encryption: Configure MongoDB to use TLS/SSL for encrypted communication between nodes and between the application and the database. This involves generating certificates and configuring mongod.conf with net.ssl options.
Regular Updates: Keep MongoDB and the underlying OS patched.

By implementing a properly configured MongoDB replica set and integrating resilient Perl application logic, you establish a solid foundation for automated failover and disaster recovery on Linode.