Disaster Recovery 101: Architecting Auto-Failovers for MongoDB and Magento 2 Deployments on Linode

Establishing a MongoDB Replica Set for High Availability

A robust disaster recovery strategy for Magento 2 hinges on a highly available MongoDB deployment. We’ll architect this using a MongoDB replica set, ensuring data redundancy and automatic failover. For this example, we’ll assume three MongoDB nodes running on separate Linode instances, each with a static IP address.

First, ensure MongoDB is installed and running on each node. The primary configuration file, typically /etc/mongod.conf, needs to be adjusted on all nodes to enable replica set functionality and network binding.

MongoDB Configuration (`mongod.conf`)

Modify the mongod.conf file on each server. Pay close attention to the replication section and ensure net.bindIp is set to allow connections from other replica set members and your application servers. Using 0.0.0.0 is common in controlled environments, but consider more restrictive IPs for enhanced security.

# mongod.conf

storage:
  dbPath: /var/lib/mongodb
  journal:
    enabled: true
systemLog:
  destination: file
  path: /var/log/mongodb/mongod.log
  logAppend: true
net:
  port: 27017
  bindIp: 0.0.0.0 # Or specific IPs of your replica set members and app servers
processManagement:
  fork: true
  pidFilePath: /var/run/mongodb/mongod.pid
replication:
  replSetName: "mgReplicaSet" # Choose a descriptive name for your replica set
  # Optional: configure election timeouts, etc.

After updating the configuration, restart the MongoDB service on each node:

sudo systemctl restart mongod
sudo systemctl enable mongod

Initializing the Replica Set

Connect to one of the MongoDB instances (it doesn’t matter which one initially, as it will become the primary). Then, initiate the replica set configuration.

# On one of the MongoDB nodes:
mongo

# Inside the mongo shell:
rs.initiate(
  {
    _id: "mgReplicaSet",
    members: [
      { _id: 0, host: "mongodb_node1_ip:27017" },
      { _id: 1, host: "mongodb_node2_ip:27017" },
      { _id: 2, host: "mongodb_node3_ip:27017" }
    ]
  }
)

Replace mongodb_nodeX_ip with the actual IP addresses of your Linode instances. After running rs.initiate(), you can check the status of the replica set on any member:

rs.status()

The output should indicate that one member is PRIMARY and the others are SECONDARY. MongoDB will automatically handle elections if the primary becomes unavailable.

Configuring Magento 2 for MongoDB High Availability

Magento 2’s database connection configuration needs to be aware of the MongoDB replica set. This is achieved by specifying the replica set name and a comma-separated list of all replica set members in the app/etc/env.php file.

`app/etc/env.php` Configuration

Locate or create the app/etc/env.php file in your Magento 2 installation’s root directory. The database configuration section should look like this:

<?php
return [
    'backend' => [
        'frontName' => 'admin_your_secret_path'
    ],
    'crypt' => [
        'key' => 'your_application_encryption_key'
    ],
    'db' => [
        'connection' => [
            'default' => [
                'host' => 'mongodb_node1_ip,mongodb_node2_ip,mongodb_node3_ip', // Comma-separated list of all members
                'dbname' => 'magento_db',
                'username' => 'magento_user',
                'password' => 'magento_password',
                'model' => 'mysqlDb', // This is for MySQL, for MongoDB it's different
                'initStatements' => 'SET NAMES utf8',
                'active' => 1,
                'persistent' => 0
            ],
            // MongoDB specific configuration
            'mongo' => [
                'host' => 'mongodb_node1_ip,mongodb_node2_ip,mongodb_node3_ip', // Same as default host
                'port' => 27017,
                'username' => 'magento_user', // If authentication is enabled on MongoDB
                'password' => 'magento_password', // If authentication is enabled on MongoDB
                'dbname' => 'magento_db',
                'replicaSet' => 'mgReplicaSet', // Crucial: your replica set name
                'options' => [
                    'replicaSet' => 'mgReplicaSet', // Redundant but good practice
                    'readPreference' => 'primaryPreferred', // Or 'secondaryPreferred', 'nearest'
                    'connectTimeoutMS' => 5000,
                    'socketTimeoutMS' => 5000
                ]
            ]
        ],
        'table_prefix' => ''
    ],
    // ... other configurations
];

Important Notes:

• The host parameter in the default connection should list all MongoDB nodes. Magento’s database adapter will use this to discover the replica set.
• The mongo section contains the specific MongoDB connection details, including the vital replicaSet parameter.
• readPreference dictates how read operations are distributed. primaryPreferred is a common choice, directing reads to the primary if available, otherwise to a secondary. Adjust this based on your application’s read/write patterns and latency tolerance.
• Ensure your Magento application servers can reach all MongoDB nodes on port 27017. Firewall rules on Linode must be configured accordingly.

Automating Failover Detection and Recovery

While MongoDB handles replica set failover automatically, your application needs to be resilient to temporary connection interruptions during the failover process. For a truly automated disaster recovery, we need external monitoring and potential intervention mechanisms.

Application-Level Health Checks

Implement periodic health checks from your application servers to the MongoDB replica set. This can be a simple script that attempts a read operation or checks the replica set status.

# Example Python script for health check
import pymongo
from pymongo.errors import ConnectionFailure, OperationFailure

MONGO_HOSTS = "mongodb_node1_ip,mongodb_node2_ip,mongodb_node3_ip"
MONGO_DBNAME = "magento_db"
MONGO_USER = "magento_user"
MONGO_PASS = "magento_password"
MONGO_REPLICA_SET = "mgReplicaSet"

def check_mongo_health():
    try:
        client = pymongo.MongoClient(
            MONGO_HOSTS,
            replicaSet=MONGO_REPLICA_SET,
            readPreference='primary', # Try to connect to primary for health check
            connectTimeoutMS=3000,
            serverSelectionTimeoutMS=5000
        )
        # The ismaster command is cheap and does not require auth.
        client.admin.command('ismaster')
        print("MongoDB replica set is healthy.")
        return True
    except (ConnectionFailure, OperationFailure) as e:
        print(f"MongoDB health check failed: {e}")
        return False
    except Exception as e:
        print(f"An unexpected error occurred: {e}")
        return False

if __name__ == "__main__":
    check_mongo_health()

This script can be run via cron on your Magento web servers. If it consistently fails, it can trigger alerts or initiate further automated actions.

External Monitoring and Alerting

Utilize Linode’s monitoring capabilities or integrate with external services like Prometheus/Grafana or Datadog. Configure alerts for:

• MongoDB service status (e.g., using systemctl is-active mongod).
• Replica set health (e.g., number of secondaries, primary availability).
• Network connectivity to MongoDB ports.

When alerts fire, they should ideally trigger an automated response. This could involve:

• Notifying the operations team.
• Initiating a diagnostic script.
• In severe cases, potentially triggering a pre-defined failover procedure for other components (e.g., load balancer re-routing).

Load Balancer Integration (Nginx/HAProxy)

If you are using a load balancer in front of your Magento web servers, ensure it’s configured to perform health checks on the web servers. If a web server becomes unresponsive (potentially due to MongoDB issues), the load balancer should stop sending traffic to it.

Nginx Configuration Example

# nginx.conf or included file

upstream magento_backend {
    server 192.168.1.10; # Magento Web Server 1
    server 192.168.1.11; # Magento Web Server 2
    server 192.168.1.12; # Magento Web Server 3

    # Health check configuration (requires nginx-plus or a module like nginx-healthcheck-module)
    # For open-source nginx, you might need a separate health check tool or script.
    # Example using a hypothetical health check module:
    # health_check uri=/healthcheck.php interval=5s fails=3 passes=2;
}

server {
    listen 80;
    server_name yourdomain.com;

    location / {
        proxy_pass http://magento_backend;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    # A simple health check endpoint for Nginx to poll
    location = /healthcheck.php {
        # This script should check MongoDB connectivity and return 200 OK if healthy,
        # or a non-200 status code otherwise.
        try_files /healthcheck.php =404;
    }
}

The /healthcheck.php script on your Magento servers should execute the MongoDB health check logic (similar to the Python script above) and exit with a status code of 0 (success) or non-zero (failure).

Automated Recovery Scripts

For more advanced scenarios, consider writing shell scripts that are triggered by monitoring alerts. These scripts could:

• Attempt to restart a failed MongoDB node.
• Reconfigure DNS if a node is permanently lost.
• Trigger a full application redeploy if necessary.

These scripts should be idempotent and carefully tested to avoid unintended consequences. They are typically executed on a separate management server or a dedicated monitoring node.

Testing Your Disaster Recovery Strategy

Regular, rigorous testing is paramount. Simulate failures to validate your setup:

• Simulate Primary Failure: Stop the MongoDB service on the current primary node. Observe if a secondary is elected and if Magento remains accessible.
• Simulate Network Partition: Block network traffic between replica set members. Observe failover behavior and recovery.
• Simulate Node Failure: Shut down one of the secondary nodes. Ensure the replica set continues to operate.
• Application Impact Test: During a simulated failover, measure the downtime experienced by the Magento application.

Document all test results and use them to refine your configurations and scripts. An automated failover system is only as good as its last successful test.