Disaster Recovery 101: Architecting Auto-Failovers for MongoDB and Python Deployments on Linode
Establishing a MongoDB Replica Set for High Availability
A robust disaster recovery strategy for MongoDB hinges on a properly configured replica set. This ensures data redundancy and automatic failover in case of node failure. We’ll focus on a three-node setup for quorum and resilience, deployed on Linode instances.
First, ensure MongoDB is installed on each Linode instance. For this example, we’ll assume three nodes: `mongo-node-1` (primary candidate), `mongo-node-2` (secondary), and `mongo-node-3` (secondary). Each node should have a dedicated data directory, e.g., `/var/lib/mongodb`.
Configuring MongoDB Instances
On each MongoDB server, edit the MongoDB configuration file, typically located at `/etc/mongod.conf`. We need to enable replication and specify a replica set name.
On `mongo-node-1`, `mongo-node-2`, and `mongo-node-3`:
replication: replSetName: "myReplicaSet" net: bindIp: 0.0.0.0 port: 27017 storage: dbPath: /var/lib/mongodb systemLog: destination: file path: /var/log/mongodb/mongod.log logAppend: true processManagement: fork: true pidFilePath: /var/run/mongodb/mongod.pid
After modifying the configuration, restart the MongoDB service on each node:
sudo systemctl restart mongod
Initializing the Replica Set
Connect to one of the MongoDB instances (it doesn’t matter which one initially, as they will synchronize). We’ll use `mongo-node-1` for this step.
mongo --host mongo-node-1 --port 27017
Once connected to the MongoDB shell, initiate the replica set configuration:
rs.initiate(
{
_id: "myReplicaSet",
members: [
{ _id: 0, host: "mongo-node-1:27017" },
{ _id: 1, host: "mongo-node-2:27017" },
{ _id: 2, host: "mongo-node-3:27017" }
]
}
)
You should see output indicating the replica set has been initialized and the members are being added. You can verify the status by running:
rs.status()
Wait until all members show as `PRIMARY` or `SECONDARY` and have synchronized. The `myReplicaSet` name should be consistent across all nodes.
Architecting Python Application for MongoDB Failover
Your Python application needs to be aware of the MongoDB replica set and handle potential connection disruptions gracefully. The PyMongo driver provides built-in support for replica sets.
Connection String Configuration
The key to enabling automatic failover in your Python application is to use a connection string that specifies all members of the replica set and the replica set name. This allows PyMongo to discover the current primary and automatically switch if it becomes unavailable.
Instead of connecting to a single MongoDB instance, use a connection string like this:
from pymongo import MongoClient
# Replace with your actual Linode IP addresses or hostnames
MONGO_URI = "mongodb://mongo-node-1:27017,mongo-node-2:27017,mongo-node-3:27017/?replicaSet=myReplicaSet"
try:
client = MongoClient(MONGO_URI, serverSelectionTimeoutMS=5000) # 5 second timeout
# The ismaster command is cheap and does not require auth.
client.admin.command('ismaster')
print("Successfully connected to MongoDB replica set.")
db = client.mydatabase
# Perform your database operations here
# Example:
# result = db.mycollection.insert_one({"name": "test"})
# print(f"Inserted document with ID: {result.inserted_id}")
except Exception as e:
print(f"Error connecting to MongoDB: {e}")
# Implement your application-level error handling or retry logic here
# For critical applications, you might want to trigger alerts or enter a degraded mode.
finally:
if 'client' in locals() and client:
client.close()
print("MongoDB connection closed.")
The `serverSelectionTimeoutMS` parameter is crucial. It defines how long PyMongo will attempt to find a suitable server (the primary) before raising an exception. A value between 3-10 seconds is generally recommended for production environments.
Handling Connection Errors and Retries
While PyMongo handles the automatic failover between replica set members, your application should still implement robust error handling for transient network issues or prolonged primary unavailability.
from pymongo import MongoClient
from pymongo.errors import ServerSelectionTimeoutError, ConnectionFailure
import time
MONGO_URI = "mongodb://mongo-node-1:27017,mongo-node-2:27017,mongo-node-3:27017/?replicaSet=myReplicaSet"
MAX_RETRIES = 5
RETRY_DELAY_SECONDS = 10
def get_mongo_client():
client = None
for attempt in range(MAX_RETRIES):
try:
client = MongoClient(MONGO_URI, serverSelectionTimeoutMS=5000)
client.admin.command('ismaster') # Check connection
print(f"Successfully connected to MongoDB on attempt {attempt + 1}.")
return client
except (ServerSelectionTimeoutError, ConnectionFailure) as e:
print(f"Connection attempt {attempt + 1} failed: {e}")
if attempt < MAX_RETRIES - 1:
print(f"Retrying in {RETRY_DELAY_SECONDS} seconds...")
time.sleep(RETRY_DELAY_SECONDS)
else:
print("Max retries reached. Could not connect to MongoDB.")
# Consider sending an alert here
return None
except Exception as e:
print(f"An unexpected error occurred: {e}")
return None
return client
# Example usage:
client = get_mongo_client()
if client:
db = client.mydatabase
try:
# Perform database operations
db.mycollection.insert_one({"message": "hello world"})
print("Document inserted successfully.")
except Exception as e:
print(f"Error during database operation: {e}")
finally:
client.close()
print("MongoDB connection closed.")
else:
print("Application cannot proceed without MongoDB connection.")
# Implement fallback logic, e.g., serve cached data, return error to user.
This pattern implements a basic retry mechanism. For more sophisticated retry strategies, consider libraries like `tenacity`.
Automated Failover Monitoring and Alerting
While MongoDB and PyMongo handle the technical failover, you need to monitor the health of your replica set and be alerted when failovers occur. This allows for investigation and potential manual intervention if issues persist.
MongoDB Monitoring Tools
Several tools can help monitor your MongoDB replica set:
- `mongostat` and `mongotop`: Command-line utilities for real-time monitoring of server statistics and resource usage.
- `rs.status()`: As shown earlier, provides detailed information about the replica set state, including which member is primary and the health of others.
- MongoDB Atlas Monitoring: If you were using Atlas, this would be a built-in feature.
- Third-party Monitoring Solutions: Prometheus with the MongoDB Exporter, Datadog, New Relic, etc.
Implementing Custom Alerts
A common approach is to periodically check the replica set status and trigger alerts. This can be done with a simple script that runs via cron.
import pymongo
import requests # For sending alerts, e.g., to Slack or PagerDuty
import os
MONGO_URI = "mongodb://mongo-node-1:27017,mongo-node-2:27017,mongo-node-3:27017/?replicaSet=myReplicaSet"
SLACK_WEBHOOK_URL = os.environ.get("SLACK_WEBHOOK_URL") # Store webhook URL in environment variable
def check_replica_set_health():
client = None
try:
client = pymongo.MongoClient(MONGO_URI, serverSelectionTimeoutMS=3000)
client.admin.command('ismaster') # Check connection
rs_status = client.admin.command('replSetGetStatus')
primary_member = None
secondary_members = []
for member in rs_status['members']:
if member['stateStr'] == 'PRIMARY':
primary_member = member
else:
secondary_members.append(member)
if not primary_member:
alert_message = "MongoDB replica set has no primary member!"
send_alert(alert_message)
return False
# Check if any secondary is not in a healthy state (e.g., STARTUP, ARBITER, UNKNOWN, DOWN)
unhealthy_secondaries = [m for m in secondary_members if m['stateStr'] not in ['SECONDARY', 'SECONDARY_ைகளையும்']]
if unhealthy_secondaries:
unhealthy_details = ", ".join([f"{m['name']} ({m['stateStr']})" for m in unhealthy_secondaries])
alert_message = f"MongoDB replica set has unhealthy secondary members: {unhealthy_details}"
send_alert(alert_message)
return False
print(f"MongoDB replica set healthy. Primary: {primary_member['name']}")
return True
except pymongo.errors.ServerSelectionTimeoutError as e:
alert_message = f"MongoDB connection timeout: {e}. Replica set may be unavailable."
send_alert(alert_message)
return False
except Exception as e:
alert_message = f"An error occurred while checking MongoDB health: {e}"
send_alert(alert_message)
return False
finally:
if client:
client.close()
def send_alert(message):
if SLACK_WEBHOOK_URL:
payload = {"text": f":red_circle: {message}"}
try:
response = requests.post(SLACK_WEBHOOK_URL, json=payload)
response.raise_for_status() # Raise an exception for bad status codes
print(f"Alert sent to Slack: {message}")
except requests.exceptions.RequestException as e:
print(f"Failed to send alert to Slack: {e}")
else:
print(f"ALERT: {message} (Slack webhook URL not configured)")
if __name__ == "__main__":
if not check_replica_set_health():
print("MongoDB health check failed.")
else:
print("MongoDB health check passed.")
To use this script:
- Install the `pymongo` and `requests` libraries:
pip install pymongo requests - Set the `SLACK_WEBHOOK_URL` environment variable with your Slack incoming webhook URL.
- Schedule this script to run periodically using cron (e.g., every 5 minutes).
# Example crontab entry to run every 5 minutes */5 * * * * /usr/bin/python3 /path/to/your/monitor_script.py >> /var/log/mongodb_monitor.log 2>&1
This script checks for the presence of a primary and the health of secondary nodes. If any issues are detected, it sends an alert to Slack. You can adapt `send_alert` to use other notification services like PagerDuty or email.
Linode Specific Considerations
When deploying on Linode, pay attention to network configuration and Linode's infrastructure.
Linode Firewall and Networking
Ensure that your Linode firewall rules allow traffic on port 27017 (or your MongoDB port) between your MongoDB nodes and between your application servers and MongoDB nodes. If your MongoDB nodes are in different Linode data centers or VPCs, ensure proper network connectivity.
# Example: Allowing traffic from app server (192.168.1.100) and other mongo nodes # On mongo-node-1: sudo ufw allow from 192.168.1.100 to any port 27017 sudo ufw allow from mongo-node-2 to any port 27017 sudo ufw allow from mongo-node-3 to any port 27017 # On app server: sudo ufw allow from mongo-node-1 to any port 27017 sudo ufw allow from mongo-node-2 to any port 27017 sudo ufw allow from mongo-node-3 to any port 27017
Using private IP addresses for inter-node communication within a Linode VPC is highly recommended for security and performance.
Linode Node Balancers (Optional but Recommended)
While not strictly necessary for MongoDB's internal failover, a Linode NodeBalancer can be used to provide a single, highly available entry point for your application servers to connect to the MongoDB replica set. However, the NodeBalancer itself does not understand MongoDB's replica set topology. Your application *must* still use the replica set connection string. The NodeBalancer would primarily be for your application servers if they needed to connect to a single endpoint for *other* services, or if you were abstracting the MongoDB connection layer further.
For direct MongoDB access, relying on the replica set connection string in your application is the standard and most effective approach.
Linode Backups
Complement your replica set with Linode's automated backup service. While a replica set protects against node failures, it doesn't protect against accidental data deletion or logical corruption. Regular backups provide an essential safety net.
Configure Linode backups for your MongoDB instances, ensuring they are taken at a frequency that aligns with your Recovery Point Objective (RPO). Test your backup restoration process periodically.
Conclusion
By implementing a MongoDB replica set, configuring your Python application with the correct connection string and error handling, and setting up robust monitoring and alerting, you can achieve a highly available and resilient database layer. This architecture, combined with Linode's infrastructure and backup services, forms a solid foundation for disaster recovery.