Vengala Vinay

Having 9+ Years of Experience in Software Development

Home » Disaster Recovery 101: Architecting Auto-Failovers for MongoDB and Python Deployments on Google Cloud

Disaster Recovery 101: Architecting Auto-Failovers for MongoDB and Python Deployments on Google Cloud

Designing for Resilience: MongoDB Replica Sets and Python Application Failover on GCP

This document outlines a robust disaster recovery strategy for a Python-based application leveraging MongoDB as its primary datastore, deployed on Google Cloud Platform (GCP). The focus is on architecting automated failover mechanisms for both the database and application tiers to minimize downtime during infrastructure failures or maintenance events.

MongoDB Replica Set Configuration for High Availability

A fundamental component of MongoDB’s resilience is the replica set. We’ll configure a multi-node replica set distributed across different GCP availability zones within a single region for optimal fault tolerance against zonal failures. For a production environment, a minimum of three nodes is recommended, with an odd number of voting members to prevent split-brain scenarios.

Consider a setup with three data-bearing nodes (primary, secondary, secondary) and potentially an arbiter if an even number of data-bearing nodes are used and a tie-breaker is needed. For this example, we’ll assume three data-bearing nodes.

GCP Instance Setup

Provision three Compute Engine instances. Ensure they are in the same VPC network and have appropriate firewall rules allowing MongoDB traffic (default port 27017) between instances. Distribute these instances across different zones (e.g., us-central1-a, us-central1-b, us-central1-c).

MongoDB Installation and Configuration

On each instance, install MongoDB. The configuration file (`mongod.conf`) will be crucial. Here’s a sample configuration for one of the nodes:

systemLog:
  destination: file
  path: /var/log/mongodb/mongod.log
  logAppend: true
storage:
  dbPath: /var/lib/mongodb
  journal:
    enabled: true
net:
  port: 27017
  bindIp: 0.0.0.0 # Or specific private IPs for enhanced security
security:
  authorization: enabled
replication:
  replSetName: myReplicaSet
sharding:
  clusterRole: configsvr # If using sharding, otherwise omit
# For a standalone node before initiating replica set
# processManagement:
#   fork: true
#   pidFilePath: /var/run/mongodb/mongod.pid

Repeat this configuration on all three nodes, ensuring the `replSetName` is identical. Start the MongoDB service on each instance:

sudo systemctl start mongod
sudo systemctl enable mongod

Initiating the Replica Set

Connect to one of the MongoDB instances using the `mongo` shell. You’ll need to add at least two members to the replica set before it becomes operational. It’s best practice to add all members.

mongo --host  --port 27017

Inside the `mongo` shell:

rs.initiate(
  {
    _id: "myReplicaSet",
    members: [
      { _id: 0, host: ":27017" },
      { _id: 1, host: ":27017" },
      { _id: 2, host: ":27017" }
    ]
  }
)

Verify the replica set status:

rs.status()

You should see one node as PRIMARY and the others as SECONDARY. The election process will automatically handle failover if the primary becomes unavailable.

Automating Application Failover with GCP Load Balancing and Health Checks

For the Python application tier, we’ll employ GCP’s Load Balancing service, specifically an External HTTP(S) Load Balancer or a Network Load Balancer depending on application needs, coupled with instance groups and health checks. This ensures that traffic is automatically routed away from unhealthy application instances.

Instance Group Configuration

Create a Managed Instance Group (MIG) in GCP. This group will manage multiple identical instances of your Python application. Configure the MIG to span across the same availability zones as your MongoDB nodes, or at least across multiple zones for high availability.

Health Check Setup

Define a health check that your application instances will respond to. This should be a simple HTTP endpoint (e.g., `/healthz`) that returns a 200 OK status code if the application is healthy and able to connect to MongoDB. If the application cannot reach MongoDB, it should return a non-2xx status code (e.g., 503 Service Unavailable).

from flask import Flask, jsonify
from pymongo import MongoClient
from pymongo.errors import ConnectionFailure

app = Flask(__name__)

# Replace with your MongoDB replica set connection string
# Ensure it includes all members for robust connection
MONGO_URI = "mongodb://user:password@:27017,:27017,:27017/?replicaSet=myReplicaSet&authSource=admin"

def get_mongo_client():
    try:
        client = MongoClient(MONGO_URI, serverSelectionTimeoutMS=5000) # Timeout for connection attempts
        client.admin.command('ping') # Quick check to see if server is available
        return client
    except ConnectionFailure as e:
        print(f"Could not connect to MongoDB: {e}")
        return None

@app.route('/healthz')
def health_check():
    client = get_mongo_client()
    if client:
        return jsonify({"status": "healthy"}), 200
    else:
        return jsonify({"status": "unhealthy", "reason": "MongoDB connection failed"}), 503

@app.route('/')
def index():
    client = get_mongo_client()
    if client:
        db = client.mydatabase
        # Example: Fetch a document to ensure read capability
        try:
            doc_count = db.mycollection.count_documents({})
            return jsonify({"message": "Application is running", "document_count": doc_count}), 200
        except Exception as e:
            return jsonify({"message": "Application running, but MongoDB read failed", "error": str(e)}), 500
    else:
        return jsonify({"message": "Application running, but MongoDB connection failed"}), 503

if __name__ == '__main__':
    # Use a production-ready WSGI server like Gunicorn or uWSGI
    # For local testing:
    app.run(host='0.0.0.0', port=5000)

In GCP, create a health check resource configured to probe your application’s health endpoint (e.g., `http://[INSTANCE_IP]:5000/healthz`).

Load Balancer Configuration

Set up an External HTTP(S) Load Balancer. Configure a backend service that uses your MIG and the health check you defined. The load balancer will continuously monitor the health of instances in the MIG. If an instance fails the health check, the load balancer will stop sending traffic to it. When the instance becomes healthy again, it will be automatically added back into the rotation.

For MongoDB, ensure your application’s connection string is configured to use the replica set name and includes multiple members. This allows the MongoDB driver to automatically discover and connect to the current primary, even after a failover event.

Simulating Failures and Testing

Thorough testing is paramount. Implement a testing regimen to validate your automated failover mechanisms.

MongoDB Failover Testing

1. Primary Node Failure: Gracefully shut down the primary MongoDB instance (or simulate a network partition). Observe the replica set status from a secondary node. A new primary should be elected within seconds. Verify that your application can still connect and perform operations.

# On the current primary node:
sudo systemctl stop mongod

2. Network Partition: Block network traffic between nodes to simulate a network failure. Monitor election behavior and application connectivity.

Application Failover Testing

1. Application Instance Failure: Stop the application process on one of the instances. The health check should fail, and the load balancer should stop sending traffic to it. Verify that traffic is routed to healthy instances.

# On one of the application instances:
sudo pkill -f "python your_app.py"

2. Simulated MongoDB Unavailability: Temporarily stop the MongoDB service on all nodes (or block application access to MongoDB). Your application’s health check should start failing, and the load balancer should mark instances as unhealthy. Once MongoDB is back online, the health checks should pass, and instances will be re-added.

Advanced Considerations

Cross-Region Disaster Recovery

For true disaster recovery against region-wide outages, consider a multi-region deployment. This typically involves:

  • MongoDB: Asynchronous replication to a replica set in a different region. This introduces latency and potential data loss during failover.
  • Application: Deploying application instances in multiple regions behind global load balancing (e.g., GCP’s Global External HTTP(S) Load Balancer) with health checks that span regions.
  • Data Synchronization: Implementing robust data synchronization strategies, potentially using tools like MongoDB Atlas’s Global Clusters or custom solutions involving data pipelines.

Automated Recovery Scripts

While GCP’s managed services handle much of the automation, consider custom scripts or Cloud Functions triggered by monitoring alerts (e.g., Cloud Monitoring) to perform specific recovery actions, such as restarting a failed service or initiating a database failover process if automatic election doesn’t occur within a defined SLA.

Monitoring and Alerting

Implement comprehensive monitoring using GCP’s Cloud Monitoring and Cloud Logging. Set up alerts for:

  • MongoDB replica set health (e.g., primary availability, replication lag).
  • Application instance health checks failing.
  • High error rates in application logs.
  • Resource utilization (CPU, memory, disk).

These alerts are critical for both proactive issue detection and for triggering automated recovery workflows.