Disaster Recovery 101: Architecting Auto-Failovers for MongoDB and WooCommerce Deployments on Google Cloud

Designing for Resilience: MongoDB Replica Sets and Auto-Failover on Google Cloud

Achieving high availability for critical data stores like MongoDB is paramount for any production application. For WooCommerce, where order processing and customer data are at stake, a robust disaster recovery strategy with automated failover is non-negotiable. This section details the architecture for a self-healing MongoDB deployment on Google Cloud Platform (GCP), leveraging its managed services and native tooling.

We will focus on deploying a MongoDB replica set across multiple GCP availability zones within a single region. This provides high availability within that region. For true disaster recovery across regions, a separate strategy involving cross-region replication and failover orchestration would be required, which is beyond the scope of this initial 101 guide but is a natural next step.

MongoDB Replica Set Configuration for GCP

A MongoDB replica set consists of multiple MongoDB instances, one of which is the primary, and the others are secondaries. The primary handles all write operations, and secondaries replicate the data from the primary. If the primary becomes unavailable, the secondaries elect a new primary. We’ll deploy this on Google Compute Engine (GCE) instances, ensuring each instance resides in a different availability zone for resilience against zone-level failures.

Consider a three-node replica set for simplicity and quorum. For production, a five-node set (3 data-bearing nodes + 2 arbiters, or 5 data-bearing nodes) is recommended for better fault tolerance and election stability. We’ll use three data-bearing nodes for this example.

Instance Setup and Networking

Provision three GCE instances in the same GCP region but in distinct availability zones (e.g., `us-central1-a`, `us-central1-b`, `us-central1-c`). Ensure these instances are within the same Virtual Private Cloud (VPC) network and have static internal IP addresses. Configure firewall rules to allow MongoDB traffic (default port 27017) between these instances and any application servers that need to connect.

MongoDB Installation and Configuration

Install MongoDB Community Edition on each GCE instance. The configuration file (`mongod.conf`) needs to be adjusted for replica set operation. Here’s a sample configuration for one of the nodes:

Node 1 Configuration (`/etc/mongod.conf`)

# mongod.conf

storage:
  dbPath: /var/lib/mongodb
  journal:
    enabled: true
systemLog:
  destination: file
  path: /var/log/mongodb/mongod.log
  logAppend: true
net:
  bindIp: 0.0.0.0  # Bind to all interfaces to be accessible from other nodes and app servers
  port: 27017
processManagement:
  fork: true
  pidFilePath: /var/run/mongodb/mongod.pid
sharding:
  clusterRole: configsvr # If this were a sharded cluster, otherwise omit
replication:
  replSetName: "rs0" # The name of our replica set
security:
  keyFile: /var/lib/mongodb/mongodb-keyfile.pem # For authentication between replica set members
  authorization: enabled
setParameter:
  enableLocalhostAuthBypass: false

Repeat this configuration on all three nodes, ensuring the `bindIp` allows external access and the `replSetName` is identical. The `keyFile` is crucial for secure inter-node communication and authentication. Generate a strong key file and distribute it securely to all nodes, ensuring strict file permissions (e.g., `chmod 400`).

Replica Set Initialization

Once MongoDB is installed and configured on all nodes, start the `mongod` service on each instance. Then, connect to one of the instances using the `mongo` shell and initialize the replica set:

Initializing Replica Set (from one node)

# Connect to MongoDB
mongo

# Inside the mongo shell:
rs.initiate(
  {
    _id : "rs0",
    members: [
      { _id: 0, host: "mongo-node-1.internal:27017" },
      { _id: 1, host: "mongo-node-2.internal:27017" },
      { _id: 2, host: "mongo-node-3.internal:27017" }
    ]
  }
)

Replace `mongo-node-1.internal`, `mongo-node-2.internal`, and `mongo-node-3.internal` with the actual internal DNS names or IP addresses of your GCE instances. After running `rs.initiate()`, the replica set will form. You can check its status with `rs.status()`.

Automated Failover with GCP Load Balancing and Health Checks

While MongoDB’s replica set handles automatic failover internally, applications need a reliable endpoint to connect to. Directing application traffic to the current primary is essential. GCP’s Load Balancing services, specifically the Network Load Balancer (NLB) or Internal TCP/UDP Load Balancer (ILB), can be configured to achieve this.

Using GCP Internal TCP/UDP Load Balancer

An Internal TCP/UDP Load Balancer is ideal for directing traffic within your VPC. It can distribute traffic to backend instances based on health checks. For MongoDB, we need a health check that can determine if an instance is a primary and is healthy.

Health Check Configuration

Create a TCP health check that targets port 27017. This basic check verifies if the port is open. However, for true failover, we need to ensure the load balancer directs traffic *only* to the primary. MongoDB’s replica set protocol itself doesn’t expose a simple HTTP endpoint for “am I primary?”. A common approach is to use a small proxy or a custom health check script.

A more robust solution involves a custom health check mechanism. We can deploy a small application (e.g., a Python script) on each MongoDB node that queries the replica set status and exposes an HTTP endpoint (e.g., `/health`). This endpoint would return `200 OK` only if the instance is the primary and healthy, and `503 Service Unavailable` otherwise.

Custom Health Check Script (Python Example)

from flask import Flask, jsonify
from pymongo import MongoClient
from pymongo.errors import ConnectionFailure

app = Flask(__name__)

# Configure MongoDB connection details
MONGO_URI = "mongodb://localhost:27017/"
REPLICA_SET_NAME = "rs0" # Must match your replica set name

@app.route('/health')
def health_check():
    try:
        client = MongoClient(MONGO_URI, serverSelectionTimeoutMS=5000)
        # The ismaster command is cheap and does not require auth.
        client.admin.command('ismaster')

        # Get replica set status
        rs_status = client.admin.command('replSetGetStatus')

        # Find our member in the status
        current_member = None
        for member in rs_status['members']:
            if member['name'] == client.nodes[0][0] + ':' + str(client.nodes[0][1]): # Match hostname:port
                current_member = member
                break

        if current_member is None:
            return jsonify({"status": "error", "message": "Could not find self in replica set status"}), 503

        # Check if we are the primary and healthy
        if current_member['stateStr'] == 'PRIMARY':
            return jsonify({"status": "ok", "message": "I am the primary"}), 200
        else:
            return jsonify({"status": "secondary", "message": f"I am a {current_member['stateStr']}"}), 503

    except ConnectionFailure as e:
        return jsonify({"status": "error", "message": f"MongoDB connection failed: {e}"}), 503
    except Exception as e:
        return jsonify({"status": "error", "message": f"An unexpected error occurred: {e}"}), 503

if __name__ == '__main__':
    # Run on a port that is accessible by GCP health checks (e.g., 8080)
    # Ensure this port is open in your GCP firewall rules.
    app.run(host='0.0.0.0', port=8080)

Deploy this script on each MongoDB node, install Flask and PyMongo (`pip install Flask pymongo`), and run it. Configure your GCP Internal TCP/UDP Load Balancer with a custom health check pointing to `http://:8080/health`. The load balancer will then only send traffic to instances reporting a `200 OK`.

Backend Service and Forwarding Rule

Create a GCP Backend Service. Add your three MongoDB GCE instances as backends. Associate the custom health check with this backend service. Finally, create a Forwarding Rule that uses this backend service. Applications will connect to the IP address of this forwarding rule.

WooCommerce Integration and Application-Level Considerations

WooCommerce, typically running on PHP, needs to be configured to connect to the load balancer’s IP address. The MongoDB PHP driver (or a compatible ORM/library) should be used.

PHP MongoDB Connection String

Your application’s database connection configuration should point to the load balancer’s IP. If using authentication, include credentials.

<?php
// Example configuration in wp-config.php or a custom plugin

define('DB_HOST', 'LOAD_BALANCER_IP:27017'); // Use the ILB IP address
define('DB_NAME', 'your_database_name');
define('DB_USER', 'your_db_user');
define('DB_PASSWORD', 'your_db_password');
define('DB_CHARSET', 'utf8');
define('DB_COLLATE', '');

// MongoDB connection options for the driver
$mongo_options = [
    'replicaSet' => 'rs0', // Important for failover
    'authSource' => 'admin', // Or your auth database
    'username'   => DB_USER,
    'password'   => DB_PASSWORD,
    // Add other options like tls=true if using TLS
];

// Construct the MongoDB URI
$mongo_uri = sprintf(
    "mongodb://%s@%s/%s",
    DB_USER,
    DB_HOST,
    DB_NAME // Database to authenticate against, can be different from the one used for operations
);

// Example using the MongoDB PHP driver
try {
    $client = new MongoDB\Client($mongo_uri, $mongo_options);
    $db = $client->selectDatabase(DB_NAME); // Select the database for operations

    // Perform a simple operation to test connection
    $collection = $db->selectCollection('test_collection');
    $result = $collection->insertOne(['test' => 'connection', 'timestamp' => new MongoDB\BSON\UTCDateTime()]);

    if ($result->getInsertedCount() === 1) {
        echo "MongoDB connection successful!";
    } else {
        echo "MongoDB connection failed during test operation.";
    }

} catch (MongoDB\Driver\Exception\Exception $e) {
    echo "Error connecting to MongoDB: " . $e->getMessage();
    // Implement fallback or error logging here
}
?>

Crucially, ensure the `replicaSet` option is passed to the MongoDB driver. This tells the driver to be aware of the replica set topology and to use the replica set protocol, which is essential for automatic failover detection and reconnection.

Application Resilience Patterns

Even with automated failover, applications should implement retry mechanisms and handle transient connection errors gracefully. Network partitions or brief periods of unavailability during failover can occur. Libraries like `mongodb-php-library` often have built-in retry logic, but it’s good practice to wrap critical database operations in your own try-catch blocks with exponential backoff.

Monitoring and Alerting

A robust monitoring strategy is vital. Utilize GCP’s Cloud Monitoring to track:

• MongoDB replica set status (primary, secondary, arbiter states).
• Instance health (CPU, memory, disk I/O).
• Network latency between nodes and to the load balancer.
• Application error rates, especially those related to database connectivity.
• The health check endpoint of your custom health check application.

Configure alerts for critical conditions, such as:

• No primary available in the replica set.
• Health check failures for a significant duration.
• High error rates in application logs related to MongoDB.
• Replica lag exceeding acceptable thresholds.

This setup provides a solid foundation for an auto-failover MongoDB deployment on GCP, ensuring your WooCommerce store remains available even in the face of infrastructure failures.