Disaster Recovery 101: Architecting Auto-Failovers for MongoDB and PHP Deployments on DigitalOcean

Establishing a MongoDB Replica Set for High Availability

A robust disaster recovery strategy for MongoDB hinges on implementing a replica set. This ensures data redundancy and automatic failover in case of node failure. For this architecture, we’ll assume a three-node replica set deployed across different DigitalOcean availability zones for maximum resilience.

First, ensure MongoDB is installed on your DigitalOcean Droplets. We’ll configure each instance to be part of the replica set. The primary configuration file, typically /etc/mongod.conf, needs specific adjustments.

MongoDB Configuration for Replica Set Members

On each Droplet designated for the replica set, modify the mongod.conf file. Key parameters include:

• replication.replSetName: A unique name for your replica set.
• net.bindIp: Should be set to 0.0.0.0 to allow connections from other nodes and your application servers.
• security.keyFile: A shared secret file for authentication between replica set members.
• storage.dbPath: The directory for MongoDB data files.
• systemLog.path: The path for MongoDB log files.
• systemLog.destination: Set to file.

Here’s an example snippet for /etc/mongod.conf:

replication:
  replSetName: "rs0" # Example replica set name
storage:
  dbPath: "/var/lib/mongodb"
  journal:
    enabled: true
systemLog:
  destination: file
  path: "/var/log/mongodb/mongod.log"
  logAppend: true
net:
  bindIp: "0.0.0.0"
  port: 27017
security:
  authorization: enabled
  keyFile: "/etc/mongo-keyfile" # Path to the shared key file

Generating and Distributing the Keyfile

A secure keyfile is crucial for replica set member authentication. Generate it on one node and distribute it securely to all others. Ensure file permissions are restrictive.

# On the first node:
openssl rand -base64 756 > /etc/mongo-keyfile
chmod 400 /etc/mongo-keyfile
chown mongodb:mongodb /etc/mongo-keyfile

# Securely copy this file to all other nodes (e.g., using scp)
# On each subsequent node:
chmod 400 /etc/mongo-keyfile
chown mongodb:mongodb /etc/mongo-keyfile

Initializing the Replica Set

After configuring and restarting MongoDB on all nodes, connect to one of them using the mongo shell and initiate the replica set.

# Connect to one of the MongoDB instances
mongo

# Inside the mongo shell:
rs.initiate(
  {
    _id: "rs0",
    members: [
      { _id: 0, host: "mongo-node-1.your-domain.com:27017" },
      { _id: 1, host: "mongo-node-2.your-domain.com:27017" },
      { _id: 2, host: "mongo-node-3.your-domain.com:27017" }
    ]
  }
)

Replace mongo-node-X.your-domain.com with the actual hostnames or IP addresses of your Droplets. After initiation, you can check the replica set status with rs.status().

Architecting PHP Application for MongoDB Replica Set Connectivity

Your PHP application needs to be configured to connect to the MongoDB replica set. This involves specifying the replica set name and providing a list of potential hosts. The MongoDB PHP driver will handle discovering the primary and directing operations accordingly.

Connection String Configuration

The connection string is the most critical part. It should include all members of the replica set and the replicaSet option.

<?php

// Configuration for your MongoDB connection
$mongoConfig = [
    'host' => 'mongodb://mongo-node-1.your-domain.com:27017,mongo-node-2.your-domain.com:27017,mongo-node-3.your-domain.com:27017/?replicaSet=rs0&authSource=admin',
    'username' => 'your_mongo_user',
    'password' => 'your_mongo_password',
    'database' => 'your_app_database',
];

try {
    // Create a new MongoDB client
    $client = new MongoDB\Client($mongoConfig['host']);

    // Select the database
    $database = $client->selectDatabase($mongoConfig['database']);

    // Authenticate if necessary (often handled by the connection string with authSource)
    // If not using authSource in the URI, you might need to authenticate explicitly:
    // $database->command(['authenticate' => 1, 'username' => $mongoConfig['username'], 'password' => $mongoConfig['password']]);

    // Example: Get a collection and perform an operation
    $collection = $database->selectCollection('users');
    $result = $collection->insertOne(['name' => 'Test User', 'timestamp' => new MongoDB\BSON\UTCDateTime()]);

    echo "Successfully inserted document with ID: " . $result->getInsertedId() . "\n";

} catch (MongoDB\Driver\Exception\Exception $e) {
    // Log the error and handle it appropriately
    error_log("MongoDB Connection Error: " . $e->getMessage());
    // Depending on your application, you might want to display a user-friendly error
    // or trigger a fallback mechanism.
    die("Database connection failed. Please try again later.");
}

?>

In this example, replicaSet=rs0 tells the driver to connect to a replica set named “rs0”. The driver will discover the current primary node and connect to it. If the primary becomes unavailable, the driver will automatically switch to another available member.

Handling Failover in PHP

The MongoDB PHP driver, by default, attempts to reconnect and discover a new primary if the current one fails. However, for critical applications, you might want to implement more explicit error handling and retry mechanisms.

<?php

// ... (previous connection setup) ...

try {
    // Attempt to perform a read operation
    $user = $collection->findOne(['name' => 'Test User']);
    if ($user) {
        echo "Found user: " . $user['name'] . "\n";
    } else {
        echo "User not found.\n";
    }

    // Attempt to perform a write operation
    $updateResult = $collection->updateOne(
        ['_id' => $result->getInsertedId()],
        ['$set' => ['status' => 'active']]
    );
    echo "Matched: " . $updateResult->getMatchedCount() . ", Modified: " . $updateResult->getModifiedCount() . "\n";

} catch (MongoDB\Driver\Exception\ConnectionTimeoutException $e) {
    error_log("MongoDB Connection Timeout: " . $e->getMessage());
    // Implement retry logic or fallback
    echo "Database temporarily unavailable. Please try again shortly.\n";
    // You might want to trigger an alert here for immediate attention.
} catch (MongoDB\Driver\Exception\ServerException $e) {
    error_log("MongoDB Server Error: " . $e->getMessage());
    // Handle specific server-side errors, e.g., network issues, authentication failures
    echo "A server error occurred. Please contact support.\n";
} catch (MongoDB\Driver\Exception\Exception $e) {
    error_log("General MongoDB Error: " . $e->getMessage());
    echo "An unexpected database error occurred.\n";
}

?>

The driver will automatically attempt to reconnect to other members of the replica set if the primary is unreachable. You can configure read preferences (e.g., reading from secondaries) to distribute load and improve availability during failovers, though writes will always go to the primary.

DigitalOcean Load Balancer and Health Checks

While MongoDB’s replica set handles database-level failover, your PHP application servers need to connect to a stable endpoint. A DigitalOcean Load Balancer can abstract the MongoDB replica set, but it’s crucial to understand its limitations with stateful connections and replica set discovery.

A more robust approach for application servers is to have them directly connect to the replica set using the connection string as shown above. However, if you wish to use a load balancer for your PHP application servers themselves, or to present a single endpoint for monitoring, here’s how you might configure it.

Load Balancer Configuration for PHP Application Servers

Deploy your PHP application on multiple Droplets. Configure a DigitalOcean Load Balancer to distribute traffic across these application servers. This provides high availability for your web application layer.

• Frontend Network: Public IP address of the Load Balancer.
• Frontend Protocol: Typically HTTP or HTTPS.
• Backend Pool: The private IP addresses or hostnames of your PHP application Droplets.
• Health Check: Configure a health check endpoint on your PHP application (e.g., /healthz) that returns a 200 OK status if the application is healthy and can connect to the database.

The health check endpoint is vital. It should not only verify the application process is running but also attempt a quick database query to ensure connectivity to the MongoDB replica set.

<?php
// healthz.php

header('Content-Type: application/json');

// Assume $mongoConfig is loaded from a secure configuration file
// and $client is already instantiated and authenticated as per previous examples.
// For a health check, a simple query is sufficient.

try {
    // Attempt a quick read operation on a small collection or a known document.
    // Using a read preference that might hit a secondary can be faster for health checks.
    $client = new MongoDB\Client('mongodb://mongo-node-1.your-domain.com:27017/?replicaSet=rs0&readPreference=secondary');
    $db = $client->selectDatabase('your_app_database');
    $db->command(['ping' => 1]); // A simple ping command

    echo json_encode(['status' => 'ok', 'message' => 'Database connection successful.']);
    http_response_code(200);

} catch (MongoDB\Driver\Exception\Exception $e) {
    error_log("Health Check DB Error: " . $e->getMessage());
    echo json_encode(['status' => 'error', 'message' => 'Database connection failed.']);
    http_response_code(503); // Service Unavailable
}

?>

Configure your web server (e.g., Nginx) to serve this healthz.php file. Then, set up the DigitalOcean Load Balancer’s health check to point to this endpoint.

Nginx Configuration for Health Check Endpoint

server {
    listen 80;
    server_name your-app.com;
    root /var/www/your-app;
    index index.php index.html index.htm;

    location / {
        try_files $uri $uri/ /index.php?$query_string;
    }

    location ~ \.php$ {
        include snippets/fastcgi-php.conf;
        fastcgi_pass unix:/var/run/php/php7.4-fpm.sock; # Adjust PHP version as needed
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        include fastcgi_params;
    }

    # Health check endpoint
    location = /healthz.php {
        include snippets/fastcgi-php.conf;
        fastcgi_pass unix:/var/run/php/php7.4-fpm.sock; # Adjust PHP version as needed
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        include fastcgi_params;
        access_log off; # Don't log health check requests
    }

    # Deny access to other hidden files
    location ~ /\.ht {
        deny all;
    }
}

With this setup, the Load Balancer will automatically remove unhealthy application Droplets from rotation, ensuring traffic is only sent to functional instances. The MongoDB replica set ensures data availability even if one or more database nodes fail.

Monitoring and Alerting for Proactive Disaster Recovery

Automated failover is only effective if you are aware of failures and can react quickly. Comprehensive monitoring and alerting are non-negotiable.

MongoDB Monitoring Metrics

Utilize tools like Prometheus with the MongoDB exporter, or DigitalOcean’s built-in monitoring, to track key metrics:

• Replica Set Status: Monitor the state of each member (PRIMARY, SECONDARY, ARBITER, etc.).
• Network Latency: High latency between nodes can indicate network issues or impending failures.
• Disk I/O and Usage: Ensure nodes have sufficient resources and are not running out of disk space.
• Oplog Lag: Significant oplog lag on secondaries indicates replication issues.
• Connection Counts: Monitor active and queued connections.
• Query Performance: Slow queries can strain resources.

Application-Level Monitoring

Monitor your PHP application’s health checks. If the health check endpoint consistently returns 503, it’s a strong indicator of a problem, potentially with the database connection.

# Example using Prometheus Alertmanager rules
groups:
- name: mongodb.rules
  rules:
  - alert: MongoDBReplicaSetDown
    expr: mongodb_replica_set_state == 0 # Assuming 0 means DOWN or UNKNOWN
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "MongoDB replica set member {{ $labels.instance }} is down."
      description: "The MongoDB instance {{ $labels.instance }} on replica set {{ $labels.replset }} is reporting a state of 0 (down/unknown)."

  - alert: MongoDBOplogLagging
    expr: mongodb_replication_oplog_lag_seconds > 600 # Oplog lag greater than 10 minutes
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: "MongoDB oplog lagging on {{ $labels.instance }}"
      description: "The oplog on {{ $labels.instance }} is lagging by {{ $value }} seconds."

Configure alerts to notify your operations team via Slack, PagerDuty, or email when critical thresholds are breached. This allows for manual intervention if automatic failover doesn’t complete successfully or if there are underlying issues.