Disaster Recovery 101: Architecting Auto-Failovers for MongoDB and C Deployments on DigitalOcean
Automated MongoDB Failover with DigitalOcean Load Balancers and Kubernetes
Achieving high availability for stateful applications like MongoDB requires a robust disaster recovery strategy, with automated failover being the cornerstone of minimizing downtime. This post details an advanced architecture for deploying MongoDB on DigitalOcean Kubernetes (DOKS) with automated failover, leveraging DigitalOcean’s Load Balancers and a custom health check mechanism.
MongoDB Replica Set Configuration for High Availability
A fundamental prerequisite for automated failover is a properly configured MongoDB replica set. For production environments, a minimum of three nodes is recommended to ensure quorum and prevent split-brain scenarios. We’ll assume a replica set with at least three members, where one member is designated as the primary and the others as secondaries. The arbiter role is generally discouraged in production due to its lack of data redundancy.
The key to automated failover lies in the replica set’s ability to elect a new primary when the current primary becomes unavailable. MongoDB’s internal election process handles this automatically. Our task is to ensure that client applications can seamlessly connect to the *new* primary after a failover event.
Leveraging DigitalOcean Kubernetes for Deployment
DigitalOcean Kubernetes (DOKS) provides a managed Kubernetes environment, simplifying the deployment and management of containerized applications. We will deploy MongoDB as a StatefulSet within DOKS. StatefulSets are crucial for stateful applications as they provide stable network identifiers, persistent storage, and ordered deployment/scaling.
Configuring MongoDB StatefulSet on DOKS
The Kubernetes manifest for our MongoDB StatefulSet needs to be carefully crafted. It should define the replica count, the container image, persistent volume claims, and importantly, readiness and liveness probes. These probes are critical for Kubernetes to understand the health of our MongoDB pods.
Here’s a simplified example of a MongoDB StatefulSet manifest. Note the use of `volumeClaimTemplates` for persistent storage and the `command` to initialize the replica set. In a real-world scenario, you’d likely use a more sophisticated initialization script or an operator.
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: mongodb
spec:
serviceName: "mongodb-headless"
replicas: 3
selector:
matchLabels:
app: mongodb
template:
metadata:
labels:
app: mongodb
spec:
containers:
- name: mongodb
image: mongo:latest
ports:
- containerPort: 27017
name: mongo
command: ["mongod", "--replSet", "rs0", "--bind_ip", "0.0.0.0"]
volumeMounts:
- name: mongo-persistent-storage
mountPath: /data/db
readinessProbe:
exec:
command: ["mongo", "--eval", "db.runCommand('ping').ok"]
initialDelaySeconds: 30
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
livenessProbe:
exec:
command: ["mongo", "--eval", "db.runCommand('ping').ok"]
initialDelaySeconds: 60
periodSeconds: 20
timeoutSeconds: 10
failureThreshold: 5
volumeClaimTemplates:
- metadata:
name: mongo-persistent-storage
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 10Gi
storageClassName: "do-block-storage" # Or your preferred DigitalOcean storage class
Headless Service for Stable Network Identity
A headless service is essential for StatefulSets. It allows each pod to have a stable, unique network identifier. This is crucial for MongoDB’s replica set discovery and inter-node communication. The headless service will not have a ClusterIP and will instead provide DNS records for each pod.
apiVersion: v1
kind: Service
metadata:
name: mongodb-headless
labels:
app: mongodb
spec:
ports:
- port: 27017
targetPort: 27017
name: mongo
clusterIP: None
selector:
app: mongodb
External Access and Automated Failover with DigitalOcean Load Balancer
To enable external access and facilitate automated failover, we’ll deploy a DigitalOcean Load Balancer. This Load Balancer will point to a regular Kubernetes Service of type `LoadBalancer`. The key to automated failover here is how we configure the Load Balancer’s health checks.
Configuring the Kubernetes LoadBalancer Service
This service will expose MongoDB to the outside world. The `externalTrafficPolicy: Local` setting is important. It ensures that traffic is only sent to pods on the same node, preserving the source IP address and preventing potential issues with network address translation (NAT) that could interfere with MongoDB’s internal communication or client connections.
apiVersion: v1
kind: Service
metadata:
name: mongodb-external
annotations:
# DigitalOcean specific annotation for Load Balancer configuration
# This is where we define the health check.
# The 'path' should point to a reliable endpoint that indicates primary status.
# We'll create a custom endpoint for this.
service.beta.kubernetes.io/do-loadbalancer-healthcheck-path: "/healthz/primary"
service.beta.kubernetes.io/do-loadbalancer-healthcheck-interval: "10" # seconds
service.beta.kubernetes.io/do-loadbalancer-healthcheck-timeout: "5" # seconds
service.beta.kubernetes.io/do-loadbalancer-healthcheck-unhealthy-threshold: "2"
service.beta.kubernetes.io/do-loadbalancer-healthcheck-healthy-threshold: "2"
spec:
type: LoadBalancer
externalTrafficPolicy: Local
selector:
app: mongodb
ports:
- protocol: TCP
port: 27017
targetPort: 27017
Custom Health Check Endpoint for Primary Detection
The DigitalOcean Load Balancer needs a reliable way to determine which MongoDB instance is the current primary. Kubernetes’ built-in readiness probes are excellent for pod health but don’t directly tell us about the primary role within a replica set. We need a custom endpoint exposed by our MongoDB pods that reports this status.
We can achieve this by running a small web server (e.g., using Python Flask or Node.js Express) within the MongoDB container or as a sidecar. This web server will query the MongoDB replica set status and return an HTTP 200 OK response only if the local MongoDB instance is the primary. Otherwise, it will return a non-2xx status code (e.g., 503 Service Unavailable).
Here’s a Python Flask example that can be integrated into the MongoDB container or run as a sidecar. This script assumes MongoDB is accessible on `localhost:27017` within the pod.
from flask import Flask, Response
from pymongo import MongoClient
from pymongo.errors import ConnectionFailure
import os
app = Flask(__name__)
# Use environment variables for MongoDB connection details
MONGO_HOST = os.environ.get("MONGO_HOST", "localhost")
MONGO_PORT = int(os.environ.get("MONGO_PORT", 27017))
REPLICA_SET_NAME = os.environ.get("REPLICA_SET_NAME", "rs0")
def is_primary():
try:
client = MongoClient(MONGO_HOST, MONGO_PORT, serverSelectionTimeoutMS=2000)
# The ismaster command is cheap and does not require auth.
client.admin.command('ismaster')
rs_status = client.admin.command('replSetGetStatus', REPLICA_SET_NAME)
for member in rs_status['members']:
if member['stateStr'] == 'PRIMARY':
# Check if the current host is the primary
# This requires knowing the pod's hostname or IP, which can be tricky.
# A simpler approach is to check if the local mongod instance reports PRIMARY.
# The 'ismaster' command itself can reveal this.
master_info = client.admin.command('ismaster')
if master_info['ismaster']:
return True
return False
except ConnectionFailure:
return False
except Exception as e:
print(f"Error checking primary status: {e}")
return False
finally:
if 'client' in locals() and client:
client.close()
@app.route('/healthz/primary')
def healthz_primary():
if is_primary():
return Response("MongoDB is primary", status=200, mimetype='text/plain')
else:
return Response("MongoDB is not primary", status=503, mimetype='text/plain')
if __name__ == '__main__':
# Run on port 8080, accessible within the pod
app.run(host='0.0.0.0', port=8080)
To integrate this into your MongoDB container, you would modify the `command` or `args` in your StatefulSet manifest to start both `mongod` and the Flask application. Alternatively, a sidecar pattern is cleaner, where a separate container in the same pod runs the Flask app and shares the network namespace.
Automated Failover Workflow
1. Primary Failure: The current MongoDB primary pod becomes unresponsive (e.g., due to node failure, network partition, or application crash). Kubernetes’ liveness probes will eventually mark the pod as unhealthy.
2. Health Check Failure: The DigitalOcean Load Balancer, configured with `/healthz/primary`, will stop receiving 200 OK responses from the failed primary pod. It will then stop sending traffic to this unhealthy instance.
3. Replica Set Election: MongoDB’s internal replica set election process will automatically trigger. One of the healthy secondary nodes will be elected as the new primary.
4. New Primary Health Check: The newly elected primary pod will now respond to `/healthz/primary` with a 200 OK status. The DigitalOcean Load Balancer will detect this healthy endpoint and start directing traffic to the new primary.
5. Client Reconnection: Client applications configured to connect via the Load Balancer’s IP address will automatically attempt to connect to the new primary. MongoDB drivers typically have built-in retry mechanisms and replica set connection string support that handle this transition gracefully.
Considerations and Enhancements
Replica Set Initialization: The provided manifest assumes MongoDB is already initialized as a replica set. In a production setup, you’ll need a robust mechanism for initializing the replica set on the first deployment, potentially using an init container or a dedicated Kubernetes operator.
Security: Ensure MongoDB is configured with authentication and authorization. The health check script should ideally connect using a read-only user with appropriate permissions. TLS/SSL encryption for inter-node communication and client connections is also highly recommended.
Monitoring and Alerting: Implement comprehensive monitoring for your MongoDB replica set and DOKS cluster. Tools like Prometheus and Grafana can be integrated to track replica lag, election events, and resource utilization. Set up alerts for critical events.
Storage Class: The `do-block-storage` is a placeholder. Choose a DigitalOcean block storage class that meets your performance and durability requirements. Ensure it supports `ReadWriteOnce` for single-pod access.
Connection Strings: Client applications should use MongoDB connection strings that include all replica set members and specify the `replicaSet` option. This allows drivers to discover the replica set topology and automatically switch to the new primary during failover. Example: mongodb://user:[email protected]:27017,mongo2.example.com:27017/?replicaSet=rs0. When using a Load Balancer, the connection string would point to the LB IP.
Testing Failover: Regularly test your failover mechanism by simulating node failures (e.g., by deleting a MongoDB pod or scaling down the node it’s running on) and verifying that clients can still connect and perform operations without significant interruption.
By combining MongoDB’s native replica set capabilities with DigitalOcean’s managed Kubernetes and Load Balancer services, and a custom health check endpoint, you can architect a highly available and resilient MongoDB deployment with automated failover.