Vengala Vinay

Having 9+ Years of Experience in Software Development

Home » Server Monitoring Best Practices: Keeping Your C++ App and MongoDB Clusters Alive on DigitalOcean

Server Monitoring Best Practices: Keeping Your C++ App and MongoDB Clusters Alive on DigitalOcean

Proactive C++ Application Health Checks

For a C++ application running on DigitalOcean, robust health checking is paramount. Beyond simple process existence, we need to verify internal state and responsiveness. A common pattern is to expose an HTTP endpoint that the application itself can hit internally, or that an external monitoring system can query. This endpoint should report on critical subsystems.

Consider a C++ application that manages a connection pool to MongoDB. The health check endpoint should not only confirm the application process is running but also validate the health of its MongoDB connections.

Implementing a C++ Health Check Endpoint

We’ll use a lightweight HTTP server library like cpp-httplib for this example. The health check handler will query internal metrics and MongoDB connectivity.

Example C++ Health Check Handler

Assume you have a class MongoConnectionManager with a method isHealthy() that returns true if active connections are valid and false otherwise.

#include <iostream>
#include <string>
#include "httplib.h" // Assuming cpp-httplib is included

// Forward declaration for the MongoDB manager
class MongoConnectionManager;

// Global instance of the connection manager (for simplicity in example)
MongoConnectionManager* g_mongoManager = nullptr;

// Function to set the global manager
void setMongoManager(MongoConnectionManager* manager) {
    g_mongoManager = manager;
}

// Health check handler function
void handleHealthCheck(const httplib::Request& req, httplib::Response& res) {
    bool appHealthy = true;
    std::string statusMessage = "OK";

    // 1. Check internal application state (e.g., background tasks, caches)
    //    (Placeholder for actual application-specific checks)
    // if (!isBackgroundTaskHealthy()) {
    //     appHealthy = false;
    //     statusMessage = "BackgroundTask unhealthy";
    // }

    // 2. Check MongoDB connectivity
    if (g_mongoManager) {
        if (!g_mongoManager->isHealthy()) {
            appHealthy = false;
            statusMessage = "MongoDB connection pool unhealthy";
        }
    } else {
        appHealthy = false;
        statusMessage = "MongoDB manager not initialized";
    }

    if (appHealthy) {
        res.status = 200; // HTTP OK
        res.set_content(statusMessage, "text/plain");
    } else {
        res.status = 503; // Service Unavailable
        res.set_content(statusMessage, "text/plain");
    }
}

// In your main application setup:
int main() {
    // ... initialize your application ...

    // Initialize MongoDB manager
    // MongoConnectionManager mongoManager;
    // setMongoManager(&mongoManager);
    // mongoManager.initialize(); // Connect to MongoDB

    httplib::Server svr;

    // Register the health check endpoint
    svr.Get("/health", handleHealthCheck);

    // Start the HTTP server on a dedicated port (e.g., 8080)
    // Ensure this port is accessible for monitoring but potentially not public.
    if (!svr.listen("0.0.0.0", 8080)) {
        std::cerr << "Failed to start HTTP server on port 8080" << std::endl;
        return 1;
    }

    // ... rest of your application logic ...

    return 0;
}

This handler returns a 200 OK if all checks pass, and a 503 Service Unavailable otherwise. The status message provides a hint about the failure.

Monitoring C++ Application Health on DigitalOcean

DigitalOcean’s monitoring tools can be leveraged, but for more granular control and integration with external alerting, consider a dedicated monitoring agent or service. A common approach is to use Prometheus with the node_exporter and a custom exporter or a simple curl check.

Using Prometheus with a Blackbox Exporter

The Prometheus Blackbox Exporter is ideal for probing endpoints over various protocols, including HTTP. It allows you to monitor services from an external perspective, simulating user access.

First, deploy the Blackbox Exporter. You can run it as a Docker container or a standalone binary.

# blackbox.yml (configuration for the exporter)
modules:
  http_2xx:
    prober: http
    timeout: 5s
    http:
      method: GET
      # Expect a 200 OK status code for the /health endpoint
      valid_status_codes: [200]
      # Optionally, check for specific content in the response body
      # body_match: "OK"

Then, configure Prometheus to scrape this exporter:

# prometheus.yml (snippet)
scrape_configs:
  - job_name: 'blackbox_cpp_app'
    metrics_path: /probe
    params:
      module: [http_2xx] # Use the module defined in blackbox.yml
    static_configs:
      - targets:
        - http://your_cpp_app_ip:8080/health # Replace with your app's IP/domain and port
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox_exporter_ip:9115 # Replace with your Blackbox Exporter's IP and port

This setup will periodically query your C++ application’s `/health` endpoint. Prometheus will record the success or failure of these probes, which can then be visualized in Grafana and used for alerting.

MongoDB Cluster Monitoring Best Practices

Monitoring MongoDB clusters, especially on a cloud platform like DigitalOcean, requires a multi-faceted approach. We need to track resource utilization, query performance, replication status, and overall cluster health.

Key MongoDB Metrics to Monitor

  • Resource Utilization: CPU, Memory, Disk I/O, Network Traffic (per node).
  • Query Performance: Query execution time, slow queries, read/write operations per second.
  • Replication Status: oplog lag, replica set member states (PRIMARY, SECONDARY, ARBITER), network latency between members.
  • Connection Management: Number of active connections, connection pool usage (from your application’s perspective).
  • Storage: Disk space usage, WiredTiger cache usage, data size.
  • Operations: Inserts, updates, deletes, queries per second.
  • Errors: Network errors, authentication failures, assertion failures.

Leveraging MongoDB’s Built-in Tools

MongoDB provides several command-line tools and database commands for introspection:

# Check replica set status
rs.status()

# Get server status (resource usage, connections, etc.)
db.serverStatus()

# Get database statistics (disk usage, document counts)
db.stats()

# Get collection statistics
db.collection.stats()

# List slow operations (requires profiling to be enabled)
db.system.profile.find({ op: { $in: ["query", "update", "remove", "insert"] }, millis: { $gt: 100 } }).sort({ ts: -1 }).limit(10)

While these are invaluable for manual debugging, they need to be automated for continuous monitoring.

Automated MongoDB Monitoring with Prometheus

The most common and effective way to monitor MongoDB in a production environment is by using Prometheus with the mongodb_exporter.

1. Deploy mongodb_exporter:

You can run this as a Docker container or a standalone binary on a dedicated monitoring server or one of your DigitalOcean Droplets. It needs to be able to connect to your MongoDB cluster.

# Example command to run mongodb_exporter in Docker
docker run -d \
  --name mongodb_exporter \
  -p 9216:9216 \
  quay.io/prometheus/mongodb-exporter \
  --mongodb.uri="mongodb://user:password@your_mongodb_host:27017/admin?replicaSet=yourReplicaSetName"

Important:

  • Replace user, password, your_mongodb_host, and yourReplicaSetName with your actual MongoDB credentials and cluster details.
  • Ensure the user has sufficient privileges (e.g., `clusterMonitor`, `readAnyDatabase`).
  • For a replica set, specifying the replicaSet parameter is crucial for the exporter to gather replica set-specific metrics.
  • If your MongoDB is on DigitalOcean Managed Databases, you’ll use the provided connection string and credentials.

2. Configure Prometheus to Scrape mongodb_exporter:

# prometheus.yml (snippet)
scrape_configs:
  - job_name: 'mongodb'
    static_configs:
      - targets:
        - 'mongodb_exporter_ip:9216' # Replace with your mongodb_exporter's IP and port

This configuration tells Prometheus to fetch metrics from the deployed mongodb_exporter. The exporter will then query your MongoDB instances and expose metrics like:

mongodb_mongod_connections_current
mongodb_mongod_network_bytes_in_total
mongodb_mongod_network_bytes_out_total
mongodb_mongod_opcounters_insert
mongodb_mongod_opcounters_query
mongodb_mongod_opcounters_update
mongodb_mongod_opcounters_delete
mongodb_replset_member_state
mongodb_replset_oplog_lag_seconds
mongodb_wiredtiger_cache_bytes_used
mongodb_wiredtiger_cache_bytes_total

Alerting on MongoDB Issues

Once metrics are flowing into Prometheus, you can define alerting rules in Alertmanager. Critical alerts for MongoDB might include:

  • Replica Set Unhealthy: When a member is not PRIMARY or SECONDARY, or when mongodb_replset_member_state is not in a healthy state for a sustained period.
  • High Oplog Lag: When mongodb_replset_oplog_lag_seconds exceeds a defined threshold, indicating replication is falling behind.
  • Disk Space Critically Low: Using node_exporter metrics for disk usage on MongoDB data directories.
  • High Query Latency: Alerting on sustained high values for query execution times (requires custom metrics or profiling analysis).
  • Connection Exhaustion: When mongodb_mongod_connections_current approaches the configured maximum connections.
# alert_rules.yml (snippet for Prometheus)
groups:
- name: mongodb_alerts
  rules:
  - alert: MongoDBReplicaSetDown
    expr: mongodb_replset_member_state != 1 # 1 is PRIMARY, 2 is SECONDARY
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "MongoDB replica set member {{ $labels.instance }} is not PRIMARY or SECONDARY."
      description: "Replica set member {{ $labels.instance }} has been in an unhealthy state for 5 minutes."

  - alert: MongoDBHighOplogLag
    expr: mongodb_replset_oplog_lag_seconds > 600 # 10 minutes lag
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: "MongoDB oplog lag is high on {{ $labels.instance }}"
      description: "MongoDB oplog lag on {{ $labels.instance }} has exceeded 10 minutes for 2 minutes."

Integrating C++ App and MongoDB Monitoring

The ultimate goal is a unified view of your system’s health. By using Prometheus as the central monitoring system, you can:

  • Correlate application health with database health. For example, if your C++ app’s health check starts failing with “MongoDB connection pool unhealthy,” you can immediately pivot to the MongoDB metrics in Prometheus to diagnose if the cluster itself is experiencing issues (e.g., high load, network problems, node failures).
  • Create dashboards in Grafana that display both application-level metrics (e.g., request rates, error counts from your C++ app, potentially exposed via a custom Prometheus exporter) and MongoDB metrics side-by-side.
  • Set up unified alerting policies that consider both application and database status.

By implementing these proactive monitoring strategies for both your C++ application and your MongoDB clusters on DigitalOcean, you significantly increase your system’s resilience and reduce the mean time to recovery (MTTR) when issues inevitably arise.