Server Monitoring Best Practices: Keeping Your C App and MongoDB Clusters Alive on DigitalOcean
Proactive C Application Health Checks
Maintaining the health of a C application, especially one serving critical functions, requires more than just basic process monitoring. We need to implement application-level health checks that can be queried externally. A common pattern is to expose an HTTP endpoint that reports the application’s internal state.
For a C application, this often involves integrating a lightweight HTTP server library or, if the application already has network I/O, extending its existing listener. Let’s consider a simple approach using `libmicrohttpd` for demonstration. This allows us to expose a `/health` endpoint.
Implementing a Health Endpoint in C
The following C code snippet illustrates how to set up a basic HTTP server that responds to `/health` requests. This endpoint will check critical internal components, such as database connectivity (simulated here) and internal worker thread status.
#include <stdio.h>
#include <stdlib.h>
#include <microhttpd.h>
#include <string.h>
#include <stdbool.h>
// Simulate critical application states
volatile bool g_db_connected = true;
volatile int g_active_workers = 5;
volatile int g_max_workers = 10;
// Function to handle incoming requests
static int handle_request(void *cls, struct MHD_Connection *connection,
const char *url, const char *method,
const char *version, const char *upload_data,
size_t *upload_data_size, void *private_data) {
if (strcmp(url, "/health") == 0 && strcmp(method, "GET") == 0) {
// Check application health
bool healthy = g_db_connected && (g_active_workers > 0); // Basic check
const char *response_body;
int http_status;
if (healthy) {
response_body = "{"status": "ok", "db_connected": true, "active_workers": 5, "max_workers": 10}";
http_status = MHD_HTTP_OK;
} else {
response_body = "{"status": "degraded", "db_connected": false, "active_workers": 0, "max_workers": 10}";
http_status = MHD_HTTP_SERVICE_UNAVAILABLE; // Or MHD_HTTP_INTERNAL_SERVER_ERROR
}
struct MHD_Response *response;
response = MHD_create_response_from_buffer(strlen(response_body),
(void *)response_body,
MHD_NO_PULLUP);
MHD_add_header(response, MHD_HTTP_HEADER_CONTENT_TYPE, "application/json");
int ret = MHD_queue_response(connection, http_status, response);
MHD_destroy_response(response);
return ret;
}
// Handle other requests or return 404
const char *not_found_body = "{"error": "Not Found"}";
struct MHD_Response *response = MHD_create_response_from_buffer(strlen(not_found_body),
(void *)not_found_body,
MHD_NO_PULLUP);
MHD_add_header(response, MHD_HTTP_HEADER_CONTENT_TYPE, "application/json");
int ret = MHD_queue_response(connection, MHD_HTTP_NOT_FOUND, response);
MHD_destroy_response(response);
return ret;
}
int main(void) {
struct MHD_Daemon *daemon;
// Start the HTTP server on port 8080
daemon = MHD_start_daemon(MHD_SERVER_PORT_60000, 8080, NULL, NULL,
&handle_request, NULL, MHD_OPTION_END);
if (daemon == NULL) {
fprintf(stderr, "Failed to start HTTP daemon\n");
return 1;
}
printf("HTTP server started on port 8080. Health check at /health\n");
// Keep the server running
// In a real app, this would be integrated with the main application loop
// and health status would be updated dynamically.
getchar(); // Wait for user input to stop
MHD_stop_daemon(daemon);
return 0;
}
To compile this, you’ll need `libmicrohttpd` installed. On Debian/Ubuntu:
sudo apt-get update sudo apt-get install libmicrohttpd-dev gcc -o health_server health_server.c -lmicrohttpd
Once running, you can test it with `curl`:
curl http://localhost:8080/health
# Expected output (if healthy): {"status": "ok", "db_connected": true, "active_workers": 5, "max_workers": 10}
In a production environment, the `g_db_connected` and `g_active_workers` variables would be updated by the application’s core logic, reflecting actual system status. This health endpoint is crucial for external monitoring tools.
Leveraging DigitalOcean Monitoring and Alerting
DigitalOcean’s built-in monitoring provides essential metrics for Droplets and managed databases. For our C application’s health endpoint, we’ll use an external monitoring service or a custom script that periodically polls this endpoint. Tools like Prometheus, Datadog, or even a simple cron job with `curl` can be employed.
Let’s configure a basic alert in DigitalOcean for the C application’s Droplet. We’ll focus on CPU, memory, and disk I/O, but also set up an alert for the application’s health endpoint using a custom script.
DigitalOcean Droplet Alerts (CPU, Memory, Disk)
Navigate to your Droplet in the DigitalOcean control panel. Under the “Monitoring” tab, you can set up alerts. Key metrics to watch:
- CPU Usage: Set a threshold (e.g., > 90% for 15 minutes) to detect sustained high load.
- Memory Usage: Alert if memory usage exceeds a critical level (e.g., > 95% for 10 minutes).
- Disk I/O: Monitor read/write operations per second. Spikes can indicate performance bottlenecks or runaway processes.
- Network Traffic: High inbound/outbound traffic can signal DDoS attacks or unexpected load.
For each alert, configure notification channels (email, Slack via webhooks) to ensure timely response.
Custom Health Check Alerting for C App
Since DigitalOcean’s native monitoring doesn’t directly poll application endpoints, we’ll use a simple Bash script run via cron on a separate monitoring Droplet or even on one of the application Droplets (though a separate one is better for isolation).
Create a script, e.g., check_app_health.sh:
#!/bin/bash
APP_URL="http://YOUR_APP_DROPLET_IP:8080/health"
ALERT_THRESHOLD_SECONDS=300 # Alert if check fails for 5 minutes
LOG_FILE="/var/log/app_health_check.log"
ALERT_STATE_FILE="/tmp/app_health_alert_state"
# Get current timestamp
TIMESTAMP=$(date +"%Y-%m-%d %H:%M:%S")
# Perform the health check
RESPONSE=$(curl -s -o /dev/null -w "%{http_code}" "$APP_URL")
if [ "$RESPONSE" -eq 200 ]; then
# Application is healthy
echo "$TIMESTAMP - Health check OK (HTTP $RESPONSE)" >> "$LOG_FILE"
# If we were in an alert state, clear it
if [ -f "$ALERT_STATE_FILE" ]; then
echo "$TIMESTAMP - Health check recovered." >> "$LOG_FILE"
rm "$ALERT_STATE_FILE"
# Optionally send a recovery notification here
fi
exit 0
else
# Application is unhealthy or unreachable
echo "$TIMESTAMP - Health check FAILED (HTTP $RESPONSE)" >> "$LOG_FILE"
# Check if we are already in an alert state
if [ ! -f "$ALERT_STATE_FILE" ]; then
# First failure, record the time
echo "$TIMESTAMP" > "$ALERT_STATE_FILE"
else
# Subsequent failure, check duration
LAST_FAIL_TIME=$(cat "$ALERT_STATE_FILE")
LAST_FAIL_EPOCH=$(date -d "$LAST_FAIL_TIME" +%s)
CURRENT_EPOCH=$(date +%s)
DURATION=$((CURRENT_EPOCH - LAST_FAIL_EPOCH))
if [ "$DURATION" -ge "$ALERT_THRESHOLD_SECONDS" ]; then
echo "$TIMESTAMP - ALERT: Application unhealthy for $DURATION seconds. HTTP Status: $RESPONSE" >> "$LOG_FILE"
# Trigger an alert (e.g., send an email, Slack notification)
# Example: Send email
echo "Subject: ALERT: C App Unhealthy on $(hostname)" | sendmail [email protected] << EOF
The C application at $APP_URL is reporting HTTP status $RESPONSE.
This has been ongoing since $LAST_FAIL_TIME.
Please investigate immediately.
EOF
# To prevent repeated alerts, we can either remove the state file
# and re-trigger after recovery, or implement a cooldown.
# For simplicity here, we'll just log and let the next cron run check again.
# If you want to ensure only one alert per outage, you might need a more robust state management.
fi
fi
exit 1
fi
Make the script executable:
chmod +x check_app_health.sh
Add this script to cron. For example, to run it every minute:
crontab -e # Add the following line: * * * * * /path/to/your/check_app_health.sh
Remember to replace YOUR_APP_DROPLET_IP with the actual IP address of your C application’s Droplet and configure the email address for alerts.
Monitoring MongoDB Clusters on DigitalOcean
DigitalOcean Managed MongoDB offers built-in monitoring, which is essential for understanding cluster health, performance, and resource utilization. We’ll focus on key metrics and how to set up alerts for them.
Key MongoDB Metrics to Monitor
Within the DigitalOcean control panel for your Managed MongoDB cluster, you’ll find a “Monitoring” tab. Pay close attention to:
- Connections: The number of active client connections. High connection counts can indicate performance issues or resource exhaustion. Alert if connections approach the configured limit.
- Network Traffic: Inbound and outbound data transfer. Useful for identifying unusual activity or high query load.
- CPU Usage: CPU utilization across the cluster nodes. Sustained high CPU can point to inefficient queries or insufficient resources.
- Memory Usage: RAM consumed by MongoDB processes. Monitor for excessive usage that could lead to swapping or OOM killer events.
- Disk Usage: The amount of disk space used by your data. Essential for capacity planning.
- Operations: Read and write operations per second. Helps in understanding query load and identifying performance bottlenecks.
- Replication Lag: For replica sets, the difference in oplog application time between the primary and secondaries. Significant lag indicates replication issues that can impact read consistency and failover readiness.
Configuring MongoDB Alerts in DigitalOcean
DigitalOcean allows you to set up alerts directly on these metrics. For a production MongoDB cluster, consider the following alert configurations:
- Replication Lag: Set an alert if the replication lag exceeds a few seconds (e.g., > 10 seconds for 5 minutes). This is critical for data consistency and high availability.
- Connections: Alert if active connections exceed 80% of the configured maximum for a sustained period (e.g., > 80% for 10 minutes).
- Disk Usage: Alert when disk usage reaches a high percentage (e.g., > 85% for 1 hour) to allow time for scaling or data cleanup.
- CPU/Memory Usage: Similar to Droplet alerts, set thresholds for sustained high CPU or memory usage (e.g., > 90% for 15 minutes).
Ensure your notification channels (email, Slack) are correctly configured for these alerts.
Advanced MongoDB Monitoring with `mongotop` and `mongostat`
While DigitalOcean’s dashboard provides a good overview, real-time, granular insights often require direct interaction with the MongoDB instances. `mongotop` and `mongostat` are invaluable command-line utilities for this.
Using `mongostat` for Real-time Statistics
`mongostat` provides a live, aggregated view of MongoDB server statistics. It’s excellent for quickly assessing the overall health and performance of your cluster.
# Connect to a MongoDB node (replace with your connection string) mongostat --host YOUR_MONGODB_HOST:27017 --username YOUR_USER --password YOUR_PASSWORD --authenticationDatabase admin --ssl
Key columns to watch:
- insert, query, update, delete: Operations per second.
- get_m, qr, qw, ar, aw: Lock percentages for readers and writers. High values indicate contention.
- dirty: Percentage of dirty pages in the journal.
- used: Memory used by the WiredTiger cache.
- res: Resident memory usage.
- netIn, netOut: Network traffic in/out.
- conn: Number of active connections.
- idx: Index hit ratio (if available, often requires specific configurations or newer versions).
You can also specify intervals and counts:
# Get stats every 5 seconds, 10 times mongostat --host YOUR_MONGODB_HOST:27017 --username YOUR_USER --password YOUR_PASSWORD --authenticationDatabase admin --ssl 5 10
Using `mongotop` for Collection-Level Performance
`mongotop` provides a per-collection view of read and write activity, helping to pinpoint which collections are experiencing the most load.
# Connect to a MongoDB node mongotop --host YOUR_MONGODB_HOST:27017 --username YOUR_USER --password YOUR_PASSWORD --authenticationDatabase admin --ssl
The output shows:
- ns: Namespace (database.collection).
- total: Total time spent on operations (in ms).
- read: Time spent on read operations.
- write: Time spent on write operations.
This is invaluable for identifying slow queries or heavily accessed collections that might require indexing optimization or schema redesign.
Integrating Custom Metrics with Prometheus and Grafana
For a more comprehensive and customizable monitoring dashboard, integrating Prometheus and Grafana is a standard practice. This allows you to collect metrics from your C application, MongoDB, and Droplets into a single, unified view.
Exposing C Application Metrics for Prometheus
To expose metrics from your C application to Prometheus, you can use the official Prometheus C/C++ client library. This library allows you to define custom metrics (counters, gauges, histograms) and expose them via an HTTP endpoint, typically `/metrics`.
Here’s a conceptual outline of how you’d integrate it. You would need to:
- Include the Prometheus client library headers.
- Initialize the client library.
- Define metrics (e.g., a gauge for active workers, a counter for requests processed).
- Update these metrics within your application logic.
- Start an HTTP server (often integrated with your existing web server or a separate thread) to expose the `/metrics` endpoint.
Example (conceptual C code):
// Assuming you have the Prometheus C++ client library integrated
// Define metrics globally
prometheus_gauge_t active_workers_gauge;
prometheus_counter_t requests_processed_counter;
void initialize_metrics() {
active_workers_gauge = prometheus_gauge_new("app_active_workers", "Number of active worker threads");
requests_processed_counter = prometheus_counter_new("app_requests_total", "Total number of requests processed");
// Register metrics with the registry
// ...
}
void update_worker_count(int count) {
prometheus_gauge_set(active_workers_gauge, (double)count);
}
void increment_request_counter() {
prometheus_counter_inc(requests_processed_counter);
}
// In your request handler:
// increment_request_counter();
// ...
// In your worker management logic:
// update_worker_count(current_worker_count);
You would then configure Prometheus to scrape this `/metrics` endpoint on your C application’s Droplet.
Prometheus Exporters for MongoDB and Droplets
For MongoDB, the official mongodb_exporter is widely used. It connects to your MongoDB instances and exposes metrics in Prometheus format.
# Example of running mongodb_exporter docker run -d \ --name mongodb_exporter \ -p 9274:9274 \ -e MONGODB_URI="mongodb://user:password@host:port/admin?ssl=true" \ prom/mongodb-exporter:latest
For DigitalOcean Droplets, you can run the node_exporter on each Droplet. This exporter collects hardware and OS metrics.
# Example of running node_exporter on a Droplet docker run -d \ --name node_exporter \ --net="host" \ prom/node-exporter:latest
Configure your Prometheus server’s prometheus.yml to scrape these exporters:
scrape_configs:
- job_name: 'my_c_app'
static_configs:
- targets: ['YOUR_C_APP_DROPLET_IP:9100'] # Assuming app exposes /metrics on 9100
- job_name: 'mongodb'
static_configs:
- targets: ['YOUR_MONGODB_DROPLET_IP:9274'] # Or your mongodb_exporter service
- job_name: 'droplets'
static_configs:
- targets: ['DROPLET_1_IP:9100', 'DROPLET_2_IP:9100'] # For node_exporter on each Droplet
Setting up Grafana Dashboards
Once Prometheus is collecting data, Grafana can be used to visualize it. You can import pre-built dashboards for MongoDB and Node Exporter from the Grafana dashboards repository, or create custom dashboards for your C application’s metrics.
Key Grafana dashboards to consider:
- MongoDB: Look for dashboards that visualize connections, operations, replication lag, cache usage, and query performance.
- Node Exporter: Dashboards showing CPU, memory, disk I/O, network traffic, and system load for your Droplets.
- Custom C App Dashboard: Visualize your application-specific metrics like active workers, request rates, error counts, and latency.
Configure Grafana alerts based on these dashboards to mirror or enhance the alerts set up in DigitalOcean, providing a unified alerting experience.