Vengala Vinay

Having 9+ Years of Experience in Software Development

Home » Server Monitoring Best Practices: Keeping Your C App and MongoDB Clusters Alive on Linode

Server Monitoring Best Practices: Keeping Your C App and MongoDB Clusters Alive on Linode

Proactive C Application Health Checks

For a C application deployed on Linode, robust health checking is paramount. Beyond basic process existence, we need to verify application-level functionality. A common pattern is to expose an HTTP endpoint that signals the application’s readiness and operational status. This endpoint should perform internal checks, such as database connectivity or critical service availability.

Let’s consider a simple C application using `libmicrohttpd` to expose a health check endpoint. This example assumes you have `libmicrohttpd` installed on your Linode instance.

C Health Check Endpoint Implementation

The following C code snippet demonstrates a basic HTTP server that responds to requests on `/health`. A non-zero return code from the handler indicates an issue.

#include <microhttpd.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

#define PORT 8080
#define HEALTH_PATH "/health"

// Simulate a critical dependency check
int check_database_connection() {
    // In a real app, this would involve actual DB connection logic.
    // For demonstration, we'll randomly simulate success/failure.
    // srand(time(NULL)); // Uncomment for true randomness, but be mindful of seeding
    return rand() % 2; // 0 for success, 1 for failure
}

static int health_handler(void *cls, struct MHD_Connection *connection,
                          const char *url, const char *method,
                          const char *version, const char *upload_data,
                          size_t *upload_data_size, void *private_data) {
    if (strcmp(url, HEALTH_PATH) == 0) {
        if (check_database_connection() != 0) {
            // Database connection failed
            const char *response = "{\"status\": \"unhealthy\", \"reason\": \"database_connection_failed\"}";
            struct MHD_Response *mhd_response = MHD_create_response_from_buffer(strlen(response), (void *)response, MHD_RESPMEM_PERSISTENT);
            MHD_add_response_header(mhd_response, MHD_HTTP_HEADER_CONTENT_TYPE, "application/json");
            MHD_add_response_header(mhd_response, MHD_HTTP_HEADER_CACHE_CONTROL, "no-cache");
            MHD_queue_response(connection, 503, NULL, 0, NULL, mhd_response); // 503 Service Unavailable
            MHD_destroy_response(mhd_response);
            return MHD_YES;
        } else {
            // Application is healthy
            const char *response = "{\"status\": \"healthy\"}";
            struct MHD_Response *mhd_response = MHD_create_response_from_buffer(strlen(response), (void *)response, MHD_RESPMEM_PERSISTENT);
            MHD_add_response_header(mhd_response, MHD_HTTP_HEADER_CONTENT_TYPE, "application/json");
            MHD_add_response_header(mhd_response, MHD_HTTP_HEADER_CACHE_CONTROL, "no-cache");
            MHD_queue_response(connection, 200, NULL, 0, NULL, mhd_response); // 200 OK
            MHD_destroy_response(mhd_response);
            return MHD_YES;
        }
    }
    // Handle other paths or return 404
    const char *response = "Not Found";
    struct MHD_Response *mhd_response = MHD_create_response_from_buffer(strlen(response), (void *)response, MHD_RESPMEM_PERSISTENT);
    MHD_queue_response(connection, 404, NULL, 0, NULL, mhd_response);
    MHD_destroy_response(mhd_response);
    return MHD_YES;
}

int main() {
    struct MHD_Daemon *daemon;

    daemon = MHD_start_daemon(MHD_SERVER_PORT_HTTP, PORT, NULL, NULL,
                              &health_handler, NULL, MHD_OPTION_END);
    if (daemon == NULL) {
        fprintf(stderr, "Failed to start HTTP daemon on port %d\n", PORT);
        return 1;
    }

    printf("HTTP daemon started on port %d, listening for /health requests.\n", PORT);

    // Keep the daemon running
    getchar();

    MHD_stop_daemon(daemon);
    return 0;
}

To compile this, you’ll need the `libmicrohttpd` development headers and libraries. On Debian/Ubuntu-based systems, this is typically `libmicrohttpd-dev`.

sudo apt update
sudo apt install libmicrohttpd-dev
gcc -o health_server health_server.c -lmicrohttpd

Once compiled and running, you can test it:

./health_server
curl http://localhost:8080/health

The output will be either `{“status”: “healthy”}` or `{“status”: “unhealthy”, “reason”: “database_connection_failed”}`. This endpoint can then be polled by external monitoring tools.

Leveraging Linode’s Monitoring and External Tools

Linode provides basic server-level monitoring (CPU, RAM, Disk I/O, Network). While useful, it doesn’t understand your C application’s internal state. For comprehensive monitoring, we combine Linode’s metrics with external polling of our health endpoint and potentially application-specific metrics.

Configuring External Health Checks

Tools like Prometheus with its Blackbox Exporter, or even simpler cron jobs with `curl`, can poll your health endpoint. For production, a dedicated monitoring service is recommended.

Example using `curl` and `cron` (for basic alerting):

# Create a script to check health
cat > /usr/local/bin/check_app_health.sh << 'EOF'
#!/bin/bash
HEALTH_URL="http://localhost:8080/health"
LOG_FILE="/var/log/app_health.log"
ALERT_EMAIL="[email protected]"

RESPONSE=$(curl -s -o /dev/null -w "%{http_code}" $HEALTH_URL)

if [ "$RESPONSE" -ne 200 ]; then
    echo "$(date): Health check failed. HTTP status: $RESPONSE" >> $LOG_FILE
    # In a real scenario, you'd integrate with an alerting system here.
    # For simplicity, we'll just log and potentially send an email.
    # echo "Application health check failed on $(hostname) - HTTP $RESPONSE" | mail -s "ALERT: App Health Critical" $ALERT_EMAIL
else
    echo "$(date): Health check successful. HTTP status: $RESPONSE" >> $LOG_FILE
fi
EOF

chmod +x /usr/local/bin/check_app_health.sh

# Add to cron to run every minute
(crontab -l 2>&1 | grep -v "^\s*#" ; echo "* * * * * /usr/local/bin/check_app_health.sh") | crontab -

This basic setup logs health check results and can be extended to trigger alerts. For more sophisticated monitoring, consider Prometheus.

MongoDB Cluster Monitoring on Linode

Monitoring a MongoDB replica set or sharded cluster involves several layers: Linode’s infrastructure metrics, MongoDB’s internal operational metrics, and application-level interactions with the database.

Key MongoDB Metrics to Monitor

  • Replication Lag: Crucial for high availability. Use `rs.status()` to check `optime` and `syncedTo`.
  • Connection Counts: Monitor active and queued connections to prevent overload.
  • Query Performance: Slow queries can indicate indexing issues or heavy load. Use `db.currentOp()`, `db.setProfilingLevel()`, and the Slow Query Log.
  • Disk Usage: MongoDB can consume significant disk space. Monitor data files, journal, and oplog size.
  • Memory Usage: Track resident and virtual memory. Ensure sufficient RAM for working set.
  • Network I/O: Monitor network traffic to and from MongoDB instances.
  • CPU Usage: High CPU can indicate inefficient queries or insufficient resources.
  • Lock Contention: Monitor global and document locks.
  • Oplog Size: Essential for replication and sharding. Ensure it’s large enough to prevent oplog window issues.

Tools for MongoDB Monitoring

Several tools can be integrated with Linode for MongoDB monitoring:

  • `mongostat` and `mongotop`: Real-time command-line utilities for basic metrics.
  • MongoDB Cloud Manager / Ops Manager: Comprehensive commercial solutions.
  • Prometheus with `mongodb_exporter`: A popular open-source choice.
  • Datadog, New Relic, etc.: Commercial SaaS monitoring platforms.

Setting up Prometheus with `mongodb_exporter`

This is a powerful, cost-effective approach. We’ll assume you have a Prometheus server already running (perhaps on another Linode instance or managed service).

1. Install `mongodb_exporter` on each MongoDB node:

# Download the latest release (adjust version as needed)
wget https://github.com/mongodb-developer/mongodb_exporter/releases/download/v0.28.0/mongodb_exporter-v0.28.0.linux-amd64.tar.gz
tar -xzf mongodb_exporter-v0.28.0.linux-amd64.tar.gz
sudo mv mongodb_exporter-v0.28.0.linux-amd64/mongodb_exporter /usr/local/bin/
rm -rf mongodb_exporter-v0.28.0.linux-amd64*

2. Create a MongoDB user for the exporter:

Connect to your MongoDB instance and create a user with the `clusterMonitor` role. This role provides read-only access to cluster-wide information.

use admin
db.createUser({
  user: "exporter_user",
  pwd: "your_secure_password",
  roles: [ { role: "clusterMonitor", db: "admin" } ]
})

3. Configure `mongodb_exporter` to run as a service:

Create a systemd service file.

# /etc/systemd/system/mongodb_exporter.service
[Unit]
Description=MongoDB Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=mongodb # Or a dedicated user if MongoDB isn't running as 'mongodb'
Group=mongodb
ExecStart=/usr/local/bin/mongodb_exporter \
  --mongodb.uri="mongodb://exporter_user:your_secure_password@localhost:27017/?authSource=admin" \
  --web.listen-address=":9216" \
  --mongodb.log-dir="/var/log/mongodb" \
  --mongodb.oplog-stats
Restart=on-failure

[Install]
WantedBy=multi-user.target

Replace `localhost:27017` with your MongoDB connection string if it’s not local. Ensure the `User` and `Group` match your MongoDB installation. Start and enable the service:

sudo systemctl daemon-reload
sudo systemctl start mongodb_exporter
sudo systemctl enable mongodb_exporter
sudo systemctl status mongodb_exporter

The exporter will expose metrics on port `9216`. You can verify by curling it:

curl http://localhost:9216/metrics

4. Configure Prometheus to scrape `mongodb_exporter`:

Edit your Prometheus configuration file (`prometheus.yml`) to add a scrape job for your MongoDB nodes.

scrape_configs:
  - job_name: 'mongodb'
    static_configs:
      - targets:
        - 'mongodb-node-1:9216' # Replace with actual IPs/hostnames
        - 'mongodb-node-2:9216'
        - 'mongodb-node-3:9216'
    # If using service discovery, adapt accordingly

Reload your Prometheus configuration. You should now see MongoDB metrics appearing in Prometheus, ready for graphing in Grafana and alerting.

Integrating Linode Metrics with Monitoring Dashboards

While Prometheus and Grafana are excellent for application and database metrics, Linode’s native metrics are also valuable. You can pull these via the Linode API or use agents that integrate with Linode’s monitoring endpoints.

Example: Using `node_exporter` for Linode Metrics

The standard `node_exporter` (often used with Prometheus) collects system-level metrics that align with what Linode provides. Installing and configuring `node_exporter` on each Linode instance allows you to consolidate all metrics into Prometheus.

# Download and install node_exporter (similar to mongodb_exporter)
wget https://github.com/prometheus/node_exporter/releases/download/v1.7.0/node_exporter-1.7.0.linux-amd64.tar.gz
tar -xzf node_exporter-1.7.0.linux-amd64.tar.gz
sudo mv node_exporter-1.7.0.linux-amd64/node_exporter /usr/local/bin/
rm -rf node_exporter-1.7.0.linux-amd64*

# Create systemd service for node_exporter
cat > /etc/systemd/system/node_exporter.service << 'EOF'
[Unit]
Description=Prometheus Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=nobody
Group=nogroup
ExecStart=/usr/local/bin/node_exporter \
  --collector.textfile.directory="/var/lib/node_exporter/textfile_collector" \
  --web.listen-address=":9100"
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl start node_exporter
sudo systemctl enable node_exporter
sudo systemctl status node_exporter

Add a scrape job to your `prometheus.yml` for `node_exporter` on port `9100` for each Linode instance. This gives you CPU, RAM, disk, and network metrics directly comparable to Linode’s dashboard, but within your unified Prometheus/Grafana view.

Alerting Strategies

Effective alerting is crucial. Configure Prometheus Alertmanager to notify you of critical conditions:

  • High CPU/Memory/Disk Usage on Linode instances.
  • Application Health Check Failures (HTTP 5xx errors).
  • MongoDB Replication Lag exceeding a defined threshold (e.g., > 60 seconds).
  • MongoDB Connection Pool Exhaustion.
  • MongoDB Oplog Window Too Small.
  • High Latency for critical MongoDB queries.

Integrate Alertmanager with your preferred notification channels: Slack, PagerDuty, email, etc. Ensure alerts are actionable and avoid alert fatigue by tuning thresholds and alert rules carefully.