Vengala Vinay

Having 9+ Years of Experience in Software Development

Home » Server Monitoring Best Practices: Keeping Your C++ App and Redis Clusters Alive on Linode

Server Monitoring Best Practices: Keeping Your C++ App and Redis Clusters Alive on Linode

Proactive C++ Application Health Checks

Maintaining the health of a C++ application, especially one serving critical traffic, requires more than just basic process monitoring. We need to implement application-level health checks that can be queried externally. For a C++ application, this often involves exposing an HTTP endpoint that reports its internal state.

A common pattern is to use a lightweight HTTP server library within your C++ application. For this example, we’ll assume a simple HTTP server is integrated, capable of responding to requests on a specific port (e.g., 8080). The health endpoint, typically `/health`, should return a 200 OK status if all critical internal components are functioning, and a non-2xx status otherwise.

Implementing a Basic C++ Health Endpoint

Let’s outline a conceptual C++ implementation. This isn’t a full, production-ready HTTP server, but demonstrates the core logic for a health check endpoint. We’ll use a hypothetical `HttpServer` class.

The `is_healthy()` method would encapsulate checks for database connections, cache availability, internal queues, and any other essential dependencies. If any check fails, it returns `false`, triggering a non-200 HTTP response.

#include <iostream>
#include <string>
#include <chrono>
#include <thread>
#include <atomic>

// Hypothetical HTTP server and request handling
class HttpServer {
public:
    void start(int port) {
        std::cout << "Starting HTTP server on port " << port << std::endl;
        // In a real scenario, this would involve socket programming,
        // request parsing, and response generation.
        // For demonstration, we'll simulate a running server.
        running.store(true);
        // ... server loop ...
    }

    void stop() {
        running.store(false);
        std::cout << "Stopping HTTP server." << std::endl;
    }

    // Simulate handling an incoming request
    void handle_request(const std::string& path, std::string& response_body, int& status_code) {
        if (path == "/health") {
            if (is_healthy()) {
                response_body = "OK";
                status_code = 200;
            } else {
                response_body = "Service Unavailable";
                status_code = 503; // Service Unavailable
            }
        } else {
            response_body = "Not Found";
            status_code = 404;
        }
    }

private:
    std::atomic<bool> running{false};

    // Core health check logic
    bool is_healthy() const {
        // Simulate checks for critical components
        bool db_connected = check_database_connection();
        bool cache_available = check_cache_availability();
        bool queue_ok = check_internal_queue();

        // Return true only if all critical components are healthy
        return db_connected && cache_available && queue_ok;
    }

    bool check_database_connection() const {
        // In a real app: ping database, check connection pool status
        std::cout << "Checking DB connection..." << std::endl;
        return true; // Simulate success
    }

    bool check_cache_availability() const {
        // In a real app: ping cache server (e.g., Redis), check latency
        std::cout << "Checking cache availability..." << std::endl;
        return true; // Simulate success
    }

    bool check_internal_queue() const {
        // In a real app: check queue depth, processing rate
        std::cout << "Checking internal queue..." << std::endl;
        return true; // Simulate success
    }
};

// Example usage within main (simplified)
int main() {
    HttpServer server;
    server.start(8080);

    // In a real application, this would be a loop
    // processing incoming requests.
    // For demonstration, we'll simulate a health check call.
    std::this_thread::sleep_for(std::chrono::seconds(2)); // Give server time to "start"

    std::string response_body;
    int status_code;

    std::cout << "\nSimulating /health request:\n";
    server.handle_request("/health", response_body, status_code);
    std::cout << "Status: " << status_code << ", Body: " << response_body << std::endl;

    // Simulate a failure
    // In a real app, this would be triggered by an actual failure
    // For demo, we'll just imagine is_healthy() returned false.
    // server.simulate_failure(); // Hypothetical method

    std::cout << "\nSimulating /health request after hypothetical failure:\n";
    // Manually set to simulate failure for demo
    // In reality, the internal checks would fail.
    // For this simplified example, we can't easily trigger a failure
    // without modifying the private methods.
    // Let's assume the checks would now return false.
    // For demonstration purposes, we'll just show the expected output.
    std::cout << "Status: 503, Body: Service Unavailable" << std::endl;


    // server.stop(); // In a real app, this would be called on shutdown
    return 0;
}

External Monitoring with `curl` and `wget`

Once your application exposes a health endpoint, you can use standard command-line tools to monitor it from an external perspective. Linode’s monitoring tools or a dedicated external monitoring service can execute these checks periodically.

A simple `curl` command can verify the health endpoint. We’ll check for a 200 status code and optionally the expected response body.

# Check HTTP status code and response body
curl --fail --silent --show-error --connect-timeout 5 --max-time 10 "http://your_app_ip:8080/health"
if [ $? -eq 0 ]; then
    echo "Application health check PASSED."
else
    echo "Application health check FAILED."
fi

# More robust check: verify status code explicitly
STATUS_CODE=$(curl -s -o /dev/null -w "%{http_code}" --connect-timeout 5 --max-time 10 "http://your_app_ip:8080/health")
if [ "$STATUS_CODE" -eq 200 ]; then
    echo "Application health check PASSED (Status: $STATUS_CODE)."
else
    echo "Application health check FAILED (Status: $STATUS_CODE)."
fi

The `–fail` flag in `curl` is crucial; it makes `curl` return a non-zero exit code on HTTP errors (like 4xx or 5xx). `–connect-timeout` and `–max-time` prevent checks from hanging indefinitely.

Redis Cluster Monitoring Strategies

Monitoring a Redis cluster involves checking the health of individual nodes, the cluster’s overall state, and key performance metrics. Redis provides built-in commands and exposes metrics that are essential for this.

Individual Node Health and Cluster State

The `redis-cli` tool is your primary interface for interacting with Redis. For cluster monitoring, we’ll focus on commands that reveal node status and cluster integrity.

# Connect to a Redis node (replace with your node's IP and port)
redis-cli -h  -p 

# Check if a node is reachable and responsive
PING
# Expected output: PONG

# Get general information about the Redis instance
INFO server
INFO clients
INFO memory
INFO persistence
INFO stats
INFO replication
INFO cpu
INFO commandstats

# Check cluster status (run on any node)
CLUSTER INFO
# Look for 'cluster_state:ok' and 'cluster_slots_assigned' vs 'cluster_slots_ok'

# Get a list of all nodes in the cluster and their status
CLUSTER NODES
# This output is crucial. It shows each node, its role (master/slave),
# its status (connected, disconnected, fail), and its assigned slots.
# Look for nodes marked as 'disconnected' or 'fail'.

When using `CLUSTER NODES`, pay close attention to the flags associated with each node. A healthy cluster will have masters with slaves, and all nodes should be marked as `connected`. If a master is marked as `fail` or `disconnected`, the cluster is in a degraded state.

Key Redis Metrics for Monitoring

The `INFO` command provides a wealth of metrics. Here are some critical ones to monitor:

  • `used_memory`: Current memory usage. Monitor against configured `maxmemory` to prevent OOM errors.
  • `mem_fragmentation_ratio`: Ratio of used memory to allocated memory. High fragmentation can indicate memory inefficiency.
  • `connected_clients`: Number of active client connections. High numbers might indicate a need for scaling or a potential DoS attack.
  • `rejected_connections`: Number of connections rejected due to reaching the `maxclients` limit.
  • `sync_full` and `sync_partial_ok`: Related to replication. High `sync_full` can indicate network issues or slow disk I/O on replicas.
  • `keyspace_hits` and `keyspace_misses`: Cache hit ratio. A low hit ratio might indicate insufficient memory or an inefficient caching strategy.
  • `instantaneous_ops_per_sec`: Current operations per second. Useful for understanding load.
  • `latest_fork_usec`: Time taken for the last fork operation (used for persistence). Long forks can block the server.
  • `rdb_changes_since_last_save` and `last_save_time`: Related to RDB persistence. Ensure saves are happening and not too far apart.
  • `aof_enabled`: Whether AOF persistence is enabled.
  • `aof_last_write_status`: Status of AOF writes (e.g., `ok`, `err`).

Automating Redis Cluster Health Checks

We can script these checks to run periodically. A Bash script can connect to each node, execute `PING`, `CLUSTER INFO`, and `CLUSTER NODES`, and then parse the output for critical indicators.

#!/bin/bash

REDIS_HOSTS=("node1.example.com" "node2.example.com" "node3.example.com")
REDIS_PORT=6379
ALERT_EMAIL="[email protected]"

# Function to send alert
send_alert() {
    local message="$1"
    echo "$(date): ALERT - $message" | mail -s "Redis Cluster Alert" "$ALERT_EMAIL"
}

echo "Starting Redis cluster health check..."

# Check individual node reachability and PING response
for host in "${REDIS_HOSTS[@]}"; do
    echo "Checking node: $host:$REDIS_PORT"
    if ! redis-cli -h "$host" -p "$REDIS_PORT" PING >& /dev/null; then
        send_alert "Node $host:$REDIS_PORT is unreachable or not responding to PING."
        continue # Skip further checks for this unreachable node
    fi
    echo "  PING OK."
done

# Check cluster state from one node (assuming at least one is up)
CLUSTER_INFO_OUTPUT=$(redis-cli -h "${REDIS_HOSTS[0]}" -p "$REDIS_PORT" CLUSTER INFO 2>&1)
if echo "$CLUSTER_INFO_OUTPUT" | grep -q "cluster_state:ok"; then
    echo "Cluster state is OK."
else
    send_alert "Redis cluster state is NOT OK. CLUSTER INFO output:\n$CLUSTER_INFO_OUTPUT"
fi

# Detailed check of all nodes using CLUSTER NODES
CLUSTER_NODES_OUTPUT=$(redis-cli -h "${REDIS_HOSTS[0]}" -p "$REDIS_PORT" CLUSTER NODES 2>&1)
if echo "$CLUSTER_NODES_OUTPUT" | grep -q "fail" || echo "$CLUSTER_NODES_OUTPUT" | grep -q "disconnected"; then
    send_alert "One or more Redis nodes are marked as 'fail' or 'disconnected'. CLUSTER NODES output:\n$CLUSTER_NODES_OUTPUT"
else
    echo "All nodes in CLUSTER NODES are connected."
fi

# Example: Check memory usage against a threshold (e.g., 80% of maxmemory)
# This requires parsing INFO memory output.
# For simplicity, let's assume maxmemory is set to 1GB (1073741824 bytes)
MAX_MEMORY_BYTES=1073741824 # 1GB
MEMORY_THRESHOLD=$(echo "$MAX_MEMORY_BYTES * 0.8" | bc)

for host in "${REDIS_HOSTS[@]}"; do
    MEMORY_INFO=$(redis-cli -h "$host" -p "$REDIS_PORT" INFO memory)
    USED_MEMORY=$(echo "$MEMORY_INFO" | grep "^used_memory:" | awk -F: '{print $2}')

    if [ -n "$USED_MEMORY" ] && [ "$USED_MEMORY" -gt "$MEMORY_THRESHOLD" ]; then
        send_alert "Node $host:$REDIS_PORT is nearing max memory. Used: $(echo "$USED_MEMORY / 1024 / 1024" | bc) MB, Threshold: $(echo "$MEMORY_THRESHOLD / 1024 / 1024" | bc) MB."
    fi
done

echo "Redis cluster health check finished."

This script can be scheduled via cron on a dedicated monitoring server or one of the application servers. Ensure the `mail` command is configured on the system running the script for alerts to be sent.

Leveraging Linode’s Monitoring and Alerting

Linode Cloud offers built-in monitoring capabilities that can be integrated with your custom checks. You can use Linode’s API or agents to collect metrics and trigger alerts.

Custom Metrics with Node Exporter

For more granular metrics, especially from your C++ application, consider deploying Prometheus Node Exporter. You can write custom collectors for Node Exporter to expose application-specific metrics, including the status of your health checks.

Alternatively, you can use `curl` or `wget` commands as part of Linode’s external network monitoring to hit your application’s health endpoint. Configure these checks to run at intervals (e.g., every minute) and set up alerts to notify you via email or Slack when a check fails.

# Example of a Linode External Network Monitor check (conceptual)
# This would be configured in the Linode Cloud Manager UI.
# The check would execute a command like this from an external location.

curl --fail --silent --show-error --connect-timeout 5 --max-time 10 "http://your_app_ip:8080/health"

Redis Exporter for Prometheus

For Redis, the `redis_exporter` is a standard tool that scrapes metrics from Redis instances and exposes them in a Prometheus-compatible format. This exporter can be configured to connect to your Redis cluster nodes and collect all the `INFO` and `CLUSTER` metrics we discussed.

Once `redis_exporter` is running and scraping metrics, you can configure Prometheus to scrape these targets. Then, use Grafana to visualize the metrics and set up alerting rules within Prometheus or Grafana based on thresholds for memory usage, cluster state, client connections, etc.

# Example redis_exporter configuration (redis_exporter.yml)
# This is a simplified example. Refer to redis_exporter documentation for full options.

redis_exporter:
  redis:
    - host: "node1.example.com"
      port: 6379
      password: "" # if password protected
      mode: "cluster" # Important for cluster-aware metrics
      metrics:
        - "commandstats"
        - "keyspace"
        - "memory"
        - "replication"
        - "stats"
        - "server"
        - "clients"
        - "cpu"
        - "persistence"
        - "cluster" # Enables cluster-specific metrics

# Prometheus configuration (prometheus.yml) snippet to scrape redis_exporter
scrape_configs:
  - job_name: 'redis_cluster'
    static_configs:
      - targets:
        - 'redis_exporter_host:9121' # Assuming redis_exporter runs on port 9121

By combining application-level health checks, robust Redis cluster monitoring scripts, and Linode’s integrated alerting and external monitoring features, you can build a resilient system that proactively identifies and addresses issues before they impact your users.