Vengala Vinay

Having 9+ Years of Experience in Software Development

Home » Server Monitoring Best Practices: Keeping Your C++ App and Elasticsearch Clusters Alive on Google Cloud

Server Monitoring Best Practices: Keeping Your C++ App and Elasticsearch Clusters Alive on Google Cloud

Proactive C++ Application Health Checks

For critical C++ applications running on Google Cloud, a robust health check mechanism is paramount. This goes beyond simple process existence. We need to ensure the application is not just running, but also responsive and performing within acceptable parameters. A common pattern is to expose an HTTP endpoint that provides detailed health status.

Consider a C++ application using `libmicrohttpd` for its HTTP server. We can implement a `/healthz` endpoint that checks internal state, database connectivity, and resource utilization.

Implementing a C++ Health Endpoint

Here’s a simplified example of how to integrate a health check into a C++ application. This example assumes you have a way to access internal metrics or states.

#include <microhttpd.h>
#include <string>
#include <vector>
#include <sstream>
#include <chrono>
#include <thread>

// Assume these functions exist and provide application-specific health info
bool is_database_connected() {
    // Simulate database check
    return true;
}

bool is_cache_healthy() {
    // Simulate cache check
    return true;
}

int get_active_connections() {
    // Simulate active connections
    return 123;
}

long long get_memory_usage_bytes() {
    // Simulate memory usage
    return 512 * 1024 * 1024; // 512 MB
}

// Callback function for handling requests
static int handle_request(void *cls, struct MHD_Connection *connection,
                          const char *url, const char *method,
                          const char *version, const char *upload_data,
                          size_t *upload_data_size, void **con_cls) {
    if (std::string(url) == "/healthz") {
        std::stringstream response_body;
        int http_status = MHD_HTTP_OK;

        response_body << "{";
        response_body << "\"status\": \"OK\",";
        response_body << "\"timestamp\": " << std::chrono::duration_cast<std::chrono::seconds>(std::chrono::system_clock::now().time_since_epoch()).count() << ",";
        response_body << "\"checks\": {";

        // Database Check
        response_body << "\"database\": " << (is_database_connected() ? "\"OK\"" : "\"ERROR\"") << ",";
        if (!is_database_connected()) http_status = MHD_HTTP_SERVICE_UNAVAILABLE;

        // Cache Check
        response_body << "\"cache\": " << (is_cache_healthy() ? "\"OK\"" : "\"ERROR\"") << ",";
        if (!is_cache_healthy()) http_status = MHD_HTTP_SERVICE_UNAVAILABLE;

        // Resource Checks (example: memory)
        long long memory_usage = get_memory_usage_bytes();
        response_body << "\"memory_mb\": " << (memory_usage / (1024 * 1024));
        // Add more resource checks as needed (CPU, disk, etc.)

        response_body << "},";
        response_body << "\"metrics\": {";
        response_body << "\"active_connections\": " << get_active_connections();
        response_body << "}";

        response_body << "}";

        std::string response_str = response_body.str();
        struct MHD_Response *response = MHD_create_response_from_buffer(response_str.length(), (void *)response_str.c_str(), MHD_RESPONSE_CHUNKED);
        MHD_add_response_header(response, MHD_HTTP_HEADER_CONTENT_TYPE, "application/json");
        int ret = MHD_queue_response(connection, http_status, response);
        MHD_destroy_response(response);
        return ret;
    }

    // Handle other URLs or return 404
    const char *not_found_msg = "Not Found";
    struct MHD_Response *response = MHD_create_response_from_buffer(strlen(not_found_msg), (void *)not_found_msg, MHD_RESPONSE_CHUNKED);
    int ret = MHD_queue_response(connection, MHD_HTTP_NOT_FOUND, response);
    MHD_destroy_response(response);
    return ret;
}

int main() {
    struct MHD_Daemon *daemon;

    daemon = MHD_start_daemon(MHD_NO_PROCFS, 8080, NULL, NULL,
                              &handle_request, NULL, MHD_OPTION_END);
    if (daemon == NULL) {
        return 1;
    }

    // Keep the application running
    while (1) {
        std::this_thread::sleep_for(std::chrono::seconds(1));
    }

    MHD_stop_daemon(daemon);
    return 0;
}

This endpoint provides a JSON payload that can be easily parsed by monitoring systems. The HTTP status code itself is a critical signal: 200 OK for healthy, 503 Service Unavailable for unhealthy. This allows load balancers and orchestration systems (like Kubernetes or Google Cloud’s Managed Instance Groups) to automatically remove unhealthy instances from service.

Integrating with Google Cloud Monitoring (Cloud Monitoring)

Google Cloud Monitoring (formerly Stackdriver) is the natural choice for collecting and visualizing these health metrics. We can use the Cloud Monitoring Agent to scrape custom metrics or, more simply, configure external HTTP health checks.

Configuring External HTTP Health Checks

For applications behind a Google Cloud Load Balancer (e.g., HTTP(S) Load Balancer), configuring external health checks is straightforward. These checks are performed from Google’s infrastructure to your application’s public IP or backend service.

gcloud compute health-checks create http my-cpp-app-health-check \
    --request-path="/healthz" \
    --port=8080 \
    --check-interval=10s \
    --timeout=5s \
    --unhealthy-threshold=3 \
    --healthy-threshold=2 \
    --global # or --region=[REGION] for regional LBs

This command creates an HTTP health check that probes the `/healthz` path on port 8080. It’s configured to consider an instance unhealthy after 3 consecutive failures and healthy after 2 consecutive successes. The interval and timeout are crucial for responsiveness.

Once created, this health check needs to be associated with your backend service:

gcloud compute backend-services update my-cpp-app-backend-service \
    --health-checks=my-cpp-app-health-check \
    --global # or --region=[REGION]

Monitoring Elasticsearch Clusters

Elasticsearch clusters are complex distributed systems. Monitoring them requires a multi-faceted approach, focusing on cluster health, node status, resource utilization (CPU, memory, disk I/O), and query performance.

Cluster Health API

The Elasticsearch Cluster Health API is the first line of defense. It provides a high-level overview of the cluster’s state.

curl -X GET "localhost:9200/_cluster/health?pretty"

# Example Output:
# {
#   "cluster_name" : "my-es-cluster",
#   "status" : "green",
#   "timed_out" : false,
#   "number_of_nodes" : 3,
#   "number_of_data_nodes" : 3,
#   "active_primary_shards" : 10,
#   "active_shards" : 30,
#   "relocating_shards" : 0,
#   "initializing_shards" : 0,
#   "unassigned_shards" : 0,
#   "delayed_unassigned_shards" : 0,
#   "number_of_pending_tasks" : 0,
#   "every_other_node_is_master" : true,
#   "max_virtual_memory_node_count" : 1,
#   "max_number_of_threads_per_node" : 1024
# }

The status field is critical: green (all shards allocated), yellow (all primary shards allocated, some replicas not), red (some primary shards not allocated). A transition to yellow or red requires immediate investigation.

Node Stats API

To drill down into individual node performance, the Node Stats API is invaluable. It provides metrics on CPU usage, memory, filesystem, JVM heap, and Elasticsearch-specific statistics.

curl -X GET "localhost:9200/_nodes/stats?pretty"

# Example Snippet for a specific node:
# "nodes" : {
#   "node_id_1" : {
#     "name" : "node-1",
#     "transport_address" : "inet[/10.0.0.1:9300]",
#     "host" : "node-1",
#     "ip" : "10.0.0.1",
#     "roles" : [ "master", "data", "ingest" ],
#     "attributes" : { },
#     "os" : { ... },
#     "process" : { ... },
#     "jvm" : {
#       "mem" : {
#         "heap_used_in_bytes" : 1073741824,
#         "heap_max_in_bytes" : 1073741824
#       },
#       ...
#     },
#     "indices" : { ... },
#     "fs" : { ... },
#     "thread_pool" : { ... },
#     "network" : { ... },
#     "breaker" : { ... }
#   },
#   ...
# }

Key metrics to watch here include:

  • JVM Heap Usage: High heap usage (approaching heap_max_in_bytes) can lead to garbage collection pauses and instability. Aim to keep heap usage below 80-90%.
  • CPU Usage: High CPU on data nodes can indicate heavy indexing or search load. High CPU on master nodes might suggest issues with cluster state management.
  • Disk I/O: Slow disk performance will directly impact indexing and search latency.
  • Thread Pool Queues: Growing queues in thread pools (e.g., write, search) indicate that the node is overloaded and cannot process requests fast enough.

Google Cloud Elasticsearch Monitoring Setup

For Elasticsearch clusters deployed on Google Cloud (e.g., on GCE VMs or using Google Kubernetes Engine), Cloud Monitoring is essential. You can deploy the Cloud Monitoring Agent (Ops Agent) to collect metrics.

Ops Agent Configuration for Elasticsearch

The Ops Agent can collect system metrics and can be configured to scrape custom metrics, including those from Elasticsearch. You’ll typically configure it to collect JVM metrics, filesystem metrics, and potentially parse Elasticsearch logs for errors.

A typical Ops Agent configuration file (/etc/google-cloud-ops-agent/config.yaml) might look like this:

metrics:
  # Collects metrics from the Ops Agent's built-in collectors.
  plugins:
    # Collects system metrics like CPU, memory, disk, and network.
    - type: cpu
    - type: memory
    - type: disk
    - type: network
    # Collects JVM metrics from Elasticsearch.
    - type: jvm
      # Elasticsearch typically exposes JMX metrics on port 9010 by default.
      # Adjust host and port if your Elasticsearch setup differs.
      host: localhost
      port: 9010
      # You might need to configure JMX authentication if enabled.
      # username: "user"
      # password: "password"

logs:
  # Collects logs from Elasticsearch.
  - type: file
    include:
      # Adjust path to your Elasticsearch log files.
      - /var/log/elasticsearch/my-es-cluster.log
    # You can define parsers here to extract structured data from logs.
    # For example, to parse JSON logs:
    # format: json
    # Or for grok patterns:
    # format: single_line
    # pattern: "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:severity} \[%{DATA:thread}\] %{GREEDYDATA:message}"

# Optional: Enable Prometheus scraping if you have Prometheus exporters running.
# For Elasticsearch, you might use a Prometheus exporter that scrapes the _nodes/stats API.
# prometheus:
#   configs:
#     - url: http://localhost:9108/metrics # Example for a Prometheus exporter

After updating the configuration, restart the Ops Agent:

sudo systemctl restart google-cloud-ops-agent

These collected metrics will then be available in Cloud Monitoring, where you can create dashboards and alerting policies.

Alerting Strategies

Effective alerting is crucial for proactive issue resolution. For both C++ applications and Elasticsearch, we should set up alerts based on:

  • Health Check Failures: For C++ apps, alert immediately when external health checks start failing.
  • Cluster Status: For Elasticsearch, alert on yellow or red cluster status.
  • Resource Saturation: High CPU, low disk space, high JVM heap usage, or growing thread pool queues on Elasticsearch nodes.
  • Latency: Monitor request latency for your C++ app and search/indexing latency for Elasticsearch.
  • Error Rates: Track the rate of HTTP 5xx errors from your C++ app or Elasticsearch.

In Cloud Monitoring, you can create alerting policies that trigger notifications via Email, Slack, PagerDuty, or Pub/Sub. For example, an Elasticsearch alert could be configured for:

Metric: elasticsearch.googleapis.com/cluster/health/status
Condition: Threshold is above 1 (where 0=green, 1=yellow, 2=red)
For: 5 minutes
Trigger: Any time the condition is met

And for the C++ application:

Metric: loadbalancing.googleapis.com/https/backend_latencies (or http/backend_latencies)
Filter: health_check_status = "OFFLINE"
Condition: Count is above 0
For: 1 minute
Trigger: Any time the condition is met

By combining application-level health checks, robust infrastructure monitoring, and intelligent alerting, you can significantly improve the reliability and availability of your C++ applications and Elasticsearch clusters on Google Cloud.