Vengala Vinay

Having 9+ Years of Experience in Software Development

Home » Server Monitoring Best Practices: Keeping Your Shopify App and Elasticsearch Clusters Alive on Google Cloud

Server Monitoring Best Practices: Keeping Your Shopify App and Elasticsearch Clusters Alive on Google Cloud

Proactive Elasticsearch Health Checks with Google Cloud Monitoring

Maintaining the health and performance of Elasticsearch clusters, especially those powering critical Shopify applications, demands a robust monitoring strategy. On Google Cloud Platform (GCP), this translates to leveraging Cloud Monitoring (formerly Stackdriver) for deep visibility into your Elasticsearch nodes and the applications interacting with them. We’ll focus on actionable metrics and alerting configurations that go beyond basic CPU/memory utilization.

Key Elasticsearch Metrics for Cloud Monitoring

While GCP’s Compute Engine metrics provide foundational OS-level insights, true Elasticsearch health requires diving into its internal metrics. The Elasticsearch `_cat` APIs are invaluable for this. We’ll use a custom metrics exporter or agent to push these to Cloud Monitoring.

Node Status and JVM Health

A node’s ability to join and remain part of the cluster is paramount. JVM heap usage is a primary indicator of potential OutOfMemory errors and garbage collection pressure.

Example: Fetching Node Status and JVM Heap

You can retrieve this information via the Elasticsearch API. A simple `curl` command can illustrate:

curl -X GET "http://localhost:9200/_cat/nodes?v&h=ip,heap.percent,heap.max,load,uptime"
curl -X GET "http://localhost:9200/_nodes/stats/jvm?pretty"

To ingest these into Cloud Monitoring, consider using the OpenTelemetry Collector with an Elasticsearch receiver or a custom agent that periodically polls these endpoints and sends metrics via the Cloud Monitoring API. For instance, a Python script using the `google-cloud-monitoring` library:

import google.auth
from google.cloud import monitoring_v3
from google.protobuf.timestamp_pb2 import Timestamp
import time
import requests
import json

# Configuration
ES_HOST = "http://localhost:9200"
PROJECT_ID = "your-gcp-project-id"
METRIC_SCOPE = "your-gcp-metric-scope" # e.g., "projects/your-gcp-project-id"

client = monitoring_v3.MetricServiceClient()
project_name = client.project_path(PROJECT_ID)

def get_es_metrics():
    try:
        # Get node stats
        nodes_response = requests.get(f"{ES_HOST}/_cat/nodes?h=ip,heap.percent,heap.max,load,uptime", timeout=5)
        nodes_response.raise_for_status()
        node_lines = nodes_response.text.strip().split('\n')
        
        # Get JVM stats
        jvm_response = requests.get(f"{ES_HOST}/_nodes/stats/jvm", timeout=5)
        jvm_response.raise_for_status()
        jvm_data = jvm_response.json()

        metrics = []
        now = time.time()
        seconds = int(now)
        nanos = int((now - seconds) * 10**9)
        timestamp = Timestamp(seconds=seconds, nanos=nanos)

        for line in node_lines[1:]: # Skip header
            parts = line.split()
            ip = parts[0]
            heap_percent = float(parts[1])
            heap_max = int(parts[2].replace('b', '')) # Assuming 'b' for bytes
            load = float(parts[3])
            uptime_str = parts[4] # e.g., "1d10h" or "2h30m" - needs parsing for actual uptime in ms

            # Find corresponding JVM data for this node (assuming node name matches IP or a part of it)
            node_jvm_stats = None
            for node_id, node_info in jvm_data['nodes'].items():
                if ip in node_info['host'] or ip.split('.')[-1] in node_info['host']: # Simple heuristic
                    node_jvm_stats = node_info
                    break
            
            if node_jvm_stats:
                heap_used_bytes = node_jvm_stats['jvm']['mem']['heap_used_in_bytes']
                heap_max_bytes = node_jvm_stats['jvm']['mem']['heap_max_in_bytes']
                gc_collection_time_ms = node_jvm_stats['jvm']['gc']['collectors']['old']['collection_time_in_millis']
                gc_collection_count = node_jvm_stats['jvm']['gc']['collectors']['old']['collection_count']

                # Heap Usage Percentage (from JVM stats for accuracy)
                heap_usage_percent_jvm = (heap_used_bytes / heap_max_bytes) * 100 if heap_max_bytes > 0 else 0

                # Add metrics to list
                metrics.append({
                    "metric.type": "custom.googleapis.com/elasticsearch/node/heap_usage_percent",
                    "resource": {
                        "type": "gce_instance", # Or a custom resource type if not directly mapped
                        "labels": {
                            "project_id": PROJECT_ID,
                            "instance_name": f"es-node-{ip.replace('.', '-')}", # Example instance name
                            "zone": "us-central1-a" # Replace with actual zone
                        }
                    },
                    "points": [{"interval": {"endTime": timestamp}, "value": {"doubleValue": heap_usage_percent_jvm}}]
                })
                metrics.append({
                    "metric.type": "custom.googleapis.com/elasticsearch/node/jvm_heap_used_bytes",
                    "resource": {
                        "type": "gce_instance",
                        "labels": {
                            "project_id": PROJECT_ID,
                            "instance_name": f"es-node-{ip.replace('.', '-')}",
                            "zone": "us-central1-a"
                        }
                    },
                    "points": [{"interval": {"endTime": timestamp}, "value": {"doubleValue": heap_used_bytes}}]
                })
                metrics.append({
                    "metric.type": "custom.googleapis.com/elasticsearch/node/jvm_gc_collection_time_ms",
                    "resource": {
                        "type": "gce_instance",
                        "labels": {
                            "project_id": PROJECT_ID,
                            "instance_name": f"es-node-{ip.replace('.', '-')}",
                            "zone": "us-central1-a"
                        }
                    },
                    "points": [{"interval": {"endTime": timestamp}, "value": {"doubleValue": gc_collection_time_ms}}]
                })
                metrics.append({
                    "metric.type": "custom.googleapis.com/elasticsearch/node/jvm_gc_collection_count",
                    "resource": {
                        "type": "gce_instance",
                        "labels": {
                            "project_id": PROJECT_ID,
                            "instance_name": f"es-node-{ip.replace('.', '-')}",
                            "zone": "us-central1-a"
                        }
                    },
                    "points": [{"interval": {"endTime": timestamp}, "value": {"doubleValue": gc_collection_count}}]
                })
                # Add more metrics as needed (load, uptime, etc.)

        # Write metrics to Cloud Monitoring
        if metrics:
            write_timeseries_request = monitoring_v3.WriteTimeSeriesRequest()
            write_timeseries_request.name = project_name
            write_timeseries_request.time_series.extend(
                client.new_time_series(m) for m in metrics
            )
            client.write_time_series(request=write_timeseries_request)
            print(f"Successfully wrote {len(metrics)} time series.")
        else:
            print("No metrics to write.")

    except requests.exceptions.RequestException as e:
        print(f"Error fetching Elasticsearch metrics: {e}")
    except Exception as e:
        print(f"An unexpected error occurred: {e}")

if __name__ == "__main__":
    # Schedule this to run periodically (e.g., every 60 seconds)
    while True:
        get_es_metrics()
        time.sleep(60)

Important Considerations:

  • Resource Mapping: The example uses gce_instance with a derived instance_name. For better accuracy, ensure your custom resource or a more specific mapping is used if your Elasticsearch nodes aren’t directly GCE instances (e.g., GKE pods).
  • Authentication: If your Elasticsearch cluster requires authentication, include appropriate headers or credentials in the requests.get calls.
  • Error Handling: Robust error handling for network issues, Elasticsearch unavailability, and JSON parsing is crucial.
  • Scheduling: This script needs to be run periodically. Consider using cron, a systemd timer, or a container orchestration scheduler.

Cluster Health and Shard Status

Beyond individual node health, the cluster’s overall state and the status of its shards are critical. Unassigned or relocating shards indicate problems that can impact search performance and data availability.

Example: Fetching Cluster Health and Shard Counts

curl -X GET "http://localhost:9200/_cat/health?v"
curl -X GET "http://localhost:9200/_cat/shards?v&h=index,shard,prirep,state,unassigned.reason"

These metrics should also be exported. Focus on:

  • Cluster status (green, yellow, red).
  • Number of unassigned shards.
  • Number of relocating shards.
  • Number of initializing shards.

The Python script can be extended to include these by parsing the output of _cat/health and _cat/shards. For example, to capture the number of unassigned shards:

# ... (inside get_es_metrics function, after fetching other data)

        # Fetch cluster health
        health_response = requests.get(f"{ES_HOST}/_cat/health?h=status,unassign", timeout=5)
        health_response.raise_for_status()
        health_lines = health_response.text.strip().split('\n')
        
        if len(health_lines) > 1: # Ensure there's data beyond the header
            health_parts = health_lines[1].split()
            cluster_status = health_parts[0]
            unassigned_shards = int(health_parts[1])

            metrics.append({
                "metric.type": "custom.googleapis.com/elasticsearch/cluster/unassigned_shards",
                "resource": {
                    "type": "generic_task", # Or a custom resource type for the cluster
                    "labels": {
                        "project_id": PROJECT_ID,
                        "job_name": "elasticsearch-cluster"
                    }
                },
                "points": [{"interval": {"endTime": timestamp}, "value": {"int64Value": unassigned_shards}}]
            })
            # Add metric for cluster status (e.g., map to integers: green=0, yellow=1, red=2)
            status_map = {"green": 0, "yellow": 1, "red": 2}
            cluster_status_val = status_map.get(cluster_status, -1) # -1 for unknown
            metrics.append({
                "metric.type": "custom.googleapis.com/elasticsearch/cluster/status",
                "resource": {
                    "type": "generic_task",
                    "labels": {
                        "project_id": PROJECT_ID,
                        "job_name": "elasticsearch-cluster"
                    }
                },
                "points": [{"interval": {"endTime": timestamp}, "value": {"int64Value": cluster_status_val}}]
            })
# ... (rest of the function and script)

Shopify App Integration Monitoring

Your Shopify app’s interaction with Elasticsearch is a critical path. Monitoring this connection involves tracking API response times, error rates, and throughput from the application’s perspective.

Application-Level Metrics

Instrument your application code to emit custom metrics for Elasticsearch operations. This provides direct insight into how the application experiences Elasticsearch performance.

Example: Python (Flask/Django) with Cloud Monitoring Client

from google.cloud import monitoring_v3
from google.protobuf.timestamp_pb2 import Timestamp
import time
import requests
import functools

# Assuming you have a function to interact with Elasticsearch
def call_elasticsearch_api(query):
    start_time = time.time()
    try:
        response = requests.post("http://localhost:9200/your_index/_search", json={"query": query}, timeout=10)
        response.raise_for_status()
        return response.json()
    except requests.exceptions.RequestException as e:
        # Log the error and potentially raise a custom exception
        raise e
    finally:
        duration = time.time() - start_time
        # Send metrics to Cloud Monitoring
        send_es_operation_metrics(duration, "success" if 'response' in locals() and response.ok else "error")

def send_es_operation_metrics(duration_seconds, status):
    client = monitoring_v3.MetricServiceClient()
    project_id = "your-gcp-project-id"
    project_name = client.project_path(project_id)

    now = time.time()
    seconds = int(now)
    nanos = int((now - seconds) * 10**9)
    timestamp = Timestamp(seconds=seconds, nanos=nanos)

    # Metric for operation duration
    duration_metric = monitoring_v3.TimeSeries()
    duration_metric.metric.type = "custom.googleapis.com/shopify_app/elasticsearch/operation_duration_seconds"
    duration_metric.resource.type = "gce_instance" # Or your app's resource type
    duration_metric.resource.labels["project_id"] = project_id
    duration_metric.resource.labels["instance_name"] = "your-app-instance-name"
    duration_metric.resource.labels["zone"] = "us-central1-a"
    duration_metric.points.append(monitoring_v3.Point(
        interval={"endTime": timestamp},
        value={"doubleValue": duration_seconds}
    ))

    # Metric for operation status (count successes/errors)
    status_metric = monitoring_v3.TimeSeries()
    status_metric.metric.type = "custom.googleapis.com/shopify_app/elasticsearch/operation_status_count"
    status_metric.resource.type = "gce_instance"
    status_metric.resource.labels["project_id"] = project_id
    status_metric.resource.labels["instance_name"] = "your-app-instance-name"
    status_metric.resource.labels["zone"] = "us-central1-a"
    status_metric.resource.labels["status"] = status # e.g., "success", "error"
    status_metric.points.append(monitoring_v3.Point(
        interval={"endTime": timestamp},
        value={"int64Value": 1} # Increment count for this status
    ))

    write_timeseries_request = monitoring_v3.WriteTimeSeriesRequest()
    write_timeseries_request.name = project_name
    write_timeseries_request.time_series.extend([duration_metric, status_metric])
    
    try:
        client.write_time_series(request=write_timeseries_request)
        print(f"Sent ES metrics: duration={duration_seconds:.4f}s, status={status}")
    except Exception as e:
        print(f"Error writing metrics to Cloud Monitoring: {e}")

# Example usage in a web framework
# @app.route('/search')
# def search_route():
#     try:
#         results = call_elasticsearch_api("some_search_term")
#         return {"data": results}
#     except Exception as e:
#         # Log error, return appropriate HTTP response
#         return {"error": str(e)}, 500

This approach allows you to create dashboards and alerts in Cloud Monitoring based on:

  • Average/p95/p99 latency of Elasticsearch queries from your app.
  • Error rate of Elasticsearch operations (e.g., percentage of requests returning 5xx errors).
  • Throughput (requests per second) to Elasticsearch.

Alerting Strategies in Cloud Monitoring

Effective alerting is proactive, not reactive. Configure alerts based on thresholds that indicate impending issues rather than outright failures.

Essential Alerting Policies

  • High JVM Heap Usage: Alert when JVM heap usage exceeds 80% and 90% (warning and critical levels).
  • High GC Pause Times: Alert on sustained high garbage collection pause times, indicating potential performance degradation.
  • Unassigned Shards: Trigger an alert immediately if the number of unassigned shards becomes non-zero.
  • Cluster Status Red/Yellow: Critical alerts for these states.
  • High Application Latency: Alert when the p95 or p99 latency for Elasticsearch operations from your app exceeds a defined threshold (e.g., 500ms).
  • High Application Error Rate: Alert if the error rate for Elasticsearch operations from your app exceeds a certain percentage (e.g., 5%) over a rolling window.
  • Node Not Reporting: If custom metrics for a node stop appearing in Cloud Monitoring for a sustained period, it indicates the exporter/agent has failed or the node is down.

Example Alert Configuration (Conceptual)

Within the Google Cloud Console, navigate to Monitoring > Alerting. Create a new policy:

Alert: High JVM Heap Usage

Metric: custom.googleapis.com/elasticsearch/node/jvm_heap_used_bytes (or heap_usage_percent if directly available)

Filter: Select your Elasticsearch nodes (e.g., by instance name or metadata labels).

Transform: If using jvm_heap_used_bytes, calculate percentage: jvm_heap_used_bytes / jvm_heap_max_bytes * 100 (requires both metrics to be available).

Condition:

  • Threshold: > 80% (for warning)
  • For: 5 minutes
  • Trigger: Any time series violates

Add a second condition for > 90% with the same duration and trigger for a critical alert.

Alert: Application Elasticsearch Error Rate

Metric: custom.googleapis.com/shopify_app/elasticsearch/operation_status_count

Filter: Select your application instances.

Transform: Use a rate function on the count metric, then calculate the ratio of ‘error’ status counts to the total count (sum of ‘success’ and ‘error’).

Condition:

  • Threshold: > 0.05 (for 5% error rate)
  • For: 10 minutes
  • Trigger: Any time series violates

Infrastructure as Code for Monitoring

To ensure consistency and repeatability, manage your Cloud Monitoring configurations using Infrastructure as Code (IaC) tools like Terraform. This allows you to version control your alerts, dashboards, and metric configurations.

Terraform Example: Creating an Alert Policy

resource "google_monitoring_alert_policy" "elasticsearch_high_heap" {
  project      = "your-gcp-project-id"
  display_name = "Elasticsearch High JVM Heap Usage"
  combiner     = "OR" # Trigger if ANY condition is met

  conditions {
    display_name = "Warning: Heap Usage > 80%"
    condition_threshold {
      filter = <<-EOT
        metric.type="custom.googleapis.com/elasticsearch/node/heap_usage_percent"
        resource.type="gce_instance"
        resource.label.project_id="your-gcp-project-id"
        # Add more filters to target specific ES nodes
      EOT
      duration = "300s" # 5 minutes
      comparison = "COMPARISON_GT"
      threshold  = 80.0

      trigger {
        count = 1
      }
    }
  }

  conditions {
    display_name = "Critical: Heap Usage > 90%"
    condition_threshold {
      filter = <<-EOT
        metric.type="custom.googleapis.com/elasticsearch/node/heap_usage_percent"
        resource.type="gce_instance"
        resource.label.project_id="your-gcp-project-id"
        # Add more filters to target specific ES nodes
      EOT
      duration = "300s" # 5 minutes
      comparison = "COMPARISON_GT"
      threshold  = 90.0

      trigger {
        count = 1
      }
    }
  }

  alert_strategy {
    notification_rate_limit {
      period = "3600s" # 1 hour
    }
  }

  notification_channels = [
    "projects/your-gcp-project-id/notificationChannels/your-notification-channel-id", # e.g., PagerDuty, Slack
  ]

  documentation {
    content = "High JVM heap usage detected on Elasticsearch nodes. This can lead to performance degradation and OutOfMemory errors. Investigate node resource allocation and query patterns."
    mime_type = "text/markdown"
  }
}

resource "google_monitoring_alert_policy" "elasticsearch_unassigned_shards" {
  project      = "your-gcp-project-id"
  display_name = "Elasticsearch Unassigned Shards Detected"

  conditions {
    display_name = "Unassigned Shards > 0"
    condition_threshold {
      filter = <<-EOT
        metric.type="custom.googleapis.com/elasticsearch/cluster/unassigned_shards"
        resource.type="generic_task"
        resource.label.project_id="your-gcp-project-id"
      EOT
      duration = "60s" # Alert quickly
      comparison = "COMPARISON_GT"
      threshold  = 0.0

      trigger {
        count = 1
      }
    }
  }

  alert_strategy {
    notification_rate_limit {
      period = "1800s" # 30 minutes
    }
  }

  notification_channels = [
    "projects/your-gcp-project-id/notificationChannels/your-notification-channel-id",
  ]

  documentation {
    content = "Unassigned shards detected in the Elasticsearch cluster. This indicates potential data loss or availability issues. Investigate shard allocation and cluster health."
    mime_type = "text/markdown"
  }
}

By combining detailed internal Elasticsearch metrics, application-level performance indicators, and well-defined alerting policies managed via IaC, you can build a resilient monitoring system that keeps your Shopify app and its Elasticsearch backend healthy and performant on Google Cloud.