Vengala Vinay

Having 9+ Years of Experience in Software Development

Home » Server Monitoring Best Practices: Keeping Your Shopify App and Elasticsearch Clusters Alive on AWS

Server Monitoring Best Practices: Keeping Your Shopify App and Elasticsearch Clusters Alive on AWS

Proactive Elasticsearch Cluster Health Monitoring on AWS

Maintaining the health and performance of Elasticsearch clusters, especially those powering critical Shopify app functionalities, requires a robust monitoring strategy. On AWS, this translates to leveraging CloudWatch extensively, supplemented by direct Elasticsearch API calls and potentially third-party tools. The goal is not just to detect failures but to anticipate them by tracking key performance indicators (KPIs) and resource utilization.

Essential Elasticsearch Metrics via CloudWatch Agent

While AWS provides basic EC2 metrics, deep Elasticsearch insights necessitate custom metrics. The CloudWatch Agent is your primary tool for this. We’ll configure it to collect JVM heap usage, thread pool statistics, and node-level disk I/O. Ensure your Elasticsearch nodes are running on EC2 instances or ECS containers where the agent can be installed.

First, install the CloudWatch Agent on your Elasticsearch nodes. The configuration file (typically /opt/aws/amazon-cloudwatch-agent/bin/config.json) is crucial. Here’s a sample configuration focusing on Elasticsearch:

{
  "agent": {
    "metrics_collection_interval": 60,
    "run_as_user": "cwagent"
  },
  "metrics": {
    "namespace": "Elasticsearch/ShopifyApp",
    "append_dimensions": {
      "InstanceId": "${aws:InstanceId}",
      "ImageId": "${aws:ImageId}"
    },
    "metrics_collected": {
      "jvm": {
        "measurement": [
          "mem_used_percent",
          "threads_count"
        ],
        "metrics_collection_interval": 60,
        "resources": [
          "*"
        ]
      },
      "thread_pool": {
        "measurement": [
          "active",
          "rejected",
          "queued"
        ],
        "metrics_collection_interval": 60,
        "pools": [
          "write",
          "search",
          "get",
          "index",
          "bulk"
        ]
      },
      "disk": {
        "measurement": [
          "read_bytes",
          "write_bytes",
          "read_ops",
          "write_ops",
          "free_percent"
        ],
        "metrics_collection_interval": 300,
        "resources": [
          "*"
        ]
      }
    }
  }
}

After creating or updating the configuration file, start/restart the agent:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json -s

Custom Elasticsearch API Checks with Lambda

CloudWatch Agent provides system-level and JVM metrics. However, Elasticsearch-specific cluster health, shard status, and indexing latency are best queried directly. AWS Lambda functions, triggered on a schedule (e.g., every minute via CloudWatch Events), are ideal for this. These functions can query the Elasticsearch API and publish custom metrics to CloudWatch.

Here’s a Python Lambda function example that checks cluster health and publishes a custom metric for unhealthy shards:

import json
import boto3
import requests
import os

# --- Configuration ---
ES_ENDPOINT = os.environ.get('ES_ENDPOINT', 'YOUR_ES_CLUSTER_URL') # e.g., https://search-my-cluster-xxxx.us-east-1.es.amazonaws.com
CLOUDWATCH_NAMESPACE = 'Elasticsearch/ShopifyApp'
# For authenticated endpoints (e.g., AWS Elasticsearch Service with IAM auth)
AWS_REGION = os.environ.get('AWS_REGION', 'us-east-1')
USE_IAM_AUTH = os.environ.get('USE_IAM_AUTH', 'true').lower() == 'true'
# --- End Configuration ---

cloudwatch = boto3.client('cloudwatch', region_name=AWS_REGION)

def get_es_auth_headers():
    if not USE_IAM_AUTH:
        return {}
    
    session = boto3.Session()
    credentials = session.get_credentials()
    auth = boto3.session.Session().get_caller_identity()
    
    # For AWS Elasticsearch Service, use SigV4 signing
    # This requires the 'requests-aws4auth' library or manual signing
    # For simplicity, we'll assume manual signing or a pre-configured environment
    # In a real-world scenario, you'd likely use requests_aws4auth
    # Example using requests_aws4auth (install it: pip install requests-aws4auth)
    # from requests_aws4auth import AWS4Auth
    # aws_auth = AWS4Auth(credentials.access_key, credentials.secret_key, AWS_REGION, 'es', session_token=credentials.token)
    # return {'Authorization': aws_auth} # This is conceptual, actual implementation varies
    
    # Placeholder for IAM credentials if not using requests_aws4auth
    # This part is complex and depends on your exact setup.
    # For basic HTTP auth, you'd use Basic Auth.
    # For IAM, you need to sign the request.
    # For this example, we'll assume a simpler setup or that IAM is handled externally.
    print("IAM authentication is enabled but not fully implemented in this snippet. Ensure your Lambda execution role has ES access or configure SigV4 signing.")
    return {} # Return empty for now, needs proper SigV4 implementation

def publish_metric(metric_name, value, unit='Count'):
    try:
        cloudwatch.put_metric_data(
            Namespace=CLOUDWATCH_NAMESPACE,
            MetricData=[
                {
                    'MetricName': metric_name,
                    'Value': value,
                    'Unit': unit
                },
            ]
        )
        print(f"Published metric: {metric_name}={value}")
    except Exception as e:
        print(f"Error publishing metric {metric_name}: {e}")

def lambda_handler(event, context):
    headers = get_es_auth_headers()
    
    try:
        # 1. Check Cluster Health
        health_url = f"{ES_ENDPOINT}/_cluster/health"
        response = requests.get(health_url, auth=('user', 'password'), headers=headers, timeout=10) # Adjust auth as needed
        response.raise_for_status() # Raise HTTPError for bad responses (4xx or 5xx)
        health_data = response.json()

        status = health_data.get('status')
        unassigned_shards = health_data.get('unassigned_shards', 0)
        
        publish_metric('ClusterHealthStatus', 1 if status == 'green' else (2 if status == 'yellow' else 3))
        publish_metric('ClusterUnassignedShards', unassigned_shards)

        print(f"Cluster health: {status}, Unassigned Shards: {unassigned_shards}")

        # 2. Check Node Status (optional, can be noisy)
        # nodes_url = f"{ES_ENDPOINT}/_cat/nodes?h=ip,heap.percent,load_1m"
        # response = requests.get(nodes_url, auth=('user', 'password'), headers=headers, timeout=10)
        # response.raise_for_status()
        # nodes_data = response.text
        # print(f"Node status:\n{nodes_data}")

        # 3. Check Indexing Latency (example: _refresh interval)
        # This is a simplified example. Real latency monitoring might involve
        # tracking document ingestion times or using the _nodes/stats API.
        # For now, let's just check if the cluster is responsive.
        
        return {
            'statusCode': 200,
            'body': json.dumps('Elasticsearch health check successful!')
        }

    except requests.exceptions.RequestException as e:
        print(f"Error connecting to Elasticsearch: {e}")
        publish_metric('ElasticsearchConnectionErrors', 1)
        return {
            'statusCode': 500,
            'body': json.dumps(f'Error connecting to Elasticsearch: {e}')
        }
    except Exception as e:
        print(f"An unexpected error occurred: {e}")
        publish_metric('ElasticsearchUnexpectedErrors', 1)
        return {
            'statusCode': 500,
            'body': json.dumps(f'An unexpected error occurred: {e}')
        }

Deployment Notes:

  • Replace YOUR_ES_ENDPOINT with your actual Elasticsearch cluster URL.
  • Configure environment variables for ES_ENDPOINT, AWS_REGION, and USE_IAM_AUTH in your Lambda function settings.
  • Ensure the Lambda execution role has permissions to publish metrics to CloudWatch (cloudwatch:PutMetricData).
  • If using AWS Elasticsearch Service with IAM authentication, you’ll need to implement SigV4 signing for your requests. The requests-aws4auth library is highly recommended for this.
  • Set up a CloudWatch Events rule to trigger this Lambda function every minute.

Shopify App Backend Monitoring (PHP Example)

Your Shopify app’s backend, often built with PHP (e.g., Laravel, Symfony), needs its own monitoring. This includes request latency, error rates, database performance, and queue processing times.

Error Tracking and Logging

Implement a robust logging strategy. Tools like Monolog are standard in PHP. Forwarding logs to CloudWatch Logs is essential for centralized analysis and alerting.

<?php
require 'vendor/autoload.php';

use Monolog\Logger;
use Monolog\Handler\StreamHandler;
use Monolog\Handler\CloudWatchHandler; // Requires aws/aws-sdk-php

// Create a logger
$log = new Logger('shopify_app');

// Add a handler for local file logging (for debugging)
$log->pushHandler(new StreamHandler('php://stdout', Logger::DEBUG));

// Add a CloudWatch handler
// Ensure your Lambda execution role or EC2 instance profile has 'logs:CreateLogGroup',
// 'logs:CreateLogStream', and 'logs:PutLogEvents' permissions.
$cloudwatchHandler = new CloudWatchHandler(
    [
        'region' => 'us-east-1', // Your AWS region
        'version' => 'latest',
        'logs' => [
            'logGroupName' => '/aws/lambda/your-function-name', // Or your EC2 log group
            'logStreamName' => 'shopify-app-backend'
        ]
    ],
    Logger::DEBUG // Log level
);
$log->pushHandler($cloudwatchHandler);

// Example usage
try {
    // Simulate an operation
    $data = json_decode('invalid json', true);
    if (json_last_error() !== JSON_ERROR_NONE) {
        throw new \Exception('Failed to decode JSON: ' . json_last_error_msg());
    }
    // ... process data ...
} catch (\Exception $e) {
    $log->error('An error occurred during data processing.', ['exception' => $e]);
}

$log->info('User processed successfully.', ['user_id' => 123]);
?>

For production, consider using a dedicated log aggregation service like Datadog, Splunk, or AWS OpenSearch Service (if not already using it for your primary Elasticsearch cluster) with agents configured to forward logs.

Application Performance Monitoring (APM)

For deep insights into request tracing, database query performance, and external API calls, an APM solution is invaluable. AWS X-Ray is a good native option. Alternatively, Datadog APM, New Relic, or Dynatrace offer more comprehensive features.

Integrating AWS X-Ray with PHP typically involves using the AWS SDK for PHP and instrumenting your code. For frameworks like Laravel, packages exist to simplify this.

# Install the X-Ray SDK for PHP (if not using a framework integration)
composer require aws/aws-sdk-php
composer require pda/x-ray-sdk-php

# Example instrumentation (simplified)
use Aws\XRay\XRayClient;
use XRay\Segment;
use XRay\Emitter\UDPEmitter;

$xrayClient = new XRayClient([
    'region' => 'us-east-1',
    'version' => 'latest'
]);

$emitter = new UDPEmitter('127.0.0.1', 2000); // Default X-Ray daemon address
$segment = new Segment('ShopifyAppBackend', $emitter);
$segment->begin();

try {
    // Your application logic here
    // e.g., database queries, API calls
    $db = new PDO(...);
    $stmt = $db->prepare("SELECT * FROM orders WHERE shop_id = ?");
    $stmt->execute([$shopId]);
    $orders = $stmt->fetchAll();

    // Create a subsegment for the database query
    $subsegment = $segment->beginSubSegment('DatabaseQuery');
    $subsegment->putAnnotation('query_type', 'SELECT');
    $subsegment->putAnnotation('table', 'orders');
    $subsegment->end();

} catch (\Exception $e) {
    $segment->addException($e);
    throw $e; // Re-throw the exception
} finally {
    $segment->end();
}

Ensure the X-Ray daemon is running on your EC2 instances or that your Lambda function’s execution role has permissions to send traces to X-Ray (xray:PutTraceSegments).

Alerting Strategy

Effective alerting is crucial. Define clear thresholds for your key metrics. Use CloudWatch Alarms to trigger notifications via SNS topics, which can then fan out to Slack, PagerDuty, or email.

Elasticsearch Alerts

  • High JVM Heap Usage: Alarm if mem_used_percent exceeds 85% for more than 5 minutes.
  • High Rejected Thread Pool Tasks: Alarm if rejected count for write or search pools increases significantly over a short period (e.g., > 10 tasks in 1 minute).
  • Unassigned Shards: Alarm immediately if ClusterUnassignedShards metric is greater than 0.
  • Cluster Health Yellow/Red: Alarm immediately if ClusterHealthStatus metric is 2 or 3.
  • High Disk Usage: Alarm if free_percent on any data disk drops below 15%.

Shopify App Backend Alerts

  • High Error Rate: Monitor logs for exceptions. Set up alarms based on the rate of specific error messages or a general increase in error logs per minute.
  • High Request Latency: If using APM or custom metrics, alarm if average request latency exceeds a defined threshold (e.g., 500ms) for a sustained period.
  • Queue Backlog: If using background job queues (e.g., Redis Queue, SQS), alarm if the number of pending jobs exceeds a threshold.
  • Database Connection Errors: Monitor logs for database connection failures.

Configure SNS topics and subscriptions to route these alarms to the appropriate on-call engineers. Use different severity levels for alarms (e.g., OK, WARN, CRITICAL) to manage notification fatigue.

Regular Audits and Performance Tuning

Monitoring data is only useful if acted upon. Schedule regular reviews of your monitoring dashboards and historical data. Look for trends that indicate potential future problems, such as gradually increasing JVM heap usage or slowly degrading query performance. Use this data to proactively tune your Elasticsearch cluster (e.g., shard allocation, indexing strategies) and your application code.