Vengala Vinay

Having 12+ Years of Experience in Software Development

Home » Server Monitoring Best Practices: Keeping Your Shopify App and Redis Clusters Alive on DigitalOcean

Server Monitoring Best Practices: Keeping Your Shopify App and Redis Clusters Alive on DigitalOcean

Establishing a Robust Monitoring Foundation with DigitalOcean and Prometheus

Maintaining the health and performance of a Shopify app, especially one leveraging distributed systems like Redis clusters, demands a proactive and granular monitoring strategy. On DigitalOcean, this translates to a layered approach, combining DigitalOcean’s native insights with powerful open-source tools. We’ll focus on Prometheus as our central metrics aggregation and alerting engine, complemented by Node Exporter for system-level metrics and Redis Exporter for Redis-specific visibility.

Deploying Prometheus and Node Exporter on DigitalOcean

A dedicated Droplet is the ideal starting point for your Prometheus server. This Droplet will scrape metrics from your application servers and Redis nodes. We’ll configure Node Exporter on each server that needs system-level monitoring.

Prometheus Server Setup

Start with a fresh Ubuntu 22.04 Droplet. Install Prometheus from its official repository or by downloading the binary. For simplicity and manageability, using `apt` is recommended if available, otherwise, manual download and systemd service creation is the way to go.

Installing Prometheus (Manual Method)

First, identify the latest stable release from the Prometheus download page. Let’s assume version 2.48.0 for this example.

Downloading and Extracting

On your Prometheus server Droplet:

Command

Execute the following commands:

Shell
wget https://github.com/prometheus/prometheus/releases/download/v2.48.0/prometheus-2.48.0.linux-amd64.tar.gz
tar xvfz prometheus-2.48.0.linux-amd64.tar.gz
cd prometheus-2.48.0.linux-amd64
Creating a Prometheus User and Directories

It’s best practice to run Prometheus under a dedicated, non-root user.

Shell
sudo groupadd --system prometheus
sudo useradd --system --no-create-home --shell /bin/false prometheus
sudo mkdir /etc/prometheus
sudo mkdir /var/lib/prometheus
sudo mv prometheus.yml /etc/prometheus/prometheus.yml
sudo mv consoles/ /etc/prometheus/consoles
sudo mv console_libraries/ /etc/prometheus/console_libraries
sudo chown -R prometheus:prometheus /etc/prometheus
sudo chown -R prometheus:prometheus /var/lib/prometheus
Configuring Prometheus Systemd Service

Create a systemd service file to manage the Prometheus process.

Shell
sudo nano /etc/systemd/system/prometheus.service

Paste the following content into the file:

INI
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus/ \
    --web.console.templates=/etc/prometheus/consoles \
    --web.console.libraries=/etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.target

Make sure to copy the Prometheus binary to /usr/local/bin/:

Shell
sudo mv prometheus /usr/local/bin/
sudo mv promtool /usr/local/bin/
Enabling and Starting Prometheus

Reload systemd, enable the service, and start it.

Shell
sudo systemctl daemon-reload
sudo systemctl enable prometheus
sudo systemctl start prometheus
sudo systemctl status prometheus

Access the Prometheus UI at http://YOUR_PROMETHEUS_DROPLET_IP:9090. You should see the default targets (Prometheus itself) already configured.

Node Exporter Deployment

Node Exporter is a crucial component for collecting hardware and OS metrics. Deploy it on every Droplet that hosts your Shopify app instances and Redis nodes.

Installing Node Exporter

Similar to Prometheus, download the latest stable release. Let’s use v1.7.0 as an example.

Shell
wget https://github.com/prometheus/node_exporter/releases/download/v1.7.0/node_exporter-1.7.0.linux-amd64.tar.gz
tar xvfz node_exporter-1.7.0.linux-amd64.tar.gz
cd node_exporter-1.7.0.linux-amd64
Creating a Node Exporter User and Systemd Service

Run Node Exporter as a dedicated user.

Shell
sudo groupadd --system node_exporter
sudo useradd --system --no-create-home --shell /bin/false node_exporter
sudo mv node_exporter /usr/local/bin/
sudo mv node_exporter.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable node_exporter
sudo systemctl start node_exporter
sudo systemctl status node_exporter
Configuring Node Exporter Systemd Service

Create the service file:

Shell
sudo nano /etc/systemd/system/node_exporter.service

Paste the following content:

INI
[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter \
    --collector.textfile.directory=/var/lib/node_exporter/textfile_collector

[Install]
WantedBy=multi-user.target

Node Exporter will now be available at http://YOUR_DROPLET_IP:9100/metrics. Ensure this port is accessible from your Prometheus server (adjust DigitalOcean firewall rules as needed).

Integrating Redis Exporter for Redis Cluster Monitoring

To gain deep insights into your Redis clusters, Redis Exporter is essential. It exposes Redis metrics in a Prometheus-compatible format. For a Redis cluster, you’ll typically run an instance of Redis Exporter for each Redis node or a dedicated instance that can connect to all nodes.

Deploying Redis Exporter

The deployment process is similar to Node Exporter. Download, extract, create a user, and set up a systemd service.

Installing Redis Exporter

Let’s use v1.5.0 as an example.

Shell
wget https://github.com/oliver006/redis_exporter/releases/download/v1.5.0/redis_exporter-v1.5.0.linux-amd64.tar.gz
tar xvfz redis_exporter-v1.5.0.linux-amd64.tar.gz
cd redis_exporter-v1.5.0.linux-amd64
Creating a Redis Exporter User and Systemd Service

Create a dedicated user and directories.

Shell
sudo groupadd --system redis_exporter
sudo useradd --system --no-create-home --shell /bin/false redis_exporter
sudo mkdir /var/lib/redis_exporter
sudo mv redis_exporter /usr/local/bin/
Configuring Redis Exporter Systemd Service

Create the service file. This example assumes Redis is running on the default port (6379) on the same host. If Redis is on a different host or port, adjust the REDIS_ADDR environment variable.

Shell
sudo nano /etc/systemd/system/redis_exporter.service

Paste the following content:

INI
[Unit]
Description=Redis Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=redis_exporter
Group=redis_exporter
Type=simple
Environment="REDIS_ADDR=redis://127.0.0.1:6379"
ExecStart=/usr/local/bin/redis_exporter \
    --web.listen-address=":9121" \
    --redis.namespace="redis" \
    --check-keys="" \
    --check-keys-pattern="" \
    --check-single-keys="" \
    --check-single-keys-pattern="" \
    --check-commands="" \
    --metrics. வழங்கு="default"

[Install]
WantedBy=multi-user.target

If you have a Redis cluster, you might want to configure Redis Exporter to connect to multiple nodes or use a specific client configuration. For a cluster, you might point it to one of the nodes and let it discover the rest, or explicitly list them. The REDIS_ADDR can be a comma-separated list of Redis instances or a Redis Sentinel address.

Shell
sudo systemctl daemon-reload
sudo systemctl enable redis_exporter
sudo systemctl start redis_exporter
sudo systemctl status redis_exporter

Redis Exporter will be accessible at http://YOUR_DROPLET_IP:9121/metrics. Ensure this port is open for scraping by Prometheus.

Configuring Prometheus to Scrape Targets

Now, we need to tell Prometheus where to find the metrics. Edit the Prometheus configuration file /etc/prometheus/prometheus.yml on your Prometheus server.

Editing prometheus.yml

Add scrape configurations for your Node Exporter and Redis Exporter instances. For a Redis cluster, you’ll add an entry for each Redis node’s exporter, or a single entry if Redis Exporter is configured to discover the cluster.

Shell
sudo nano /etc/prometheus/prometheus.yml

Modify the scrape_configs section. Here’s an example assuming you have:

  • Prometheus server at 10.10.0.1
  • App server 1 (with Node Exporter) at 10.10.0.2
  • App server 2 (with Node Exporter) at 10.10.0.3
  • Redis cluster node 1 (with Redis Exporter) at 10.10.0.4
  • Redis cluster node 2 (with Redis Exporter) at 10.10.0.5
YAML
global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.

scrape_configs:
  # Scrape Prometheus itself
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  # Scrape Node Exporter from App Servers
  - job_name: 'node_exporter_app'
    static_configs:
      - targets:
          - '10.10.0.2:9100'
          - '10.10.0.3:9100'

  # Scrape Redis Exporter from Redis Cluster Nodes
  - job_name: 'redis_exporter_cluster'
    static_configs:
      - targets:
          - '10.10.0.4:9121'
          - '10.10.0.5:9121'
          # If Redis Exporter is configured to discover the cluster,
          # you might only need one target and it will fetch metrics for all nodes.
          # Alternatively, list all nodes if each Redis Exporter instance
          # is configured to monitor a specific node.

  # Example for a single Redis instance (if not a cluster)
  # - job_name: 'redis_exporter_single'
  #   static_configs:
  #     - targets: ['YOUR_REDIS_DROPLET_IP:9121']

  # Service discovery for dynamic environments (e.g., Kubernetes, Consul)
  # For DigitalOcean Droplets, static_configs are often sufficient initially.
  # If using DigitalOcean Kubernetes, you'd use kubernetes_sd_configs.

After saving the changes, restart the Prometheus service to apply them:

Shell
sudo systemctl restart prometheus

Check the “Targets” page in the Prometheus UI (http://YOUR_PROMETHEUS_DROPLET_IP:9090/targets) to ensure your new targets are being scraped successfully and are in an “UP” state.

Essential Metrics to Monitor for Shopify Apps and Redis

Beyond basic availability, focus on metrics that indicate performance bottlenecks and potential issues.

System-Level Metrics (Node Exporter)

  • CPU Usage: node_cpu_seconds_total (rate over time) – High CPU can impact app responsiveness.
  • Memory Usage: node_memory_MemAvailable_bytes, node_memory_SwapFree_bytes – Low available memory or excessive swapping is a critical sign.
  • Disk I/O: node_disk_io_time_seconds_total (rate) – High disk latency can slow down operations.
  • Network Traffic: node_network_receive_bytes_total, node_network_transmit_bytes_total (rate) – Monitor for saturation or unusual spikes.
  • File Descriptors: node_filefd_allocated, node_filefd_maximum – Ensure you’re not hitting limits.

Redis Metrics (Redis Exporter)

  • Memory Usage: redis_memory_used_bytes, redis_memory_peak_bytes – Monitor against configured limits.
  • Connected Clients: redis_connected_clients – High client counts can indicate connection leaks or performance issues.
  • Commands Processed: redis_commands_processed_total (rate) – Overall workload.
  • Latency: redis_instantaneous_ops_per_sec, redis_latency_percentiles (if available/configured) – Crucial for real-time performance.
  • Cache Hit Rate: redis_keyspace_hits_total / (redis_keyspace_hits_total + redis_keyspace_misses_total) (rate) – A low hit rate means Redis is less effective as a cache.
  • Replication Lag: redis_master_repl_offset vs redis_slave_repl_offset (for master/replica setups) – Ensure replicas are up-to-date.
  • Evicted Keys: redis_evicted_keys_total (rate) – Indicates memory pressure and data loss.
  • Keyspace Size: redis_db_keys – Monitor growth.

Application-Specific Metrics

Beyond system and Redis metrics, instrument your Shopify app to expose custom metrics:

  • Request Latency: Track P95/P99 latency for key API endpoints.
  • Error Rates: Monitor HTTP 5xx and 4xx error counts.
  • Queue Depths: If using background job queues (e.g., Sidekiq, Resque), monitor queue lengths.
  • Database Connection Pool Usage: Track active and idle connections.
  • Shopify API Call Durations/Errors: Monitor interactions with the Shopify API.

These custom metrics can be exposed via a simple HTTP endpoint (e.g., using a Prometheus client library for your app’s language) and scraped by Prometheus.

Alerting with Prometheus Alertmanager

Metrics are only useful if they trigger action. Prometheus Alertmanager handles deduplication, grouping, and routing of alerts.

Setting up Alertmanager

Deploy Alertmanager similarly to Prometheus (download binary, create user, systemd service). Configure it to receive alerts from Prometheus and route them to your desired notification channels (email, Slack, PagerDuty).

Alertmanager Configuration (alertmanager.yml)

YAML
global:
  resolve_timeout: 5m
  slack_api_url: ''

route:
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: 'slack-notifications'

receivers:
  - name: 'slack-notifications'
    slack_configs:
      - channel: '#your-alerts-channel'
        send_resolved: true
        title: '{{ template "slack.default.title" . }}'
        text: '{{ template "slack.default.text" . }}'

# Example of silencing alerts during maintenance
#inhibit_rules:
#  - target_match:
#      severity: 'critical'
#    source_match:
#      severity: 'warning'
#    equal: ['alertname', 'cluster', 'service']

Configuring Prometheus to Send Alerts to Alertmanager

Update your /etc/prometheus/prometheus.yml to include the Alertmanager configuration:

YAML
global:
  scrape_interval: 15s
  evaluation_interval: 15s

alerting:
  alertmanagers:
    - static_configs:
        - targets:
           # Assuming Alertmanager is on the same Prometheus server for simplicity
           - 'localhost:9093'
           # If Alertmanager is on a different Droplet:
           # - 'YOUR_ALERTMANAGER_DROPLET_IP:9093'

scrape_configs:
  # ... (your existing scrape configs) ...
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node_exporter_app'
    static_configs:
      - targets:
          - '10.10.0.2:9100'
          - '10.10.0.3:9100'

  - job_name: 'redis_exporter_cluster'
    static_configs:
      - targets:
          - '10.10.0.4:9121'
          - '10.10.0.5:9121'

Defining Alerting Rules

Create rule files (e.g., /etc/prometheus/rules/redis_alerts.yml, /etc/prometheus/rules/app_alerts.yml) and reference them in prometheus.yml under the rule_files directive.

Example Redis Alert Rule

YAML
- alert: HighRedisMemoryUsage
  expr: |
    sum(redis_memory_used_bytes) by (instance) / sum(redis_memory_max_bytes) by (instance) * 100 > 85
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "High memory usage on Redis instance {{ $labels.instance }}"
    description: "Redis instance {{ $labels.instance }} is using {{ $value | printf \"%.2f\" }}% of its memory limit."

- alert: RedisEvictionsOccurred
  expr: |
    rate(redis_evicted_keys_total[5m]) > 0
  for: 1m
  labels:
    severity: critical
  annotations:
    summary: "Redis evictions detected on {{ $labels.instance }}"
    description: "Redis instance {{ $labels.instance }} is evicting keys, indicating memory pressure."

- alert: RedisHighLatency
  expr: |
    histogram_quantile(0.99, sum(rate(redis_command_duration_seconds_bucket[5m])) by (le, instance, command)) > 0.1
  for: 2m
  labels:
    severity: warning
  annotations:
    summary: "High Redis command latency on {{ $labels.instance }}"
    description: "99th percentile latency for Redis command '{{ $labels.command }}' on instance {{ $labels.instance }} is {{ $value | printf \"%.3f\" }}s."

Example Application Alert Rule

YAML
- alert: HighAppCpuUsage
  expr: |
    avg by (instance) (rate(node_cpu_seconds_total{mode!="idle"}[5m])) * 100 > 80
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "High CPU usage on application instance {{ $labels.instance }}"
    description: "Application instance {{ $labels.instance }} is experiencing high CPU load ({{ $value | printf \"%.2f\" }}%)."

- alert: AppInstanceDown
  expr: |
    up{job="node_exporter_app"} == 0
  for: 1m
  labels:
    severity: critical
  annotations:
    summary: "Application instance {{ $labels.instance }} is down"
    description: "The application instance {{ $labels.instance }} is unreachable by Prometheus."

Ensure your prometheus.yml includes:

YAML
rule_files:
  - "/etc/prometheus/rules/*.yml"

Restart Prometheus after adding rules and Alertmanager after configuration changes.

DigitalOcean Specific Considerations

When operating on DigitalOcean, leverage its features for a more resilient setup:

  • Firewall Rules: Configure DigitalOcean Cloud Firewalls to allow traffic only from necessary sources (e.g., Prometheus server to Node/Redis Exporters on ports 9100/9121, Alertmanager to Prometheus). Restrict SSH access to trusted IPs.
  • Private Networking: Utilize DigitalOcean’s private networking for Droplets within the same datacenter. This reduces egress costs and improves performance for inter-Droplet communication (e.g., Prometheus scraping, app communicating with Redis).
  • Load Balancers: For your Shopify app, use DigitalOcean Load Balancers to distribute traffic across multiple app server Droplets. Ensure the load balancer health checks are configured appropriately.
  • Managed Databases: While this guide focuses on self-hosted Redis, consider DigitalOcean Managed Databases for Redis if you want to offload operational overhead. Monitoring would then integrate with their provided metrics.
  • Snapshots and Backups: Regularly snapshot your Prometheus and Alertmanager Droplets, and ensure your Redis data is backed up.

Advanced Monitoring Techniques

To further enhance your monitoring:

  • Grafana Integration: Use Grafana as a visualization layer on top of Prometheus. Numerous pre-built dashboards exist for Node Exporter and Redis Exporter, providing richer insights than the basic Prometheus UI.
  • Blackbox Exporter: Monitor external services (like the Shopify API endpoints your app relies on) by simulating requests from different locations.
  • Service Discovery: For more dynamic environments or larger deployments, explore Prometheus’s service discovery integrations (e.g., Consul, EC2, Kubernetes SD) to automatically discover and scrape targets.
  • Distributed Tracing: For deep application performance analysis, integrate distributed tracing tools (like Jaeger or Zipkin) alongside your metrics.

By implementing this comprehensive monitoring strategy, you establish a resilient system capable of detecting, diagnosing, and alerting on issues before they impact your Shopify app’s availability and performance.

Recent Posts

Top Categories

Our Products

Our Services