Vengala Vinay

Having 9+ Years of Experience in Software Development

Home » Server Monitoring Best Practices: Keeping Your Shopify App and MongoDB Clusters Alive on Linode

Server Monitoring Best Practices: Keeping Your Shopify App and MongoDB Clusters Alive on Linode

Establishing a Robust Monitoring Foundation with Linode and Prometheus

Maintaining high availability for a Shopify app and its associated MongoDB clusters on Linode necessitates a proactive, data-driven approach to monitoring. We’ll leverage Prometheus for metrics collection and Grafana for visualization, deploying them strategically to capture critical system and application-level insights. This setup is designed for production environments, emphasizing low overhead and actionable alerts.

Deploying Prometheus on Linode

A dedicated Linode instance for Prometheus is recommended to ensure its performance isn’t impacted by application traffic. We’ll start with a minimal Debian/Ubuntu setup and install Prometheus from its official binary releases for maximum control and minimal dependencies.

First, create a dedicated user and directory for Prometheus:

sudo useradd --no-create-home --shell /bin/false prometheus
sudo mkdir /etc/prometheus
sudo mkdir /var/lib/prometheus
sudo chown prometheus:prometheus /etc/prometheus
sudo chown prometheus:prometheus /var/lib/prometheus

Download the latest Prometheus binary. Replace `X.Y.Z` with the current version.

wget https://github.com/prometheus/prometheus/releases/download/vX.Y.Z/prometheus-X.Y.Z.linux-amd64.tar.gz
tar xvfz prometheus-X.Y.Z.linux-amd64.tar.gz
sudo mv prometheus-X.Y.Z.linux-amd64/prometheus prometheus-X.Y.Z.linux-amd64/promtool /usr/local/bin/
sudo mv prometheus-X.Y.Z.linux-amd64/consoles /etc/prometheus/
sudo mv prometheus-X.Y.Z.linux-amd64/console_libraries /etc/prometheus/
sudo mv prometheus-X.Y.Z.linux-amd64/prometheus.yml /etc/prometheus/prometheus.yml
sudo chown -R prometheus:prometheus /etc/prometheus/consoles
sudo chown -R prometheus:prometheus /etc/prometheus/console_libraries
rm prometheus-X.Y.Z.linux-amd64.tar.gz
rm -rf prometheus-X.Y.Z.linux-amd64

Configure Prometheus to scrape itself and other targets. The default prometheus.yml needs to be adjusted. For initial setup, we’ll focus on scraping the Prometheus server itself and node exporters on other machines.

global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.

scrape_configs:
  # Scrape Prometheus itself
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  # Scrape Node Exporter for system metrics
  - job_name: 'node_exporter'
    static_configs:
      - targets:
        - '192.168.1.10:9100' # Replace with your Linode IP for app server
        - '192.168.1.11:9100' # Replace with your Linode IP for MongoDB primary
        - '192.168.1.12:9100' # Replace with your Linode IP for MongoDB secondary
        # Add more node_exporter targets as needed

Create a systemd service file for Prometheus:

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus/ \
    --web.console.templates-dir=/etc/prometheus/consoles \
    --web.console.libraries-dir=/etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.target

Enable and start the Prometheus service:

sudo systemctl daemon-reload
sudo systemctl enable prometheus
sudo systemctl start prometheus
sudo systemctl status prometheus

Deploying Node Exporter for System Metrics

Node Exporter is essential for gathering hardware and OS-level metrics from your Linode instances. Install it on each server that hosts your Shopify app or MongoDB instances.

Download the Node Exporter binary:

wget https://github.com/prometheus/node_exporter/releases/download/vX.Y.Z/node_exporter-X.Y.Z.linux-amd64.tar.gz
tar xvfz node_exporter-X.Y.Z.linux-amd64.tar.gz
sudo mv node_exporter-X.Y.Z.linux-amd64/node_exporter /usr/local/bin/
rm node_exporter-X.Y.Z.linux-amd64.tar.gz
rm -rf node_exporter-X.Y.Z.linux-amd64

Create a systemd service file for Node Exporter:

[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target

Create the user and group, then enable and start the service:

sudo useradd --no-create-home --shell /bin/false node_exporter
sudo systemctl daemon-reload
sudo systemctl enable node_exporter
sudo systemctl start node_exporter
sudo systemctl status node_exporter

Ensure that port 9100 is open in your Linode firewall for each server running Node Exporter, allowing Prometheus to scrape it.

Monitoring MongoDB with the MongoDB Exporter

To gain visibility into your MongoDB clusters, we’ll deploy the official MongoDB Prometheus exporter. This exporter connects to MongoDB instances and exposes metrics like connection counts, query performance, replication status, and more.

Download the MongoDB Exporter binary. Again, replace `X.Y.Z` with the latest version.

wget https://github.com/mongodb/mongodb-prometheus-exporter/releases/download/vX.Y.Z/mongodb_exporter-X.Y.Z.linux-amd64.tar.gz
tar xvfz mongodb_exporter-X.Y.Z.linux-amd64.tar.gz
sudo mv mongodb_exporter-X.Y.Z.linux-amd64/mongodb_exporter /usr/local/bin/
rm mongodb_exporter-X.Y.Z.linux-amd64.tar.gz
rm -rf mongodb_exporter-X.Y.Z.linux-amd64

Create a dedicated user and systemd service file for the MongoDB Exporter. This exporter requires MongoDB connection details, typically via a connection string.

[Unit]
Description=MongoDB Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=mongodb_exporter
Group=mongodb_exporter
Type=simple
# Replace with your actual MongoDB connection string
# Ensure this user has read-only access to relevant MongoDB databases
ExecStart=/usr/local/bin/mongodb_exporter --mongodb.uri="mongodb://user:password@mongo-primary:27017/admin?replicaSet=rs0"

[Install]
WantedBy=multi-user.target

Create the user and group, then configure the service. It’s highly recommended to use a dedicated MongoDB user with minimal privileges (e.g., read-only access to `local` and `admin` databases) for the exporter.

sudo useradd --no-create-home --shell /bin/false mongodb_exporter
sudo systemctl daemon-reload
sudo systemctl enable mongodb_exporter
sudo systemctl start mongodb_exporter
sudo systemctl status mongodb_exporter

Add the MongoDB exporter’s target to your Prometheus configuration (/etc/prometheus/prometheus.yml). This should be done on the Prometheus server.

scrape_configs:
  # ... other jobs ...

  - job_name: 'mongodb'
    static_configs:
      - targets:
        - '192.168.1.11:9216' # IP and port of your MongoDB exporter instance
        # Add more MongoDB exporter targets if you have multiple clusters or instances

After updating prometheus.yml, reload Prometheus:

sudo systemctl reload prometheus

Monitoring Your Shopify App

Monitoring a Shopify app involves a multi-faceted approach. Beyond basic system metrics from Node Exporter, we need to track application-specific performance and errors. For a PHP-based Shopify app, this typically means:

  • Web Server Metrics (Nginx/Apache): Request rates, error rates (4xx, 5xx), request duration.
  • Application Performance Metrics (APM): Response times, error rates within the PHP application, database query performance.
  • Background Job/Queue Metrics: If your app uses background jobs (e.g., for processing webhooks or sending emails), monitor queue lengths and processing times.
  • Custom Business Metrics: Number of orders processed, API call success/failure rates to Shopify.

For web server metrics, Prometheus can scrape Nginx or Apache directly if configured with the appropriate exporter or built-in metrics endpoint. For PHP APM, consider integrating a library like New Relic, Datadog, or a self-hosted solution like OpenTelemetry with a Prometheus exporter.

Let’s assume your Shopify app runs on a server with Nginx. We can configure Nginx to expose metrics. First, ensure the ngx_http_stub_status_module is compiled into your Nginx. Then, add a location to your Nginx configuration:

http {
    # ... other http configurations ...

    server {
        listen 8080; # Or any other port not in use
        server_name localhost;

        location /nginx_metrics {
            stub_status;
            allow 127.0.0.1; # Allow access only from localhost
            deny all;
        }
    }
}

Reload Nginx:

sudo systemctl reload nginx

Add this to your Prometheus configuration (/etc/prometheus/prometheus.yml) on the Prometheus server:

scrape_configs:
  # ... other jobs ...

  - job_name: 'nginx'
    static_configs:
      - targets:
        - '192.168.1.10:8080' # IP and port of your app server's Nginx metrics endpoint

For PHP application-level metrics, you’d typically instrument your code. For example, using a custom Prometheus client library for PHP. Here’s a conceptual example of how you might expose a custom metric:

<?php
require 'vendor/autoload.php'; // Assuming you use Composer

use Prometheus\CollectorRegistry;
use Prometheus\Render\CallbackRenderer;
use Prometheus\Storage\InMemory;

$registry = new CollectorRegistry(new InMemory());

// Create a counter for successful Shopify API calls
$counter = $registry->registerCounter(
    'shopify_api_calls_total',
    'Total number of successful Shopify API calls',
    ['endpoint']
);

// When a successful API call is made:
// $counter->inc(['/admin/api/2023-10/orders.json']);

// To expose these metrics for Prometheus to scrape, you'd typically have a dedicated endpoint
// For example, in your framework's routing:
// if ($_SERVER['REQUEST_URI'] === '/metrics') {
//     header('Content-Type: text/plain');
//     $renderer = new CallbackRenderer($registry);
//     echo $renderer->render();
//     exit;
// }
?>

You would then add this endpoint to your Prometheus configuration:

scrape_configs:
  # ... other jobs ...

  - job_name: 'shopify_app'
    static_configs:
      - targets:
        - '192.168.1.10:80' # Assuming your app is on port 80 and exposes /metrics

Alerting with Alertmanager

Prometheus alone collects metrics; Alertmanager handles deduplicating, grouping, and routing alerts to the correct receivers (e.g., Slack, PagerDuty, email). We’ll deploy Alertmanager on a separate Linode instance or alongside Prometheus if resources are constrained.

Download the Alertmanager binary:

wget https://github.com/prometheus/alertmanager/releases/download/vX.Y.Z/alertmanager-X.Y.Z.linux-amd64.tar.gz
tar xvfz alertmanager-X.Y.Z.linux-amd64.tar.gz
sudo mv alertmanager-X.Y.Z.linux-amd64/alertmanager /usr/local/bin/
sudo mv alertmanager-X.Y.Z.linux-amd64/alertmanager.yml /etc/alertmanager/alertmanager.yml
sudo mkdir /var/lib/alertmanager
sudo chown -R alertmanager:alertmanager /etc/alertmanager
sudo chown -R alertmanager:alertmanager /var/lib/alertmanager
rm alertmanager-X.Y.Z.linux-amd64.tar.gz
rm -rf alertmanager-X.Y.Z.linux-amd64

Configure Alertmanager (/etc/alertmanager/alertmanager.yml). This example shows basic routing to a Slack channel.

global:
  # The default SMTP From header field.
  smtp_from: '[email protected]'
  # How to resolve the smarthost for email.
  smtp_smarthost: 'smtp.example.com:587'
  # The default is to not use SMTP AUTH unless an SMTP username is provided.
  smtp_auth_username: '[email protected]'
  # If an SMTP password is provided, it must be plain text.
  smtp_auth_password: 'your_smtp_password'

route:
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h

  receiver: 'slack-notifications' # Default receiver

receivers:
  - name: 'slack-notifications'
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX' # Replace with your Slack webhook URL
        channel: '#alerts'
        send_resolved: true

  - name: 'email-notifications'
    email_configs:
      - to: '[email protected]'
        send_resolved: true

Create a systemd service file for Alertmanager:

[Unit]
Description=Alertmanager
Wants=network-online.target
After=network-online.target

[Service]
User=alertmanager
Group=alertmanager
Type=simple
ExecStart=/usr/local/bin/alertmanager \
    --config.file /etc/alertmanager/alertmanager.yml \
    --storage.tsdb.path /var/lib/alertmanager/ \
    --web.listen-address=0.0.0.0:9093

[Install]
WantedBy=multi-user.target

Create the user and group, then enable and start the service:

sudo useradd --no-create-home --shell /bin/false alertmanager
sudo systemctl daemon-reload
sudo systemctl enable alertmanager
sudo systemctl start alertmanager
sudo systemctl status alertmanager

Finally, configure Prometheus to send alerts to Alertmanager. Edit /etc/prometheus/prometheus.yml on the Prometheus server:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
          - 'localhost:9093' # Address of your Alertmanager instance

scrape_configs:
  # ... your scrape jobs ...

Reload Prometheus after this change.

sudo systemctl reload prometheus

Essential Prometheus Alerting Rules

Define meaningful alerts in Prometheus. These rules are typically stored in separate files and referenced in prometheus.yml. Create a directory for rules, e.g., /etc/prometheus/rules/.

Example alert rule for high CPU usage:

- alert: HighCpuUsage
  expr: 100 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) > 85
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "High CPU usage on {{ $labels.instance }}"
    description: "Instance {{ $labels.instance }} has been running at over 85% CPU for 5 minutes."

Example alert rule for MongoDB replication lag:

- alert: MongoReplicationLag
  expr: |
    (
      mongodb_replset_member_state{state="SECONDARY", member="0"} == 1
      and
      mongodb_replset_member_oplog_window_seconds{member="0"} - mongodb_replset_member_oplog_window_seconds{member="1"} > 600
    )
    or
    (
      mongodb_replset_member_state{state="SECONDARY", member="1"} == 1
      and
      mongodb_replset_member_oplog_window_seconds{member="1"} - mongodb_replset_member_oplog_window_seconds{member="0"} > 600
    )
  for: 10m
  labels:
    severity: critical
  annotations:
    summary: "MongoDB replication lag detected on {{ $labels.instance }}"
    description: "Replica set member {{ $labels.member }} is lagging behind the primary by more than 10 minutes."

Add your rule files to /etc/prometheus/prometheus.yml:

rule_files:
  - "/etc/prometheus/rules/*.yml"

Reload Prometheus for the rules to take effect.

Grafana for Visualization

While Prometheus provides the data and Alertmanager handles alerts, Grafana offers a user-friendly interface for visualizing metrics. Install Grafana on a dedicated Linode instance or on the Prometheus server if it has sufficient resources.

Install Grafana (example for Debian/Ubuntu):

sudo apt-get update
sudo apt-get install -y apt-transport-https software-properties-common wget
wget -q -O - https://apt.grafana.com/gpg.key | sudo apt-key add -
echo "deb https://apt.grafana.com stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
sudo apt-get update
sudo apt-get install grafana

Enable and start the Grafana service:

sudo systemctl daemon-reload
sudo systemctl enable grafana-server
sudo systemctl start grafana-server
sudo systemctl status grafana-server

Access Grafana in your browser at http://your-linode-ip:3000. The default credentials are admin/admin. You’ll be prompted to change the password.

Add Prometheus as a data source in Grafana:

  • Navigate to Configuration (gear icon) > Data Sources.
  • Click Add data source.
  • Select Prometheus.
  • Set the URL to http://localhost:9090 (if Grafana and Prometheus are on the same server) or the IP address of your Prometheus Linode followed by :9090.
  • Click Save & Test.

Import pre-built dashboards or create your own. Search for “Node Exporter Full” and “MongoDB” dashboards on Grafana.com/dashboards. You’ll need to select your Prometheus data source when importing.

Security Considerations

When setting up monitoring, security is paramount:

  • Firewall Rules: Restrict access to Prometheus (9090), Alertmanager (9093), Grafana (3000), and exporter ports (9100, 9216) to only necessary IP addresses (e.g., your monitoring server, your IP for Grafana access).
  • TLS: For production, configure TLS for Prometheus, Alertmanager, and Grafana to encrypt traffic.
  • Authentication: Enable authentication for Grafana and consider basic auth for Prometheus/Alertmanager endpoints if they are exposed externally.
  • MongoDB Credentials: Use a dedicated, least-privilege MongoDB user for the exporter. Store credentials securely, ideally using environment variables or a secrets management system, rather than hardcoding them in service files.

Conclusion

This comprehensive setup provides a robust foundation for monitoring your Shopify app and MongoDB clusters on Linode. By integrating Prometheus, Node Exporter, MongoDB Exporter, Alertmanager, and Grafana, you gain deep visibility into system health, application performance, and database operations, enabling you to detect and resolve issues proactively, ensuring high availability and a smooth user experience.