Vengala Vinay

Having 9+ Years of Experience in Software Development

Home » Server Monitoring Best Practices: Keeping Your PHP App and MongoDB Clusters Alive on DigitalOcean

Server Monitoring Best Practices: Keeping Your PHP App and MongoDB Clusters Alive on DigitalOcean

Proactive MongoDB Cluster Health Checks with `mongostat` and `mongotop`

Maintaining the health of a MongoDB cluster, especially in a production environment on DigitalOcean, requires more than just reactive alerts. Proactive, granular monitoring of key performance indicators (KPIs) is crucial for identifying potential bottlenecks before they impact application availability. Two indispensable tools for this are mongostat and mongotop. These command-line utilities provide real-time insights into MongoDB’s operational status.

mongostat offers a snapshot of current operations, including network traffic, query performance, and resource utilization. It’s excellent for a quick, high-level overview. mongotop, on the other hand, focuses specifically on read and write operations per collection, helping to pinpoint performance issues at the collection level.

Real-time MongoDB Statistics with `mongostat`

Running mongostat without arguments provides a default set of metrics. However, for a more targeted view, especially when monitoring multiple nodes in a replica set or sharded cluster, specifying the connection string and desired interval is key. We’ll focus on monitoring a replica set member.

To monitor a specific MongoDB instance (e.g., a replica set member) every 5 seconds, you would use:

First, ensure you have the MongoDB database tools installed on your monitoring server or directly on a MongoDB node. On Debian/Ubuntu systems, this is typically done via:

sudo apt-get update
sudo apt-get install -y mongodb-database-tools

Then, execute mongostat. Replace mongodb-replica-member-ip:27017 with your actual MongoDB host and port. If authentication is enabled, you’ll need to provide credentials.

mongostat --host mongodb-replica-member-ip:27017 --username your_mongo_user --password your_mongo_password --authenticationDatabase admin --interval 5

Key metrics to watch:

  • insert, query, update, delete: Operations per second. Spikes or consistently high numbers here might indicate heavy load.
  • getmore: Number of getMore operations. High values can suggest inefficient queries or large result sets being fetched in batches.
  • dirty %: Percentage of dirty pages in the WiredTiger cache. A consistently high percentage (e.g., > 90%) can indicate insufficient cache size or heavy write activity.
  • used %: Percentage of WiredTiger cache used.
  • netIn, netOut: Network traffic in bytes/sec. Useful for identifying network saturation.
  • res: Resident memory usage.
  • qrw, qread, qwrite: Queue lengths for read and write operations. Non-zero values indicate operations are waiting.
  • idxMiss: Index miss ratio. High values suggest inefficient indexing or missing indexes.

Collection-Level Performance with `mongotop`

mongotop provides a more granular view, showing read and write activity per collection. This is invaluable for identifying which specific collections are causing performance degradation.

To run mongotop on a specific database and monitor every 10 seconds:

mongotop --host mongodb-replica-member-ip:27017 --username your_mongo_user --password your_mongo_password --authenticationDatabase admin --db your_database_name --interval 10

The output will show:

  • ns: Namespace (database.collection).
  • total: Total time spent in seconds by the mongotop process.
  • read: Time spent on read operations (in ms).
  • write: Time spent on write operations (in ms).

Consistently high read or write times for a specific collection indicate that it’s a hot spot. This might necessitate query optimization, adding or modifying indexes, or even schema redesign.

PHP Application Performance Monitoring with New Relic

For PHP applications running on DigitalOcean, robust application performance monitoring (APM) is non-negotiable. New Relic is a powerful, albeit commercial, solution that provides deep visibility into your application’s behavior, from transaction traces to database query analysis.

Setting up New Relic APM for PHP

The process involves installing the New Relic agent and configuring your PHP environment.

1. Install the New Relic Agent:

New Relic provides an automated installer script. SSH into your DigitalOcean droplet running your PHP application and execute the following command. Ensure you have root privileges or use sudo.

curl -Ls https://download.newrelic.com/install/newrelic-php5/scripts/newrelic-install.sh | sudo bash

This script will detect your PHP version and web server (e.g., Apache, Nginx with PHP-FPM) and attempt to install the agent. It will prompt you for your New Relic license key. You can find this key in your New Relic account under Account Settings > License.

2. Configure PHP:

The installer typically modifies your php.ini file (or a dedicated New Relic configuration file). You might need to manually verify or adjust settings. The primary configuration file is usually located at /etc/php/[php_version]/mods-available/newrelic.ini or similar. Ensure the following lines are present and correctly set:

extension=newrelic.so
newrelic.license = "YOUR_LICENSE_KEY"
newrelic.appname = "Your-PHP-App-Name"
newrelic.distributed_tracing_enabled = true
newrelic.framework = "laravel" ; Or "symfony", "custom", etc. if applicable

Replace YOUR_LICENSE_KEY and Your-PHP-App-Name with your actual values. Setting newrelic.appname is crucial for organizing your applications in the New Relic UI. If you’re using a framework like Laravel or Symfony, specifying it can enable more tailored insights.

3. Restart Web Server:

After installation and configuration, restart your web server to load the New Relic extension.

sudo systemctl restart apache2
# OR
sudo systemctl restart php[php_version]-fpm
sudo systemctl restart nginx

4. Verify Installation:

Create a simple PHP file (e.g., info.php) in your web root with the following content:

<?php
phpinfo();
?>

Access this file via your browser (e.g., http://your_droplet_ip/info.php). Search for “newrelic” on the page. You should see a section detailing the New Relic agent’s configuration. If it’s present, the agent is loaded. Data should start appearing in your New Relic dashboard within a few minutes.

Leveraging New Relic for MongoDB Insights

New Relic APM automatically instruments many common PHP database extensions, including those used to connect to MongoDB (like the native MongoDB driver or libraries like Moloquent). Within the New Relic UI, navigate to your application’s dashboard, then select “Databases.” You’ll see a breakdown of database calls, including:

  • Slowest Queries: Identify specific MongoDB queries that are taking the longest to execute.
  • Query Throughput: See the volume of queries to MongoDB over time.
  • Error Rates: Monitor any database-related errors.
  • Transaction Traces: Drill down into individual web transactions to see how much time is spent in MongoDB operations.

This data directly correlates with the metrics you’d observe using mongostat and mongotop, but presented within the context of your application’s requests. If New Relic highlights slow MongoDB queries, you can then use mongotop to confirm the load on specific collections and mongostat to check overall cluster health.

System-Level Monitoring with Prometheus and Grafana

While New Relic excels at application-level insights and MongoDB tools provide database-specific metrics, a comprehensive monitoring strategy also requires robust system-level monitoring. Prometheus, an open-source monitoring and alerting toolkit, coupled with Grafana, a leading visualization platform, forms a powerful combination for observing your DigitalOcean droplets and MongoDB cluster infrastructure.

Deploying Prometheus Node Exporter

Prometheus collects metrics via exporters. For system-level metrics on your droplets (CPU, memory, disk I/O, network), the node_exporter is essential. We’ll deploy it as a systemd service.

1. Download Node Exporter:

On each droplet you wish to monitor (including those running MongoDB and your PHP app), download the latest release of node_exporter.

NODE_EXPORTER_VERSION="1.7.0" # Check for the latest version
wget https://github.com/prometheus/node_exporter/releases/download/v${NODE_EXPORTER_VERSION}/node_exporter-${NODE_EXPORTER_VERSION}.linux-amd64.tar.gz
tar xvfz node_exporter-${NODE_EXPORTER_VERSION}.linux-amd64.tar.gz
sudo mv node_exporter-${NODE_EXPORTER_VERSION}.linux-amd64/node_exporter /usr/local/bin/

2. Create Systemd Service:

Create a systemd service file to manage the node_exporter process.

[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=nobody
Group=nogroup
Type=simple
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target

Save this content to /etc/systemd/system/node_exporter.service. Then, enable and start the service:

sudo systemctl daemon-reload
sudo systemctl enable node_exporter
sudo systemctl start node_exporter

By default, node_exporter listens on port 9100. You can verify it’s running and accessible:

curl http://localhost:9100/metrics

Configuring Prometheus Server

Your Prometheus server (which could be on a separate DigitalOcean droplet) needs to be configured to scrape metrics from your node_exporter instances and potentially MongoDB itself.

1. Install Prometheus Server:

Follow the official Prometheus installation guide for your operating system. For Debian/Ubuntu, this often involves downloading the binary or using a package manager.

2. Configure Prometheus Scrape Targets:

Edit your Prometheus configuration file (typically /etc/prometheus/prometheus.yml). Add scrape configurations for your droplets.

global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.

scrape_configs:
  # Scrape Prometheus itself
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  # Scrape Node Exporter on PHP App Droplet
  - job_name: 'php_app_node_exporter'
    static_configs:
      - targets: ['php_app_droplet_ip:9100']

  # Scrape Node Exporter on MongoDB Replica Set Member 1
  - job_name: 'mongodb_node_exporter_1'
    static_configs:
      - targets: ['mongodb_replica_member_1_ip:9100']

  # Scrape Node Exporter on MongoDB Replica Set Member 2
  - job_name: 'mongodb_node_exporter_2'
    static_configs:
      - targets: ['mongodb_replica_member_2_ip:9100']

  # Add more jobs for other MongoDB nodes, load balancers, etc.

  # MongoDB Exporter (requires separate setup, see below)
  - job_name: 'mongodb'
    static_configs:
      - targets: ['mongodb_exporter_ip:9274'] # Default port for mongodb_exporter

Replace php_app_droplet_ip and mongodb_replica_member_X_ip with the actual IP addresses of your DigitalOcean droplets. After modifying prometheus.yml, restart the Prometheus service:

sudo systemctl restart prometheus

Integrating MongoDB Exporter

To get detailed MongoDB metrics into Prometheus, you’ll need the mongodb_exporter. This exporter runs as a separate service and queries MongoDB for metrics, exposing them in Prometheus format.

1. Deploy MongoDB Exporter:

You can download the binary or run it via Docker. For a direct installation:

# Download binary (example for Linux amd64)
MON_EXP_VERSION="0.35.0" # Check for latest
wget https://github.com/dblock/mongodb_exporter/releases/download/v${MON_EXP_VERSION}/mongodb_exporter-${MON_EXP_VERSION}.linux-amd64.tar.gz
tar xvfz mongodb_exporter-${MON_EXP_VERSION}.linux-amd64.tar.gz
sudo mv mongodb_exporter-${MON_EXP_VERSION}.linux-amd64/mongodb_exporter /usr/local/bin/

2. Configure MongoDB Exporter:

The exporter needs MongoDB connection details. It’s best practice to create a dedicated monitoring user in MongoDB with read-only access to the admin database.

# Example MongoDB user creation (run in mongo shell)
use admin
db.createUser({
  user: "monitor_user",
  pwd: "your_secure_password",
  roles: [ { role: "clusterMonitor", db: "admin" }, { role: "readAnyDatabase", db: "admin" } ]
})

Create a systemd service for mongodb_exporter, specifying the connection string:

[Unit]
Description=MongoDB Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=nobody
Group=nogroup
Type=simple
Environment="MONGODB_URI=mongodb://monitor_user:your_secure_password@mongodb-host:27017/admin?replicaSet=yourReplicaSetName"
ExecStart=/usr/local/bin/mongodb_exporter --mongodb.uri $MONGODB_URI --web.listen-address=":9274"

[Install]
WantedBy=multi-user.target

Replace mongodb-host, your_secure_password, and yourReplicaSetName. If you have multiple MongoDB nodes, you might run multiple exporters or configure one to connect to a replica set. Start the service:

sudo systemctl daemon-reload
sudo systemctl enable mongodb_exporter
sudo systemctl start mongodb_exporter

Ensure your Prometheus server can reach the exporter’s port (9274).

Visualizing with Grafana

Grafana allows you to create dashboards to visualize the metrics collected by Prometheus.

1. Install Grafana:

Follow the official Grafana installation guide for your OS. Typically involves adding their repository and installing the grafana package.

2. Add Prometheus Data Source:

In the Grafana UI (usually at port 3000), navigate to Configuration > Data Sources > Add data source. Select “Prometheus” and enter the URL of your Prometheus server (e.g., http://your_prometheus_server_ip:9090). Save and test.

3. Import Dashboards:

Grafana has a rich community of pre-built dashboards. You can import dashboards for:

  • Node Exporter Full (ID: 1860): Comprehensive system metrics.
  • MongoDB Dashboard (ID: 740): Various MongoDB metrics from mongodb_exporter.

Go to Dashboards > Browse > Import, and enter the dashboard IDs. Configure the Prometheus data source for each imported dashboard.

These dashboards will provide real-time graphs of CPU usage, memory, disk I/O, network traffic on your droplets, and detailed MongoDB performance metrics like oplog lag, query latency, cache usage, and more. This visual representation is crucial for spotting trends and anomalies.

Alerting Strategies with Alertmanager

Monitoring is only half the battle; you need to be alerted when things go wrong. Prometheus integrates with alertmanager for robust alerting.

Configuring Prometheus Alerting Rules

Define alerting rules in a separate file (e.g., /etc/prometheus/alert.rules.yml) and reference it in prometheus.yml.

groups:
- name: general.rules
  rules:
  - alert: HostDown
    expr: up == 0
    for: 1m
    labels:
      severity: critical
    annotations:
      summary: "Host {{ $labels.instance }} is down"
      description: "{{ $labels.instance }} of job {{ $labels.job }} has been unreachable for more than 1 minute."

  - alert: HighCpuLoad
    expr: 100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 85
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "High CPU load on {{ $labels.instance }}"
      description: "CPU load on {{ $labels.instance }} is above 85% for 5 minutes."

  - alert: HighMemoryUsage
    expr: (node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes * 100 > 90
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "High memory usage on {{ $labels.instance }}"
      description: "Memory usage on {{ $labels.instance }} is above 90% for 5 minutes."

- name: mongodb.rules
  rules:
  - alert: MongoOplogLag
    # Assumes you have a metric like 'mongodb_oplog_oplog_stats_oplog_max_lag_seconds' from mongodb_exporter
    expr: mongodb_oplog_oplog_stats_oplog_max_lag_seconds > 600 # 10 minutes lag
    for: 2m
    labels:
      severity: critical
    annotations:
      summary: "MongoDB Oplog Lag on {{ $labels.instance }}"
      description: "MongoDB oplog lag on {{ $labels.instance }} is greater than 10 minutes."

  - alert: MongoHighCacheDirty
    # Assumes metric like 'mongodb_mongod_wiredtiger_cache_bytes_dirty' and 'mongodb_mongod_wiredtiger_cache_bytes_used'
    expr: mongodb_mongod_wiredtiger_cache_bytes_dirty / mongodb_mongod_wiredtiger_cache_bytes_used * 100 > 75
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "High MongoDB WiredTiger cache dirty ratio on {{ $labels.instance }}"
      description: "WiredTiger cache dirty ratio on {{ $labels.instance }} is above 75% for 5 minutes."

Add the rule file to your prometheus.yml:

rule_files:
  - "/etc/prometheus/alert.rules.yml"

Restart Prometheus after updating the configuration.

Setting up Alertmanager

alertmanager receives alerts from Prometheus and routes them to various receivers (email, Slack, PagerDuty, etc.).

1. Install Alertmanager:

Similar to Prometheus, download the binary or use a package manager.

2. Configure Alertmanager:

The main configuration file is alertmanager.yml. Here’s a sample configuration for Slack notifications:

global:
  # The default receiver for all alerts
  # The default is to send alerts to the 'default' receiver.
  resolve_timeout: 5m

route:
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: 'slack-notifications' # Default receiver

  routes:
  - match:
      severity: 'critical'
    receiver: 'slack-critical'
    continue: true # Allows matching multiple routes if needed

receivers:
- name: 'slack-notifications'
  slack_configs:
  - api_url: 'YOUR_SLACK_WEBHOOK_URL'
    channel: '#alerts'
    send_resolved: true
    title: '[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .CommonLabels.alertname }} for {{ .CommonLabels.job }}'
    text: '{{ range .Alerts }}*Alert:* {{ .Annotations.summary }}\n*Description:* {{ .Annotations.description }}\n*Details:* {{ range .Labels.SortedPairs }} {{ .Name }}={{ .Value }}{{ end }}\n{{ end }}'

- name: 'slack-critical'
  slack_configs:
  - api_url: 'YOUR_SLACK_WEBHOOK_URL'
    channel: '#alerts-critical'
    send_resolved: true
    title: '[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .CommonLabels.alertname }} for {{ .CommonLabels.job }}'
    text: '{{ range .Alerts }}*Alert:* {{ .Annotations.summary }}\n*Description:* {{ .Annotations.description }}\n*Details:* {{ range .Labels.SortedPairs }} {{ .Name }}={{ .Value }}{{ end }}\n{{ end }}'

Replace YOUR_SLACK_WEBHOOK_URL with your actual Slack incoming webhook URL. Configure Prometheus to send alerts to Alertmanager:

alerting:
  alertmanagers:
  - static_configs:
    - targets:
      - 'alertmanager_ip:9093' # Default Alertmanager port

Restart both Prometheus and Alertmanager services.

Conclusion: A Layered Approach to Reliability

Achieving high availability for your PHP applications and MongoDB clusters on DigitalOcean is a multi-faceted endeavor. It requires a layered monitoring strategy that encompasses:

  • Application Performance Monitoring (APM): Tools like New Relic provide deep insights into your PHP code’s execution, identifying slow transactions and database interactions.
  • Database-Specific Monitoring: Utilities like mongostat and mongotop, along with dedicated exporters like mongodb_exporter, offer granular control and visibility into MongoDB’s internal state.
  • System-Level Monitoring: Prometheus and Node Exporter provide essential metrics on the underlying infrastructure (CPU, RAM, disk, network) of your droplets.
  • Centralized Visualization: Grafana transforms raw metrics into actionable dashboards, making it easier to understand system behavior at a glance.
  • Proactive Alerting: Prometheus and Alertmanager ensure you are notified of critical issues before they escalate, enabling rapid response.

By combining these tools and techniques, you can build a robust monitoring framework that not only detects problems but also helps you understand their root causes, optimize performance, and ultimately keep your critical services running smoothly on DigitalOcean.