Server Monitoring Best Practices: Keeping Your Laravel App and MySQL Clusters Alive on DigitalOcean

Proactive MySQL Cluster Health Checks with `pt-heartbeat`

Maintaining the health and synchronization of a MySQL cluster, especially in a high-availability setup on DigitalOcean, is paramount. Relying solely on basic `SHOW SLAVE STATUS` or `SHOW REPLICA STATUS` can be reactive. We need a tool that actively measures replication lag and alerts us *before* it becomes a critical issue. Percona Toolkit’s `pt-heartbeat` is an indispensable utility for this. It writes a timestamp to a dedicated table on the primary and reads it from the replicas, calculating the lag.

First, ensure Percona Toolkit is installed on all your MySQL nodes. On Debian/Ubuntu, this is typically:

sudo apt-get update
sudo apt-get install percona-toolkit

Next, create a dedicated table on your primary MySQL server to store the heartbeat timestamp. This table should be replicated to all replicas.

-- On your MySQL Primary
CREATE DATABASE IF NOT EXISTS heartbeat;
USE heartbeat;
CREATE TABLE IF NOT EXISTS ping (
  id INT NOT NULL AUTO_INCREMENT PRIMARY KEY,
  server_id INT UNSIGNED NOT NULL,
  time DATETIME NOT NULL,
  INDEX(time)
) ENGINE=InnoDB;

Now, configure `pt-heartbeat` to run on the primary, writing to this table. A cron job is the standard approach. We’ll use a dedicated MySQL user with minimal privileges.

Create a MySQL user for `pt-heartbeat` on the primary:

-- On your MySQL Primary
CREATE USER 'heartbeat_user'@'localhost' IDENTIFIED BY 'your_strong_password';
GRANT INSERT, SELECT ON heartbeat.ping TO 'heartbeat_user'@'localhost';
FLUSH PRIVILEGES;

Set up the cron job on the primary. This example runs every 10 seconds.

# On your MySQL Primary
echo "*/10 * * * * /usr/bin/pt-heartbeat --host=127.0.0.1 --user=heartbeat_user --password='your_strong_password' --database=heartbeat --table=ping --update-primary --server-id=$(grep server-id /etc/mysql/my.cnf | awk '{print $3}')" | sudo crontab -

On each replica, `pt-heartbeat` will read from the `heartbeat.ping` table and report the lag. We’ll configure a cron job to run `pt-heartbeat` in read-only mode and pipe its output to a monitoring system or log file. For simplicity here, we’ll log it.

Create a MySQL user for `pt-heartbeat` on the replicas (read-only access to the heartbeat table):

-- On your MySQL Replicas
CREATE USER 'heartbeat_reader'@'localhost' IDENTIFIED BY 'your_strong_password';
GRANT SELECT ON heartbeat.ping TO 'heartbeat_reader'@'localhost';
FLUSH PRIVILEGES;

Set up the cron job on each replica. This example runs every 10 seconds and logs the lag to a file.

# On your MySQL Replicas
echo "*/10 * * * * /usr/bin/pt-heartbeat --host=127.0.0.1 --user=heartbeat_reader --password='your_strong_password' --database=heartbeat --table=ping --monitor --interval=1 --log=/var/log/mysql/heartbeat.log" | sudo crontab -

You can then parse `/var/log/mysql/heartbeat.log` on each replica to extract the lag value and send it to Prometheus, Datadog, or trigger alerts if the lag exceeds a defined threshold (e.g., 60 seconds).

Laravel Application Health Checks with Healthchecks.io

For your Laravel application, a simple “is the web server responding?” check isn’t enough. You need to verify that critical background jobs are running and that the application itself is processing requests end-to-end. Healthchecks.io is a fantastic, simple service for this. It provides unique URLs that your application pings periodically. If a ping is missed, Healthchecks.io alerts you.

Sign up for Healthchecks.io and create a new “Check”. You’ll get a unique URL like https://hc-ping.com/your-uuid-here.

On your Laravel application, you’ll schedule tasks to ping this URL. The most common pattern is to have a scheduled job that pings the health check URL, and then have another scheduled job that *should* be pinged by a background worker (like a queue worker). If the background worker fails, the second health check will eventually fail.

First, add the health check URL to your Laravel application’s configuration. A dedicated config file is best.

<?php
// config/healthchecks.php
return [
    'app_ping_url' => env('HEALTHCHECKS_APP_PING_URL'),
    'queue_worker_ping_url' => env('HEALTHCHECKS_QUEUE_WORKER_PING_URL'),
];
?>

Add these to your .env file:

HEALTHCHECKS_APP_PING_URL=https://hc-ping.com/your-app-uuid-here
HEALTHCHECKS_QUEUE_WORKER_PING_URL=https://hc-ping.com/your-queue-uuid-here

Now, create a scheduled command to ping the application’s health check URL. This command should run frequently, e.g., every minute.

<?php
// app/Console/Commands/PingAppHealthcheck.php
namespace App\Console\Commands;

use Illuminate\Console\Command;
use Illuminate\Support\Facades\Http;
use Illuminate\Support\Facades\Log;

class PingAppHealthcheck extends Command
{
    protected $signature = 'healthcheck:ping-app';
    protected $description = 'Pings the application health check URL';

    public function handle()
    {
        $url = config('healthchecks.app_ping_url');

        if (empty($url)) {
            Log::warning('HEALTHCHECKS_APP_PING_URL is not set.');
            return;
        }

        try {
            $response = Http::get($url);
            if ($response->successful()) {
                Log::info('App health check pinged successfully.');
            } else {
                Log::error("App health check ping failed: {$response->status()} - {$response->body()}");
            }
        } catch (\Exception $e) {
            Log::error("Error pinging app health check: {$e->getMessage()}");
        }
    }
}
?>

Register this command in your app/Console/Kernel.php:

<?php
// app/Console/Kernel.php
// ...
protected $commands = [
    // ...
    \App\Console\Commands\PingAppHealthcheck::class,
];

protected function schedule(Schedule $schedule)
{
    $schedule->command('healthcheck:ping-app')->everyMinute();
    // ... other schedules
}
// ...
?>

For the queue worker health check, you’ll create a separate command that is *dispatched* by a background job. This ensures that if your queue workers are stalled or dead, this health check won’t be pinged.

<?php
// app/Console/Commands/PingQueueWorkerHealthcheck.php
namespace App\Console\Commands;

use Illuminate\Console\Command;
use Illuminate\Support\Facades\Http;
use Illuminate\Support\Facades\Log;

class PingQueueWorkerHealthcheck extends Command
{
    protected $signature = 'healthcheck:ping-queue-worker';
    protected $description = 'Pings the queue worker health check URL';

    public function handle()
    {
        $url = config('healthchecks.queue_worker_ping_url');

        if (empty($url)) {
            Log::warning('HEALTHCHECKS_QUEUE_WORKER_PING_URL is not set.');
            return;
        }

        try {
            $response = Http::get($url);
            if ($response->successful()) {
                Log::info('Queue worker health check pinged successfully.');
            } else {
                Log::error("Queue worker health check ping failed: {$response->status()} - {$response->body()}");
            }
        } catch (\Exception $e) {
            Log::error("Error pinging queue worker health check: {$e->getMessage()}");
        }
    }
}
?>

Register this command in app/Console/Kernel.php as well, but do NOT schedule it directly. Instead, dispatch it from a job that runs periodically or is triggered by your queue processing.

<?php
// app/Console/Kernel.php
// ...
protected $commands = [
    // ...
    \App\Console\Commands\PingAppHealthcheck::class,
    \App\Console\Commands\PingQueueWorkerHealthcheck::class, // Add this
];

protected function schedule(Schedule $schedule)
{
    $schedule->command('healthcheck:ping-app')->everyMinute();
    // Do NOT schedule 'healthcheck:ping-queue-worker' here.
    // It will be dispatched by a job.
}
// ...
?>

Now, create a simple job that dispatches the queue worker health check command. You can schedule this job to run, for example, every 5 minutes.

<?php
// app/Jobs/DispatchQueueWorkerHealthcheck.php
namespace App\Jobs;

use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\DispatchesJobs;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\SerializesModels;
use App\Console\Commands\PingQueueWorkerHealthcheck; // Import the command

class DispatchQueueWorkerHealthcheck implements ShouldQueue
{
    use Queueable, InteractsWithQueue, SerializesModels, DispatchesJobs;

    public function handle()
    {
        // Dispatch the command to be executed by the scheduler's command runner
        // This ensures it uses the same logic as running via CLI
        $this->dispatchSync(new PingQueueWorkerHealthcheck());
    }
}
?>

Schedule this job in app/Console/Kernel.php:

<?php
// app/Console/Kernel.php
// ...
protected function schedule(Schedule $schedule)
{
    $schedule->command('healthcheck:ping-app')->everyMinute();
    $schedule->job(new DispatchQueueWorkerHealthcheck())
             ->everyFiveMinutes(); // Run this job every 5 minutes
}
// ...
?>

With this setup, if your queue workers stop processing jobs, the DispatchQueueWorkerHealthcheck job won’t run, and the queue worker health check URL will eventually time out on Healthchecks.io, triggering an alert.

DigitalOcean Droplet & Nginx Monitoring with Prometheus & Node Exporter

For infrastructure-level monitoring on DigitalOcean, Prometheus is a robust choice. We’ll deploy Node Exporter on each Droplet to expose system metrics, which Prometheus will then scrape.

First, install Prometheus on a dedicated server (or a management Droplet). A common approach is to use Docker.

# On your Prometheus server
mkdir ~/prometheus
cd ~/prometheus
cat <<EOF > prometheus.yml
global:
  scrape_interval: 15s # By default, scrape targets every 15 seconds.

scrape_configs:
  # Scrape Prometheus itself
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  # Scrape Node Exporter on your Laravel App Droplet
  - job_name: 'laravel_app_node_exporter'
    static_configs:
      - targets: ['YOUR_LARAVEL_APP_DROPLET_IP:9100']

  # Scrape Node Exporter on your MySQL Primary Droplet
  - job_name: 'mysql_primary_node_exporter'
    static_configs:
      - targets: ['YOUR_MYSQL_PRIMARY_DROPLET_IP:9100']

  # Scrape Node Exporter on your MySQL Replica Droplet(s)
  - job_name: 'mysql_replica_node_exporter'
    static_configs:
      - targets: ['YOUR_MYSQL_REPLICA_DROPLET_IP:9100']

  # Scrape Nginx exporter (if deployed)
  - job_name: 'nginx_exporter'
    static_configs:
      - targets: ['YOUR_LARAVEL_APP_DROPLET_IP:9113'] # Assuming Nginx exporter runs on port 9113
EOF

cat <<EOF > docker-compose.yml
version: '3.7'
services:
  prometheus:
    image: prom/prometheus:v2.40.0
    container_name: prometheus
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
volumes:
  prometheus_data:
EOF

docker-compose up -d

Now, install Node Exporter on each Droplet you want to monitor (Laravel app, MySQL nodes).

# On each Droplet to be monitored
wget https://github.com/prometheus/node_exporter/releases/download/v1.6.0/node_exporter-1.6.0.linux-amd64.tar.gz
tar xvfz node_exporter-1.6.0.linux-amd64.tar.gz
cd node_exporter-1.6.0.linux-amd64
sudo ./node_exporter --web.listen-address=":9100" &

To make Node Exporter run as a service, create a systemd unit file:

# Create /etc/systemd/system/node_exporter.service
[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=nobody
Group=nogroup
Type=simple
ExecStart=/usr/local/bin/node_exporter # Adjust path if necessary

[Install]
WantedBy=multi-user.target

Then, enable and start the service:

sudo mv node_exporter /usr/local/bin/
sudo systemctl daemon-reload
sudo systemctl enable node_exporter
sudo systemctl start node_exporter
sudo systemctl status node_exporter

Ensure your DigitalOcean firewall rules allow inbound traffic on port 9100 from your Prometheus server’s IP address.

For Nginx monitoring, you can use the official Nginx exporter. Install it on your Laravel app Droplet.

# On your Laravel App Droplet
wget https://github.com/nginxinc/nginx-prometheus-exporter/releases/download/v0.10.0/nginx-prometheus-exporter_0.10.0_linux_amd64.tar.gz
tar xvfz nginx-prometheus-exporter_0.10.0_linux_amd64.tar.gz
sudo mv nginx-prometheus-exporter /usr/local/bin/

# Create systemd service for Nginx exporter
cat <<EOF | sudo tee /etc/systemd/system/nginx-exporter.service
[Unit]
Description=Nginx Prometheus Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=nobody
Group=nogroup
Type=simple
ExecStart=/usr/local/bin/nginx-prometheus-exporter --nginx.scrape-format prometheus --web.listen-address=":9113"
# If you have multiple Nginx configs, you might need to specify them:
# ExecStart=/usr/local/bin/nginx-prometheus-exporter --nginx.config "/etc/nginx/nginx.conf" --nginx.scrape-format prometheus --web.listen-address=":9113"

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable nginx-exporter
sudo systemctl start nginx-exporter
sudo systemctl status nginx-exporter

Update your Prometheus prometheus.yml to include the Nginx exporter target (as shown in the Prometheus setup section). Again, ensure firewall rules allow access to port 9113.

Alerting with Alertmanager

Prometheus itself doesn’t send alerts; it relies on Alertmanager. You’ll need to set up Alertmanager, typically alongside Prometheus, and configure alerting rules in Prometheus that fire alerts to Alertmanager.

Add Alertmanager to your Docker Compose file on the Prometheus server:

# ~/prometheus/docker-compose.yml (add to existing file)
version: '3.7'
services:
  prometheus:
    # ... existing prometheus config ...
  alertmanager:
    image: prom/alertmanager:v0.25.0
    container_name: alertmanager
    ports:
      - "9093:9093"
    volumes:
      - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml
      - alertmanager_data:/data
    depends_on:
      - prometheus

volumes:
  prometheus_data:
  alertmanager_data:

Create an alertmanager.yml file for basic configuration (e.g., Slack integration):

# ~/prometheus/alertmanager.yml
global:
  resolve_timeout: 5m

route:
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: 'slack-notifications' # Default receiver

receivers:
  - name: 'slack-notifications'
    slack_configs:
      - api_url: 'YOUR_SLACK_WEBHOOK_URL'
        channel: '#alerts'
        send_resolved: true
        title: '[{{ .Status | toUpper }}{{ if .ExternalURL }} - {{ .ExternalURL }}{{ end }}] {{ .CommonLabels.alertname }} for {{ .CommonLabels.job }}'
        text: >-
          {{ range .Alerts }}
            *Alert:* {{ .Annotations.summary }} - `{{ .Labels.severity }}`
            *Description:* {{ .Annotations.description }}
            *Details:*
            {{ range .Labels.SortedPairs }} • *{{ .Name }}:* `{{ .Value }}`
            {{ end }}
          {{ end }}

Update your Prometheus prometheus.yml to point to Alertmanager:

# ~/prometheus/prometheus.yml (add to existing file)
# ... global and scrape_configs ...

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['alertmanager:9093'] # Use the service name from docker-compose

Run cd ~/prometheus && docker-compose up -d again to start Alertmanager and reload Prometheus.

Finally, define alerting rules in Prometheus. Create a new file, e.g., ~/prometheus/rules.yml:

# ~/prometheus/rules.yml
groups:
  - name: general.rules
    rules:
      # Alert if replication lag exceeds 5 minutes
      - alert: HighReplicationLag
        expr: time() - mysql_slave_lag_seconds > 300 # Assuming mysql_slave_lag_seconds is exposed by a MySQL exporter or similar
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "High MySQL replication lag detected on {{ $labels.instance }}"
          description: "Replication lag on {{ $labels.instance }} has exceeded 5 minutes."

      # Alert if Node Exporter is down
      - alert: NodeExporterDown
        expr: up{job="laravel_app_node_exporter"} == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Node Exporter down on Laravel App Droplet"
          description: "Prometheus failed to scrape Node Exporter on {{ $labels.instance }} for over 1 minute."

      # Alert if Nginx exporter is down
      - alert: NginxExporterDown
        expr: up{job="nginx_exporter"} == 0
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: "Nginx Exporter down on Laravel App Droplet"
          description: "Prometheus failed to scrape Nginx Exporter on {{ $labels.instance }} for over 1 minute."

      # Alert if application health check is missed for more than 10 minutes
      - alert: AppHealthcheckMissed
        expr: absent(up{job="healthchecks_io_app"}) # This requires a Prometheus exporter for Healthchecks.io, or a custom exporter
        for: 10m
        labels:
          severity: critical
        annotations:
          summary: "Laravel App Healthcheck Missed"
          description: "The Laravel application health check has not been pinged for 10 minutes."

Add this rules file to your prometheus.yml:

# ~/prometheus/prometheus.yml (add to existing file)
# ... global, scrape_configs, alerting ...

rule_files:
  - "rules.yml" # Add this line

Reload Prometheus configuration by sending a SIGHUP signal or restarting the container: docker-compose restart prometheus.

Conclusion: A Layered Approach

Effective server monitoring is a multi-layered strategy. By combining application-level checks (Healthchecks.io), database replication monitoring (`pt-heartbeat`), and infrastructure metrics (Prometheus/Node Exporter), you build a resilient system. This approach allows you to detect and resolve issues proactively, minimizing downtime and ensuring the stability of your Laravel applications and MySQL clusters on DigitalOcean.