Server Monitoring Best Practices: Keeping Your Laravel App and PostgreSQL Clusters Alive on Linode
Establishing Core Metrics for Laravel & PostgreSQL on Linode
Effective server monitoring hinges on a curated set of metrics that provide actionable insights into application health and infrastructure stability. For a Laravel application running on Linode, with PostgreSQL as its data backbone, we need to track system-level, application-level, and database-specific indicators. This isn’t about collecting every possible metric; it’s about focusing on those that directly correlate with performance degradation, potential failures, and resource exhaustion.
System-Level Monitoring with Node Exporter
Prometheus’s Node Exporter is the de facto standard for collecting hardware and OS metrics. Deploying it on each Linode instance provides a foundational layer of visibility. Key metrics to prioritize include:
- CPU Utilization: High CPU usage can indicate inefficient code, runaway processes, or insufficient resources.
- Memory Usage: Swapping to disk is a performance killer. Monitor both RAM usage and swap space.
- Disk I/O: Bottlenecks here can cripple application and database performance. Track read/write operations and latency.
- Network Traffic: Monitor inbound and outbound bandwidth to detect saturation or unusual traffic patterns.
- Filesystem Usage: Running out of disk space is a common cause of application failure.
Installation and basic configuration on a Debian/Ubuntu-based Linode:
Installing Node Exporter
Download the latest release from the official Prometheus GitHub repository. Replace `[VERSION]` with the current stable version (e.g., `1.7.0`).
wget https://github.com/prometheus/node_exporter/releases/download/v[VERSION]/node_exporter-[VERSION].linux-amd64.tar.gz tar xvfz node_exporter-[VERSION].linux-amd64.tar.gz sudo mv node_exporter-[VERSION].linux-amd64/node_exporter /usr/local/bin/
Systemd Service for Node Exporter
Create a systemd service file to manage the Node Exporter process. This ensures it starts on boot and can be easily controlled.
[Unit] Description=Node Exporter Wants=network-online.target After=network-online.target [Service] User=prometheus Group=prometheus Type=simple ExecStart=/usr/local/bin/node_exporter [Install] WantedBy=multi-user.target
Create the user and group, then enable and start the service:
sudo useradd -rs /bin/false prometheus sudo mv node_exporter.service /etc/systemd/system/ sudo systemctl daemon-reload sudo systemctl start node_exporter sudo systemctl enable node_exporter
Ensure your Linode firewall (or an external firewall) allows traffic on port 9100 for Prometheus to scrape metrics.
Application-Level Monitoring with Prometheus Client Libraries
To gain insight into your Laravel application’s behavior, instrument it using Prometheus client libraries. This allows you to track request latency, error rates, queue lengths, and other application-specific metrics. For PHP, the promphp/prometheus_client_php library is a robust choice.
Instrumenting Laravel with Prometheus
First, add the library to your project via Composer:
composer require promphp/prometheus_client_php
Create a Prometheus metrics endpoint in your Laravel application. A common approach is to register a route that exposes the collected metrics.
<?php
namespace App\Http\Controllers;
use Illuminate\Http\Request;
use Prometheus\CollectorRegistry;
use Prometheus\Render\RenderTextFormat;
use Prometheus\Storage\InMemory; // Or Redis for persistence
class PrometheusController extends Controller
{
public function metrics()
{
// Initialize the registry. Use Redis for production to avoid data loss on restarts.
// For simplicity here, we use InMemory.
$adapter = new InMemory();
$registry = new CollectorRegistry($adapter);
// --- Example: Registering a Counter for HTTP Requests ---
$counter = new \Prometheus\Counter(
$registry,
'http_requests_total',
'Total number of HTTP requests received',
['method', 'path', 'status_code']
);
// In a real application, you'd increment this in your middleware.
// --- Example: Registering a Histogram for Request Latency ---
$histogram = new \Prometheus\Histogram(
$registry,
'http_request_duration_seconds',
'HTTP request latency in seconds',
['method', 'path']
);
// In a real application, you'd measure and observe this in your middleware.
// --- Example: Gauge for Active Users (e.g., from session count) ---
$gauge = new \Prometheus\Gauge(
$registry,
'active_users',
'Number of currently active users'
);
// In a real application, you'd update this based on session activity.
// --- Example: Summary for API Response Times ---
$summary = new \Prometheus\Summary(
$registry,
'api_response_time_seconds',
'API response time in seconds',
['endpoint']
);
// In a real application, you'd measure and observe this for specific API calls.
// --- Example: Custom Metric for Laravel Queue Jobs ---
$queueJobsCounter = new \Prometheus\Counter(
$registry,
'laravel_queue_jobs_total',
'Total number of Laravel queue jobs processed',
['job_name', 'status'] // e.g., 'success', 'failed'
);
// This would be updated by a custom listener or within your job processing logic.
// --- Render the metrics ---
$renderer = new RenderTextFormat();
header('Content-Type: ' . $renderer->getMimeType());
echo $renderer->render($registry->getMetricFamilySamples());
exit;
}
}
Register this method in your routes/web.php or routes/api.php:
use App\Http\Controllers\PrometheusController;
Route::get('/metrics', [PrometheusController::class, 'metrics']);
To make these metrics persistent across application restarts and available to Prometheus, especially if you’re not using Redis as your storage adapter, you’ll need to configure Prometheus to scrape this endpoint. For production, using Redis as the storage adapter for promphp/prometheus_client_php is highly recommended to maintain metric state.
Middleware for Automatic Metric Collection
To automatically capture request counts and durations, create a middleware. This is where you’d typically increment counters and observe histograms.
php artisan make:middleware TrackMetrics
namespace App\Http\Middleware;
use Closure;
use Illuminate\Http\Request;
use Prometheus\CollectorRegistry;
use Prometheus\Storage\Redis as RedisStorage; // Or InMemory
use Symfony\Component\HttpFoundation\Response;
class TrackMetrics
{
/**
* Handle an incoming request.
*
* @param \Closure(\Illuminate\Http\Request): (\Symfony\Component\HttpFoundation\Response) $next
*/
public function handle(Request $request, Closure $next): Response
{
// Initialize registry (ensure this matches your PrometheusController setup)
// For production, use Redis.
$redis = new \Redis();
$redis->connect('127.0.0.1', 6379); // Adjust if Redis is on a different host/port
$adapter = new RedisStorage($redis);
$registry = new CollectorRegistry($adapter);
// Get or create the HTTP requests counter
$counter = $registry->getOrRegister(
new \Prometheus\Counter(
null, // Registry is passed to constructor
'http_requests_total',
'Total number of HTTP requests received',
['method', 'path', 'status_code']
)
);
// Get or create the request duration histogram
$histogram = $registry->getOrRegister(
new \Prometheus\Histogram(
null,
'http_request_duration_seconds',
'HTTP request latency in seconds',
['method', 'path']
)
);
$startTime = microtime(true);
$response = $next($request);
$duration = microtime(true) - $startTime;
// Observe the duration
$histogram->observe($duration, [$request->method(), $request->path()]);
// Increment the counter
$counter->inc([$request->method(), $request->path(), $response->getStatusCode()]);
return $response;
}
}
Register this middleware in your app/Http/Kernel.php‘s $middleware or $middlewareGroups array.
PostgreSQL Cluster Monitoring with Prometheus Exporter
Monitoring your PostgreSQL cluster is critical for database performance and availability. The postgres_exporter is a widely adopted tool for exposing PostgreSQL metrics in a Prometheus-compatible format.
Installing and Configuring Postgres Exporter
Download the latest release from the official postgres_exporter GitHub repository. Replace `[VERSION]` with the current stable version (e.g., `0.13.0`).
wget https://github.com/prometheus-community/postgres_exporter/releases/download/v[VERSION]/postgres_exporter-v[VERSION].linux-amd64.tar.gz tar xvfz postgres_exporter-v[VERSION].linux-amd64.tar.gz sudo mv postgres_exporter-v[VERSION].linux-amd64/postgres_exporter /usr/local/bin/
The exporter needs to connect to your PostgreSQL instances. Create a .pgpass file for the user running the exporter, or use environment variables. For security, using a dedicated monitoring user with read-only permissions on `pg_stat_activity` and other relevant views is best practice.
# Create a dedicated monitoring user in PostgreSQL CREATE USER monitor WITH PASSWORD 'your_secure_password'; GRANT pg_read_all_stats TO monitor; -- For older versions or specific needs, you might need to grant access to specific views. -- GRANT SELECT ON pg_stat_activity TO monitor; -- GRANT SELECT ON pg_stat_database TO monitor; -- etc. # On the Linode server, create .pgpass for the user running postgres_exporter (e.g., 'postgres' or a dedicated 'monitor' user) echo "your_linode_ip_or_hostname:5432:*:monitor:your_secure_password" | sudo tee /home/your_user/.pgpass sudo chmod 600 /home/your_user/.pgpass
Create a systemd service file for postgres_exporter. You’ll need to specify the connection string via an environment variable or command-line argument. Using a configuration file is also an option.
[Unit] Description=PostgreSQL Exporter Wants=network-online.target After=network-online.target [Service] User=postgres # Or the user running the exporter Group=postgres Type=simple Environment="DATA_SOURCE_NAME=postgresql://monitor:your_secure_password@your_linode_ip_or_hostname:5432/postgres?sslmode=disable" # Or use PGHOST, PGPORT, PGDATABASE, PGUSER, PGPASSWORD environment variables ExecStart=/usr/local/bin/postgres_exporter --web.listen-address=":9187" --extend.queries-path=/etc/postgres_exporter/queries.yaml [Install] WantedBy=multi-user.target
Create the necessary directories and files, then enable and start the service:
sudo mkdir -p /etc/postgres_exporter sudo chown -R postgres:postgres /etc/postgres_exporter sudo mv postgres_exporter.service /etc/systemd/system/ sudo systemctl daemon-reload sudo systemctl start postgres_exporter sudo systemctl enable postgres_exporter
The postgres_exporter will expose metrics on port 9187 by default. You can configure it to monitor multiple PostgreSQL instances by providing different DATA_SOURCE_NAME values or using a configuration file.
Key PostgreSQL Metrics to Monitor
pg_stat_activitymetrics: Number of active connections, idle connections, queries in progress, blocked queries.- Replication Lag: For high availability setups, monitor
pg_stat_replicationfor lag. - Database Size and Table Sizes: Track growth to anticipate storage needs.
- Cache Hit Ratios: Monitor
pg_stat_databasefor buffer cache hit rates. - Transaction Throughput:
xact_commitandxact_rollback. - Lock Wait Times: Identify contention issues.
- Replication Slots: Ensure they are not growing indefinitely.
You can extend postgres_exporter with custom queries defined in queries.yaml to capture application-specific or more detailed performance indicators.
Centralized Monitoring with Prometheus and Grafana
With exporters running on your Linode instances, the next step is to set up a central Prometheus server to scrape these metrics and Grafana to visualize them. This provides a unified dashboard for your entire stack.
Prometheus Configuration
Your Prometheus configuration file (prometheus.yml) will define scrape targets. For a dynamic environment, consider using service discovery (e.g., file-based discovery, Consul, or Kubernetes service discovery if applicable).
global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
scrape_configs:
# Scrape Prometheus itself
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']
# Scrape Node Exporter on each Linode instance
- job_name: 'node_exporter'
static_configs:
- targets:
- 'linode-app-1:9100'
- 'linode-app-2:9100'
- 'linode-db-1:9100'
- 'linode-db-2:9100'
# If using service discovery, this section would be different.
# Scrape Laravel application metrics endpoint
- job_name: 'laravel_app'
static_configs:
- targets:
- 'linode-app-1:80' # Assuming your Laravel app runs on port 80
- 'linode-app-2:80'
metrics_path: '/metrics' # The endpoint you registered in Laravel
# Scrape PostgreSQL Exporter on each database Linode instance
- job_name: 'postgres_exporter'
static_configs:
- targets:
- 'linode-db-1:9187'
- 'linode-db-2:9187'
Ensure your Prometheus server can reach all target ports (9100, 80, 9187) on your Linode instances. Adjust IP addresses or hostnames as necessary.
Grafana Dashboards
Grafana provides powerful visualization capabilities. You can import pre-built dashboards for Node Exporter and PostgreSQL Exporter, or create custom ones tailored to your specific needs.
- Node Exporter Full Dashboard (ID 1860): Excellent for system-level metrics.
- PostgreSQL Overview Dashboard (ID 721): Provides comprehensive database insights.
When setting up Grafana, add Prometheus as a data source, pointing it to your Prometheus server’s URL (e.g., http://your-prometheus-server-ip:9090). Then, import the desired dashboards by their IDs or JSON files.
Alerting with Prometheus Alertmanager
Collecting metrics is only half the battle; you need to be alerted when things go wrong. Prometheus Alertmanager handles alerts sent by Prometheus server instances. Configure alerting rules in Prometheus and define notification channels (e.g., Slack, PagerDuty, email) in Alertmanager.
Example Alerting Rules
Add these rules to your Prometheus configuration or a separate rules file referenced in prometheus.yml.
- alert: HighCpuUsage
expr: 100 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) > 90
for: 5m
labels:
severity: warning
annotations:
summary: "High CPU usage on {{ $labels.instance }}"
description: "CPU usage on {{ $labels.instance }} is above 90% for the last 5 minutes."
- alert: LowDiskSpace
expr: node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"} * 100 < 10
for: 10m
labels:
severity: critical
annotations:
summary: "Low disk space on {{ $labels.instance }}"
description: "Filesystem on {{ $labels.instance }} has less than 10% free space."
- alert: HighPostgresReplicationLag
expr: pg_replication_lag_seconds > 600 # Lag greater than 10 minutes
for: 2m
labels:
severity: critical
annotations:
summary: "PostgreSQL replication lag detected on {{ $labels.instance }}"
description: "Replication lag on {{ $labels.instance }} is {{ $value }} seconds, exceeding the 600-second threshold."
- alert: LaravelQueueStuck
# This requires a custom metric for queue job age or processing time.
# Example: Assuming you have a metric 'laravel_queue_job_age_seconds'
# expr: laravel_queue_job_age_seconds > 3600 # Jobs older than 1 hour
# For demonstration, let's use a placeholder:
expr: up{job="laravel_app"} == 0 # If the Laravel app endpoint is down
for: 2m
labels:
severity: critical
annotations:
summary: "Laravel application endpoint is down on {{ $labels.instance }}"
description: "The /metrics endpoint for the Laravel application on {{ $labels.instance }} is unreachable."
Configure Alertmanager to route these alerts to appropriate channels. For example, a Slack integration can provide immediate notifications to your team.
Advanced Considerations: Health Checks and Tracing
Beyond basic metrics, consider implementing more sophisticated monitoring techniques:
- Application Health Checks: Implement dedicated health check endpoints in your Laravel app (e.g.,
/health) that verify database connectivity, cache status, and other critical dependencies. Prometheus can scrape these endpoints. - Distributed Tracing: For complex microservice architectures or deep performance analysis within your Laravel app, tools like Jaeger or Zipkin, integrated via OpenTelemetry, can provide end-to-end request tracing. This helps pinpoint latency bottlenecks across different services or layers of your application.
- Log Aggregation: Centralize logs from all your Linode instances using tools like Elasticsearch/Logstash/Kibana (ELK stack) or Loki. This is invaluable for debugging and correlating events.
By combining system-level metrics, application instrumentation, database-specific monitoring, robust alerting, and advanced techniques like tracing and health checks, you can build a resilient monitoring strategy that keeps your Laravel and PostgreSQL clusters on Linode healthy and performant.