Server Monitoring Best Practices: Keeping Your PHP App and Elasticsearch Clusters Alive on Linode
Proactive PHP Application Health Checks
Maintaining the health of a PHP application goes beyond simply checking if the web server is responding. We need to ensure the application itself is functioning correctly, processing requests efficiently, and not succumbing to common pitfalls like memory leaks or database connection exhaustion. This involves implementing granular health checks that can be queried by your monitoring system.
A robust approach is to create a dedicated health check endpoint within your PHP application. This endpoint should perform several critical checks:
- Database connectivity: Verify that the application can establish and query a connection to its primary database.
- Cache connectivity: If using an external cache like Redis or Memcached, confirm a successful connection and a simple read/write operation.
- External API dependencies: For critical external services, attempt a basic, non-intrusive call.
- Internal application state: Check for critical background processes or queues that might be stalled.
- Resource utilization: While often handled by system-level monitoring, a quick check for abnormally high memory usage within the PHP process itself can be beneficial.
Implementing a PHP Health Check Endpoint
Let’s craft a simple yet effective health check script. This example assumes a common MVC structure where you might have a `public/healthcheck.php` file or a route defined in your framework.
Example: `public/healthcheck.php`
<?php
// public/healthcheck.php
// Configuration for external services (ideally loaded from environment variables or config files)
define('DB_HOST', getenv('DB_HOST') ?: 'localhost');
define('DB_PORT', getenv('DB_PORT') ?: '3306');
define('DB_NAME', getenv('DB_NAME') ?: 'app_db');
define('DB_USER', getenv('DB_USER') ?: 'app_user');
define('DB_PASS', getenv('DB_PASS') ?: 'secret');
define('CACHE_HOST', getenv('CACHE_HOST') ?: 'localhost');
define('CACHE_PORT', getenv('CACHE_PORT') ?: '6379');
// --- Health Check Functions ---
function checkDatabase() {
$dsn = "mysql:host=" . DB_HOST . ";port=" . DB_PORT . ";dbname=" . DB_NAME . ";charset=utf8mb4";
$options = [
PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION,
PDO::ATTR_DEFAULT_FETCH_MODE => PDO::FETCH_ASSOC,
PDO::ATTR_EMULATE_PREPARES => false,
PDO::ATTR_TIMEOUT => 2, // 2-second timeout
];
try {
$pdo = new PDO($dsn, DB_USER, DB_PASS, $options);
// Perform a simple query to ensure connection is active and functional
$stmt = $pdo->query("SELECT 1");
if ($stmt->fetchColumn() !== '1') {
return ['status' => 'error', 'message' => 'Database query failed.'];
}
return ['status' => 'ok', 'message' => 'Database connected successfully.'];
} catch (PDOException $e) {
return ['status' => 'error', 'message' => 'Database connection failed: ' . $e->getMessage()];
}
}
function checkCache() {
try {
$redis = new Redis();
// Use a short timeout for connection
if ($redis->connect(CACHE_HOST, CACHE_PORT, 1)) { // 1-second timeout
// Perform a simple PING command
if ($redis->ping() === '+PONG') {
// Optionally, set and get a test key
$testKey = 'healthcheck_' . uniqid();
if ($redis->set($testKey, 'test', 5)) { // Expires in 5 seconds
if ($redis->get($testKey) === 'test') {
$redis->del($testKey); // Clean up
return ['status' => 'ok', 'message' => 'Cache (Redis) connected and operational.'];
} else {
return ['status' => 'error', 'message' => 'Cache (Redis) SET/GET operation failed.'];
}
} else {
return ['status' => 'error', 'message' => 'Cache (Redis) SET operation failed.'];
}
} else {
return ['status' => 'error', 'message' => 'Cache (Redis) PING failed.'];
}
} else {
return ['status' => 'error', 'message' => 'Cache (Redis) connection failed.'];
}
} catch (RedisException $e) {
return ['status' => 'error', 'message' => 'Cache (Redis) exception: ' . $e->getMessage()];
}
}
// --- Execute Checks ---
$results = [];
$overallStatus = 'ok';
$dbResult = checkDatabase();
$results['database'] = $dbResult;
if ($dbResult['status'] === 'error') {
$overallStatus = 'error';
}
$cacheResult = checkCache();
$results['cache'] = $cacheResult;
if ($cacheResult['status'] === 'error') {
$overallStatus = 'error';
}
// Add more checks here (e.g., external APIs, queue status)
// --- Output Response ---
header('Content-Type: application/json');
http_response_code($overallStatus === 'ok' ? 200 : 503); // 200 OK, 503 Service Unavailable
echo json_encode([
'status' => $overallStatus,
'checks' => $results,
'timestamp' => date('c')
]);
exit;
?>
Key Considerations for the PHP Health Check:
- Timeouts: Crucially, set aggressive timeouts for all external connections (database, cache, APIs). A health check that hangs indefinitely is worse than no health check.
- Error Handling: Catch exceptions and return meaningful error messages. This aids in debugging.
- HTTP Status Codes: Use appropriate HTTP status codes (200 for healthy, 5xx for unhealthy) so that load balancers and monitoring tools can easily interpret the application’s state.
- Idempotency: Ensure the health check is read-only and doesn’t alter application state.
- Security: Protect this endpoint. It should not be accessible from the public internet without authentication or IP whitelisting, especially if it reveals sensitive information. For internal monitoring, consider placing it behind a firewall or using a private network.
- Dependencies: Ensure necessary PHP extensions (like `pdo_mysql`, `redis`) are installed and enabled.
Monitoring PHP Processes with `php-fpm` and `systemd`
For PHP applications running with PHP-FPM (FastCGI Process Manager), `systemd` is the standard service manager on most modern Linux distributions (like Ubuntu, Debian, CentOS). We need to ensure `php-fpm` itself is running and healthy.
`systemd` Service File for PHP-FPM
The `php-fpm` service file typically resides in `/etc/systemd/system/php*-fpm.service` (the asterisk depends on your PHP version, e.g., `php8.1-fpm.service`). You can inspect its status and manage it using `systemctl`.
# Check the status of PHP-FPM sudo systemctl status php8.1-fpm # If it's not running, start it sudo systemctl start php8.1-fpm # Ensure it starts on boot sudo systemctl enable php8.1-fpm # Reload configuration after changes (e.g., pool settings) sudo systemctl reload php8.1-fpm
`systemd` Monitoring Integration
Monitoring tools can query `systemd`’s status. For example, using `systemd-analyze` or by parsing the output of `systemctl status`. A more robust integration involves using tools that can directly query `systemd`’s D-Bus API or by setting up `systemd-journald` forwarding to your central logging system.
A common pattern is to have your monitoring agent (e.g., Prometheus Node Exporter with a `systemd` collector, or a custom script) check the `systemctl status` output. If the service is not `active (running)`, an alert is triggered.
# Example script to check PHP-FPM status and exit with non-zero code if down
PHP_FPM_SERVICE="php8.1-fpm.service" # Adjust to your service name
if ! systemctl is-active --quiet $PHP_FPM_SERVICE; then
echo "PHP-FPM service ($PHP_FPM_SERVICE) is not running."
exit 1
else
echo "PHP-FPM service ($PHP_FPM_SERVICE) is running."
exit 0
fi
This script can be run periodically by `cron` or by a monitoring agent. The exit code is crucial for alerting systems.
Elasticsearch Cluster Health Monitoring
Elasticsearch clusters are complex distributed systems. Monitoring their health requires looking at multiple layers: node status, cluster health (red, yellow, green), shard allocation, JVM heap usage, and performance metrics.
Elasticsearch Cluster Health API
The most fundamental check is the Cluster Health API. It provides a quick overview of the cluster’s state.
curl -X GET "http://localhost:9200/_cluster/health?pretty"
The output will look something like this:
{
"cluster_name" : "my-es-cluster",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 3,
"number_of_data_nodes" : 3,
"active_primary_shards" : 10,
"active_shards" : 30,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"max_in_flight_fetch" : 0,
"last_ccr_checked_exception" : 0,
"task_max_waiting_in_queue" : 0,
"number_of_nodes_with_errors" : 0,
"nodes" : {
"node1" : {
"name" : "node1",
"transport_address" : "192.168.1.10:9300",
"http_address" : "192.168.1.10:9200",
"roles" : [ "master", "data", "ingest" ]
},
"node2" : {
"name" : "node2",
"transport_address" : "192.168.1.11:9300",
"http_address" : "192.168.1.11:9200",
"roles" : [ "data", "ingest" ]
},
"node3" : {
"name" : "node3",
"transport_address" : "192.168.1.12:9300",
"http_address" : "192.168.1.12:9200",
"roles" : [ "data", "ingest" ]
}
}
}
Interpreting the `status` field:
- green: All primary and replica shards are allocated and active. The cluster is healthy.
- yellow: All primary shards are allocated and active, but some replica shards are not yet allocated. This is often a temporary state during cluster changes or if there aren’t enough nodes to satisfy replica requirements. It indicates a potential risk of data loss if a node fails.
- red: One or more primary shards are not allocated. This means data is unavailable for those shards, and the cluster is in a critical state.
Your monitoring system should alert immediately if the status is anything other than `green`. You can script this check:
#!/bin/bash
ES_HOST="localhost:9200"
STATUS=$(curl -s "http://${ES_HOST}/_cluster/health?pretty" | grep -A 1 '"status" :' | tail -n 1 | awk '{print $2}' | tr -d '",')
if [ "$STATUS" != "green" ]; then
echo "Elasticsearch cluster status is $STATUS. Alerting!"
# Add your alerting mechanism here (e.g., send to Slack, PagerDuty)
exit 1
else
echo "Elasticsearch cluster status is green."
exit 0
fi
Node-Level Monitoring and JVM Metrics
Beyond cluster health, individual node health and resource utilization are critical. Elasticsearch exposes extensive metrics via its Nodes Stats API and JVM stats.
# Get stats for all nodes curl -X GET "http://localhost:9200/_nodes/stats?pretty" # Get JVM stats for all nodes curl -X GET "http://localhost:9200/_nodes/jvm?pretty"
Key metrics to monitor include:
- JVM Heap Usage: High heap usage (consistently above 80-90%) can lead to frequent garbage collection pauses and eventual OutOfMemory errors. Monitor `nodes.stats.jvm.mem.heap_used_percent`.
- CPU Usage: High CPU can indicate heavy indexing or search load. Monitor `nodes.stats.os.cpu.percent`.
- Disk I/O and Space: Elasticsearch is I/O intensive. Monitor disk read/write operations and ensure sufficient free disk space. Monitor `nodes.stats.fs.data.free_in_bytes` and `nodes.stats.fs.data.total_in_bytes`.
- Indexing and Search Latency: Monitor `indices.stats.indexing.index_time_in_millis`, `search_time_in_millis`, and `query_cache.get_time_in_millis`.
- Shard Allocation: Monitor `cluster.stats.indices.shards.total`, `initializing_shards`, `relocating_shards`, and `unassigned_shards` from the cluster health API.
Integrating Elasticsearch Metrics with Prometheus
Prometheus is a popular choice for monitoring Elasticsearch. The official `prometheus-community/elasticsearch-exporter` is highly recommended. It scrapes Elasticsearch’s APIs and exposes metrics in Prometheus format.
Setting up the Elasticsearch Exporter
1. Download the Exporter: Obtain the latest release binary from the project’s GitHub repository.
# Example for Linux AMD64 wget https://github.com/prometheus-community/elasticsearch-exporter/releases/download/vX.Y.Z/elasticsearch_exporter-X.Y.Z.linux-amd64.tar.gz tar -xzf elasticsearch_exporter-X.Y.Z.linux-amd64.tar.gz sudo mv elasticsearch_exporter-X.Y.Z.linux-amd64/elasticsearch_exporter /usr/local/bin/ rm -rf elasticsearch_exporter-X.Y.Z.linux-amd64*
2. Create a `systemd` Service:
# /etc/systemd/system/elasticsearch_exporter.service [Unit] Description=Prometheus Exporter for Elasticsearch Wants=network-online.target After=network-online.target [Service] User=prometheus # Or a dedicated user Group=prometheus # Or a dedicated group Type=simple ExecStart=/usr/local/bin/elasticsearch_exporter \ --es.uri=http://localhost:9200 \ --es.timeout=5s \ --es.version=8.x # Specify your ES version if known # Add --es.username and --es.password if authentication is enabled Restart=always RestartSec=10 [Install] WantedBy=multi-user.target
3. Start and Enable the Service:
sudo systemctl daemon-reload sudo systemctl start elasticsearch_exporter sudo systemctl enable elasticsearch_exporter sudo systemctl status elasticsearch_exporter
4. Configure Prometheus: Add a scrape job to your `prometheus.yml`:
scrape_configs:
- job_name: 'elasticsearch'
static_configs:
- targets: ['localhost:9114'] # Default port for elasticsearch-exporter
5. Configure Grafana Dashboards: Import pre-built Elasticsearch dashboards (e.g., from Grafana.com) that utilize these metrics for visualization.
Linode Specific Considerations
When running on Linode, several Linode-specific features and limitations come into play for monitoring:
Linode NodeBalancers and Health Checks
If you’re using Linode NodeBalancers to distribute traffic to your PHP application servers, configure their health checks to point to your application’s health check endpoint (e.g., `http://
# Linode NodeBalancer Health Check Configuration Example Protocol: HTTP Port: 80 Check Path: /healthcheck.php Check Interval: 10 (seconds) Check Timeout: 5 (seconds) Response Timeout: 5 (seconds) Healthy Threshold: 3 Unhealthy Threshold: 3
This ensures that NodeBalancer stops sending traffic to unhealthy PHP instances automatically.
Linode Metrics and Alerts
Linode provides basic infrastructure metrics (CPU, network, disk I/O) through its Cloud Manager and API. While useful for overall server health, they are not sufficient for application-level monitoring.
Leverage Linode Alerts:
- Configure Linode Alerts for critical infrastructure metrics like CPU utilization exceeding 90% for extended periods, high network traffic, or disk I/O saturation.
- Set up email or webhook notifications for these alerts. Webhooks can be integrated with tools like Alertmanager or directly with Slack/Discord for immediate notification.
However, remember that Linode’s built-in monitoring is reactive and infrastructure-focused. For proactive application and service monitoring (like the PHP health check and Elasticsearch exporter), you’ll need to deploy your own monitoring stack (Prometheus, Grafana, Alertmanager) on your Linode instances or use a managed service.
Network Configuration and Firewalls
Ensure your Linode firewall rules (both `iptables`/`ufw` on the instance and Linode’s Cloud Firewall) allow traffic for:
- Your PHP application’s web port (e.g., 80, 443).
- Your Elasticsearch cluster’s transport port (9300) and HTTP port (9200) – restrict access to only necessary internal IPs or monitoring servers.
- The port your monitoring exporter is listening on (e.g., 9114 for Elasticsearch exporter).
For Elasticsearch, it’s critical to restrict access to ports 9200 and 9300. Ideally, these should only be accessible from within your private network or from specific monitoring/application servers, not the public internet.
Conclusion: A Layered Monitoring Strategy
Effective server monitoring for a PHP application and its Elasticsearch backend on Linode requires a multi-layered strategy:
- Application-Level: Implement granular health check endpoints in your PHP app.
- Service-Level: Monitor PHP-FPM (`systemd`) and Elasticsearch exporter processes.
- Cluster/System-Level: Utilize Elasticsearch’s APIs and Prometheus exporters for deep insights into cluster and node health.
- Infrastructure-Level: Leverage Linode’s built-in metrics and alerts for foundational server health.
- Network-Level: Configure firewalls and NodeBalancer health checks to manage traffic flow and isolate services.
By combining these approaches, you gain visibility into every layer of your stack, enabling proactive issue detection, faster troubleshooting, and ultimately, a more stable and reliable service for your users.