Server Monitoring Best Practices: Keeping Your Laravel App and DynamoDB Clusters Alive on OVH
Proactive Laravel & DynamoDB Monitoring on OVH: Beyond Basic Metrics
Maintaining high availability for a Laravel application backed by AWS DynamoDB, especially when hosted on OVH infrastructure, demands a robust, multi-layered monitoring strategy. This isn’t about simply checking CPU and memory; it’s about understanding application-level performance, database throughput, network latency between OVH and AWS, and implementing actionable alerts that prevent outages before they impact users.
I. Laravel Application Performance Monitoring (APM)
For Laravel, we need to go beyond basic web server logs. APM tools provide deep insights into request tracing, database query performance, and external service calls. While commercial solutions like New Relic or Datadog are powerful, a cost-effective and highly capable open-source alternative is Jaeger, integrated with Prometheus for metrics collection.
A. Jaeger Integration with Laravel
Jaeger’s OpenTracing instrumentation allows us to trace requests as they flow through your Laravel application. This involves instrumenting your code to send trace data to a Jaeger agent or collector.
First, install the necessary packages:
composer require jaegertracing/jaeger-client php-http/guzzle6-adapter
Next, configure the Jaeger client in your Laravel application. This typically involves creating a service provider and binding the tracer instance.
<?php
namespace App\Providers;
use Illuminate\Support\ServiceProvider;
use Jaeger\Config;
use Jaeger\Reporter\RemoteReporter;
use Jaeger\Sampler\ConstSampler;
use OpenTracing\Tracer;
use GuzzleHttp\Client as HttpClient;
class JaegerServiceProvider extends ServiceProvider
{
/**
* Register services.
*
* @return void
*/
public function register()
{
$this->app->singleton(Tracer::class, function ($app) {
$config = Config::getInstance();
$config->initializeTracer(
env('JAEGER_SERVICE_NAME', 'laravel-app'),
env('JAEGER_AGENT_HOST', 'localhost'),
env('JAEGER_AGENT_PORT', 6831),
env('JAEGER_SAMPLER_TYPE', 'const'),
env('JAEGER_SAMPLER_PARAM', 1) // 1 for always sample, 0 for never
);
// If you need to configure a specific reporter or sampler:
// $reporter = new RemoteReporter(new UdpTransport($config->getAgentHost(), $config->getAgentPort()));
// $sampler = new ConstSampler(true); // Always sample
// $tracer = new Tracer($reporter, $sampler);
return $config->tracer();
});
}
/**
* Bootstrap services.
*
* @return void
*/
public function boot()
{
//
}
}
Register this provider in config/app.php.
Now, you can inject the Tracer into your controllers or services to create spans. For example, to trace an incoming request:
<?php
namespace App\Http\Controllers;
use Illuminate\Http\Request;
use OpenTracing\Tracer;
use OpenTracing\Span;
class ExampleController extends Controller
{
protected $tracer;
public function __construct(Tracer $tracer)
{
$this->tracer = $tracer;
}
public function show(Request $request)
{
// Start a new span for the incoming request
$span = $this->tracer->startSpan('http_request');
$span->setTag('http.method', $request->method());
$span->setTag('http.url', $request->fullUrl());
try {
// Your controller logic here...
// Example: Trace a database query (if not automatically instrumented)
$dbSpan = $this->tracer->startSpan('db_query', ['child_of' => $span->getContext()]);
// Execute your Eloquent query
$users = \App\Models\User::all();
$dbSpan->finish();
// Finish the request span
$span->finish();
return response()->json($users);
} catch (\Exception $e) {
// Log the error and set span status to error
$span->setTag('error', true);
$span->log(['event' => 'error', 'message' => $e->getMessage()]);
$span->finish();
throw $e;
}
}
}
For automatic instrumentation of common libraries (like Guzzle, PDO), consider using libraries like opentracing-php/opentracing-php with specific integrations or a more comprehensive APM agent that handles this automatically.
B. Prometheus Metrics Collection
While Jaeger excels at tracing, Prometheus is ideal for collecting time-series metrics. We can expose custom metrics from Laravel and scrape them using Prometheus.
Install the Prometheus client library:
composer require promphp/prometheus_client
Create a route to expose metrics:
<?php
namespace App\Http\Controllers;
use Illuminate\Http\Request;
use Prometheus\Render\CallbackRenderer;
use Prometheus\Storage\InMemory;
use Prometheus\Registry;
class MetricsController extends Controller
{
public function index()
{
$adapter = new InMemory();
$registry = new Registry($adapter);
// Example: Expose a custom counter for successful API calls
$counter = $registry->registerCounter(
'myapp', 'api_calls_total', 'Total number of API calls', ['method', 'endpoint']
);
// In your API routes or controllers, you would increment this counter:
// $counter->incBy(1, ['GET', '/api/users']);
// Example: Expose a gauge for active users
$gauge = $registry->registerGauge(
'myapp', 'active_users', 'Number of currently active users', ['status']
);
// $gauge->set(count($activeUsers), ['logged_in']);
$renderer = new CallbackRenderer($registry);
return response($renderer->render(), 200, ['Content-Type' => 'text/plain']);
}
}
Add this route to routes/api.php or routes/web.php:
// routes/api.php
Route::get('/metrics', [App\Http\Controllers\MetricsController::class, 'index']);
Configure Prometheus to scrape this endpoint. Assuming your Laravel app is accessible at http://your-laravel-app.ovh/api/metrics:
scrape_configs:
- job_name: 'laravel_app'
static_configs:
- targets: ['your-laravel-app.ovh:80'] # Or your specific OVH IP/domain
metrics_path: '/api/metrics'
scheme: 'http' # or 'https' if you have SSL configured
II. DynamoDB Performance and Health Monitoring
DynamoDB is a managed service, so we don’t monitor server health directly. Instead, we focus on throughput, latency, and errors reported by AWS CloudWatch.
A. Key DynamoDB CloudWatch Metrics
Essential metrics to monitor for each DynamoDB table and global secondary index (GSI):
- ConsumedReadCapacityUnits / ConsumedWriteCapacityUnits: Track actual usage against provisioned capacity. Spikes indicate potential throttling.
- ProvisionedReadCapacityUnits / ProvisionedWriteCapacityUnits: Monitor your configured capacity.
- ReadThrottleEvents / WriteThrottleEvents: Crucial for identifying throttling. Any non-zero value here is a problem.
- SuccessfulRequestLatency: Average, 90th, 95th, and 99th percentile latency for read and write operations. High latency directly impacts application performance.
- SystemErrors: Server-side errors within DynamoDB.
- UserErrors: Client-side errors (e.g., validation errors).
- ReturnedItemCount: Useful for understanding the efficiency of scan/query operations.
B. Integrating CloudWatch Metrics with Prometheus/Grafana
To bring AWS metrics into your existing Prometheus/Grafana stack, you can use the cloudwatch_exporter. This exporter runs as a separate service, queries CloudWatch, and exposes the metrics in Prometheus format.
1. **Deploy cloudwatch_exporter:** This can be run on a VM within OVH or even on an EC2 instance in the same AWS region as your DynamoDB tables. Ensure it has appropriate IAM permissions to access CloudWatch metrics.
# Example cloudwatch_exporter configuration (config.yml)
# This configuration queries specific DynamoDB metrics.
aws:
region: 'us-east-1' # Replace with your AWS region
access_key_id: 'YOUR_ACCESS_KEY_ID' # Or use IAM roles
secret_access_key: 'YOUR_SECRET_ACCESS_KEY'
# Define which metrics to scrape
metrics:
- namespace: 'AWS/DynamoDB'
name: 'ConsumedReadCapacityUnits'
dimensions:
- name: 'TableName'
value: 'your-dynamodb-table-name' # Replace with your table name
statistics:
- Average
- Maximum
period: 60 # seconds
- namespace: 'AWS/DynamoDB'
name: 'ConsumedWriteCapacityUnits'
dimensions:
- name: 'TableName'
value: 'your-dynamodb-table-name'
statistics:
- Average
- Maximum
period: 60
- namespace: 'AWS/DynamoDB'
name: 'ReadThrottleEvents'
dimensions:
- name: 'TableName'
value: 'your-dynamodb-table-name'
statistics:
- Sum
period: 60
- namespace: 'AWS/DynamoDB'
name: 'WriteThrottleEvents'
dimensions:
- name: 'TableName'
value: 'your-dynamodb-table-name'
statistics:
- Sum
period: 60
- namespace: 'AWS/DynamoDB'
name: 'SuccessfulRequestLatency'
dimensions:
- name: 'TableName'
value: 'your-dynamodb-table-name'
statistics:
- Average
- p95 # 95th percentile
period: 60
2. **Configure Prometheus to scrape cloudwatch_exporter:**
scrape_configs:
- job_name: 'cloudwatch_dynamodb'
static_configs:
- targets: ['cloudwatch-exporter-host:9100'] # IP/hostname of your cloudwatch_exporter
3. **Create Grafana Dashboards:** Use the scraped metrics to build dashboards visualizing DynamoDB performance. Key panels include:
- Provisioned vs. Consumed Capacity (Bar/Line chart)
- Throttle Events (Stat/Gauge showing current count, alert if > 0)
- Request Latency (Graph showing p95/p99 latency over time)
- Error Rates (Graph for SystemErrors and UserErrors)
III. Network Latency Monitoring (OVH <-> AWS)
The physical and network distance between your OVH-hosted Laravel application and AWS DynamoDB can introduce latency. Proactive monitoring of this link is crucial.
A. Ping and Traceroute from OVH
Regularly pinging AWS endpoints from your OVH infrastructure can reveal network degradation. You can automate this using a simple Bash script.
#!/bin/bash
# AWS endpoint to ping (e.g., a regional DynamoDB endpoint)
# Find your specific endpoint at https://docs.aws.amazon.com/general/latest/gr/rande.html
TARGET_ENDPOINT="dynamodb.us-east-1.amazonaws.com"
LOG_FILE="/var/log/network_monitor.log"
ALERT_THRESHOLD_MS=150 # Adjust based on your acceptable latency
ALERT_FILE="/tmp/dynamodb_latency_alert"
# Get current timestamp
TIMESTAMP=$(date +"%Y-%m-%d %H:%M:%S")
# Perform ping
PING_OUTPUT=$(ping -c 5 "$TARGET_ENDPOINT" | grep "avg/min/max/mdev")
LATENCY_AVG=$(echo "$PING_OUTPUT" | awk -F'/' '{print $5}' | awk -F'.' '{print $1}')
echo "$TIMESTAMP - Pinging $TARGET_ENDPOINT: $PING_OUTPUT" >> "$LOG_FILE"
if [[ -n "$LATENCY_AVG" && "$LATENCY_AVG" -gt "$ALERT_THRESHOLD_MS" ]]; then
echo "$TIMESTAMP - ALERT: High latency detected ($LATENCY_AVG ms) to $TARGET_ENDPOINT" >> "$LOG_FILE"
# Trigger an alert (e.g., send an email, Slack notification, or trigger a Prometheus alert)
touch "$ALERT_FILE" # Simple flag file for demonstration
# Example: curl -X POST -H 'Content-type: application/json' --data '{"text":"High DynamoDB latency detected: '"$LATENCY_AVG"'ms"}' YOUR_SLACK_WEBHOOK_URL
else
# Clear alert if latency is back to normal
if [ -f "$ALERT_FILE" ]; then
echo "$TIMESTAMP - Latency back to normal." >> "$LOG_FILE"
rm "$ALERT_FILE"
fi
fi
# Optional: Traceroute for more detailed path analysis if latency is consistently high
# traceroute "$TARGET_ENDPOINT" >> "$LOG_FILE"
Schedule this script to run periodically (e.g., every 5 minutes) using cron.
# crontab -e */5 * * * * /path/to/your/network_monitor.sh
You can then use Prometheus’s blackbox_exporter to scrape these ping results or the presence of the alert file for centralized alerting.
B. Application-Level Latency Measurement
The most accurate measure of network impact is the latency experienced by your application when interacting with DynamoDB. This should be captured by your APM (Jaeger) and DynamoDB CloudWatch metrics (SuccessfulRequestLatency).
Correlate high SuccessfulRequestLatency in CloudWatch with periods of high network latency detected by ping/traceroute from OVH. If application-level latency is high but network ping is low, the issue might be within DynamoDB itself (e.g., hot partitions, inefficient queries) or within the AWS network fabric closer to the service.
IV. Alerting Strategy
Effective alerting is about reducing noise and ensuring critical issues are addressed promptly. Use Prometheus Alertmanager for centralized alerting.
A. Alerting Rules (Prometheus)
Example alerting rules in Prometheus:
groups:
- name: laravel_dynamodb_alerts
rules:
- alert: HighDynamoDBReadLatency
expr: avg by (table) (aws_dynamodb_successfulrequestlatency_p95{job="cloudwatch_dynamodb"}) > 100 # Latency in ms
for: 5m
labels:
severity: warning
annotations:
summary: "High 95th percentile read latency for DynamoDB table {{ $labels.table }}"
description: "95th percentile read latency for table {{ $labels.table }} is {{ $value }}ms, exceeding the threshold of 100ms for 5 minutes."
- alert: DynamoDBReadThrottling
expr: sum by (table) (aws_dynamodb_readthrottleevents_sum{job="cloudwatch_dynamodb"}) > 0
for: 1m
labels:
severity: critical
annotations:
summary: "DynamoDB read throttling detected on table {{ $labels.table }}"
description: "Table {{ $labels.table }} is experiencing read throttling events."
- alert: HighLaravelRequestLatency
expr: avg by (job) (http_request_duration_seconds_bucket{le="1.0", job="laravel_app"}) > 0.95 # 95% of requests take longer than 1s
for: 5m
labels:
severity: warning
annotations:
summary: "High latency for Laravel app requests"
description: "95% of requests to {{ $labels.job }} are taking longer than 1s."
- alert: HighNetworkLatencyToDynamoDB
expr: vector(1) # This is a placeholder; ideally, use blackbox_exporter or the alert file status
# For the alert file method:
# expr: absent(time() - file_mtime("/tmp/dynamodb_latency_alert")) < 300 # Alert if alert file is older than 5 mins
for: 10m
labels:
severity: critical
annotations:
summary: "Potential network issue to DynamoDB endpoint"
description: "Network latency monitoring script has not updated the alert file in 10 minutes, or high latency was detected."
B. Alert Routing and Notification
Configure Alertmanager to route alerts to appropriate channels (PagerDuty, Slack, email) based on severity and team responsibility. Ensure that critical alerts trigger immediate notifications, while warnings might be batched or sent to less urgent channels.
V. Log Aggregation and Analysis
Centralized logging is indispensable for post-mortem analysis and identifying recurring issues. Use tools like Elasticsearch, Fluentd, and Kibana (EFK stack) or Loki, Promtail, and Grafana.
Ensure your Laravel application logs:
- Application errors (exceptions, stack traces)
- Slow database queries (if not fully covered by APM)
- Authentication failures
- Significant events (e.g., cache invalidation, background job failures)
Configure Fluentd or Promtail on your OVH servers to collect these logs and forward them to your aggregation backend. In Kibana or Grafana, create dashboards to visualize error rates, log volume, and search for specific error messages.
VI. Regular Audits and Capacity Planning
Monitoring isn’t a set-and-forget solution. Regularly review your dashboards and alerts:
- Analyze historical trends: Are read/write capacity units consistently increasing? Is latency creeping up?
- Tune provisioned capacity: Adjust DynamoDB provisioned throughput based on observed usage patterns to balance cost and performance. Consider using DynamoDB Auto Scaling.
- Optimize queries: Identify inefficient queries from APM traces or logs and refactor them.
- Review alert thresholds: Are alerts firing too often (false positives) or not often enough (missed incidents)? Adjust thresholds accordingly.
By implementing this comprehensive monitoring strategy, you gain deep visibility into your Laravel application and DynamoDB cluster’s health, enabling proactive issue resolution and ensuring a stable, high-performing environment on OVH.