Real-time Observability for Laravel Applications on Kubernetes: Mastering Prometheus, Grafana, and Loki
Instrumenting Laravel for Prometheus Metrics
To achieve real-time observability, we first need to expose application-level metrics from our Laravel application. Prometheus is the de facto standard for this. We’ll leverage the prometheus_client_php library for this purpose. This library allows us to define and expose various metric types like counters, gauges, and histograms.
First, install the library via Composer:
composer require promphp/prometheus_client_php
Next, create a service provider to register the Prometheus exporter and its routes. This will typically be placed in app/Providers/AppServiceProvider.php or a dedicated MetricsServiceProvider.
<?php
namespace App\Providers;
use Illuminate\Support\ServiceProvider;
use Prometheus\CollectorRegistry;
use Prometheus\Render\CallbackRenderer;
use Prometheus\Storage\InMemory;
class AppServiceProvider extends ServiceProvider
{
/**
* Register any application services.
*
* @return void
*/
public function register()
{
$this->app->singleton(CollectorRegistry::class, function ($app) {
// For production, consider using Redis or APCu for persistent storage
// if you need metrics across multiple pods or restarts.
return new CollectorRegistry(new InMemory());
});
$this->app->singleton('prometheus.renderer', function ($app) {
return new CallbackRenderer($app->make(CollectorRegistry::class));
});
}
/**
* Bootstrap any application services.
*
* @return void
*/
public function boot()
{
// Register the route to expose metrics
if ($this->app->runningInConsole()) {
return;
}
$router = $this->app->make(\Illuminate\Contracts\Routing\Registrar::class);
$router->get('/metrics', function () {
$renderer = $this->app->make('prometheus.renderer');
return response($renderer->render(), 200, ['Content-Type' => CallbackRenderer::MIME_TYPE]);
})->name('prometheus.metrics');
// Example: Instrumenting HTTP request duration
$this->app->make(\Illuminate\Contracts\Http\Kernel::class)
->prependMiddleware(\App\Http\Middleware\PrometheusMetrics::class);
}
}
Now, create the middleware to capture request metrics. This middleware will record the duration of each request and increment a counter for successful requests.
namespace App\Http\Middleware;
use Closure;
use Illuminate\Http\Request;
use Prometheus\CollectorRegistry;
use Prometheus\Counter;
use Prometheus\Histogram;
class PrometheusMetrics
{
private CollectorRegistry $registry;
public function __construct(CollectorRegistry $registry)
{
$this->registry = $registry;
}
/**
* Handle an incoming request.
*
* @param \Illuminate\Http\Request $request
* @param \Closure(\Illuminate\Http\Request): \Illuminate\Http\Response|\Illuminate\Http\JsonResponse $next
* @return \Illuminate\Http\Response|\Illuminate\Http\JsonResponse
*/
public function handle(Request $request, Closure $next)
{
// Initialize metrics if they don't exist
$requestCounter = $this->registry->getOrRegister(
'app', 'http_requests_total', 'Total number of HTTP requests',
['method', 'path', 'status_code']
);
$requestDuration = $this->registry->getOrRegister(
'app', 'http_request_duration_seconds', 'HTTP request duration in seconds',
['method', 'path']
);
// Start timer
$startTime = microtime(true);
$response = $next($request);
// Stop timer and record duration
$duration = microtime(true) - $startTime;
$requestDuration->observe($duration, [$request->method(), $request->path()]);
// Increment request counter
$statusCode = $response->status();
$requestCounter->inc([$request->method(), $request->path(), $statusCode]);
return $response;
}
}
Ensure the middleware is registered in app/Http/Kernel.php within the $middleware or $middlewareGroups array, or as shown in the AppServiceProvider, prepended to the HTTP kernel.
Deploying Prometheus and Grafana to Kubernetes
We’ll use the official Prometheus and Grafana Helm charts for deployment. These charts are well-maintained and provide a robust starting point.
First, add the Prometheus community Helm repository:
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm repo update
Next, install Prometheus. We’ll configure it to scrape our Laravel application pods. This requires setting up a ServiceMonitor or PodMonitor (if using the Prometheus Operator) to tell Prometheus where to find the /metrics endpoint.
Create a values.yaml file for the Prometheus Helm chart:
server:
global:
scrape_interval: 15s # How frequently to scrape targets by default.
evaluation_interval: 15s # How frequently to evaluate rules.
# Enable the Prometheus Operator if you are using it
# prometheusOperator:
# enabled: true
# Configure scrape configs if not using Prometheus Operator
extraScrapeConfigs:
- job_name: 'laravel-app'
kubernetes_sd_configs:
- role: pod
relabel_configs:
# Only scrape pods with the 'app=my-laravel-app' label
- source_labels: [__meta_kubernetes_pod_label_app]
action: keep
regex: my-laravel-app
# Use the pod's IP and the 'metrics' port (defined in the pod spec)
- source_labels: [__address__]
regex: (.*):(\d+)
target_label: __address__
replacement: ${1}:8080 # Assuming your metrics port is 8080
# Add the metrics path
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
action: replace
target_label: __metrics_path__
regex: (.*)
replacement: /metrics # Default path if annotation is not present
# Add the pod name as a label
- source_labels: [__meta_kubernetes_pod_name]
action: replace
target_label: pod
# Add the namespace as a label
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: namespace
Deploy Prometheus:
helm install prometheus prometheus-community/prometheus \ --namespace monitoring \ --create-namespace \ -f values.yaml
Now, install Grafana. We’ll configure it to use Prometheus as a data source.
helm install grafana grafana/grafana \ --namespace monitoring \ --set adminPassword='your_strong_password' \ --set persistence.enabled=true \ --set persistence.storageClassName='your-storage-class' \ --set persistence.size=10Gi
To access Grafana, you can port-forward the service:
kubectl port-forward svc/grafana 3000:80 -n monitoring
Log in to Grafana with username admin and the password you set. Navigate to Configuration -> Data Sources, click Add data source, select Prometheus, and enter the Prometheus service URL (e.g., http://prometheus-server.monitoring.svc.cluster.local:9090). Save and test the connection.
Configuring Laravel Pods for Prometheus Scraping
For Prometheus to scrape your Laravel application pods, you need to ensure two things:
- The pods have a label that Prometheus can use to identify them (e.g.,
app: my-laravel-app). - The pods expose the metrics endpoint on a specific port and path.
In your Kubernetes deployment manifest (e.g., deployment.yaml), add the necessary labels:
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-laravel-app
spec:
selector:
matchLabels:
app: my-laravel-app
template:
metadata:
labels:
app: my-laravel-app
# Add annotations for Prometheus scraping
annotations:
prometheus.io/scrape: "true"
prometheus.io/path: "/metrics"
prometheus.io/port: "8080" # The port your application exposes metrics on
spec:
containers:
- name: app
image: your-laravel-app-image:latest
ports:
- containerPort: 8080 # The port your application listens on for metrics
# ... other container configurations
You also need a Kubernetes Service that targets these pods. If you are using the Prometheus Operator, a ServiceMonitor is the preferred way to configure scraping. If not, Prometheus’s kubernetes_sd_configs in its configuration will handle discovery based on pod labels and annotations.
Example ServiceMonitor (if using Prometheus Operator):
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: laravel-app-monitor
namespace: default # Namespace where your Laravel app is deployed
labels:
release: prometheus # Label to match Prometheus Operator's discovery
spec:
selector:
matchLabels:
app: my-laravel-app # Label on your Laravel pods
namespaceSelector:
matchNames:
- default # Namespace where your Laravel app is deployed
endpoints:
- port: metrics # Name of the port in your Service, or the port number
path: /metrics
interval: 15s
If you are not using the Prometheus Operator, the extraScrapeConfigs in the Prometheus Helm chart configuration (shown previously) will handle discovery based on pod annotations.
Integrating Loki for Log Aggregation
For log aggregation, we’ll use Loki, a horizontally scalable, highly available, multi-tenant log aggregation system inspired by Prometheus. We’ll deploy Loki and Promtail (its log collection agent) using Helm.
Add the Grafana Helm repository (if you haven’t already):
helm repo add grafana https://grafana.github.io/helm-charts helm repo update
Create a loki-values.yaml file for Loki and Promtail:
loki:
persistence:
enabled: true
storageClassName: "your-storage-class" # e.g., gp2, standard
size: 50Gi
promtail:
enabled: true
config:
scrape_configs:
- job_name: kubernetes-pods
kubernetes_sd_configs:
- role: pod
relabel_configs:
# Keep only pods with the 'app=my-laravel-app' label
- source_labels: [__meta_kubernetes_pod_label_app]
action: keep
regex: my-laravel-app
# Extract namespace and pod name for labels
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: namespace
- source_labels: [__meta_kubernetes_pod_name]
action: replace
target_label: pod
# Use container name as stream label
- source_labels: [__meta_kubernetes_pod_container_name]
action: replace
target_label: container
# Filter logs by container name if needed
# - source_labels: [__meta_kubernetes_pod_container_name]
# action: keep
# regex: app # Or your specific container name
# Set log file path based on container and pod
- source_labels: [__meta_kubernetes_pod_name, __meta_kubernetes_pod_container_name]
target_label: __path__
replacement: /var/log/pods/${1}/${2}/*log # Adjust path as per your container logging driver
# Add labels for log stream, e.g., application name
static_configs:
- labels:
job: kubernetes-application # Or any other identifier
app: my-laravel-app
Deploy Loki and Promtail:
helm install loki grafana/loki \ --namespace monitoring \ -f loki-values.yaml
Promtail will automatically discover and tail logs from your Laravel application pods based on the kubernetes_sd_configs and relabel_configs. Ensure your Laravel application is configured to log to standard output (stdout) or a file that Promtail can access within the container’s filesystem.
Visualizing Metrics and Logs in Grafana
With Prometheus and Loki deployed and configured, we can now visualize the data in Grafana.
First, add Loki as a data source in Grafana. Navigate to Configuration -> Data Sources, click Add data source, select Loki, and enter the Loki service URL (e.g., http://loki.monitoring.svc.cluster.local:3100). Save and test.
Now, create a new dashboard in Grafana. You can import pre-built dashboards or create your own panels.
Example Prometheus Panel (HTTP Request Rate):
Query Type: Prometheus
rate(http_requests_total{job="laravel-app"}[5m])
Example Prometheus Panel (HTTP Request Duration Histogram):
histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{job="laravel-app"}[5m])) by (le, method, path))
Example Loki Panel (Laravel Logs):
{job="kubernetes-application", namespace="default", app="my-laravel-app"} | json | level="error"
This query will display all log lines from your Laravel application that are JSON formatted and have a log level of “error”. You can adjust the labels and filters to narrow down your search.
To link metrics and logs, you can use Grafana’s “Explore” view. Select your Prometheus data source, run a metric query, and then switch to your Loki data source. Grafana will often pre-fill the Loki query with labels derived from the metric context, allowing you to jump directly from a spike in errors on a graph to the corresponding log entries.
Advanced Considerations and Best Practices
Persistent Storage: For production environments, ensure that both Prometheus and Loki have persistent storage configured using appropriate Kubernetes StorageClasses. This prevents data loss during pod restarts or upgrades.
Alerting: Configure Prometheus Alertmanager to set up alerts based on your metrics. For example, you can alert on high error rates, increased request latency, or low request throughput.
Resource Management: Properly configure resource requests and limits for Prometheus, Grafana, Loki, and Promtail pods in Kubernetes to ensure stability and prevent resource contention.
Security: Secure access to Grafana and Prometheus UIs. Consider using Ingress controllers with authentication or network policies to restrict access.
Metric Granularity: For high-traffic applications, consider using more efficient storage backends for Prometheus (e.g., Thanos, Cortex) and Loki (e.g., Cassandra, S3) to handle scale and long-term retention.
Laravel Logging: Configure Laravel’s logging to output in a structured format (e.g., JSON) to make parsing logs in Loki more straightforward. The monolog/monolog library, which Laravel uses, supports JSON formatting.
// config/logging.php
'channels' => [
// ...
'loki' => [
'driver' => 'single',
'path' => storage_path('logs/laravel.log'),
'level' => env('LOG_LEVEL', 'debug'),
'formatter' => \Monolog\Formatter\JsonFormatter::class, // Use JSON formatter
],
// ...
],
By implementing these steps, you establish a robust, real-time observability stack for your Laravel applications running on Kubernetes, enabling proactive issue detection and faster debugging.