Vengala Vinay

Having 9+ Years of Experience in Software Development

Home » Server Monitoring Best Practices: Keeping Your Laravel App and Redis Clusters Alive on OVH

Server Monitoring Best Practices: Keeping Your Laravel App and Redis Clusters Alive on OVH

Proactive Redis Cluster Health Checks with `redis-cli` and Custom Scripts

Maintaining the health of a Redis cluster, especially one powering a critical Laravel application, requires more than just basic uptime checks. We need to monitor internal cluster state, replication lag, and memory usage. OVH’s infrastructure, while robust, doesn’t absolve us of the responsibility for deep application-level monitoring. A common pitfall is relying solely on external `ping` checks, which tell us nothing about Redis’s internal operational status.

We’ll start by leveraging the `redis-cli` tool to gather essential metrics. For a Redis Sentinel setup, monitoring the master’s status and the health of its replicas is paramount. A simple script can automate these checks and alert us to potential issues before they impact the Laravel application.

Automated Redis Sentinel Master/Replica Monitoring

This Python script connects to a Redis Sentinel instance and queries the status of the master and its replicas. It checks for the number of available replicas and the replication lag. If any replica is down or the lag exceeds a defined threshold, it triggers an alert.

First, ensure you have the `redis-py` library installed:

pip install redis

Here’s the monitoring script:

import redis
import time
import sys

# --- Configuration ---
SENTINEL_HOST = 'your_sentinel_host'  # e.g., '192.168.1.10'
SENTINEL_PORT = 26379
MASTER_NAME = 'mymaster'  # The name of your master set in Sentinel config
MAX_REPLICATION_LAG_SECONDS = 5  # Alert if replication lag exceeds this
MIN_REPLICAS_REQUIRED = 2  # Alert if fewer than this many replicas are available
ALERT_EMAIL_RECIPIENT = '[email protected]'
ALERT_EMAIL_SUBJECT = f'ALERT: Redis Cluster unhealthy ({MASTER_NAME})'
# --- End Configuration ---

def send_alert(message):
    """
    Placeholder for your alerting mechanism.
    This could be sending an email, posting to Slack, PagerDuty, etc.
    For simplicity, we'll just print to stderr.
    """
    print(f"ALERT: {message}", file=sys.stderr)
    # Example for sending email (requires smtplib and email.mime modules)
    # from email.mime.text import MIMEText
    # import smtplib
    # msg = MIMEText(message)
    # msg['Subject'] = ALERT_EMAIL_SUBJECT
    # msg['From'] = '[email protected]'
    # msg['To'] = ALERT_EMAIL_RECIPIENT
    # try:
    #     with smtplib.SMTP('your_smtp_server', 587) as server:
    #         server.starttls()
    #         server.login('your_smtp_user', 'your_smtp_password')
    #         server.sendmail('[email protected]', [ALERT_EMAIL_RECIPIENT], msg.as_string())
    # except Exception as e:
    #     print(f"Failed to send email alert: {e}", file=sys.stderr)

def check_redis_cluster():
    try:
        sentinel = redis.Sentinel([(SENTINEL_HOST, SENTINEL_PORT)], socket_timeout=5)
        master = sentinel.master_for(MASTER_NAME, socket_timeout=5)
        replicas = sentinel.slaves_for(MASTER_NAME, socket_timeout=5)

        # Check master status (basic connectivity)
        try:
            master.ping()
            print(f"Master '{MASTER_NAME}' is reachable.")
        except redis.exceptions.ConnectionError as e:
            send_alert(f"Master '{MASTER_NAME}' is unreachable: {e}")
            return

        # Check replica count
        if len(replicas) < MIN_REPLICAS_REQUIRED:
            send_alert(f"Insufficient replicas for '{MASTER_NAME}'. Required: {MIN_REPLICAS_REQUIRED}, Available: {len(replicas)}")

        # Check replication lag for each replica
        for replica in replicas:
            try:
                replica.ping() # Ensure replica is responsive
                # Get master's current replication offset
                master_repl_offset = master.info('replication').get('master_repl_offset')
                if not master_repl_offset:
                    send_alert(f"Could not retrieve master replication offset for '{MASTER_NAME}'.")
                    continue

                master_repl_offset = int(master_repl_offset)

                # Get replica's slave_repl_offset
                replica_info = replica.info('replication')
                replica_repl_offset = replica_info.get('slave_repl_offset')

                if not replica_repl_offset:
                    # This can happen if the replica just connected or is in a weird state
                    print(f"Warning: Replica {replica.connection_pool.connection_kwargs['host']}:{replica.connection_pool.connection_kwargs['port']} has no slave_repl_offset. Assuming it's syncing.")
                    continue

                replica_repl_offset = int(replica_repl_offset)

                lag = master_repl_offset - replica_repl_offset

                if lag > MAX_REPLICATION_LAG_SECONDS:
                    send_alert(f"High replication lag on replica {replica.connection_pool.connection_kwargs['host']}:{replica.connection_pool.connection_kwargs['port']}. Lag: {lag}s (Master Offset: {master_repl_offset}, Replica Offset: {replica_repl_offset})")
                else:
                    print(f"Replica {replica.connection_pool.connection_kwargs['host']}:{replica.connection_pool.connection_kwargs['port']} lag is acceptable ({lag}s).")

            except redis.exceptions.ConnectionError as e:
                send_alert(f"Replica {replica.connection_pool.connection_kwargs['host']}:{replica.connection_pool.connection_kwargs['port']} is unreachable: {e}")
            except Exception as e:
                send_alert(f"Error checking replica {replica.connection_pool.connection_kwargs['host']}:{replica.connection_pool.connection_kwargs['port']}: {e}")

    except redis.exceptions.ConnectionError as e:
        send_alert(f"Could not connect to Sentinel at {SENTINEL_HOST}:{SENTINEL_PORT}: {e}")
    except Exception as e:
        send_alert(f"An unexpected error occurred: {e}")

if __name__ == "__main__":
    check_redis_cluster()

To integrate this into your monitoring system (e.g., Nagios, Zabbix, Prometheus with `node_exporter`’s textfile collector, or a simple cron job), you can schedule this script to run at regular intervals. For instance, using cron:

Edit your crontab:

crontab -e

Add the following line to run the script every minute:

* * * * * /usr/bin/python3 /path/to/your/redis_monitor.py >> /var/log/redis_monitor.log 2>&1

Remember to replace your_sentinel_host, mymaster, and /path/to/your/redis_monitor.py with your actual configuration. The output to stderr will be captured by your cron daemon or system logger, which should be configured to forward alerts.

Laravel Application Performance Monitoring (APM) with New Relic

For the Laravel application itself, deep visibility into its performance is crucial. This includes tracking request latency, database query times, external service calls, and error rates. New Relic is a powerful APM tool that integrates seamlessly with PHP applications.

On your OVH server, you’ll need to install the New Relic PHP agent. The exact installation steps can vary slightly based on your PHP version and OS distribution (e.g., Ubuntu, Debian, CentOS). Always refer to the official New Relic documentation for the most up-to-date instructions.

Generally, the process involves:

  • Downloading the New Relic agent installer script.
  • Running the installer script, providing your New Relic license key.
  • Configuring the agent by editing the newrelic.ini file.
  • Restarting your web server (e.g., Apache, Nginx with PHP-FPM) and PHP-FPM service.

Here’s a typical command sequence for Ubuntu/Debian:

# Download the installer
wget https://download.newrelic.com/install/newrelic-php5-linux.tar.gz
tar xvf newrelic-php5-linux.tar.gz
cd newrelic-install

# Run the installer (replace YOUR_LICENSE_KEY)
sudo bash newrelic-install.sh install YOUR_LICENSE_KEY

# After installation, configure the agent
sudo nano /etc/php/7.4/fpm/conf.d/newrelic.ini # Adjust PHP version as needed
# Or for CLI: sudo nano /etc/php/7.4/cli/conf.d/newrelic.ini

# Ensure these settings are correct in newrelic.ini:
# license = YOUR_LICENSE_KEY
# appname = "Your Laravel App Name"
# enabled = true
# log.level = "info"
# high_security = false # Set to true for stricter security if needed

# Restart PHP-FPM and web server
sudo systemctl restart php7.4-fpm # Adjust PHP version
sudo systemctl restart nginx # Or apache2

Once installed and configured, New Relic will automatically start collecting data for your Laravel application. You can then access the New Relic dashboard to monitor:

  • Transaction Traces: Identify slow requests and bottlenecks.
  • Database Performance: Monitor query times, count, and identify slow queries.
  • External Services: Track latency and errors for API calls.
  • Error Reporting: Get detailed stack traces for exceptions.
  • Throughput and Response Time: Understand overall application health and user experience.

For Laravel-specific insights, New Relic automatically instruments many common framework components. You can also add custom instrumentation using the New Relic API within your Laravel code to track specific business logic or critical code paths.

OVH Infrastructure Monitoring with `ovh-monitoring-agent` (Conceptual)

While New Relic and custom scripts cover application and Redis cluster health, monitoring the underlying OVH infrastructure is also vital. OVH provides its own monitoring tools, often accessible via their control panel or API. For deeper, OS-level metrics, you might deploy a dedicated agent.

OVH offers a service called “Public Cloud Archive” and “Monitoring” which can be configured through the OVHcloud Control Panel. This allows you to set up alerts based on CPU usage, network traffic, disk I/O, and memory usage for your Public Cloud instances.

For more granular control and integration with existing monitoring stacks (like Prometheus/Grafana), you can deploy agents like node_exporter. This agent exposes system metrics in a Prometheus-compatible format.

# Example: Installing Prometheus node_exporter on Ubuntu
sudo apt update
sudo apt install prometheus-node-exporter

# Configure node_exporter to run as a service (systemd)
# Create a file like /etc/systemd/system/node_exporter.service
# Example content:
# [Unit]
# Description=Node Exporter
# Wants=network-online.target
# After=network-online.target
#
# [Service]
# User=prometheus
# Group=prometheus
# Type=simple
# ExecStart=/usr/bin/prometheus-node-exporter \
#     --web.listen-address="0.0.0.0:9100" \
#     --collector.filesystem.mount-points-exclude="^/(sys|proc|dev|host|etc)($$|/.*)"
#
# [Install]
# WantedBy=multi-user.target

# Reload systemd and start the service
sudo systemctl daemon-reload
sudo systemctl start prometheus-node-exporter
sudo systemctl enable prometheus-node-exporter

# Verify it's running and accessible
curl http://localhost:9100/metrics

You would then configure your Prometheus server to scrape metrics from this node_exporter endpoint. This provides a unified view of your server’s resource utilization, which can be correlated with application performance issues identified by New Relic or Redis cluster alerts.

Log Aggregation and Analysis with ELK Stack or Loki

Centralized logging is indispensable for debugging and understanding the behavior of distributed systems. For your Laravel application and Redis cluster, aggregating logs into a central location allows for easier searching, correlation, and alerting on specific events.

Popular choices include the ELK Stack (Elasticsearch, Logstash, Kibana) or Grafana Loki. For simplicity and resource efficiency, Loki is often preferred for containerized environments, but it’s also effective on bare-metal servers.

For Laravel: Configure Laravel’s Monolog handler to send logs to your chosen aggregation system. This might involve using a custom handler or a pre-built package.

// config/logging.php

'channels' => [
    // ... other channels
    'loki' => [
        'driver' => 'monolog',
        'handler' => \Monolog\Handler\GelfHandler::class, // Or a custom handler for Loki
        'with' => [
            'host' => env('LOKI_HOST', 'loki.yourdomain.com'),
            'port' => env('LOKI_PORT', 12201), // Default GELF UDP port
            'level' => env('LOG_LEVEL', 'debug'),
            // Add labels for Loki
            'extra_fields' => [
                'environment' => env('APP_ENV', 'production'),
                'app' => 'laravel-app',
            ],
        ],
    ],
    // ...
    'daily' => [
        'driver' => 'daily',
        'path' => storage_path('logs/laravel.log'),
        'level' => env('LOG_LEVEL', 'debug'),
        'days' => env('LOG_RETENTION', 14),
    ],
    // ...
],

// Set default channel to loki or a combination
'default' => env('LOG_CHANNEL', 'loki'), // Or 'stack' to combine daily and loki

You’ll need a log shipping agent like Promtail (for Loki) or Filebeat (for Elasticsearch) running on your OVH server to tail these log files and send them to your central logging system. Configure Promtail to watch storage/logs/laravel.log and add appropriate labels (e.g., app="laravel", env="production").

For Redis: Redis logs can be configured in redis.conf. Ensure you’re logging errors and slow commands (if enabled). Use your log shipping agent (Promtail/Filebeat) to tail these Redis log files and send them to your central system, adding labels like service="redis", role="master/replica".

Conclusion: A Multi-Layered Approach

Effective server monitoring for a Laravel application and its Redis cluster on OVH requires a multi-layered strategy. This involves:

  • Application Performance Monitoring (APM): Tools like New Relic provide deep insights into Laravel’s runtime behavior.
  • Database/Cache Cluster Health: Custom scripts using redis-cli or libraries like redis-py to monitor Redis Sentinel status, replication lag, and availability.
  • Infrastructure Metrics: OS-level monitoring via agents like node_exporter, integrated with Prometheus, and leveraging OVH’s own cloud monitoring features.
  • Centralized Logging: Aggregating logs from both Laravel and Redis using ELK or Loki for debugging and historical analysis.

By implementing these practices, you move from reactive firefighting to proactive system management, ensuring the stability and performance of your critical applications.