Vengala Vinay

Having 9+ Years of Experience in Software Development

Home » Server Monitoring Best Practices: Keeping Your Python App and Redis Clusters Alive on OVH

Server Monitoring Best Practices: Keeping Your Python App and Redis Clusters Alive on OVH

Establishing a Robust Monitoring Foundation

Effective server monitoring for production Python applications and Redis clusters on OVH requires a multi-layered approach. We’ll focus on actionable strategies, starting with essential system-level metrics and progressing to application-specific and cluster-aware monitoring.

System-Level Metrics: The Pulse of Your Servers

Before diving into application specifics, we need to ensure the underlying infrastructure is healthy. This involves monitoring CPU, memory, disk I/O, and network traffic. For this, we’ll leverage tools like node_exporter, which exposes system metrics in a Prometheus-compatible format.

Installation on a Debian/Ubuntu-based OVH instance:

sudo apt update
sudo apt install -y prometheus-node-exporter

# Start and enable the service
sudo systemctl start prometheus-node-exporter
sudo systemctl enable prometheus-node-exporter

# Verify status
sudo systemctl status prometheus-node-exporter

By default, node_exporter runs on port 9100. Ensure this port is accessible from your Prometheus server (or the server running Prometheus if it’s co-located). If you’re using OVH’s firewall, you’ll need to open this port.

Monitoring Python Applications with Prometheus Client Libraries

Instrumenting your Python application is crucial for understanding its behavior. The prometheus_client library is the standard for this. We’ll focus on key metrics: request latency, error rates, and custom business metrics.

First, install the library:

pip install prometheus_client

Here’s a basic example of how to instrument a Flask application:

from flask import Flask, request, Response
from prometheus_client import Counter, Histogram, Gauge, push_to_gateway, CollectorRegistry, write_to_textfile
import time
import os

# Initialize registry
# If you're pushing to a Pushgateway, use a dedicated registry.
# For simple textfile collection, the default registry is fine.
# registry = CollectorRegistry()

# Define metrics
REQUEST_COUNT = Counter('http_requests_total', 'Total HTTP Requests', ['method', 'endpoint', 'status_code'])
REQUEST_LATENCY = Histogram('http_request_duration_seconds', 'HTTP Request Latency', ['method', 'endpoint'])
ACTIVE_USERS = Gauge('app_active_users', 'Number of currently active users')

app = Flask(__name__)

# Middleware to track requests
@app.before_request
def before_request():
    request.start_time = time.time()
    # Increment active users (simplified example)
    ACTIVE_USERS.inc()

@app.after_request
def after_request(response):
    request_latency = time.time() - request.start_time
    REQUEST_LATENCY.labels(method=request.method, endpoint=request.path).observe(request_latency)
    REQUEST_COUNT.labels(method=request.method, endpoint=request.path, status_code=response.status_code).inc()
    ACTIVE_USERS.dec()
    return response

@app.route('/')
def hello_world():
    # Simulate some work
    time.sleep(0.1)
    return 'Hello, World!'

@app.route('/error')
def trigger_error():
    # Simulate an error
    time.sleep(0.05)
    return Response("Internal Server Error", status=500)

# Example of pushing metrics to Pushgateway (optional, for distributed systems)
# PUSHGATEWAY_URL = os.environ.get('PUSHGATEWAY_URL', 'http://localhost:9091')
# JOB_NAME = 'my_python_app'

# def push_metrics():
#     try:
#         push_to_gateway(PUSHGATEWAY_URL, job=JOB_NAME, registry=registry)
#         print("Metrics pushed successfully.")
#     except Exception as e:
#         print(f"Failed to push metrics: {e}")

# If using textfile collector for Prometheus
METRICS_DIR = '/var/lib/prometheus/node-exporter/' # Ensure this directory exists and Prometheus user has read access
if not os.path.exists(METRICS_DIR):
    os.makedirs(METRICS_DIR)

def write_metrics():
    try:
        write_to_textfile(os.path.join(METRICS_DIR, 'my_python_app.prom'), CollectorRegistry(auto_describe=True))
        print("Metrics written to textfile.")
    except Exception as e:
        print(f"Failed to write metrics to textfile: {e}")

if __name__ == '__main__':
    # For development, run Flask directly
    # app.run(host='0.0.0.0', port=5000)

    # For production, use a WSGI server like Gunicorn
    # Example: gunicorn --bind 0.0.0.0:5000 --worker-class eventlet -w 1 your_module:app
    # And run metrics collection in a separate thread or process
    from threading import Thread
    # If using Pushgateway:
    # metrics_thread = Thread(target=lambda: (time.sleep(30), push_metrics(), metrics_thread.start()))
    # metrics_thread.daemon = True
    # metrics_thread.start()

    # If using textfile collector:
    metrics_writer_thread = Thread(target=lambda: (time.sleep(30), write_metrics(), metrics_writer_thread.start()))
    metrics_writer_thread.daemon = True
    metrics_writer_thread.start()

    # Run the Flask app (e.g., with Gunicorn in production)
    # For demonstration, we'll just start the app here.
    app.run(host='0.0.0.0', port=5000)

In this example:

  • http_requests_total: A counter to track the number of requests, labeled by HTTP method, endpoint, and status code.
  • http_request_duration_seconds: A histogram to measure the distribution of request latencies.
  • app_active_users: A gauge to track a dynamic value, like the number of concurrent users.

For Prometheus to scrape these metrics, you need to configure it. If using the textfile collector, ensure Prometheus is configured to read from the specified directory (e.g., /var/lib/prometheus/node-exporter/). If using Pushgateway, Prometheus scrapes the Pushgateway itself.

Monitoring Redis Clusters with Redis Exporter

Redis clusters require specialized monitoring to understand node health, replication status, memory usage, and command performance. redis_exporter is an excellent choice for this.

Download and install redis_exporter. The easiest way is often to download a pre-compiled binary from its GitHub releases page.

# Example for Linux AMD64
wget https://github.com/oliver006/redis_exporter/releases/download/v1.45.0/redis_exporter-v1.45.0.linux-amd64.tar.gz
tar xvfz redis_exporter-v1.45.0.linux-amd64.tar.gz
sudo mv redis_exporter-v1.45.0.linux-amd64/redis_exporter /usr/local/bin/
rm -rf redis_exporter-v1.45.0.linux-amd64*

Create a systemd service file for redis_exporter:

[Unit]
Description=Redis Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=redis # Or a dedicated monitoring user
Group=redis
ExecStart=/usr/local/bin/redis_exporter \
  --redis.addr=redis://localhost:6379 \ # Adjust if your Redis is not on localhost or default port
  --redis.password=your_redis_password \ # If Redis requires a password
  --cluster.enabled=true \ # Crucial for cluster monitoring
  --check-keyspace=true \ # Enable keyspace metrics
  --check-single-keys=mykey \ # Optional: monitor specific keys
  --check-slave-nodes=true \ # Monitor slave health
  --check-master-node=true # Monitor master health

Restart=always

[Install]
WantedBy=multi-user.target

Replace your_redis_password with your actual Redis password. If you have multiple Redis instances or a complex cluster setup, you might need to adjust the --redis.addr and potentially run multiple instances of redis_exporter or configure it to connect to a sentinel.

Start and enable the service:

sudo systemctl daemon-reload
sudo systemctl start redis_exporter
sudo systemctl enable redis_exporter
sudo systemctl status redis_exporter

redis_exporter typically runs on port 9121. Ensure this port is accessible from your Prometheus server.

Configuring Prometheus for Scraping

Your Prometheus configuration (prometheus.yml) needs to include scrape jobs for your Python app and Redis exporter.

global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.

scrape_configs:
  # Job for node_exporter (system metrics)
  - job_name: 'node_exporter'
    static_configs:
      - targets: ['your_server_ip:9100'] # Replace with your server's IP or hostname

  # Job for Python application metrics (if using textfile collector)
  # This assumes Prometheus is running on the same server or has access to the metrics directory.
  # If using Pushgateway, this job would target the Pushgateway.
  - job_name: 'python_app_textfile'
    static_configs:
      - targets: ['your_server_ip:9100'] # Node exporter often serves textfile collector metrics too
    # If your textfile collector is configured to write to a specific path that Prometheus scrapes directly:
    # scrape_configs:
    #   - job_name: 'python_app_metrics'
    #     static_configs:
    #       - targets: ['your_server_ip:9100'] # Or a dedicated HTTP endpoint if you expose it
    #     metric_relabel_configs:
    #       - source_labels: [__address__]
    #         target_label: __address__
    #         regex: '(.*):9100'
    #         replacement: '$1:9100/metrics/node' # Example if node_exporter serves textfiles

  # Job for Redis exporter
  - job_name: 'redis_exporter'
    static_configs:
      - targets: ['your_server_ip:9121'] # Replace with your server's IP or hostname

  # Example for Python app if exposing metrics via HTTP endpoint (e.g., using Flask-Prometheus-Metrics)
  # - job_name: 'python_app_http'
  #   static_configs:
  #     - targets: ['your_python_app_ip:5000'] # Assuming Flask app runs on port 5000 and exposes /metrics

  # Example for Redis Pushgateway (if you chose that over textfile/direct scrape)
  # - job_name: 'redis_pushgateway'
  #   static_configs:
  #     - targets: ['your_pushgateway_ip:9091']

Remember to replace your_server_ip with the actual IP address or hostname of your OVH server where these exporters are running. If Prometheus is on a different machine, ensure network connectivity and firewall rules are in place.

Alerting Strategies with Alertmanager

Effective monitoring is incomplete without alerting. Prometheus integrates with Alertmanager to handle alerts. Here are some critical alerts for your Python app and Redis cluster:

Python Application Alerts

groups:
- name: python_app_alerts
  rules:
  - alert: HighRequestLatency
    expr: histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, method, endpoint)) > 2
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "High request latency detected for {{ $labels.endpoint }}"
      description: "95th percentile latency for {{ $labels.method }} {{ $labels.endpoint }} is {{ $value }}s, exceeding the threshold."

  - alert: HighErrorRate
    expr: sum(rate(http_requests_total{status_code=~"5..|4.."}[5m])) by (method, endpoint) / sum(rate(http_requests_total[5m])) by (method, endpoint) * 100 > 5
    for: 10m
    labels:
      severity: critical
    annotations:
      summary: "High HTTP error rate for {{ $labels.endpoint }}"
      description: "Error rate for {{ $labels.method }} {{ $labels.endpoint }} is {{ $value }}%, exceeding the threshold."

  - alert: AppInstanceDown
    expr: up{job="python_app_http"} == 0 # Adjust job name if using different scraping method
    for: 1m
    labels:
      severity: critical
    annotations:
      summary: "Python application instance is down"
      description: "The Python application instance at {{ $labels.instance }} is unreachable."

  - alert: LowActiveUsers
    expr: app_active_users < 10
    for: 15m
    labels:
      severity: info
    annotations:
      summary: "Low number of active users"
      description: "The number of active users has dropped below 10 for the last 15 minutes."

Redis Cluster Alerts

groups:
- name: redis_cluster_alerts
  rules:
  - alert: RedisMasterDown
    expr: redis_master_down > 0
    for: 2m
    labels:
      severity: critical
    annotations:
      summary: "Redis Master Node Down"
      description: "A Redis master node is down. Check cluster status."

  - alert: RedisSlaveNotConnected
    expr: redis_connected_slaves == 0
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "Redis Slave Node Not Connected"
      description: "A Redis master node has no connected slaves."

  - alert: HighRedisMemoryUsage
    expr: redis_memory_used_bytes / redis_total_system_memory_bytes * 100 > 85
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: "High Redis Memory Usage"
      description: "Redis instance {{ $labels.instance }} is using {{ $value }}% of its memory."

  - alert: RedisKeyspaceHitRateLow
    expr: 1 - (rate(redis_keyspace_misses[5m]) / rate(redis_keyspace_hits[5m])) * 100 < 90
    for: 10m
    labels:
      severity: info
    annotations:
      summary: "Low Redis Keyspace Hit Rate"
      description: "Redis instance {{ $labels.instance }} has a low keyspace hit rate ({{ $value }}%)."

Configure Alertmanager (alertmanager.yml) to receive these alerts and route them to your desired notification channels (email, Slack, PagerDuty, etc.). Ensure Prometheus is configured to point to your Alertmanager instance.

OVH Specific Considerations

When deploying on OVH, keep these points in mind:

  • Network Security Groups/Firewalls: Ensure that ports used by Prometheus, node_exporter (9100), redis_exporter (9121), and your Python application are open to your Prometheus server’s IP address. Restrict access as much as possible.
  • Instance Sizing: Monitor resource utilization (CPU, RAM, Disk I/O) of your OVH instances. Prometheus and its exporters themselves consume resources. Ensure your instances are adequately sized for your application load and monitoring infrastructure.
  • Persistent Storage for Prometheus: If running Prometheus on OVH, configure persistent storage for its time-series data. This is crucial to avoid data loss upon instance restarts or upgrades.
  • Managed Services: OVH offers managed databases and Redis services. If you are using these, the monitoring setup might differ slightly, particularly for Redis, as you may not have direct access to install exporters on the managed Redis instances. In such cases, you might need to rely on OVH’s provided metrics or use client-side monitoring within your application.
  • Logging: While this post focuses on metrics, robust logging is equally important. Centralize logs from your Python application and Redis instances using tools like ELK stack, Graylog, or Loki.

Advanced Techniques and Next Steps

For more advanced setups:

  • Distributed Tracing: Integrate distributed tracing (e.g., Jaeger, Zipkin) into your Python application to track requests across multiple services and identify bottlenecks.
  • Application Performance Monitoring (APM): Consider dedicated APM tools (e.g., Datadog, New Relic, Sentry) for deeper insights into application performance, error tracking, and profiling.
  • Kubernetes/Containerization: If your Python app and Redis are containerized (e.g., on OVH’s Managed Kubernetes Service), leverage Kubernetes-native monitoring solutions like Prometheus Operator, Grafana Operator, and specific exporters for containerized environments.
  • Synthetic Monitoring: Implement synthetic checks to proactively test critical user flows and API endpoints, simulating user behavior from different geographical locations.

By implementing these monitoring best practices, you can significantly improve the reliability and performance of your Python applications and Redis clusters running on OVH infrastructure.