The Ultimate DevOps Playbook: Tuning Nginx, Gunicorn/FPM, and PostgreSQL on OVH for Shopify

Nginx as a High-Performance Frontend for Gunicorn/PHP-FPM

When deploying applications that leverage Python (via Gunicorn) or PHP (via PHP-FPM) on OVH infrastructure, Nginx serves as the de facto standard for a robust and performant frontend. Its event-driven architecture excels at handling a high volume of concurrent connections, making it ideal for offloading static file serving, SSL termination, and request routing. The key to unlocking Nginx’s full potential lies in meticulous configuration tuning.

Optimizing Nginx Worker Processes and Connections

The `worker_processes` directive dictates how many worker processes Nginx will spawn. A common best practice is to set this to the number of CPU cores available on the server. This allows Nginx to fully utilize the available processing power for handling requests. The `worker_connections` directive, on the other hand, defines the maximum number of simultaneous connections that each worker process can handle. This value should be set high enough to accommodate peak traffic, but not so high that it exhausts system resources. A good starting point is often 1024 or higher, depending on the server’s RAM and the expected load.

Consider the following Nginx configuration snippet for tuning:

worker_processes auto; # Or set to the number of CPU cores
events {
    worker_connections 4096; # Adjust based on system resources and expected load
    multi_accept on;
    use epoll; # For Linux systems
}

http {
    # ... other http configurations ...

    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 65;
    types_hash_max_size 2048;
    server_tokens off; # Important for security

    # ... gzip compression and caching directives ...
}

The `auto` setting for `worker_processes` is generally preferred as Nginx will automatically detect and use the number of available CPU cores. `multi_accept on` allows workers to accept multiple new connections at once, and `epoll` is the highly efficient I/O event notification mechanism on Linux. `sendfile on` and `tcp_nopush on` optimize the transfer of files, while `keepalive_timeout` reduces the overhead of establishing new connections.

Configuring Gunicorn for Optimal Performance

Gunicorn (Green Unicorn) is a Python WSGI HTTP Server. Its performance is heavily influenced by the number of worker processes and the worker class used. For CPU-bound applications, a synchronous worker class is often sufficient. However, for I/O-bound applications, asynchronous workers like `gevent` or `eventlet` can significantly improve concurrency and throughput by allowing workers to handle multiple requests concurrently without blocking.

The number of worker processes is typically set to `(2 * number_of_cores) + 1`. This formula aims to keep CPU cores busy while accounting for potential I/O waits. The `threads` parameter is relevant for synchronous workers and can be tuned based on the application’s I/O characteristics.

Here’s an example of how to launch Gunicorn with optimized settings:

gunicorn --workers 5 \
         --worker-class sync \
         --bind 0.0.0.0:8000 \
         --timeout 120 \
         your_app.wsgi:application

For asynchronous workers, you might use:

gunicorn --workers 3 \
         --worker-class gevent \
         --bind 0.0.0.0:8000 \
         --timeout 120 \
         your_app.wsgi:application

The `–timeout` parameter is crucial for preventing long-running requests from holding up worker processes indefinitely. Adjust this value based on your application’s typical request durations.

Tuning PHP-FPM for Scalability

PHP-FPM (FastCGI Process Manager) is the standard for running PHP applications. Its performance hinges on the process management settings. The `pm` (process manager) directive can be set to `static`, `dynamic`, or `ondemand`. `dynamic` is often a good balance, allowing FPM to manage the number of child processes based on demand, while `static` pre-forks a fixed number of processes, which can be beneficial if you have predictable traffic and ample resources.

Key directives to tune within `php-fpm.conf` (or its included pool configuration files, typically in `pool.d/`):

• pm.max_children: The maximum number of child processes to be created when pm is set to dynamic or static. This should be set considering available RAM.
• pm.start_servers: The number of child processes to be created when the FPM master process starts.
• pm.min_spare_servers: The desired minimum number of idle supervisor processes.
• pm.max_spare_servers: The desired maximum number of idle supervisor processes.
• pm.max_requests: The number of requests each child process should execute before respawning. Setting this to a moderate value (e.g., 500) helps prevent memory leaks from accumulating over time.

A sample `www.conf` (or similar pool configuration) for PHP-FPM:

[www]
user = www-data
group = www-data
listen = /run/php/php7.4-fpm.sock # Adjust version as needed
listen.owner = www-data
listen.group = www-data
listen.mode = 0660

pm = dynamic
pm.max_children = 50
pm.start_servers = 5
pm.min_spare_servers = 2
pm.max_spare_servers = 10
pm.max_requests = 500

request_terminate_timeout = 120
slowlog = /var/log/php/php-fpm/slow.log

The `request_terminate_timeout` directive is crucial for preventing runaway scripts from consuming resources. Adjust `pm.max_children` carefully based on your server’s RAM. A common rule of thumb is to leave enough RAM for the OS and other services, and then calculate the maximum number of PHP processes that can run concurrently without swapping.

PostgreSQL Performance Tuning on OVH

PostgreSQL’s performance is highly dependent on its configuration parameters, particularly those related to memory management and query execution. On OVH, where you might be managing your own PostgreSQL instances or using managed services, tuning these parameters is critical for application responsiveness.

Key PostgreSQL Configuration Parameters

The primary configuration file is `postgresql.conf`. Here are some of the most impactful parameters:

• shared_buffers: This is arguably the most important parameter. It defines the amount of memory dedicated to PostgreSQL for caching data and indexes. A common recommendation is 25% of the total system RAM, but this can be tuned higher for dedicated database servers.
• work_mem: The maximum amount of memory that can be used for internal sort operations and hash tables before spilling to disk. This is per operation, so it’s important not to set it too high if you have many concurrent queries.
• maintenance_work_mem: The maximum memory to be used for maintenance operations like VACUUM, CREATE INDEX, and ALTER TABLE ADD FOREIGN KEY. Larger values can significantly speed up these operations.
• effective_cache_size: This parameter informs the query planner about how much memory is available for disk caching by the operating system and PostgreSQL’s shared buffers. Setting this to 50-75% of total RAM is a good starting point.
• wal_buffers: The amount of memory used for WAL (Write-Ahead Logging) data before writing to disk. A value of -1 (auto) is often sufficient, but tuning can be beneficial.
• max_connections: The maximum number of concurrent connections allowed. This should be set based on application needs and server resources.

Example tuning for a server with 32GB RAM:

# postgresql.conf
shared_buffers = 8GB          # ~25% of 32GB RAM
work_mem = 64MB               # Adjust based on query complexity and concurrency
maintenance_work_mem = 1GB    # For faster maintenance tasks
effective_cache_size = 24GB   # ~75% of 32GB RAM
wal_buffers = 16MB            # Default is often fine, but can be increased
max_connections = 200         # Adjust based on application needs
random_page_cost = 1.1        # Lowering this can favor sequential scans if disk I/O is fast
seq_page_cost = 1.0           # Default

After modifying `postgresql.conf`, a PostgreSQL service restart is required for the changes to take effect.

Query Optimization and Indexing

Even with optimal server configuration, poorly written queries can cripple performance. Regularly analyze slow queries using `pg_stat_statements` or by enabling the `log_min_duration_statement` parameter in `postgresql.conf`.

-- Example of enabling logging for slow queries
log_min_duration_statement = '5s' -- Log queries taking longer than 5 seconds

-- Example of checking for missing indexes
EXPLAIN ANALYZE SELECT * FROM users WHERE email = '[email protected]';
-- Look for sequential scans on large tables where an index would be beneficial.

-- Creating an index
CREATE INDEX idx_users_email ON users (email);

Understanding your application’s data access patterns is crucial for effective indexing. Avoid over-indexing, as indexes add overhead to write operations and consume disk space.

Integrating Nginx, Gunicorn/PHP-FPM, and PostgreSQL on OVH

The typical deployment architecture on OVH involves Nginx acting as the reverse proxy, forwarding requests to either Gunicorn (for Python apps) or PHP-FPM (for PHP apps). PostgreSQL serves as the persistent data store.

Nginx Configuration for Backend Proxies

Nginx needs to be configured to proxy requests to your application servers. This involves setting up `upstream` blocks for Gunicorn/PHP-FPM pools and using `proxy_pass` directives.

# For Gunicorn
upstream gunicorn_app {
    server 127.0.0.1:8000; # Address where Gunicorn is listening
    # Add more servers for load balancing if needed
}

# For PHP-FPM
upstream php_fpm_pool {
    server unix:/run/php/php7.4-fpm.sock; # Or IP:Port if not using sockets
}

server {
    listen 80;
    server_name yourdomain.com;

    location / {
        proxy_pass http://gunicorn_app; # Or http://php_fpm_pool;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_read_timeout 300s; # Increase for long-running requests
        proxy_connect_timeout 75s;
    }

    # Serve static files directly from Nginx for better performance
    location /static/ {
        alias /path/to/your/static/files/;
        expires 30d;
        access_log off;
    }

    # Handle PHP files if using PHP-FPM
    location ~ \.php$ {
        include snippets/fastcgi-php.conf;
        fastcgi_pass unix:/run/php/php7.4-fpm.sock; # Match your PHP-FPM socket
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        include fastcgi_params;
    }
}

The `proxy_read_timeout` and `proxy_connect_timeout` directives are crucial for handling potentially long-running requests from your application. Adjust these based on your application’s behavior. Serving static files directly via Nginx is a significant performance win, offloading this task from your application servers.

Monitoring and Iterative Tuning

Performance tuning is not a one-time event. Continuous monitoring is essential. Utilize tools like:

• Nginx: `stub_status` module for connection metrics, Nginx Amplify, or Prometheus exporters.
• Gunicorn/PHP-FPM: Application performance monitoring (APM) tools like New Relic, Datadog, or custom logging.
• PostgreSQL: `pg_stat_activity`, `pg_stat_statements`, Prometheus exporters (e.g., `postgres_exporter`), and OVH’s monitoring dashboards.
• System-level: top, htop, iostat, vmstat, and Prometheus Node Exporter.

By systematically monitoring these components and iteratively adjusting the configurations outlined above, you can build a highly performant and scalable infrastructure on OVH for your Shopify-related applications.