The Ultimate DevOps Playbook: Tuning Nginx, Gunicorn/FPM, and MySQL on OVH for Shopify

Nginx as a High-Performance Frontend Proxy

For a Shopify backend application, Nginx serves as the primary entry point, handling SSL termination, static file serving, and reverse proxying to your application servers (Gunicorn for Python/Django/Flask, or PHP-FPM for PHP applications). Optimizing Nginx is crucial for low latency and high throughput.

We’ll focus on tuning key directives within the http and server blocks. The goal is to maximize concurrent connections while minimizing resource consumption.

Core Nginx Configuration Tuning

Locate your main Nginx configuration file, typically /etc/nginx/nginx.conf, and the relevant site configuration, often in /etc/nginx/sites-available/your_shopify_app. We’ll adjust parameters that directly impact connection handling and buffering.

nginx.conf Adjustments

Within the http block, these directives are paramount:

# /etc/nginx/nginx.conf

user www-data;
worker_processes auto; # Set to the number of CPU cores for optimal performance
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;

events {
    worker_connections 1024; # Adjust based on expected concurrent users and server RAM
    multi_accept on;
}

http {
    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 65;
    types_hash_max_size 2048;
    server_tokens off; # Crucial for security, hides Nginx version

    # Gzip compression for text-based assets
    gzip on;
    gzip_vary on;
    gzip_proxied any;
    gzip_comp_level 6;
    gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

    # Buffering and timeouts for upstream connections
    proxy_connect_timeout 60s;
    proxy_send_timeout 60s;
    proxy_read_timeout 60s;
    proxy_buffer_size 16k;
    proxy_buffers 4 32k;
    proxy_busy_buffers_size 64k;
    proxy_temp_file_write_size 64k;

    # Client request body limits (adjust as needed for Shopify webhook payloads, etc.)
    client_max_body_size 50M;
    client_body_buffer_size 128k;

    include /etc/nginx/mime.types;
    default_type application/octet-stream;

    # Logging settings (consider a more optimized format for production)
    access_log /var/log/nginx/access.log;
    error_log /var/log/nginx/error.log;

    # Include site-specific configurations
    include /etc/nginx/sites-enabled/*;
}

Explanation:

• worker_processes auto;: Dynamically scales Nginx worker processes to match your CPU cores.
• worker_connections: The maximum number of simultaneous connections a single worker process can handle. This is a critical tuning parameter. A common starting point is 1024, but it can be increased significantly if your server has ample RAM and you expect high concurrency. Ensure your OS limits (ulimit -n) are also set high enough.
• sendfile on;: Allows Nginx to send files directly from disk to the client without passing through user space, significantly improving performance for static file delivery.
• tcp_nopush on;: Instructs Nginx to send header information and the first part of a file in one packet, reducing the number of TCP packets.
• tcp_nodelay on;: Disables the Nagle algorithm, which can reduce latency by sending small packets immediately.
• keepalive_timeout 65;: Sets the timeout for persistent connections. A moderate value balances resource usage with connection efficiency.
• gzip_* directives: Enable and configure Gzip compression for text-based responses, reducing bandwidth and improving load times.
• proxy_* directives: These control how Nginx interacts with your upstream application servers. Tuning buffer sizes and timeouts prevents Nginx from being blocked by slow application responses and ensures efficient data transfer.
• client_max_body_size: Essential for handling large uploads or webhook payloads from Shopify.

Site Configuration (sites-enabled/your_shopify_app)

This block defines how Nginx proxies requests to your application server. Pay close attention to the location / block.

# /etc/nginx/sites-available/your_shopify_app

server {
    listen 80;
    listen [::]:80;
    server_name your_domain.com www.your_domain.com;

    # Redirect HTTP to HTTPS
    location / {
        return 301 https://$host$request_uri;
    }
}

server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    server_name your_domain.com www.your_domain.com;

    # SSL Configuration (ensure you have valid certificates)
    ssl_certificate /etc/letsencrypt/live/your_domain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/your_domain.com/privkey.pem;
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_prefer_server_ciphers on;
    ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;
    ssl_session_cache shared:SSL:10m; # Adjust size based on RAM
    ssl_session_timeout 10m;
    ssl_session_tickets off; # Consider security implications

    # HSTS Header (optional but recommended for security)
    # add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload" always;

    # Static file serving (configure paths to your static assets)
    location ~ ^/(assets|images|javascripts|stylesheets)/ {
        root /var/www/your_shopify_app/public; # Adjust this path
        expires 30d;
        add_header Cache-Control "public";
        access_log off;
    }

    # Proxy to application server (Gunicorn or PHP-FPM)
    location / {
        proxy_pass http://127.0.0.1:8000; # For Gunicorn, adjust port if needed
        # proxy_pass http://unix:/var/run/php/php7.4-fpm.sock; # For PHP-FPM, adjust socket path

        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_redirect off;
    }

    # Deny access to hidden files
    location ~ /\. {
        deny all;
    }

    # Error pages
    error_page 500 502 503 504 /50x.html;
    location = /50x.html {
        root /usr/share/nginx/html; # Default Nginx error page location
    }
}

Explanation:

• listen 443 ssl http2;: Enables SSL/TLS and HTTP/2 for faster multiplexed connections.
• ssl_certificate & ssl_certificate_key: Paths to your SSL certificates.
• ssl_protocols & ssl_ciphers: Modern and secure SSL/TLS configurations.
• ssl_session_cache: Caches SSL session parameters to speed up subsequent connections from the same client.
• location ~ ^/(assets|images|javascripts|stylesheets)/: This block is crucial for performance. It tells Nginx to serve static assets directly from disk, bypassing your application server entirely. Ensure the root directive points to your application’s static file directory. Setting long expires headers and Cache-Control allows browsers and intermediate caches to store these assets, reducing load on your server.
• location /: This is the main proxy block.
• proxy_pass: The upstream server address. For Gunicorn, it’s typically a TCP socket (e.g., http://127.0.0.1:8000). For PHP-FPM, it’s usually a Unix domain socket (e.g., http://unix:/var/run/php/php7.4-fpm.sock).
• proxy_set_header directives: These pass essential client information (like the original IP address and protocol) to your application server, which is vital for logging and security.

Applying Nginx Changes

After modifying configuration files, always test your Nginx configuration for syntax errors:

sudo nginx -t

If the test passes, reload Nginx to apply the changes gracefully:

sudo systemctl reload nginx
# or
sudo service nginx reload

Gunicorn Tuning for Python Applications

Gunicorn (Green Unicorn) is a popular WSGI HTTP Server for Python. Its performance is heavily influenced by the number of worker processes and threads it spawns.

Worker Processes and Threads

The optimal number of workers depends on your application’s nature (CPU-bound vs. I/O-bound) and your server’s resources. A common recommendation is to use the “sync” worker type and set the number of workers to (2 * number_of_cores) + 1. For I/O-bound applications, you might consider the “gevent” or “event” worker types, which use asynchronous I/O and can handle more concurrent connections with fewer processes.

Starting Gunicorn with Optimized Settings

You can start Gunicorn directly from the command line or, more commonly, manage it via a systemd service file.

Command Line Example (for testing):

# Assuming your Django app is in a directory with wsgi.py
# For a Django app:
gunicorn --workers 4 --threads 2 --bind 127.0.0.1:8000 your_project.wsgi:application

# For a Flask app (assuming app instance is named 'app' in 'main.py'):
gunicorn --workers 4 --threads 2 --bind 127.0.0.1:8000 main:app

Explanation:

• --workers 4: Sets the number of worker processes. If your server has 2 cores, (2 * 2) + 1 = 5 would be a good starting point. Adjust based on monitoring.
• --threads 2: For the ‘sync’ worker type, this specifies the number of threads each worker process will use. This helps handle multiple requests concurrently within a single worker process, especially for I/O-bound tasks.
• --bind 127.0.0.1:8000: The address and port Gunicorn listens on. Nginx will proxy to this.

Systemd Service File for Production

Create a service file, e.g., /etc/systemd/system/gunicorn.service:

[Unit]
Description=Gunicorn instance to serve your_shopify_app
After=network.target

[Service]
User=your_app_user
Group=www-data
WorkingDirectory=/path/to/your_shopify_app
Environment="PATH=/path/to/your_venv/bin"
ExecStart=/path/to/your_venv/bin/gunicorn --workers 4 --threads 2 --bind unix:/run/gunicorn.sock your_project.wsgi:application
# Or for TCP socket:
# ExecStart=/path/to/your_venv/bin/gunicorn --workers 4 --threads 2 --bind 127.0.0.1:8000 your_project.wsgi:application

Restart=always
RestartSec=5s

[Install]
WantedBy=multi-user.target

Explanation:

• User & Group: Run Gunicorn under a non-root user for security.
• WorkingDirectory: The directory where your application code resides.
• Environment="PATH=...": Ensures Gunicorn can find your virtual environment’s Python executable.
• ExecStart: The command to start Gunicorn. Using a Unix domain socket (unix:/run/gunicorn.sock) is generally faster than a TCP socket for local communication between Nginx and Gunicorn. Ensure Nginx has permissions to access this socket.
• Restart=always: Ensures Gunicorn restarts automatically if it crashes.

After creating the service file, enable and start it:

sudo systemctl daemon-reload
sudo systemctl enable gunicorn.service
sudo systemctl start gunicorn.service
sudo systemctl status gunicorn.service

Gunicorn Worker Types

If your application is heavily I/O bound (e.g., making many external API calls, database queries), consider asynchronous workers:

# Using gevent workers (install gevent: pip install gevent)
gunicorn --worker-class gevent --workers 2 --bind 127.0.0.1:8000 your_project.wsgi:application

# Using event workers (built-in, but often less performant than gevent)
gunicorn --worker-class event --workers 2 --bind 127.0.0.1:8000 your_project.wsgi:application

With asynchronous workers, the concept of --threads changes or is not applicable. The number of --workers can often be lower, as each worker can handle many concurrent connections asynchronously. Monitor your application’s performance closely when switching worker types.

PHP-FPM Tuning for PHP Applications

PHP-FPM (FastCGI Process Manager) is the standard way to run PHP applications with Nginx. Its performance is governed by how it manages its pool of PHP worker processes.

PHP-FPM Pool Configuration

The primary configuration file for PHP-FPM pools is typically located at /etc/php/X.Y/fpm/pool.d/www.conf (replace X.Y with your PHP version, e.g., 7.4 or 8.1).

; /etc/php/X.Y/fpm/pool.d/www.conf

[www]
user = www-data
group = www-data
listen = /var/run/php/phpX.Y-fpm.sock ; Or use a TCP socket like 127.0.0.1:9000
listen.owner = www-data
listen.group = www-data
listen.mode = 0660

pm = dynamic
pm.max_children = 50       ; Max number of child processes at any time
pm.start_servers = 5       ; Number of servers to start upon system startup
pm.min_spare_servers = 5   ; Min number of servers to maintain in idle state
pm.max_spare_servers = 10  ; Max number of servers to maintain in idle state
pm.process_idle_timeout = 10s ; The timeout after which a child process will be killed
pm.max_requests = 500      ; Max number of requests each child process will serve before respawning

request_terminate_timeout = 60s ; Timeout for script execution
request_slowlog_timeout = 10s   ; Timeout for slow scripts, logs them

; Other useful settings
; rlimit_files = 1024
; rlimit_nofile = 65536
; chdir = /

Explanation:

• listen: The socket PHP-FPM listens on. Match this in your Nginx configuration. Unix sockets are generally preferred for local communication.
• pm: Process Manager control.
• dynamic: PHP-FPM will dynamically adjust the number of child processes based on load, within the defined limits.
• static: PHP-FPM will maintain a fixed number of child processes (defined by pm.max_children). This can be more predictable but less resource-efficient if traffic fluctuates wildly.
• pm.max_children: This is the most critical setting. It defines the absolute maximum number of PHP processes that can run concurrently. Setting this too high can exhaust server RAM and cause instability. Setting it too low will lead to request queues and slow responses. A common starting point is (number_of_cores * 5), but this needs to be tuned based on your application’s memory footprint and server RAM.
• pm.start_servers, pm.min_spare_servers, pm.max_spare_servers: These control the dynamic scaling behavior. They define how many processes are started initially and how many are kept ready to handle spikes in traffic.
• pm.process_idle_timeout: How long an idle process waits before being terminated to save resources.
• pm.max_requests: Respawning processes after a certain number of requests can help prevent memory leaks from accumulating over time.
• request_terminate_timeout: Prevents a single script from running indefinitely and hogging a PHP process.
• request_slowlog_timeout: Useful for identifying performance bottlenecks in your PHP code.

Applying PHP-FPM Changes

After modifying the PHP-FPM pool configuration, you need to restart the PHP-FPM service:

sudo systemctl restart phpX.Y-fpm.service
# or
sudo service phpX.Y-fpm restart

Ensure your Nginx configuration’s proxy_pass directive correctly points to the PHP-FPM socket or TCP address defined in www.conf.

MySQL/MariaDB Tuning for Shopify Data

Shopify applications often rely heavily on a robust database. Optimizing MySQL/MariaDB is essential for fast data retrieval and writes. We’ll focus on key parameters within my.cnf or mysqld.cnf.

Key MySQL Configuration Parameters

Locate your MySQL configuration file, typically /etc/mysql/my.cnf, /etc/mysql/mysql.conf.d/mysqld.cnf, or similar. The following parameters are crucial:

[mysqld]
# General Settings
user                    = mysql
pid-file                = /var/run/mysqld/mysqld.pid
socket                  = /var/run/mysqld/mysqld.sock
port                    = 3306
basedir                 = /usr
datadir                 = /var/lib/mysql
tmpdir                  = /tmp
lc-messages-dir         = /usr/share/mysql
skip-external-locking

# Performance Tuning
key_buffer_size         = 128M  ; For MyISAM tables (less relevant for InnoDB)
innodb_buffer_pool_size = 2G    ; CRITICAL for InnoDB. Set to 50-70% of available RAM on a dedicated DB server.
innodb_log_file_size    = 512M  ; Larger log files reduce I/O frequency. Requires careful restart procedure.
innodb_log_buffer_size  = 16M   ; Buffer for transaction logs.
innodb_flush_log_at_trx_commit = 1 ; ACID compliance (1=safest, 0=fastest, 2=balance)
innodb_flush_method     = O_DIRECT ; Recommended for Linux with hardware RAID/SSDs
innodb_file_per_table   = 1       ; Recommended for better space management and performance
innodb_io_capacity      = 2000    ; Adjust based on your disk I/O capabilities (e.g., SSDs)
innodb_io_capacity_max  = 4000    ; Max I/O capacity for InnoDB background tasks

max_connections         = 200   ; Adjust based on expected concurrent connections from your app servers
thread_cache_size       = 16    ; Cache threads for reuse
table_open_cache        = 2048  ; Cache open table file descriptors
table_definition_cache  = 1024  ; Cache table definitions
sort_buffer_size        = 2M    ; For sorting operations
read_buffer_size        = 1M    ; For sequential scans
read_rnd_buffer_size    = 2M    ; For reading rows after sorting
join_buffer_size        = 2M    ; For joins without indexes

# Query Cache (Deprecated in MySQL 5.7, removed in 8.0. Use application-level caching instead.)
# query_cache_type        = 0
# query_cache_size        = 0

# Logging
log_error               = /var/log/mysql/error.log
slow_query_log          = 1
slow_query_log_file     = /var/log/mysql/mysql-slow.log
long_query_time         = 2     ; Log queries taking longer than 2 seconds
log_queries_not_using_indexes = 1 ; Log queries that don't use indexes

# Character Set
character-set-server    = utf8mb4
collation-server        = utf8mb4_unicode_ci

# Replication (if applicable)
# server-id = 1
# log_bin = /var/log/mysql/mysql-bin.log
# binlog_format = ROW

Explanation:

• innodb_buffer_pool_size: The most critical setting for InnoDB. It’s the memory area where InnoDB caches table data and indexes. Setting this to 50-70% of your server’s RAM is a common recommendation for dedicated database servers.
• innodb_log_file_size: Larger log files reduce the frequency of flushing dirty pages to disk, improving write performance. Changing this requires a specific shutdown and startup procedure.
• innodb_flush_log_at_trx_commit: Controls the trade-off between ACID compliance and performance. 1 is safest (log flushed on every commit), 2 is a good balance (log written on commit, flushed to disk ~once per second), 0 is fastest but risks data loss on crash. For Shopify, 1 or 2 is usually appropriate.
• innodb_flush_method = O_DIRECT: Bypasses the OS file system cache, which can prevent double buffering and improve performance, especially on systems with fast storage.
• innodb_io_capacity: Helps InnoDB understand your storage subsystem’s capabilities for background flushing. Tune this based on your disk speed (e.g., higher for NVMe SSDs).
• max_connections: The maximum number of simultaneous client connections. Ensure this is high enough for your application servers but not so high that it exhausts server memory.
• thread_cache_size: Caching threads reduces the overhead of creating new threads for each connection.
• table_open_cache & table_definition_cache: Caching these reduces the overhead of opening and parsing table structures.
• sort_buffer_size, read_buffer_size, read_rnd_buffer_size, join_buffer_size: These are per-connection buffers. Increasing them can help complex queries, but setting them too high can quickly exhaust memory if max_connections is also high. Tune these based on observed query performance.
• slow_query_log & long_query_time: Essential for identifying and optimizing slow queries. Regularly review this log.
• character-set-server & collation-server: Use utf8mb4 for full Unicode support, which is essential for modern web applications.

Applying MySQL Changes

After modifying your MySQL configuration file (e.g., /etc/mysql/mysql.conf.d/mysqld.cnf), you must restart the MySQL service:

sudo systemctl restart mysql
# or
sudo service mysql restart

Important Note on innodb_log_file_size: If you change innodb_log_file_size, you must perform a clean shutdown, remove the existing log files (ib_logfile*) from your MySQL data directory, and then restart MySQL. Failure to do so can result in a corrupted database. Always back up your data before making such changes.

Monitoring and Iteration

Performance tuning is an iterative process. Use monitoring tools (e.g., Prometheus with Node Exporter and MySQL Exporter, Datadog, New Relic) to track key metrics:

• Nginx: Active connections, requests per second, error rates (5xx, 4xx), upstream response times.
• Gunicorn/PHP-FPM: Worker utilization, request queue lengths, response times, memory usage per worker.
• MySQL: Query throughput, slow queries, connection usage, buffer pool hit rate, I/O wait times, replication lag (if applicable).
• System: CPU utilization, memory usage, disk I/O, network traffic.

Start with conservative settings, monitor the impact, and gradually adjust parameters. Focus on the most impactful settings first (e.g., innodb_buffer_pool_size, worker_connections, pm.max_children) before fine-tuning less critical ones.