The Ultimate DevOps Playbook: Tuning Nginx, Gunicorn/FPM, and MySQL on DigitalOcean for Python
Nginx as a High-Performance Frontend Proxy
For Python web applications, Nginx serves as an indispensable frontend proxy, efficiently handling static file serving, SSL termination, and request routing to your application server (Gunicorn for WSGI or PHP-FPM for PHP). Optimizing Nginx is crucial for maximizing throughput and minimizing latency.
Nginx Worker Processes and Connections
The `worker_processes` directive dictates how many worker processes Nginx will spawn. A common best practice is to set this to the number of CPU cores available on your server. The `worker_connections` directive sets the maximum number of simultaneous connections that each worker process can handle. The total theoretical maximum connections is `worker_processes * worker_connections`.
On a DigitalOcean droplet with 4 vCPUs, a good starting point for nginx.conf (typically located at /etc/nginx/nginx.conf) would be:
user www-data;
worker_processes 4; # Set to the number of CPU cores
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;
events {
worker_connections 4096; # Adjust based on expected load and system limits
multi_accept on;
}
http {
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
# SSL Settings
ssl_protocols TLSv1.2 TLSv1.3;
ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;
ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384';
# Gzip Compression
gzip on;
gzip_vary on;
gzip_proxied any;
gzip_comp_level 6;
gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;
# Logging
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log warn;
# Include virtual host configurations
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}
After modifying nginx.conf, always test the configuration and reload Nginx:
sudo nginx -t sudo systemctl reload nginx
Optimizing Static File Serving
Nginx excels at serving static assets. Ensure your server block (within /etc/nginx/sites-available/your_app) is configured to leverage this:
server {
listen 80;
server_name your_domain.com www.your_domain.com;
# Serve static files directly
location /static/ {
alias /path/to/your/app/static/;
expires 30d; # Cache static assets for 30 days
access_log off;
add_header Cache-Control "public";
}
location /media/ {
alias /path/to/your/app/media/;
expires 30d;
access_log off;
add_header Cache-Control "public";
}
# Proxy requests to your application server
location / {
proxy_pass http://unix:/run/gunicorn.sock; # Or http://127.0.0.1:8000;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_read_timeout 300s; # Increase timeout for long-running requests
proxy_connect_timeout 75s;
}
# Optional: Handle Let's Encrypt renewals
location ~ /.well-known/acme-challenge/ {
allow all;
root /var/www/certbot; # Ensure this directory exists and is writable by certbot
}
# Optional: Redirect HTTP to HTTPS
# listen 443 ssl;
# ssl_certificate /etc/letsencrypt/live/your_domain.com/fullchain.pem;
# ssl_certificate_key /etc/letsencrypt/live/your_domain.com/privkey.pem;
# include /etc/letsencrypt/options-ssl-nginx.conf;
# ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
# return 301 https://$host$request_uri;
}
Gunicorn Tuning for WSGI Applications
Gunicorn (Green Unicorn) is a popular WSGI HTTP Server for Python. Its performance is heavily influenced by the number of worker processes and threads.
Worker Processes and Threads
The recommended number of worker processes is typically (2 * number_of_cores) + 1. For threads, Gunicorn’s default is 1. If your application is I/O bound, increasing the number of threads per worker can improve concurrency. However, for CPU-bound tasks, more threads might not help and could even increase context-switching overhead.
A common Gunicorn startup command or systemd service file configuration:
# Example command line
gunicorn --workers 5 \
--threads 2 \
--bind unix:/run/gunicorn.sock \
--access-logfile /var/log/gunicorn/access.log \
--error-logfile /var/log/gunicorn/error.log \
your_project.wsgi:application
# Example systemd service file (/etc/systemd/system/gunicorn.service)
[Unit]
Description=gunicorn daemon for your_project
After=network.target
[Service]
User=your_user
Group=www-data
WorkingDirectory=/path/to/your/project
ExecStart=/path/to/your/venv/bin/gunicorn \
--workers 5 \
--threads 2 \
--bind unix:/run/gunicorn.sock \
--access-logfile /var/log/gunicorn/access.log \
--error-logfile /var/log/gunicorn/error.log \
your_project.wsgi:application
[Install]
WantedBy=multi-user.target
Remember to create the necessary log directories and ensure permissions are set correctly:
sudo mkdir -p /var/log/gunicorn sudo chown your_user:www-data /var/log/gunicorn sudo chmod 755 /var/log/gunicorn
PHP-FPM Tuning for PHP Applications
If you’re running PHP applications, PHP-FPM (FastCGI Process Manager) is the standard. Its performance is tuned via the pm (process manager) settings.
Process Manager Settings
The pm directive can be set to static, dynamic, or ondemand. For servers with consistent traffic, static is often best. For variable traffic, dynamic or ondemand can save resources.
Edit the PHP-FPM pool configuration file (e.g., /etc/php/8.1/fpm/pool.d/www.conf or a custom pool file):
; Example for a pool named 'www' [www] user = www-data group = www-data listen = /run/php/php8.1-fpm.sock ; Or a TCP socket like 127.0.0.1:9000 ; Process Manager Settings pm = dynamic pm.max_children = 50 ; Max number of children at any one time pm.start_servers = 5 ; Number of children when pm = dynamic pm.min_spare_servers = 2 ; Min number of idle/spare children pm.max_spare_servers = 10 ; Max number of idle/spare children pm.process_idle_timeout = 10s ; Timeout for idle processes (used with ondemand) pm.max_requests = 500 ; Max requests per child process before respawning ; For static process management (less common for variable loads) ; pm = static ; pm.max_children = 50 ; For ondemand process management (saves memory when idle) ; pm = ondemand ; pm.max_children = 50 ; pm.process_idle_timeout = 10s ; Other important settings request_terminate_timeout = 300 ; Timeout for script execution ; rlimit_files = 1024 ; rlimit_nofile = 65536
After changes, reload PHP-FPM:
sudo systemctl reload php8.1-fpm
MySQL/MariaDB Performance Tuning
Database performance is often the bottleneck. Tuning MySQL/MariaDB involves adjusting key configuration variables, primarily in my.cnf or mysqld.cnf (often found in /etc/mysql/mysql.conf.d/mysqld.cnf or /etc/my.cnf).
Key Configuration Variables
Focus on memory allocation and buffer sizes. These values are highly dependent on your server’s RAM and workload. Start with conservative values and monitor performance.
[mysqld] # General Settings user = mysql pid-file = /var/run/mysqld/mysqld.pid socket = /var/run/mysqld/mysqld.sock datadir = /var/lib/mysql log-error = /var/log/mysql/error.log # log_bin = /var/log/mysql/mysql-bin.log # Enable for replication/point-in-time recovery # InnoDB Settings (Most common storage engine) innodb_buffer_pool_size = 1G # Crucial: ~70-80% of available RAM for dedicated DB servers innodb_log_file_size = 256M # Larger logs can improve write performance but increase recovery time innodb_log_buffer_size = 16M innodb_flush_log_at_trx_commit = 1 # ACID compliance (0 or 2 for higher performance, but riskier) innodb_flush_method = O_DIRECT # Avoid double buffering with OS cache # Connection and Thread Settings max_connections = 200 # Adjust based on application needs and server resources thread_cache_size = 16 # Cache threads for reuse table_open_cache = 2000 # Cache open table file descriptors table_definition_cache = 1000 # Cache table definitions # Query Cache (Deprecated in MySQL 8.0, consider alternatives if using older versions) # query_cache_type = 1 # query_cache_size = 64M # Sort and Join Buffers sort_buffer_size = 2M join_buffer_size = 2M read_buffer_size = 1M read_rnd_buffer_size = 2M # Temporary Tables tmp_table_size = 64M max_heap_table_size = 64M # Network Settings # skip-networking # Uncomment if only local access is needed # Character Set character-set-server = utf8mb4 collation-server = utf8mb4_unicode_ci
After modifying the configuration file, restart the MySQL service:
sudo systemctl restart mysql
Monitoring and Iteration
Tuning is an iterative process. Use monitoring tools to observe the impact of your changes. Key metrics include:
- Nginx: Active connections, requests per second, error rates (4xx, 5xx), worker connections usage. Tools like
netstat,htop, and Nginx’s status module are useful. - Gunicorn/PHP-FPM: Worker utilization, request latency, error logs, CPU/memory usage per worker.
- MySQL: Slow query log, connections, buffer pool hit rate, disk I/O, CPU/memory usage. Tools like
mysqltuner.pl(use with caution, it provides suggestions, not definitive answers),pt-query-digest, and Prometheus/Grafana with the MySQL exporter are invaluable.
Regularly review your application’s performance characteristics and adjust these configurations accordingly. A DigitalOcean droplet with 4GB RAM might require significantly different settings than one with 16GB RAM.