Vengala Vinay

Having 9+ Years of Experience in Software Development

Home » The Ultimate DevOps Playbook: Tuning Nginx, Gunicorn/FPM, and PostgreSQL on DigitalOcean for Shopify

The Ultimate DevOps Playbook: Tuning Nginx, Gunicorn/FPM, and PostgreSQL on DigitalOcean for Shopify

Nginx as a High-Performance Frontend for Gunicorn/PHP-FPM

When deploying applications that utilize Python (via Gunicorn) or PHP (via PHP-FPM) on DigitalOcean, Nginx serves as the de facto standard for a robust, high-performance frontend. Its event-driven architecture excels at handling concurrent connections, buffering slow client requests, and efficiently serving static assets. The key to unlocking Nginx’s full potential lies in meticulous configuration, particularly around worker processes, connection limits, and caching strategies.

Nginx Worker Processes and Connections

The worker_processes directive dictates how many worker processes Nginx will spawn. Setting this to auto is generally recommended, allowing Nginx to detect the number of CPU cores available and scale accordingly. The worker_connections directive, on the other hand, defines the maximum number of simultaneous connections that each worker process can handle. This value, combined with worker_processes, determines the total connection capacity. A common starting point for worker_connections on a DigitalOcean droplet with a reasonable number of cores (e.g., 4-8) is 4096. Remember that the total number of open file descriptors on the system also plays a role; ensure your system’s limits are set appropriately.

Tuning nginx.conf

Edit your main Nginx configuration file, typically located at /etc/nginx/nginx.conf. Focus on the events block.

events {
    worker_connections 4096;
    multi_accept on; # Allows workers to accept multiple connections at once
    use epoll;       # Linux-specific, high-performance event notification mechanism
}

http {
    # ... other http configurations ...

    sendfile on;             # Optimize file transfers by using OS's sendfile()
    tcp_nopush on;           # Improves efficiency of sending data over TCP
    tcp_nodelay on;          # Disables Nagle's algorithm for lower latency
    keepalive_timeout 65;    # Timeout for keep-alive connections
    keepalive_requests 1000; # Max requests per keep-alive connection

    # ... server blocks ...
}

Proxying to Gunicorn/PHP-FPM

When Nginx acts as a reverse proxy, it forwards requests to your application server (Gunicorn for Python, PHP-FPM for PHP). The configuration here is critical for efficient communication. For Gunicorn, you’ll typically proxy to a Unix socket or a local TCP port. For PHP-FPM, it’s usually a Unix socket or a local TCP port as well. Key directives include proxy_pass, proxy_set_header, and timeouts.

Nginx to Gunicorn Configuration

Assuming Gunicorn is running and listening on a Unix socket /run/gunicorn.sock.

server {
    listen 80;
    server_name your_domain.com;

    location / {
        proxy_pass http://unix:/run/gunicorn.sock;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        proxy_connect_timeout 60s;
        proxy_send_timeout 60s;
        proxy_read_timeout 60s;
    }

    location /static/ {
        alias /path/to/your/static/files/;
        expires 30d; # Cache static files for 30 days
    }
}

Nginx to PHP-FPM Configuration

Assuming PHP-FPM is listening on a Unix socket /var/run/php/php7.4-fpm.sock.

server {
    listen 80;
    server_name your_domain.com;
    root /var/www/your_app;
    index index.php index.html index.htm;

    location / {
        try_files $uri $uri/ /index.php?$query_string;
    }

    location ~ \.php$ {
        include snippets/fastcgi-php.conf;
        fastcgi_pass unix:/var/run/php/php7.4-fpm.sock;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        include fastcgi_params;

        fastcgi_connect_timeout 60s;
        fastcgi_send_timeout 60s;
        fastcgi_read_timeout 60s;
    }

    location ~ /\.ht {
        deny all;
    }

    location /static/ {
        alias /path/to/your/static/files/;
        expires 30d;
    }
}

Gunicorn and PHP-FPM: Application Server Tuning

The performance of your application server is directly tied to how efficiently it can process incoming requests. For Gunicorn, this involves tuning the number of worker processes and their type. For PHP-FPM, it’s about managing child process pools.

Gunicorn Worker Processes

Gunicorn’s performance is heavily influenced by its worker class and the number of workers. The default sync worker class is simple but can block under heavy load. The gevent or eventlet worker classes, which are asynchronous, are generally preferred for I/O-bound applications. A common recommendation for the number of workers is (2 * number_of_cores) + 1. However, this can vary based on your application’s CPU vs. I/O bound nature.

Gunicorn Command Line/Systemd Service

Here’s an example of how you might configure Gunicorn, assuming a 4-core CPU:

# Example command line
gunicorn --workers 9 --worker-class gevent --bind unix:/run/gunicorn.sock myapp.wsgi:application

# Example Systemd service file (/etc/systemd/system/gunicorn.service)
[Unit]
Description=Gunicorn instance to serve myapp
After=network.target

[Service]
User=your_user
Group=your_group
WorkingDirectory=/path/to/your/app
ExecStart=/path/to/your/venv/bin/gunicorn \
    --workers 9 \
    --worker-class gevent \
    --bind unix:/run/gunicorn.sock \
    myapp.wsgi:application

Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

PHP-FPM Child Process Management

PHP-FPM uses process pools to manage child workers. The configuration for these pools is typically found in /etc/php/X.Y/fpm/pool.d/www.conf (replace X.Y with your PHP version). The key directives are pm (process manager), pm.max_children, pm.start_servers, pm.min_spare_servers, and pm.max_spare_servers.

Tuning www.conf

For a DigitalOcean droplet with 4 CPU cores, a reasonable starting point for a moderately trafficked site might be:

; /etc/php/7.4/fpm/pool.d/www.conf

[www]
user = www-data
group = www-data
listen = /var/run/php/php7.4-fpm.sock
listen.owner = www-data
listen.group = www-data
listen.mode = 0660

; Process Manager settings
pm = dynamic
pm.max_children = 50       ; Maximum number of child processes that can be started.
pm.start_servers = 5       ; Number of child processes to start when pm is 'dynamic'.
pm.min_spare_servers = 2   ; Minimum number of spare servers.
pm.max_spare_servers = 10  ; Maximum number of spare servers.
pm.process_idle_timeout = 10s; The timeout for a child process to become idle.

; Request handling
request_terminate_timeout = 60s ; Timeout for script execution
; request_slowlog_timeout = 10s ; Uncomment and configure if you need to log slow requests

; Other settings
; php_admin_value[memory_limit] = 256M
; php_admin_value[upload_max_filesize] = 64M
; php_admin_value[post_max_size] = 64M

Note: The optimal values for pm.max_children and related settings depend heavily on your application’s memory footprint per request and the available RAM on your droplet. Monitor your server’s memory usage closely after applying these changes.

PostgreSQL Performance Tuning on DigitalOcean

PostgreSQL is a powerful relational database, and its performance can be significantly enhanced through careful configuration. On DigitalOcean, you have direct access to the PostgreSQL configuration file (postgresql.conf) and can leverage tools like pgtune to get a baseline. Key areas to focus on are memory allocation, connection pooling, and query optimization.

Memory Allocation

The most impactful parameters relate to how PostgreSQL utilizes system memory. The primary ones are:

  • shared_buffers: This is the most critical parameter. It defines the amount of memory dedicated to PostgreSQL’s shared memory buffers. A common recommendation is 25% of your total system RAM, but this can be pushed higher (up to 40%) on dedicated database servers with ample RAM.
  • work_mem: This controls the amount of memory that can be used for internal sort operations and hash tables before spilling to disk. It’s allocated per operation, so setting it too high can lead to excessive memory consumption if many operations run concurrently.
  • maintenance_work_mem: Used for maintenance operations like VACUUM, CREATE INDEX, and ALTER TABLE. It can be set higher than work_mem as these operations are typically not run concurrently.

Tuning postgresql.conf

Locate your postgresql.conf file (often in /etc/postgresql/X.Y/main/). Here’s an example for a 4GB RAM droplet:

# /etc/postgresql/12/main/postgresql.conf

# Memory settings
shared_buffers = 1GB             # ~25% of 4GB RAM
work_mem = 32MB                  # Adjust based on query complexity and concurrency
maintenance_work_mem = 256MB     # For VACUUM, CREATE INDEX etc.

# Connection settings
max_connections = 100            # Adjust based on application needs and droplet RAM
; listen_addresses = '*'       # Uncomment and set to '*' if PostgreSQL is not on localhost and needs external access (ensure firewall is configured)

# WAL settings (Write-Ahead Logging)
wal_buffers = 16MB               # Default is -1 (auto-tuned based on shared_buffers)
wal_writer_delay = 200ms         # Default is 200ms
wal_checkpoint_timeout = 30min   # Default is 5min
wal_checkpoint_completion_target = 0.9 # Default is 0.5

# Background writer settings
bgwriter_delay = 10ms            # Default is 10ms
bgwriter_lru_maxpages = 1000     # Default is 100
bgwriter_lru_multiplier = 1.0    # Default is 1.0

# Autovacuum settings
autovacuum = on
autovacuum_max_workers = 3
autovacuum_naptime = 15s         # Default is 1min
autovacuum_vacuum_threshold = 50
autovacuum_analyze_threshold = 50

# Logging
log_destination = 'stderr'
logging_collector = on
log_directory = 'pg_log'
log_filename = 'postgresql-%Y-%m-%d_%H%M%S.log'
log_min_duration_statement = 250ms # Log queries taking longer than 250ms
log_checkpoints = on
log_connections = on
log_disconnections = on
log_lock_waits = on
log_temp_files = 0               # Log temporary files larger than this size (0 to disable)
log_autovacuum_min_duration = 0  # Log all autovacuum actions

After modifying postgresql.conf, you must restart the PostgreSQL service:

sudo systemctl restart postgresql

Connection Pooling

Opening and closing database connections is an expensive operation. For applications that frequently connect and disconnect, a connection pooler like PgBouncer can dramatically improve performance by maintaining a pool of open connections that applications can borrow from. This is especially crucial for web applications with many concurrent users.

Setting up PgBouncer

1. **Install PgBouncer:**

sudo apt update
sudo apt install pgbouncer

2. **Configure PgBouncer (/etc/pgbouncer/pgbouncer.ini):**

[databases]
mydatabase = host=127.0.0.1 port=5432 dbname=mydatabase

[pgbouncer]
; Listen on a different port for the pooler
listen_port = 6432
listen_address = *
auth_type = md5
auth_file = /etc/pgbouncer/userlist.txt
pool_mode = session ; 'transaction' or 'session'. 'session' is generally safer for complex apps.
max_client_conn = 1000 ; Max concurrent clients connecting to pgbouncer
default_pool_size = 20 ; Default pool size per database
; reserve_pool_size = 5 ; Number of reserved connections for superusers
; reserve_pool_timeout = 5s

; Log settings
logfile = /var/log/postgresql/pgbouncer.log
pidfile = /var/run/pgbouncer/pgbouncer.pid

3. **Configure Userlist (/etc/pgbouncer/userlist.txt):**

"your_db_user" "md5_hashed_password_for_your_db_user"

You can generate the MD5 hash using psql: SELECT md5('your_db_password');. Ensure the user exists in your PostgreSQL database.

4. **Restart PgBouncer:**

sudo systemctl restart pgbouncer

5. **Update your application’s database connection string** to point to localhost:6432 instead of the default PostgreSQL port (5432).

Query Optimization and Indexing

While not strictly a configuration parameter, effective indexing and query optimization are paramount. Regularly analyze slow queries logged by PostgreSQL (using log_min_duration_statement) and ensure appropriate indexes are in place. Use EXPLAIN ANALYZE to understand query execution plans.

-- Example: Analyze a slow query
EXPLAIN ANALYZE SELECT * FROM orders WHERE customer_id = 123 AND order_date > '2023-01-01';

-- Example: Add an index
CREATE INDEX idx_orders_customer_date ON orders (customer_id, order_date);

Regularly run VACUUM ANALYZE (or rely on autovacuum) to keep table statistics up-to-date, which is crucial for the query planner.