Vengala Vinay

Having 9+ Years of Experience in Software Development

Home » The Ultimate DevOps Playbook: Tuning Nginx, Gunicorn/FPM, and MongoDB on DigitalOcean for C

The Ultimate DevOps Playbook: Tuning Nginx, Gunicorn/FPM, and MongoDB on DigitalOcean for C

Nginx as a High-Performance Frontend Proxy

For a robust web application stack, Nginx serves as an exceptional frontend proxy, efficiently handling static assets, SSL termination, and load balancing. Its asynchronous, event-driven architecture makes it ideal for high concurrency. We’ll focus on tuning Nginx for optimal performance in a DigitalOcean environment, assuming a typical setup with a single or multi-core Droplet.

Core Nginx Configuration Tuning

The primary configuration file, typically located at /etc/nginx/nginx.conf, contains directives that significantly impact performance. Let’s break down the essential ones:

Worker Processes and Connections

The worker_processes directive dictates how many worker processes Nginx will spawn. A common best practice is to set this to the number of CPU cores available on your server. This allows Nginx to fully utilize your hardware for handling requests.

The worker_connections directive sets the maximum number of simultaneous connections that each worker process can handle. The total maximum connections will be worker_processes * worker_connections. Ensure this value is set high enough to accommodate your expected traffic, but not so high that it exhausts system resources (like file descriptors).

Here’s an example snippet from nginx.conf:

# /etc/nginx/nginx.conf

user www-data;
worker_processes auto; # Or set to the number of CPU cores, e.g., worker_processes 4;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;

events {
    worker_connections 1024; # Adjust based on expected concurrent connections per worker
    multi_accept on;
}

http {
    # ... other http directives ...
}

Keepalive Connections

Enabling keepalive connections reduces the overhead of establishing new TCP connections for each HTTP request. This is particularly beneficial for clients making multiple requests to the same server.

# Inside the http block of nginx.conf

http {
    # ... other http directives ...

    keepalive_timeout 65; # Time to keep a connection open after the last request
    keepalive_requests 100; # Max requests per keepalive connection
    tcp_nodelay on; # Improves latency by reducing packet delays
    tcp_nopush on; # Reduces the number of packets sent by sending data in fewer, larger chunks
}

Gzip Compression

Compressing responses with Gzip can significantly reduce bandwidth usage and improve page load times for clients. It’s crucial to enable this for text-based content like HTML, CSS, and JavaScript.

# Inside the http block of nginx.conf

http {
    # ... other http directives ...

    gzip on;
    gzip_vary on; # Adds the "Vary: Accept-Encoding" header to responses
    gzip_proxied any; # Compresses responses for proxied requests
    gzip_comp_level 6; # Compression level (1-9, 6 is a good balance)
    gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;
    gzip_min_length 256; # Minimum response length to compress
    gzip_disable "msie6"; # Disable gzip for older IE versions if necessary
}

Caching Static Assets

Leveraging browser caching for static assets (CSS, JS, images) is a fundamental optimization. Nginx can set appropriate cache-control headers to instruct browsers to cache these resources.

# Inside the http block of nginx.conf, or in a separate conf.d file

http {
    # ... other http directives ...

    location ~* \.(css|js|jpg|jpeg|png|gif|ico|svg|woff|woff2|ttf|eot)$ {
        expires 365d; # Cache for 1 year
        add_header Cache-Control "public, no-transform";
        access_log off; # Optionally disable access logs for static files
    }
}

Configuring Nginx to Proxy to Gunicorn/PHP-FPM

Nginx excels at proxying requests to application servers. For Python applications using Gunicorn or PHP applications using PHP-FPM, Nginx acts as the gateway, forwarding dynamic requests and serving static content directly.

Proxying to Gunicorn (Python/WSGI)

Assuming Gunicorn is running and listening on a Unix socket (e.g., /run/gunicorn.sock) or a local TCP port (e.g., 127.0.0.1:8000), configure your Nginx site’s server block accordingly.

# /etc/nginx/sites-available/your_app

server {
    listen 80;
    server_name your_domain.com www.your_domain.com;

    # Serve static files directly
    location /static/ {
        alias /path/to/your/app/static/;
        expires 365d;
        add_header Cache-Control "public, no-transform";
    }

    # Proxy dynamic requests to Gunicorn
    location / {
        proxy_pass http://unix:/run/gunicorn.sock; # Or http://127.0.0.1:8000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_read_timeout 300s; # Increase timeout for long-running requests
        proxy_connect_timeout 75s;
    }
}

Proxying to PHP-FPM

For PHP applications, Nginx communicates with PHP-FPM via a Unix socket or TCP port. The fastcgi_pass directive is key here.

# /etc/nginx/sites-available/your_php_app

server {
    listen 80;
    server_name your_php_domain.com;
    root /var/www/your_php_app;
    index index.php index.html index.htm;

    location / {
        try_files $uri $uri/ /index.php?$query_string;
    }

    location ~ \.php$ {
        include snippets/fastcgi-php.conf;
        # Assuming PHP-FPM is listening on a Unix socket
        fastcgi_pass unix:/run/php/php7.4-fpm.sock; # Adjust PHP version as needed
        # Or if PHP-FPM is listening on TCP:
        # fastcgi_pass 127.0.0.1:9000;
    }

    # Deny access to .htaccess files, if Apache's document root
    # concurs with nginx's one
    location ~ /\.ht {
        deny all;
    }
}

Gunicorn Performance Tuning

Gunicorn (Green Unicorn) is a Python WSGI HTTP Server. Its performance is heavily influenced by its worker configuration and settings.

Worker Types and Count

Gunicorn supports several worker types. The most common and recommended for I/O-bound applications are gevent or event (which uses Python’s asyncio or selectors). For CPU-bound tasks, sync workers might be simpler but less scalable.

The number of workers is crucial. A common starting point is (2 * number_of_cores) + 1. However, for I/O-bound applications with asynchronous workers, you might need more workers to keep the CPU busy while waiting for I/O operations.

Here’s how to start Gunicorn with specific settings:

# Example using gevent workers
gunicorn --workers 3 --worker-class gevent --bind unix:/run/gunicorn.sock myapp.wsgi:application

# Example using event workers (default for Python 3.7+)
gunicorn --workers 3 --bind unix:/run/gunicorn.sock myapp.wsgi:application

# Example with more workers and a TCP bind
gunicorn --workers 5 --bind 127.0.0.1:8000 myapp.wsgi:application

Worker Timeout and Max Requests

--timeout specifies how long Gunicorn will wait for a worker to respond before considering it dead. This should be set higher than your longest expected request processing time but not excessively high to prevent hanging workers from blocking resources.

--max-requests is a crucial setting for preventing memory leaks in long-running applications. It defines how many requests a worker will process before it’s restarted. A value between 1000 and 10000 is common.

gunicorn --workers 4 --worker-class gevent --timeout 120 --max-requests 5000 --bind unix:/run/gunicorn.sock myapp.wsgi:application

Gunicorn Configuration File

For more complex configurations, using a Python configuration file is recommended. Create a file (e.g., gunicorn_config.py):

# gunicorn_config.py

import multiprocessing

bind = "unix:/run/gunicorn.sock"
workers = multiprocessing.cpu_count() * 2 + 1
worker_class = "gevent" # Or "event"
timeout = 120
max_requests = 5000
loglevel = "info"
accesslog = "-" # Log to stdout
errorlog = "-"  # Log to stderr

Then, run Gunicorn using this configuration:

gunicorn -c gunicorn_config.py myapp.wsgi:application

PHP-FPM Performance Tuning

PHP-FPM (FastCGI Process Manager) is the de facto standard for running PHP applications with web servers like Nginx. Its performance hinges on its process management and pool configuration.

Process Management Modes

PHP-FPM offers three primary process management modes:

  • Static: A fixed number of child processes are spawned when the master process starts. This offers the best performance but can be less flexible.
  • Dynamic: The number of child processes varies dynamically based on load. It starts with a minimum number and spawns more up to a maximum as needed.
  • On-demand: Child processes are spawned only when a request is received and are terminated after a period of inactivity. This saves resources but can introduce latency for the first request after idle periods.

For most production environments, static or dynamic modes offer the best balance of performance and resource utilization. Static is generally preferred if you have a predictable load and sufficient RAM.

Pool Configuration

PHP-FPM pool configurations are typically found in /etc/php/[version]/fpm/pool.d/www.conf (or a custom pool file). Key directives to tune:

; /etc/php/7.4/fpm/pool.d/www.conf

; Choose your process management and tune accordingly
; pm = static
; pm.max_children = 50       ; Number of processes to keep active (for static)
; pm.start_servers = 5       ; Number of servers to start on boot (for dynamic)
; pm.min_spare_servers = 2   ; Minimum number of servers to maintain (for dynamic)
; pm.max_spare_servers = 10  ; Maximum number of servers to maintain (for dynamic)
; pm.max_requests = 500      ; Max requests per child process before respawn (similar to Gunicorn's max_requests)

pm = dynamic
pm.max_children = 100
pm.start_servers = 10
pm.min_spare_servers = 5
pm.max_spare_servers = 20
pm.max_requests = 1000

; Listen on a Unix socket (recommended for Nginx on the same server)
listen = /run/php/php7.4-fpm.sock

; Or listen on a TCP port
; listen = 127.0.0.1:9000

; Other important settings
request_terminate_timeout = 120s ; Timeout for script execution
; process_idle_timeout = 10s     ; For on-demand mode
; pm.process_idle_timeout = 10s  ; For dynamic mode, time before idle process is killed

Tuning Strategy:

  • Start with pm = dynamic.
  • Set pm.max_children based on your server’s RAM. A rough guideline is (Total RAM - RAM for OS/Nginx/DB) / Average PHP-FPM Child RAM Usage. Monitor RAM usage closely.
  • Adjust pm.start_servers, pm.min_spare_servers, and pm.max_spare_servers to balance responsiveness and resource consumption.
  • Set pm.max_requests to prevent memory leaks.
  • If you have consistent high traffic and sufficient RAM, consider switching to pm = static and setting pm.max_children to a value that keeps your CPU busy but doesn’t cause swapping.

PHP Settings within PHP-FPM

Some PHP settings can also impact performance. These are often configured within the php.ini file associated with your PHP-FPM installation (e.g., /etc/php/7.4/fpm/php.ini).

; /etc/php/7.4/fpm/php.ini

memory_limit = 256M
max_execution_time = 120
upload_max_filesize = 64M
post_max_size = 64M
date.timezone = UTC
opcache.enable=1
opcache.memory_consumption=128
opcache.interned_strings_buffer=16
opcache.max_accelerated_files=10000
opcache.revalidate_freq=60
opcache.save_comments=1
opcache.load_comments=1
opcache.enable_cli=1

OPcache is critical for PHP performance. Ensure it’s enabled and configured with adequate memory. opcache.revalidate_freq controls how often PHP checks for updated files; a value of 0 disables checking (use with caution, requires manual cache clearing on deploy), while a higher value like 60 (seconds) is a good balance for production.

MongoDB Performance Tuning on DigitalOcean

MongoDB’s performance is influenced by hardware, configuration, and query patterns. On DigitalOcean, choosing the right Droplet type (CPU-optimized, memory-optimized) and disk type (SSD is essential) is foundational. Beyond that, server configuration and indexing are key.

MongoDB Configuration File

The primary configuration file is typically /etc/mongod.conf. Key parameters for performance tuning include:

# /etc/mongod.conf

systemLog:
  destination: file
  path: /var/log/mongodb/mongod.log
  logAppend: true
  verbosity: 0 # 0 is default, higher values for more detailed logging

storage:
  dbPath: /var/lib/mongodb
  journal:
    enabled: true # Essential for durability and performance
  engine: wiredTiger # Default and recommended storage engine
  wiredTiger:
    collectionConfig:
      cacheRootPath: /var/lib/mongodb/journal
      # For WiredTiger, the cache size is the most critical setting.
      # It defaults to 50% of RAM for WiredTiger.
      # Explicitly setting it can be beneficial.
      # Example: 75% of RAM for WiredTiger cache
      # cacheSizeGB: 0.75 # If Droplet has 1GB RAM, this would be ~750MB

# network interfaces
net:
  port: 27017
  bindIp: 0.0.0.0 # Or specific IPs for security

# Process management
processManagement:
  fork: true
  pidFilePath: /var/run/mongodb/mongod.pid

# Sharding (if applicable)
# sharding:
#   clusterRole: configsvr
#   configDB: cluster0/mongo1.example.net:27019,mongo2.example.net:27019,mongo3.example.net:27019

# Security (essential for production)
# security:
#   keyFile: /path/to/keyfile
#   authorization: enabled

WiredTiger Cache Size

The storage.wiredTiger.collectionConfig.cacheSizeGB (or cacheSizeGB in older versions) is paramount. WiredTiger uses a portion of RAM to cache data and index blocks, significantly speeding up reads. The default is 50% of system RAM. For dedicated MongoDB Droplets, you might increase this to 70-80% if other processes have minimal RAM requirements. Monitor RAM usage to avoid excessive swapping.

Monitoring and Diagnostics

Regular monitoring is key to identifying performance bottlenecks. Use MongoDB’s built-in tools and system monitoring.

Slow Query Analysis

Enable the slow query log to identify queries that take too long to execute. A common threshold is 100ms.

# Add to mongod.conf
operationProfiling:
  mode: "slowOp" # or "all" for more verbose profiling
  slowOpThresholdMs: 100 # Log operations taking longer than 100ms

After restarting mongod, slow queries will be logged. Analyze these logs to determine which queries need indexing.

Indexing Strategy

Proper indexing is the single most effective way to improve MongoDB query performance. Use the explain() method on your queries to understand their execution plan and identify missing indexes.

// Example: Analyze a find query
db.collection.find({ user_id: 123, status: "active" }).explain("executionStats")

// Example: Create a compound index
db.collection.createIndex({ user_id: 1, status: 1 })

Always test the impact of new indexes in a staging environment before deploying to production. Over-indexing can negatively impact write performance and increase storage overhead.

Connection Pooling

Ensure your application’s MongoDB driver is configured with an appropriate connection pool size. Too few connections can lead to request queuing, while too many can exhaust server resources.

For example, in Python with PyMongo:

from pymongo import MongoClient

# Default pool size is 20. Adjust as needed.
client = MongoClient('mongodb://localhost:27017/', maxPoolSize=50)
db = client.mydatabase
collection = db.mycollection

System-Level Tuning

Ensure your Droplet’s system is tuned for database performance:

  • Swappiness: Reduce swappiness to prevent the OS from swapping out MongoDB’s memory. Edit /etc/sysctl.conf and add/modify vm.swappiness=1. Apply with sudo sysctl -p.
  • File Descriptors: MongoDB requires a high number of open file descriptors. Ensure limits are set appropriately in /etc/security/limits.conf or systemd service files.
# Example for systemd service file (e.g., /etc/systemd/system/mongod.service)
[Service]
LimitNOFILE=65536

By systematically tuning Nginx, your application server (Gunicorn/PHP-FPM), and MongoDB, you can build a highly performant and scalable web application stack on DigitalOcean.