Vengala Vinay

Having 12+ Years of Experience in Software Development

Home » The Ultimate DevOps Playbook: Tuning Nginx, Gunicorn/FPM, and MongoDB on DigitalOcean for Shopify

The Ultimate DevOps Playbook: Tuning Nginx, Gunicorn/FPM, and MongoDB on DigitalOcean for Shopify

Nginx as a High-Performance Frontend Proxy

For a Shopify-powered application, Nginx serves as the critical entry point, handling SSL termination, static file serving, and reverse proxying to your application servers (Gunicorn for Python/Django/Flask, or PHP-FPM for PHP applications). Optimizing Nginx is paramount for low latency and high throughput.

Core Nginx Configuration Tuning

We’ll focus on key directives within nginx.conf or a dedicated site configuration file (e.g., /etc/nginx/sites-available/your_shopify_app). The goal is to maximize concurrent connections and efficient request handling.

Worker Processes and Connections

The worker_processes directive should ideally be set to the number of CPU cores available on your DigitalOcean droplet. worker_connections defines the maximum number of simultaneous connections a worker process can handle. The total maximum connections will be worker_processes * worker_connections.

Example: nginx.conf Snippet
# Determine the number of CPU cores dynamically
daemon off; # For running Nginx in the foreground, useful for Docker/systemd
master_process off; # For running Nginx in the foreground

events {
    worker_connections 4096; # Adjust based on droplet RAM and expected load
    multi_accept on; # Allows a worker to accept multiple connections at once
    use epoll; # Linux-specific, highly efficient I/O event notification mechanism
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    sendfile        on; # Efficiently transfer files from one file descriptor to another
    tcp_nopush      on; # Improves performance for sending data over TCP
    tcp_nodelay     on; # Disables the Nagle algorithm, reducing latency for small packets
    keepalive_timeout 65; # Time to keep persistent connections open
    keepalive_requests 1000; # Max requests per keepalive connection

    # Gzip compression for text-based assets
    gzip on;
    gzip_vary on;
    gzip_proxied any;
    gzip_comp_level 6;
    gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript image/svg+xml;

    # Buffering and timeouts for upstream connections
    proxy_connect_timeout 60s;
    proxy_send_timeout 60s;
    proxy_read_timeout 60s;
    proxy_buffer_size 16k;
    proxy_buffers 4 32k;
    proxy_busy_buffers_size 64k;

    # Client request body limits (adjust as needed)
    client_max_body_size 100M;

    # Logging configuration
    access_log /var/log/nginx/access.log;
    error_log /var/log/nginx/error.log warn;

    # Include site-specific configurations
    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*;
}

Note: For production, you’d typically run Nginx as a daemon with daemon on; and master_process on;. The daemon off; and master_process off; are shown for easier integration with process managers like systemd or Docker.

SSL/TLS Optimization

Offloading SSL/TLS to Nginx is standard practice. Ensure you’re using modern cipher suites and enabling HTTP/2 for improved performance.

Example: SSL Configuration Snippet
server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;

    server_name your-shopify-app.com;

    ssl_certificate /etc/letsencrypt/live/your-shopify-app.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/your-shopify-app.com/privkey.pem;

    # Modern TLS configuration (consider using Mozilla SSL Configuration Generator)
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_prefer_server_ciphers on;
    ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;
    ssl_session_cache shared:SSL:10m; # Adjust size based on memory
    ssl_session_timeout 10m;
    ssl_session_tickets off; # Consider enabling if performance is critical and security implications understood

    # OCSP Stapling
    ssl_stapling on;
    ssl_stapling_verify on;
    resolver 8.8.8.8 8.8.4.4 valid=300s; # Google DNS, adjust to your preferred DNS resolver
    resolver_timeout 5s;

    # HSTS Header
    add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload" always;

    # ... rest of your proxy configuration ...
    location / {
        proxy_pass http://unix:/run/gunicorn.sock; # Or http://127.0.0.1:8000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    # Serve static files directly
    location /static/ {
        alias /path/to/your/app/static/;
        expires 30d;
        access_log off;
    }

    # Deny access to hidden files
    location ~ /\. {
        deny all;
    }
}

# Redirect HTTP to HTTPS
server {
    listen 80;
    listen [::]:80;
    server_name your-shopify-app.com;
    return 301 https://$host$request_uri;
}

Static File Serving

Nginx excels at serving static assets (CSS, JS, images). Ensure your application’s static files are collected and served directly by Nginx, bypassing your application server.

Example: Static File Configuration
location /static/ {
    alias /var/www/your-shopify-app/static/; # Ensure this path is correct
    expires 30d; # Cache static assets aggressively
    access_log off; # Reduce logging for static files
    add_header Cache-Control "public";
}

location /media/ { # If your app uses a media directory
    alias /var/www/your-shopify-app/media/;
    expires 30d;
    access_log off;
    add_header Cache-Control "public";
}

Gunicorn/PHP-FPM: Application Server Tuning

The application server is responsible for executing your application code. Tuning Gunicorn (for Python) or PHP-FPM (for PHP) is crucial for handling application-level requests efficiently.

Gunicorn Tuning (Python)

Gunicorn is a WSGI HTTP Server. Its performance is heavily influenced by the number of worker processes and threads.

Worker Processes and Threads

A common starting point is to use the sync worker class with a number of workers equal to (2 * number_of_cores) + 1. For I/O-bound applications, consider the gevent or eventlet worker classes, which use asynchronous I/O and can handle more concurrent requests with fewer processes.

Example: Gunicorn Command Line / Systemd Service

Assuming a 4-core droplet and a Python/Django app:

# Using sync workers (CPU-bound friendly)
gunicorn --workers 9 \
         --worker-class sync \
         --bind unix:/run/gunicorn.sock \
         --timeout 120 \
         --log-level info \
         your_project.wsgi:application

# Using gevent workers (I/O-bound friendly)
# Install gevent: pip install gevent
gunicorn --workers 5 \
         --worker-class gevent \
         --bind unix:/run/gunicorn.sock \
         --timeout 120 \
         --log-level info \
         your_project.wsgi:application

Systemd Service File (/etc/systemd/system/gunicorn.service):

[Unit]
Description=Gunicorn instance to serve your_project
After=network.target

[Service]
User=your_user
Group=www-data
WorkingDirectory=/path/to/your/project
ExecStart=/path/to/your/venv/bin/gunicorn \
          --workers 9 \
          --worker-class sync \
          --bind unix:/run/gunicorn.sock \
          --timeout 120 \
          --log-level info \
          your_project.wsgi:application

[Install]
WantedBy=multi-user.target

Gunicorn Timeout

The --timeout setting is crucial. It defines how long Gunicorn will wait for a worker to process a request before killing it. Set this high enough to accommodate slow operations but not so high that it masks persistent issues.

PHP-FPM Tuning (PHP)

PHP-FPM (FastCGI Process Manager) manages PHP processes. Its configuration is typically found in /etc/php/X.Y/fpm/pool.d/www.conf (replace X.Y with your PHP version).

Process Management (pm)

PHP-FPM offers several process management modes:

  • static: Manages a fixed number of processes. Good for predictable loads.
  • dynamic: Manages processes dynamically based on load.
  • ondemand: Spawns processes only when a request arrives.

For most Shopify applications, dynamic or static with careful tuning is recommended. ondemand can lead to higher initial latency.

Example: PHP-FPM Pool Configuration (www.conf)
; /etc/php/8.1/fpm/pool.d/www.conf

[www]
user = www-data
group = www-data
listen = /run/php/php8.1-fpm.sock ; Or a TCP socket like 127.0.0.1:9000

; Process Manager Settings
pm = dynamic
pm.max_children = 50       ; Max number of children at any one time
pm.start_servers = 5       ; Number of servers started when pm becomes active
pm.min_spare_servers = 2   ; Number of servers to maintain in idle state
pm.max_spare_servers = 10  ; Number of servers to maintain in idle state
pm.max_requests = 500      ; Max requests a child process should execute before respawning

; Adjust based on your droplet's CPU and RAM.
; A common starting point for pm.max_children is (CPU cores * 5) to (CPU cores * 10)
; but this is highly dependent on your application's memory footprint per process.

; Request handling settings
request_terminate_timeout = 120s ; Corresponds to Gunicorn's timeout
; request_slowlog_timeout = 10s ; Enable slowlog for debugging

; Other settings
; php_admin_value[memory_limit] = 256M
; php_admin_value[upload_max_filesize] = 100M
; php_admin_value[post_max_size] = 100M

Tuning Strategy: Start with conservative values for pm.max_children and monitor your server’s memory usage. Gradually increase it while observing performance. If you experience out-of-memory errors, reduce the value. The pm.max_requests directive helps prevent memory leaks by respawning processes after a certain number of requests.

MongoDB Performance Tuning

MongoDB is often used for storing application data, analytics, or logs. Optimizing its performance involves indexing, query optimization, and server configuration.

Indexing Strategy

This is the single most impactful optimization for MongoDB. Analyze your application’s query patterns and create indexes accordingly. Use explain() on slow queries to identify missing indexes.

Example: Identifying and Creating Indexes

Connect to your MongoDB instance using the mongo shell:

// Connect to your database
use your_database_name;

// Find slow queries (e.g., from application logs or MongoDB profiler)
// Example query: Find products by category and sort by price
db.products.find({ category: "electronics" }).sort({ price: 1 });

// Analyze the query execution plan
db.products.find({ category: "electronics" }).sort({ price: 1 }).explain("executionStats");

// Based on the explain output, if 'totalKeysExamined' is high and 'totalDocsExamined' is high,
// an index is likely needed.
// Create a compound index for the query
db.products.createIndex({ category: 1, price: 1 });

// Verify the index
db.products.getIndexes();

Best Practices:

  • Index fields used in find(), sort(), and aggregate() stages.
  • Use compound indexes for queries that filter and sort on multiple fields. The order of fields in the index matters.
  • Avoid indexing every field; indexes consume memory and slow down writes.
  • Regularly review index usage and remove unused indexes.

MongoDB Server Configuration (mongod.conf)

Key settings in /etc/mongod.conf (or similar path) can significantly impact performance.

Example: Relevant Configuration Snippets
# /etc/mongod.conf

storage:
  dbPath: /var/lib/mongodb
  journal:
    enabled: true # Essential for durability and performance
  engine: wiredTiger # Default and recommended engine

# WiredTiger specific cache settings
# The default is 50% of RAM, which is usually a good starting point.
# Adjust based on your droplet size and other services running.
# Example: If you have a 16GB droplet and MongoDB is the primary service,
# you might allocate 8-10GB to the cache.
# Ensure enough RAM is left for the OS and other processes.
# For a 4GB droplet, the default 2GB might be appropriate.
# If you need to override:
# wiredTiger:
#   engineConfig:
#     cacheSizeGB: 2 # Example: Allocate 2GB for cache

# Network settings
net:
  port: 27017
  bindIp: 127.0.0.1 # Bind to localhost if only accessed by local app servers
                    # Use droplet's private IP if accessed from other droplets
                    # Use 0.0.0.0 to bind to all interfaces (use with caution and firewall)

# Logging settings
systemLog:
  destination: file
  path: /var/log/mongodb/mongod.log
  logAppend: true
  logRotate: reopen
  verbosity: 0 # 0 is default, increase for debugging

# Operation Profiling (useful for identifying slow queries)
# profile: 0 # 0=off, 1=slow operations, 2=all operations
# slowms: 100 # Operations slower than 100ms will be logged if profile is 1 or 2

# Sharding settings (if applicable)
# sharding:
#   clusterRole: configsvr
#   configsvr: true
#   heartbeatFrequencyInSecs: 60
#   replSetName: myReplicaSetName

Key Considerations:

  • RAM Allocation: The WiredTiger cache is critical. Ensure it’s sized appropriately to hold your working set (frequently accessed data and indexes) in memory. Monitor db.serverStatus().wiredTiger.cache.
  • Disk I/O: Use SSDs on DigitalOcean for MongoDB. Slow disk I/O is a common bottleneck.
  • Network: Ensure your application servers can connect to MongoDB with low latency, ideally over private networking.
  • Replication/Sharding: For production, consider setting up replica sets for high availability and potential read scaling. Sharding is for very large datasets and high write throughput.

Monitoring and Iteration

Performance tuning is an ongoing process. Implement robust monitoring to track key metrics and identify regressions.

Key Metrics to Monitor

  • Nginx: Requests per second, active connections, error rates (4xx, 5xx), latency.
  • Gunicorn/PHP-FPM: Worker utilization, request queue length, response times, error rates.
  • MongoDB: Query latency, operations per second (reads/writes), cache hit ratio, disk I/O wait times, network traffic, CPU/Memory utilization.
  • System: CPU load, memory usage, disk I/O, network bandwidth.

Tools like Prometheus with Grafana, Datadog, or DigitalOcean’s built-in monitoring are invaluable. Regularly analyze logs and performance metrics to identify bottlenecks and areas for further optimization.

Recent Posts

Top Categories

Our Products

Our Services