Vengala Vinay

Having 9+ Years of Experience in Software Development

Home » The Ultimate DevOps Playbook: Tuning Nginx, Gunicorn/FPM, and MongoDB on AWS for Magento 2

The Ultimate DevOps Playbook: Tuning Nginx, Gunicorn/FPM, and MongoDB on AWS for Magento 2

Nginx Configuration for Magento 2 on AWS EC2

Optimizing Nginx for a high-traffic Magento 2 instance on AWS requires a multi-pronged approach, focusing on efficient static file serving, robust caching, and secure proxying to your application servers. We’ll assume a standard setup with Nginx acting as a reverse proxy to Gunicorn (for custom modules/APIs) or PHP-FPM (for the core Magento application).

Start with a baseline Nginx configuration, typically found in /etc/nginx/nginx.conf and site-specific configurations in /etc/nginx/sites-available/. For Magento 2, the key is to leverage Nginx’s strengths for static assets and delegate dynamic requests appropriately.

Static File Serving and Caching

Magento 2 generates a significant number of static assets (CSS, JS, images). Nginx is exceptionally good at serving these directly from disk or from its own cache. Configure long cache expiry headers for these assets. This reduces load on your application servers and speeds up page loads for returning visitors.

In your Magento 2 Nginx virtual host configuration (e.g., /etc/nginx/sites-available/magento2.conf), add directives like these:

# Serve static files directly
location ~ ^/(media|static)/ {
    expires 365d;
    add_header Cache-Control "public";
    access_log off; # Optional: reduce log noise for static assets
    try_files $uri $uri/ /index.php?$args; # Fallback for potential missing files, though ideally they exist
}

# Serve robots.txt and favicon.ico
location ~ /(robots\.txt|favicon\.ico) {
    expires 1h;
    add_header Cache-Control "public";
}

# Cache static assets for a long time
location ~* \.(css|js|jpg|jpeg|png|gif|ico|svg|woff|woff2|ttf|eot)$ {
    expires 365d;
    add_header Cache-Control "public";
    access_log off;
    # Ensure these files are served from the correct Magento directory
    root /var/www/magento2/public_html; # Adjust this path to your Magento root
}

Gzip Compression and Brotli

Compressing text-based assets (HTML, CSS, JS) significantly reduces bandwidth. Nginx supports Gzip out-of-the-box. For even better compression, consider Brotli, which offers superior compression ratios at a slightly higher CPU cost during compression (which can be done offline or on-the-fly).

Enable Gzip in your main nginx.conf or within your site’s configuration:

# In nginx.conf or your site config
http {
    # ... other http settings ...

    gzip on;
    gzip_vary on;
    gzip_proxied any;
    gzip_comp_level 6; # Compression level (1-9)
    gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript image/svg+xml;

    # Optional: Brotli support (requires Nginx compiled with brotli module)
    # brotli on;
    # brotli_comp_level 6;
    # brotli_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript image/svg+xml;
}

Reverse Proxy Configuration

This is where Nginx forwards requests to your application servers (Gunicorn or PHP-FPM). Key parameters include connection timeouts, buffer sizes, and load balancing if you have multiple application instances.

For PHP-FPM:

location ~ ^/index\.php$ {
    try_files $uri =404;
    fastcgi_split_path_info ^(.+\.php)(/.+)$;
    fastcgi_pass unix:/var/run/php/php7.4-fpm.sock; # Adjust to your PHP-FPM version and socket path
    fastcgi_index index.php;
    include fastcgi_params;
    fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
    fastcgi_param PATH_INFO $fastcgi_path_info;

    # Performance tuning for FastCGI
    fastcgi_connect_timeout 60s;
    fastcgi_send_timeout 300s;
    fastcgi_read_timeout 300s;
    fastcgi_buffer_size 128k;
    fastcgi_buffers 4 256k;
    fastcgi_busy_buffers_size 256k;
}

For Gunicorn (assuming it’s listening on localhost:8000):

location / {
    proxy_pass http://127.0.0.1:8000; # Or your Gunicorn cluster IP/port
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;

    # Performance tuning for proxying
    proxy_connect_timeout 60s;
    proxy_send_timeout 300s;
    proxy_read_timeout 300s;
    proxy_buffer_size 128k;
    proxy_buffers 4 256k;
    proxy_busy_buffers_size 256k;
    proxy_http_version 1.1; # Use HTTP/1.1 for keep-alive connections
    proxy_set_header Connection ""; # Clear Connection header for upstream
}

Security and Rate Limiting

Implement basic security measures and rate limiting to protect against brute-force attacks and excessive requests.

# Example: Limit requests per IP
limit_req_zone $binary_remote_addr zone=mylimit:10m rate=5r/s; # 5 requests per second per IP

server {
    # ... other server settings ...
    limit_req zone=mylimit burst=20 nodelay; # Allow bursts of 20 requests, process immediately

    # ... location blocks ...
}

# Block common malicious patterns
location ~* \.(bak|config|sql|fla|psd|ini|log|sh|inc|swp|dist|tpl|env)$ {
    deny all;
}

# Deny access to .htaccess files
location ~ /\.ht {
    deny all;
}

Gunicorn Tuning for Magento 2 (or Python Applications)

If you’re using Gunicorn to serve custom Magento 2 modules, APIs, or a separate Python application, tuning its worker processes and timeouts is crucial. Gunicorn’s performance is heavily influenced by the number of worker processes and the type of worker class used.

Worker Processes and Type

The recommended worker class for I/O-bound applications (like web applications) is gevent or event. The number of workers is typically calculated as (2 * number_of_cores) + 1. However, for applications that are CPU-bound or have significant memory footprints, you might need to adjust this based on monitoring.

Example Gunicorn command line for production:

gunicorn --workers 4 \
         --worker-class gevent \
         --bind 0.0.0.0:8000 \
         --timeout 120 \
         --graceful-timeout 120 \
         --keep-alive 60 \
         --log-level info \
         --access-logfile /var/log/gunicorn/access.log \
         --error-logfile /var/log/gunicorn/error.log \
         your_app.wsgi:application # Replace with your actual WSGI application

Explanation:

  • --workers 4: Adjust based on your EC2 instance’s vCPU count. Start with (2 * vCPUs) + 1 and monitor CPU utilization.
  • --worker-class gevent: Uses greenlets for asynchronous I/O. event is another good option.
  • --bind 0.0.0.0:8000: Listen on all interfaces on port 8000. Nginx will proxy to this.
  • --timeout 120: Maximum time (in seconds) a worker can spend handling a request before being killed. Magento operations can be lengthy, so a higher timeout is often necessary.
  • --graceful-timeout 120: Time to wait for existing requests to finish during a reload.
  • --keep-alive 60: Number of requests a worker can handle before it’s recycled.

Resource Management and Monitoring

Monitor Gunicorn’s memory usage. If workers are consuming too much RAM, you might need to reduce the worker count or optimize your Python code. Tools like htop, psutil, and Gunicorn’s own logging are essential.

Consider using a process manager like systemd or supervisor to manage Gunicorn processes, ensuring they restart automatically if they crash.

# Example systemd service file (/etc/systemd/system/gunicorn.service)
[Unit]
Description=Gunicorn instance to serve myapp
After=network.target

[Service]
User=www-data
Group=www-data
WorkingDirectory=/path/to/your/app
ExecStart=/usr/local/bin/gunicorn --workers 4 --worker-class gevent --bind unix:/path/to/your/app/gunicorn.sock --timeout 120 --graceful-timeout 120 --keep-alive 60 your_app.wsgi:application

[Install]
WantedBy=multi-user.target

PHP-FPM Tuning for Magento 2

PHP-FPM (FastCGI Process Manager) is the standard way to run PHP applications like Magento 2. Tuning its process management and memory limits is critical for performance and stability.

Process Manager Settings

The primary configuration file is typically /etc/php/X.Y/fpm/pool.d/www.conf (replace X.Y with your PHP version). The pm (process manager) setting determines how PHP-FPM handles worker processes.

Common pm settings:

  • static: Keeps a fixed number of child processes available. Good for predictable loads.
  • dynamic: Starts processes as needed, up to a defined maximum. More flexible.
  • ondemand: Starts processes only when a request comes in. Can save memory but introduces latency.

For Magento 2, dynamic is often a good balance. Adjust the following parameters:

; /etc/php/X.Y/fpm/pool.d/www.conf

; Choose how the process manager (pm) will manage the processes of this pool.
; available values: static, dynamic, ondemand
pm = dynamic

; If pm is 'dynamic', these are the intervals for the process manager.
; Number of seconds after which an idle process will be killed.
; Default Value: 10s
pm.idle_timeout = 20s

; If pm is 'dynamic', this is the maximum number of processes that will be started.
; Default Value: max_children (usually 256)
pm.max_children = 100 ; Adjust based on available RAM and expected concurrency

; If pm is 'dynamic', this is the number of *additional* processes that will be spawned
; when the number of *currently running* processes reaches this limit.
; Default Value: max_children * 0.25
pm.max_requests = 500 ; Recycle processes after a certain number of requests to prevent memory leaks

; If pm is 'dynamic', these are the intervals for the process manager.
; Start a new process when the number of processes reaches this number.
; Default Value: max_children * 0.25
pm.start_servers = 10

; If pm is 'dynamic', these are the intervals for the process manager.
; Spare processes will be kept in the background to answer incoming requests.
; Default Value: max_children * 0.5
pm.min_spare_servers = 5

; If pm is 'dynamic', these are the intervals for the process manager.
; The maximum number of spare servers. The total number of servers will not exceed pm.max_children.
; Default Value: max_children
pm.max_spare_servers = 20

Tuning `pm.max_children`: This is the most critical setting. Calculate it based on your server’s RAM. Each PHP-FPM worker can consume a certain amount of memory. A Magento 2 request can be memory-intensive. A common starting point is to estimate average memory per worker (e.g., 50-100MB) and divide your total available RAM by this figure, leaving room for the OS and other services.

PHP Memory Limits and Execution Time

Magento 2 requires significant memory and can have long-running processes, especially during cron jobs or complex product imports/exports. Adjust memory_limit and max_execution_time in php.ini (or the FPM configuration).

; /etc/php/X.Y/fpm/php.ini

memory_limit = 512M ; Or higher, depending on your needs and server RAM
max_execution_time = 300 ; 5 minutes for long-running tasks
max_input_vars = 3000 ; Important for large forms and configurations
upload_max_filesize = 64M
post_max_size = 64M

Remember to restart PHP-FPM after making changes: sudo systemctl restart phpX.Y-fpm.

MongoDB Performance Tuning on AWS

MongoDB is often used for Magento 2’s session storage, caching, and sometimes for catalog data. Optimizing MongoDB on AWS involves instance selection, storage configuration, and server-side tuning.

Instance and Storage Selection

Choose an EC2 instance type that balances CPU, RAM, and Network I/O. For MongoDB, instances with local NVMe SSDs (e.g., i3, i4i families) can offer superior performance due to low latency and high throughput compared to EBS. If using EBS, opt for gp3 or io2 volumes, provisioned IOPS as needed, and ensure sufficient throughput.

MongoDB Configuration (`mongod.conf`)

The primary configuration file is typically /etc/mongod.conf. Key parameters to tune:

# /etc/mongod.conf

storage:
  dbPath: /var/lib/mongodb
  journal:
    enabled: true
    commitInterval: 100 # Adjust based on write load and durability needs (lower = more frequent commits)
  engine: wiredTiger # Default and recommended
  wiredTiger:
    collectionConfig:
      blockCompressor: snappy # Or zstd for better compression, if CPU allows
    engineConfig:
      cacheSizeGB: 0.75 # Allocate 75% of available RAM to the WiredTiger cache. Adjust based on instance RAM.
      directoryForTempFiles: /var/tmp # Ensure this path is on fast storage

# network:
#   bindIp: 127.0.0.1 # Bind to localhost if only accessed locally, or specific IPs for replication

operationProfiling:
  mode: slowOp # Enable profiling for slow operations
  slowOpThresholdMs: 100 # Log operations slower than 100ms

# sharding: # If using sharded clusters
#   clusterRole: configsvr
#   configsvrFilePerDB: true

# replication: # If using replica sets
#   replSetName: rs0

# processManagement:
#   fork: true
#   pidFilePath: /var/run/mongodb/mongod.pid
#   logPath: /var/log/mongodb/mongod.log
#   logRotate:
#     sizeThresholdMB: 256
#     timeThresholdMin: 60

# systemLog:
#   destination: file
#   path: /var/log/mongodb/mongod.log
#   logAppend: true
#   verbosity: 0

cacheSizeGB: This is crucial. WiredTiger uses RAM for its cache. Allocate a significant portion of your instance’s RAM to it (e.g., 50-75%). Monitor cache hit rates using db.serverStatus().

journal.commitInterval: Lowering this value increases durability but also write I/O. Default is 100ms. For high-write workloads, you might experiment with values like 50ms, but monitor performance.

Indexing and Query Optimization

Ensure your MongoDB collections used by Magento (sessions, cache, etc.) have appropriate indexes. Use explain() on slow queries to identify missing indexes or inefficient query plans.

// Example: Check query performance for sessions collection
db.sessions.find({ updated_at: { $gt: ISODate("2023-10-27T00:00:00Z") } }).explain("executionStats")

// If slow, consider adding an index
db.sessions.createIndex({ updated_at: 1 })

Regularly review slow query logs (enabled via operationProfiling) to proactively identify and fix performance bottlenecks.

Monitoring and Maintenance

Use MongoDB’s built-in tools and AWS CloudWatch to monitor key metrics:

  • MongoDB Metrics: db.serverStatus() (cache hit rate, operations, connections, network), db.stats(), db.collection.stats().
  • AWS CloudWatch Metrics: CPU Utilization, Disk I/O (Read/Write Ops, Latency), Network I/O, Memory Utilization (if using CloudWatch Agent).

Perform regular backups and consider using AWS RDS for MongoDB if you prefer a managed solution, though self-hosting on EC2 often provides more granular control and potentially lower costs for high-performance needs.