Vengala Vinay

Having 9+ Years of Experience in Software Development

Home » The Ultimate DevOps Playbook: Tuning Nginx, Gunicorn/FPM, and Elasticsearch on DigitalOcean for PHP

The Ultimate DevOps Playbook: Tuning Nginx, Gunicorn/FPM, and Elasticsearch on DigitalOcean for PHP

Nginx as a High-Performance Frontend Proxy

Nginx is the de facto standard for serving static assets and acting as a reverse proxy for dynamic applications. For a PHP application, this typically means proxying requests to a PHP process manager like Gunicorn (if using a Python-based framework like Django/Flask with PHP integration) or PHP-FPM. Our primary goal here is to optimize Nginx for low latency and high concurrency.

Core Nginx Configuration Tuning

We’ll focus on key directives within the nginx.conf file or within a specific site’s configuration file (e.g., /etc/nginx/sites-available/your_app.conf).

Worker Processes and Connections

The worker_processes directive determines how many worker processes Nginx will spawn. Setting this to auto is generally recommended, allowing Nginx to detect the number of CPU cores. The worker_connections directive sets the maximum number of simultaneous connections that each worker process can handle. A common starting point is 1024, but this can be increased significantly based on server resources and expected load.

Event Handling (epoll)

For Linux systems, using the epoll event handling mechanism is crucial for high concurrency. Ensure your events block is configured as follows:

events {
    worker_connections 4096; # Increased from default 1024
    multi_accept on;
    use epoll;
}

Keepalive Connections

Enabling keepalive connections reduces the overhead of establishing new TCP connections for subsequent requests from the same client. The keepalive_timeout and keepalive_requests directives control this behavior. A reasonable timeout is 65 seconds, and a limit of 100 requests per connection is a good balance.

http {
    # ... other http directives ...

    keepalive_timeout 65;
    keepalive_requests 100;

    # ... server blocks ...
}

Buffering and Timeouts

Tuning buffer sizes and timeouts can prevent issues with large requests or slow backend responses. client_body_buffer_size, client_header_buffer_size, and large_client_header_buffers are important. For timeouts, client_header_timeout, client_body_timeout, and send_timeout should be set appropriately to avoid premature connection closures but also to free up resources from stalled clients.

http {
    # ... other http directives ...

    client_body_buffer_size       100k;
    client_header_buffer_size     10k;
    large_client_header_buffers   4 32k;

    client_header_timeout         60s;
    client_body_timeout           60s;
    send_timeout                  60s;

    # ... server blocks ...
}

Gzip Compression

Enabling Gzip compression significantly reduces bandwidth usage and improves perceived load times for text-based assets (HTML, CSS, JS, JSON). Configure it within the http block or specific server blocks.

http {
    # ... other http directives ...

    gzip on;
    gzip_vary on;
    gzip_proxied any;
    gzip_comp_level 6; # Compression level (1-9)
    gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

    # ... server blocks ...
}

Proxying to PHP-FPM

When using PHP-FPM, Nginx acts as a reverse proxy to the FPM process. The location block for your PHP files will handle this. Ensure fastcgi parameters are correctly set for optimal performance.

server {
    # ... server configuration ...

    location ~ \.php$ {
        include snippets/fastcgi-php.conf;
        # If using TCP socket:
        # fastcgi_pass 127.0.0.1:9000;
        # If using Unix socket (recommended for performance):
        fastcgi_pass unix:/var/run/php/php7.4-fpm.sock; # Adjust version as needed
        fastcgi_index index.php;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        include fastcgi_params;
    }

    # ... other locations ...
}

Proxying to Gunicorn (for Python/PHP integration)

If your PHP application relies on a Python backend managed by Gunicorn, Nginx will proxy requests to Gunicorn. This is less common for pure PHP but can occur in hybrid architectures.

server {
    # ... server configuration ...

    location / {
        proxy_pass http://127.0.0.1:8000; # Assuming Gunicorn is running on port 8000
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_read_timeout 300s; # Increase timeout for potentially long-running Python tasks
        proxy_connect_timeout 75s;
    }

    # ... other locations ...
}

PHP-FPM Optimization for High Throughput

PHP-FPM (FastCGI Process Manager) is the standard way to run PHP applications in production. Its configuration directly impacts application performance and resource utilization.

php-fpm.conf and Pool Configuration

The main configuration file is typically /etc/php/X.Y/fpm/php-fpm.conf (where X.Y is your PHP version). Pool configurations are in /etc/php/X.Y/fpm/pool.d/www.conf (or a custom pool name). We’ll focus on tuning the pool configuration.

Process Management (pm)

The pm directive controls how PHP-FPM manages worker processes. The most common and recommended settings are:

  • pm = dynamic: PHP-FPM will dynamically manage the number of child processes based on pm.max_children, pm.start_servers, pm.min_spare_servers, and pm.max_spare_servers. This is generally the best option for most web applications.
  • pm = static: PHP-FPM will maintain a fixed number of child processes defined by pm.max_children. This can offer slightly more predictable performance but might be less efficient if traffic fluctuates wildly.
  • pm = ondemand: Processes are spawned only when a request is received and killed after a certain idle time. This saves memory but can introduce latency for the first request after an idle period.

For dynamic management, the key parameters are:

; /etc/php/X.Y/fpm/pool.d/www.conf
pm = dynamic
pm.max_children = 100       ; Max number of children that can be alive at the same time.
pm.start_servers = 10       ; Number of children created at first run.
pm.min_spare_servers = 5    ; Number of children that should be kept active on average.
pm.max_spare_servers = 20   ; Number of children that should be kept active on average.
pm.process_idle_timeout = 10s; Value '0' means recursive delete. Default value is 10s.
pm.max_requests = 500       ; Max number of requests each child process should execute. Set to 0 to disable.

Tuning Strategy: Start with values based on your server’s RAM. A common rule of thumb is that each PHP-FPM worker can consume 20-50MB of RAM. If you have 4GB RAM, you might aim for pm.max_children around 80-100. Monitor memory usage and adjust. pm.max_requests is crucial for preventing memory leaks in long-running applications or poorly written code; a value of 500-1000 is typical.

Process Manager Settings for Static

If using pm = static, the configuration simplifies:

; /etc/php/X.Y/fpm/pool.d/www.conf
pm = static
pm.max_children = 100       ; Fixed number of children
pm.max_requests = 0         ; Disable max_requests for static if you are confident in memory usage

Listen Configuration

Ensure the listen directive matches what Nginx is configured to use. Using a Unix socket is generally faster than a TCP socket for local communication.

; /etc/php/X.Y/fpm/pool.d/www.conf
listen = /var/run/php/php7.4-fpm.sock ; Use Unix socket (recommended)
; listen = 127.0.0.1:9000          ; Or use TCP socket

Environment Variables

For performance-critical applications, setting environment variables directly in the pool configuration can be more efficient than relying on Nginx or other mechanisms.

; /etc/php/X.Y/fpm/pool.d/www.conf
env[MY_APP_ENV] = production
env[DATABASE_URL] = mysql://user:pass@host:port/dbname

PHP Opcode Caching

Opcode caching is non-negotiable for PHP performance. OPcache is built into PHP and should be enabled and tuned in your php.ini file (e.g., /etc/php/X.Y/fpm/php.ini).

; /etc/php/X.Y/fpm/php.ini
opcache.enable=1
opcache.enable_cli=1 ; Enable for CLI scripts too
opcache.memory_consumption=128 ; MB, adjust based on your application's code size
opcache.interned_strings_buffer=16
opcache.max_accelerated_files=10000 ; Number of files to cache. Adjust based on your project size.
opcache.revalidate_freq=60 ; Check for file updates every 60 seconds (0 to disable, not recommended for development)
opcache.validate_timestamps=1 ; Set to 0 in production for maximum performance if you deploy manually and know when code changes.
opcache.save_comments=1
opcache.load_comments=1
opcache.enable_file_override=0
opcache.optimization_level=0xFFFFFFFF ; Default optimization level

Tuning Strategy: opcache.memory_consumption should be large enough to hold all your application’s PHP files. opcache.max_accelerated_files should also be sufficient. Monitor opcache_get_status() output to see cache hit rates and memory usage.

Elasticsearch Performance Tuning on DigitalOcean

Elasticsearch is a powerful search and analytics engine. Optimizing it involves JVM tuning, shard management, and indexing strategies.

JVM Heap Size

The most critical JVM setting is the heap size. It should be set in jvm.options (e.g., /etc/elasticsearch/jvm.options). A common recommendation is to set it to 50% of available RAM, but not exceeding 30-32GB due to compressed ordinary object pointers (compressed oops).

-Xms4g
-Xmx4g

Tuning Strategy: For a DigitalOcean droplet, if you have a 16GB RAM droplet, start with 8GB heap. If you have 64GB, you might go up to 30GB. Monitor heap usage closely. If the JVM is constantly garbage collecting (high GC activity), you might need to increase the heap or optimize your queries/indexing.

Shard Allocation and Sizing

The number and size of shards significantly impact performance. Too many small shards or too few large shards can be detrimental.

Number of Shards

A general guideline is to aim for shards between 10GB and 50GB. The number of primary shards should ideally be a multiple of the number of nodes in your cluster, but for a single-node setup, this is less critical. Avoid having hundreds of thousands of shards.

Shard Routing

For specific use cases, you might want to control shard routing. For example, if you frequently query documents by a specific user ID, you could route all documents for that user to the same shard.

{
  "settings": {
    "index": {
      "number_of_shards": 3,
      "number_of_replicas": 1,
      "routing.allocation.require.box_type": "hot"
    }
  }
}

The above example sets 3 primary shards and 1 replica, and enforces that documents with box_type “hot” must be on specific nodes (if you have node attributes configured).

Indexing Performance

Optimizing indexing involves controlling refresh intervals and bulk API usage.

Refresh Interval

The refresh interval controls how often new documents become visible for search. A shorter interval means near real-time search but higher indexing overhead. For bulk indexing, it’s often beneficial to increase this.

PUT /my_index/_settings
{
  "index": {
    "refresh_interval": "30s"
  }
}

During heavy indexing, you might set it to -1 and then manually refresh when needed.

Bulk API

Always use the Bulk API for indexing multiple documents. The optimal bulk size depends on your data and cluster, but starting with 5-15MB per bulk request is common.

POST _bulk
{ "index" : { "_index" : "test", "_id" : "1" } }
{ "field1" : "value1" }
{ "index" : { "_index" : "test", "_id" : "2" } }
{ "field1" : "value2" }

Search Performance

Search performance is heavily influenced by query complexity, data structure, and caching.

Fielddata and Doc Values

For aggregations and sorting on text fields, Elasticsearch uses either fielddata (memory-intensive, deprecated) or doc values (disk-based, default and recommended). Ensure doc values are enabled for fields you intend to aggregate or sort on.

PUT /my_index
{
  "mappings": {
    "properties": {
      "my_text_field": {
        "type": "text",
        "fielddata": false,  // Ensure fielddata is off
        "doc_values": true   // Ensure doc values are on
      }
    }
  }
}

Query Optimization

Avoid leading wildcards in queries (e.g., *term). Use appropriate query types (e.g., term for exact matches, match for full-text search). Profile slow queries using the Profile API.

Monitoring and Diagnostics

Regular monitoring is key. Use tools like:

  • Nginx: nginx -s reload, tail -f /var/log/nginx/access.log, tail -f /var/log/nginx/error.log, Nginx Amplify, Prometheus + Grafana.
  • PHP-FPM: systemctl status php7.4-fpm, tail -f /var/log/php7.4-fpm.log, php-fpm -t (for syntax check), php-fpm -y /etc/php/X.Y/fpm/pool.d/www.conf -i (to inspect pool configuration).
  • Elasticsearch: Elasticsearch’s own monitoring APIs (_cat APIs, _nodes/stats, _cluster/health), Metricbeat, Kibana’s Stack Monitoring.

On DigitalOcean, leverage their monitoring tools for droplet-level CPU, RAM, and network usage. Correlate these with application-specific metrics.