Tuning Nginx Reverse Proxy caching, buffering, and timeouts on Rocky Linux 9 for Discourse platform proxying

Nginx Reverse Proxy Tuning for Discourse on Rocky Linux 9

This document details advanced Nginx configuration strategies for optimizing caching, buffering, and timeouts when acting as a reverse proxy for a Discourse instance on Rocky Linux 9. These settings are critical for ensuring high availability, responsiveness, and efficient resource utilization under heavy load.

Understanding Nginx Buffering

Nginx uses buffers to temporarily store data received from upstream servers or sent to clients. Proper buffer sizing and management are crucial for handling large responses, preventing client timeouts, and optimizing network throughput. For Discourse, which often deals with dynamic content, large attachments, and long-polling requests, these settings are paramount.

`proxy_buffering` Directive

The `proxy_buffering` directive controls whether Nginx buffers responses from the upstream server. By default, it’s `on`. For Discourse, keeping it `on` is generally beneficial as it allows Nginx to handle slow upstream responses without blocking the client connection. However, in scenarios with extremely large, infrequently accessed static assets, one might consider turning it off for specific locations to stream directly, but this is rarely optimal for Discourse’s dynamic nature.

`proxy_buffer_size` and `proxy_buffers`

These directives define the size and number of buffers used for proxying. `proxy_buffer_size` sets the size of the first buffer, which is critical for receiving the response headers. `proxy_buffers` defines the number and size of subsequent buffers for the response body. For Discourse, which can have complex headers and potentially large JSON payloads or even file uploads/downloads proxied, increasing these values can prevent `502 Bad Gateway` errors or incomplete responses.

A common starting point for a busy Discourse instance might be:

proxy_buffer_size   128k;
proxy_buffers     4 256k;
proxy_busy_buffers_size 256k;

Explanation:

• proxy_buffer_size 128k;: Allocates a larger initial buffer for headers, accommodating potentially verbose HTTP headers from Discourse or its underlying components.
• proxy_buffers 4 256k;: Configures Nginx to use up to 4 buffers, each 256KB in size, for the response body. This provides a total of 1MB of buffer space for the response body.
• proxy_busy_buffers_size 256k;: Sets the maximum size for buffers that are busy processing data. This prevents Nginx from allocating too many buffers when the upstream is slow, which could lead to memory exhaustion.

Adjust these values based on observed traffic patterns and response sizes. Monitoring Nginx error logs and using tools like netstat or ss to inspect buffer usage can guide further tuning.

Nginx Caching Strategies

While Discourse itself employs sophisticated caching mechanisms, Nginx can provide an additional layer of caching for static assets and even certain dynamic responses. This can significantly offload the Discourse application servers.

`proxy_cache` Directive

The `proxy_cache` directive enables Nginx’s proxy caching. You’ll need to define cache zones and keys.

# Define cache zone
proxy_cache_path /var/cache/nginx/discourse levels=1:2 keys_zone=discourse_cache:100m inactive=60m max_size=10g;

# In your server block or location block for Discourse
proxy_cache discourse_cache;
proxy_cache_key "$scheme$request_method$host$request_uri";
proxy_cache_valid 200 302 10m;  # Cache successful responses for 10 minutes
proxy_cache_valid 404 1m;       # Cache 404s for 1 minute
proxy_cache_use_stale error timeout invalid_header updating http_500 http_502 http_503 http_504;
proxy_cache_lock on; # Prevent multiple requests for the same uncached resource from hitting upstream simultaneously
proxy_cache_lock_timeout 5s;

Explanation:

• proxy_cache_path: Defines the directory for cache files, cache levels, a shared memory zone name (`discourse_cache`) with a size of 100MB, and cache expiration (inactive entries removed after 60 minutes). `max_size=10g` limits the total disk usage.
• proxy_cache discourse_cache;: Activates the defined cache zone.
• proxy_cache_key: A unique key for each cached item, typically based on the request details.
• proxy_cache_valid: Specifies how long different HTTP status codes should be cached. For Discourse, caching successful responses (200, 302) for a short duration (e.g., 10 minutes) can be effective.
• proxy_cache_use_stale: Instructs Nginx to serve stale cached content if the upstream server is unavailable or returns specific errors. This is crucial for maintaining availability.
• proxy_cache_lock: Prevents “thundering herd” problems where multiple identical requests for a missing cache item hit the upstream simultaneously.

Important Considerations for Discourse Caching:

• Dynamic Content: Avoid caching highly dynamic or personalized content. Focus on static assets (images, CSS, JS) and potentially public forum pages that don’t change frequently.
• Cache Invalidation: Discourse has its own cache invalidation mechanisms. Nginx caching should complement, not conflict with, these. Consider using `X-Accel-Expires` or `Cache-Control` headers from Discourse to control Nginx’s caching behavior.
• Cache Purging: Implement a strategy for purging Nginx cache when Discourse content is updated. This can be done via Nginx’s `proxy_cache_purge` directive (requires a separate module or specific Nginx Plus features) or by manually clearing the cache directory (less ideal for production).

Timeout Management

Timeouts are critical for preventing Nginx from holding onto connections indefinitely and for ensuring clients receive timely responses or appropriate error messages. For a Discourse instance, especially one with background jobs or long-running operations, these need careful tuning.

`proxy_connect_timeout`, `proxy_send_timeout`, `proxy_read_timeout`

These directives control how long Nginx waits for different stages of the proxy connection.

proxy_connect_timeout 60s;
proxy_send_timeout    60s;
proxy_read_timeout    300s; # Increased for potentially long-running requests like report generation or large uploads

Explanation:

• proxy_connect_timeout 60s;: The timeout for establishing a connection with the upstream server.
• proxy_send_timeout 60s;: The timeout for transmitting a request to the upstream server.
• proxy_read_timeout 300s;: The timeout for reading a response from the upstream server. This is often the most critical for Discourse. Some operations, like generating large reports or processing complex queries, can take longer than the default (often 60s). Increasing this to 5 minutes (300s) can prevent premature timeouts for legitimate long-running requests.

Caution: Setting `proxy_read_timeout` too high can tie up worker processes and potentially mask underlying performance issues in the Discourse application itself. Monitor your Discourse application’s performance and error logs to determine an appropriate value.

`client_header_timeout` and `client_body_timeout`

These control timeouts for client requests.

client_header_timeout 60s;
client_body_timeout   60s;

These are generally less critical for Discourse unless you’re experiencing issues with clients sending malformed or excessively slow requests. The default values are often sufficient.

Rocky Linux 9 Specifics and System-Level Tuning

While Nginx configuration is largely platform-agnostic, the underlying operating system can impact performance. Ensure your Rocky Linux 9 system is optimized.

File Descriptors and Network Stack

High traffic can exhaust file descriptors or overwhelm network buffers. Adjusting system limits is essential.

# Check current limits
ulimit -n

# Edit /etc/security/limits.conf to increase limits for the Nginx user (e.g., 'nginx')
# Add these lines (or adjust existing ones):
# nginx soft nofile 65536
# nginx hard nofile 131072

# Edit /etc/nginx/nginx.conf to set worker_rlimit_nofile
# Inside the 'events' block:
# events {
#     worker_connections 4096; # Adjust based on your system's RAM and CPU
#     worker_rlimit_nofile 131072; # Match or exceed the user limit
# }

# Apply changes by restarting Nginx and potentially logging out/in for user limits
sudo systemctl restart nginx

Additionally, tune network parameters in /etc/sysctl.conf:

# Example sysctl.conf additions for network performance
net.core.somaxconn = 4096
net.ipv4.tcp_max_syn_backlog = 2048
net.ipv4.tcp_fin_timeout = 30
net.ipv4.tcp_tw_reuse = 1
net.ipv4.ip_local_port_range = 1024 65535

# Apply changes
sudo sysctl -p

Monitoring and Iteration

Tuning is an iterative process. Continuously monitor your Nginx access and error logs, Discourse application logs, and system metrics (CPU, memory, network I/O, disk I/O). Use tools like Prometheus with `nginx-exporter`, Grafana, and Discourse’s built-in admin dashboards to identify bottlenecks and validate the impact of your changes.

Key metrics to watch:

• Nginx 5xx error rates (especially 502, 504)
• Nginx cache hit/miss ratios
• Upstream response times
• Client request latency
• System resource utilization

By carefully configuring Nginx buffering, caching, and timeouts, and complementing it with system-level optimizations on Rocky Linux 9, you can significantly enhance the performance and reliability of your Discourse platform.