Upgrading Apache 2.4 with HTTP/2 support on Rocky Linux 9: Resolving protocol negotiation timeouts

Prerequisites and Initial Setup

This guide assumes a fresh Rocky Linux 9 installation with root or sudo privileges. We’ll be upgrading Apache from its default version (likely 2.4.53 or similar) to a version that robustly supports HTTP/2, and crucially, addressing potential protocol negotiation timeouts that can plague initial deployments. Ensure your system is up-to-date:

• Run system updates:

sudo dnf update -y

We will leverage the official Apache Software Foundation (ASF) repositories for the most recent stable releases, which often include backported features and security fixes not yet in the default Rocky Linux repositories. First, install the necessary EPEL (Extra Packages for Enterprise Linux) repository, which is a prerequisite for the ASF Apache repository.

• Install EPEL:

sudo dnf install -y epel-release

Next, add the ASF Apache repository. This ensures we get the latest Apache httpd packages.

• Add ASF Apache repository:

sudo dnf install -y httpd httpd-tools mod_ssl mod_http2
sudo dnf config-manager --set-enabled crb

The `mod_http2` module is critical for HTTP/2 functionality. `mod_ssl` is required for HTTPS, which is the standard for HTTP/2 deployments. The `crb` repository is often a dependency for newer packages on RHEL-based systems.

Apache Configuration for HTTP/2

With Apache and its necessary modules installed, we need to configure Apache to enable HTTP/2. This primarily involves enabling the `mod_http2` module and configuring virtual hosts to use it. The default Apache configuration file is typically located at /etc/httpd/conf/httpd.conf, and virtual host configurations are in /etc/httpd/conf.d/.

First, ensure `mod_http2` is loaded. This is usually handled automatically by the package installation, but it’s good practice to verify. The configuration for `mod_http2` is typically in /etc/httpd/conf.modules.d/00-base.conf or a similar file. You should see a line like:

LoadModule http2_module modules/mod_http2.so

Now, let’s configure a virtual host for HTTP/2. Create or edit a virtual host configuration file, for example, /etc/httpd/conf.d/your_domain.conf. For HTTP/2 to function, you must use TLS/SSL. The key directive is Protocols h2 http/1.1, which tells Apache to prefer HTTP/2 (h2) and fall back to HTTP/1.1.

<VirtualHost *:443>
    ServerName your_domain.com
    DocumentRoot /var/www/your_domain

    SSLEngine on
    SSLCertificateFile /etc/pki/tls/certs/your_domain.crt
    SSLCertificateKeyFile /etc/pki/tls/private/your_domain.key
    # Optional: Add intermediate certificate if applicable
    # SSLCertificateChainFile /etc/pki/tls/certs/your_domain_chain.crt

    # Enable HTTP/2
    Protocols h2 http/1.1

    # Other SSL/TLS configurations (e.g., HSTS, cipher suites)
    Header always set Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    SSLProtocol             all -SSLv3 -TLSv1 -TLSv1.1
    SSLCipherSuite          ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
    SSLHonorCipherOrder     off
    SSLSessionTickets       off

    ErrorLog /var/log/httpd/your_domain-error.log
    CustomLog /var/log/httpd/your_domain-access.log combined

    # Ensure your DocumentRoot exists and has correct permissions
    <Directory /var/www/your_domain>
        Options Indexes FollowSymLinks
        AllowOverride All
        Require all granted
    </Directory>
</VirtualHost>

# Optional: Redirect HTTP to HTTPS
<VirtualHost *:80>
    ServerName your_domain.com
    Redirect permanent / https://your_domain.com/
</VirtualHost>

Replace your_domain.com, /var/www/your_domain, and the certificate paths with your actual domain and file locations. Ensure the certificate files are readable by the Apache user (typically `apache`).

Resolving Protocol Negotiation Timeouts

A common issue after enabling HTTP/2 is intermittent or persistent protocol negotiation timeouts. This often stems from network intermediaries (like load balancers, firewalls, or even client-side proxies) that are not fully compatible with or correctly configured for HTTP/2’s ALPN (Application-Layer Protocol Negotiation) handshake. Apache’s `mod_http2` relies on ALPN to establish an HTTP/2 connection over TLS.

If you observe errors like “The server sent an invalid HTTP/2 preface” or clients failing to connect with HTTP/2, the first step is to diagnose the network path. However, if you control the network path or suspect an issue with Apache’s handling of the handshake, you can adjust `mod_http2`’s behavior.

The `H2Direct` directive can be used to force HTTP/2 without ALPN, but this is generally discouraged as it bypasses standard TLS negotiation and can lead to compatibility issues. A more robust approach is to ensure your TLS configuration is optimal and that intermediaries are correctly configured.

However, for troubleshooting or specific scenarios, you can influence the ALPN negotiation. Apache’s `mod_http2` has a directive H2EarlyHints which can sometimes help. It enables sending early hints over HTTP/1.1 before the HTTP/2 connection is fully established. While not directly solving negotiation timeouts, it can improve perceived performance and sometimes smooth out handshake issues.

# In your httpd.conf or a conf.d file
<IfModule http2_module>
    H2Direct off # Default is off, ensure it's not set to on unless you know why
    H2StreamMaxMemSize 1M # Adjust if you have very large headers
    H2Push off # Consider disabling push if it causes issues, though it's a key HTTP/2 feature
    H2EarlyHints on # Enable early hints
</IfModule>

A more direct approach to mitigate negotiation timeouts, especially when dealing with older or misconfigured load balancers, is to ensure Apache’s TLS configuration is robust and compliant. The `SSLCipherSuite` and `SSLProtocol` directives shown in the virtual host example are crucial. Ensure you are using modern, secure cipher suites and protocols.

If timeouts persist, and you suspect an issue with ALPN negotiation specifically, you might need to investigate the client or intermediary. Tools like openssl s_client can help diagnose TLS handshake issues:

openssl s_client -connect your_domain.com:443 -alpn h2

If this command fails to negotiate `h2` or times out, the problem lies outside of Apache’s direct configuration and points to network infrastructure or client capabilities. If it successfully negotiates `h2`, the issue might be within Apache’s request handling after the initial connection.

Testing and Verification

After applying the configuration changes, restart Apache and test your setup. First, check Apache’s configuration syntax:

sudo apachectl configtest

If the syntax is OK, restart Apache:

sudo systemctl restart httpd

Verify Apache is running and listening on the correct ports:

sudo systemctl status httpd
sudo ss -tulnp | grep :80\|:443

Access your website via HTTPS in a modern browser (Chrome, Firefox, Edge). Most browsers will display an indicator (e.g., a padlock icon) and developer tools (Network tab) will show the protocol used for each resource. Look for “HTTP/2” or “h2” in the protocol column.

You can also use online tools like “HTTP/2 Test” by KeyCDN or “HTTP/2 Check” by Geekflare to verify HTTP/2 is enabled and functioning correctly. These tools simulate client requests and report on the protocol negotiation.

Advanced Considerations and Troubleshooting

Load Balancer Configuration: If Apache is behind a load balancer (e.g., AWS ELB/ALB, Nginx, HAProxy), ensure the load balancer is configured to support and negotiate HTTP/2 with the clients. Some older load balancer versions might only support HTTP/1.1 to the backend, or have specific settings for ALPN. For instance, AWS ALB supports HTTP/2 from the client to the ALB, but typically uses HTTP/1.1 to the backend instances unless specifically configured otherwise. Ensure your load balancer’s SSL termination and ALPN settings are correct.

Firewall Rules: Ensure that any firewalls between the client and the server (including cloud security groups or on-premise firewalls) allow traffic on port 443 and do not perform deep packet inspection that might interfere with the TLS handshake or ALPN negotiation.

Client-Side Issues: While less common in production, some older clients or specific network configurations might have issues with HTTP/2. Testing from different networks and devices can help isolate this.

Apache Worker MPM: For high-concurrency scenarios with HTTP/2, consider the `mpm_event` module over `mpm_prefork`. `mpm_event` is designed for asynchronous I/O and is generally more performant for keep-alive connections and concurrent requests, which are hallmarks of HTTP/2. Ensure `mpm_event` is enabled and `mpm_prefork` is disabled.

# Check current MPM
sudo httpd -V | grep "Server MPM"

# If prefork is enabled, disable it and enable event
sudo dnf module disable httpd:latest-freeworld # Or your specific stream
sudo dnf module enable httpd:latest # Or the stream that provides event
sudo dnf install httpd-devel # May be needed to rebuild modules
sudo systemctl restart httpd

Logging: Increase Apache’s log level for `mod_http2` if you are experiencing persistent issues. You can add LogLevel debug ssl:info to your main Apache configuration or virtual host to get more verbose output, which can be invaluable for diagnosing handshake problems.

# In httpd.conf or your vhost conf
LogLevel debug ssl:info

Remember to restart Apache after changing the LogLevel and check /var/log/httpd/error_log for detailed messages.