Troubleshooting SSL Certificate expiration and automatic renewal failures using Let’s Encrypt and certbot on RHEL 9

Diagnosing Let’s Encrypt Expiration and Renewal Failures on RHEL 9

When SSL certificates managed by Let’s Encrypt and `certbot` unexpectedly expire or fail to renew automatically on RHEL 9 systems, it often points to subtle configuration drift, permission issues, or environmental changes. This guide provides a systematic approach to diagnosing and resolving these critical failures, ensuring uninterrupted HTTPS service.

Verifying Current Certificate Status and Expiration Dates

The first step is to confirm the actual expiration status of your certificates. `certbot` stores certificate information in /etc/letsencrypt/live/. We can use OpenSSL to inspect the certificate details directly.

To check the expiration date of a specific certificate (e.g., for example.com), execute the following command:

sudo openssl x509 -in /etc/letsencrypt/live/example.com/fullchain.pem -noout -dates

This will output the notBefore and notAfter dates. If notAfter is in the past, the certificate has expired. If it’s within the next 30 days, an automatic renewal should have been triggered.

Examining Certbot Renewal Logs

certbot logs its renewal attempts and any errors encountered. These logs are invaluable for pinpointing the root cause of failure. The primary log file is located at /var/log/letsencrypt/letsencrypt.log.

To view recent renewal attempts and potential errors, you can use grep and tail:

sudo tail -n 500 /var/log/letsencrypt/letsencrypt.log | grep -i "renewal\|error\|fail"

Look for specific error messages such as:

Timeout during connect
ACME challenge failed
Rate limit exceeded
Permissions denied
Could not bind to IPv4 or IPv6.

Additionally, individual certificate renewal logs can be found in /var/log/letsencrypt/, often named like /var/log/letsencrypt/YYYYMMDD_HHMMSS.log. These provide more granular details for specific renewal runs.

Manually Triggering a Renewal and Debugging

To simulate the renewal process and capture real-time output, run certbot renew with the --dry-run flag. This attempts renewal without actually issuing new certificates, making it safe for testing.

sudo certbot renew --dry-run

If the dry run also fails, the output will provide immediate feedback. Common issues revealed here include:

Web Server Configuration Issues

certbot often relies on the web server (Nginx or Apache) to respond to ACME challenges (HTTP-01 or TLS-ALPN-01). If the web server is not running, misconfigured, or not accessible on the required ports (80/443), renewals will fail.

For Nginx:

sudo systemctl status nginx
sudo nginx -t

Ensure Nginx is active and its configuration syntax is valid. Verify that the server_name directive in your Nginx configuration correctly matches the domain(s) for which you are requesting a certificate, and that it’s listening on port 80 for HTTP-01 challenges.

server {
    listen 80;
    server_name example.com www.example.com;

    # ... other directives ...

    location ~ /.well-known/acme-challenge/ {
        allow all;
        root /usr/share/nginx/html; # Or the directory certbot is configured to use
    }
}

For Apache:

sudo systemctl status httpd
sudo apachectl configtest

Similar to Nginx, confirm Apache is running and its configuration is sound. The VirtualHost for port 80 must correctly resolve the domain and have a DocumentRoot or alias pointing to a location where certbot can place the challenge files.

Listen 80
<VirtualHost *:80>
    ServerName example.com
    ServerAlias www.example.com
    DocumentRoot /var/www/html

    <Directory /var/www/html>
        AllowOverride None
        Require all granted
    </Directory>

    Alias /.well-known/acme-challenge/ /var/www/certbot/
    <Directory "/var/www/certbot">
        AllowOverride None
        Require all granted
    </Directory>
</VirtualHost>

Firewall and Network Accessibility

Let’s Encrypt’s servers must be able to reach your server on port 80 (for HTTP-01) or port 443 (for TLS-ALPN-01). Ensure that your firewall (firewalld on RHEL 9) and any external network firewalls or security groups allow inbound traffic on these ports from the Let’s Encrypt IP ranges.

sudo firewall-cmd --list-all
sudo firewall-cmd --permanent --add-service=http
sudo firewall-cmd --permanent --add-service=https
sudo firewall-cmd --reload

If your server is behind a load balancer or NAT, ensure that port 80/443 traffic is correctly forwarded to the `certbot` host.

Permissions and Ownership

certbot needs read access to its configuration and certificate files (/etc/letsencrypt/) and write access to the web server’s document root for the HTTP-01 challenge. The renewal process also involves writing new certificate files.

ls -ld /etc/letsencrypt/
ls -l /etc/letsencrypt/live/example.com/
ls -ld /usr/share/nginx/html/ # Or your web server's document root

Ensure the user running the certbot renewal process (typically root via cron or systemd timer) has appropriate permissions. If using the webroot authenticator, the web server user (e.g., nginx or apache) must be able to write to the challenge directory if `certbot` is configured to do so, or `certbot` must have read access to the directory where it places challenge files.

Rate Limits

Let’s Encrypt enforces rate limits to prevent abuse. The most common limit is 50 certificates per registered domain per week. If you’ve been issuing or renewing certificates frequently, you might hit this limit. The logs will usually indicate this with a message like “Rate limit exceeded.”

If you suspect a rate limit issue, wait for the limit to reset (typically 7 days) or consult the Let’s Encrypt rate limit documentation.

Automated Renewal Configuration (Systemd Timers/Cron Jobs)

On RHEL 9, certbot typically installs a systemd timer or a cron job to handle automatic renewals. It’s crucial to verify that this mechanism is active and running correctly.

Systemd Timer:

sudo systemctl list-timers certbot.timer
sudo systemctl status certbot.timer
sudo journalctl -u certbot.timer

The certbot.timer unit is responsible for triggering the renewal. If it’s inactive or failed, check its status and logs. The associated service unit is usually certbot.service.

Cron Job:

sudo crontab -l -u root

Look for a line similar to:

0 3,15 * * * certbot renew --quiet

Ensure the cron job is present and that the cron daemon is running.

Advanced Troubleshooting: Specific Authenticator Issues

The method certbot uses to prove control over a domain (the authenticator) can also be a source of failure.

Webroot Authenticator Problems

If you use the --webroot flag, certbot places challenge files in a specified directory (e.g., /var/www/html/.well-known/acme-challenge/) and expects your web server to serve them. Ensure:

The webroot-path specified in your certbot configuration (or command line) is correct.
The web server is configured to serve files from that path.
The directory and its parent directories exist and have correct permissions for the web server to read from.
No redirects or access controls are preventing Let’s Encrypt’s servers from accessing the challenge files.

You can test this manually by creating a dummy file:

sudo mkdir -p /var/www/html/.well-known/acme-challenge/
echo "test" | sudo tee /var/www/html/.well-known/acme-challenge/test.txt
sudo curl http://example.com/.well-known/acme-challenge/test.txt

If this curl command fails or returns an error, your web server is not serving the file correctly.

Standalone/Nginx/Apache Plugin Issues

When using plugins like --standalone, --nginx, or --apache, certbot temporarily modifies your web server configuration or starts its own server. Ensure:

Port 80 (for standalone/http-01) or 443 (for tls-alpn-01) is not already in use by another process.
The plugin correctly identifies and modifies your web server configuration.
The web server is able to restart or reload gracefully after certbot makes changes.

For --standalone, check if port 80 is free:

sudo ss -tulnp | grep :80

If another process is using port 80, you’ll need to stop it or use a different authenticator method.

Resolving Specific Error Scenarios

“Could not bind to IPv4 or IPv6”

This error, often seen with the standalone authenticator, means another process is already listening on the required port (usually 80). Use ss -tulnp | grep :80 to identify and stop the conflicting process.

“Timeout during connect” or “Connection refused”

This indicates network reachability issues. Verify:

Firewall rules (firewalld, cloud provider security groups, network firewalls) allow inbound traffic on port 80/443.
The web server is running and listening on the correct interfaces and ports.
DNS records for the domain correctly resolve to the server’s public IP address.
No proxies or load balancers are interfering with direct access to port 80/443.

Test connectivity from an external network using tools like telnet or nc:

telnet example.com 80

“ACME challenge failed”

This is a generic error that can stem from many underlying causes. Re-examine the detailed logs from /var/log/letsencrypt/ and the output of certbot renew --dry-run. Common culprits include incorrect web server configuration for serving challenge files, firewall blocks, or DNS propagation delays.

Final Checks and Best Practices

After resolving an issue, always perform a manual renewal dry run and then a full manual renewal to confirm the fix:

sudo certbot renew --dry-run
sudo certbot renew

Monitor your certificate expiration dates regularly. Consider setting up external monitoring that checks certificate expiry dates via HTTP headers or by querying the certificate itself. Ensure your certbot installation is kept up-to-date with sudo dnf update certbot.