Upgrading RHEL 8 to RHEL 9: Migrating High-Throughput Perl Dancer2 Legacy Apps without Core API Breaking Changes

Pre-Upgrade Assessment and Preparation

Migrating a production RHEL 8 environment, especially one hosting high-throughput Perl Dancer2 applications, to RHEL 9 requires meticulous planning. The primary concern for legacy Dancer2 applications is the potential for breaking changes in core Perl modules or system libraries that these applications depend on. RHEL 9, while offering significant advancements, also introduces updated package versions. A thorough assessment of the current RHEL 8 environment is paramount before initiating any upgrade process.

Begin by cataloging all installed Perl modules, particularly those critical to your Dancer2 application’s functionality. Tools like cpanm or perl -MApp::cpanminus -e 'App::cpanminus::list()' can provide a comprehensive list. Pay close attention to modules that might have C extensions or rely on specific system libraries (e.g., OpenSSL, libxml2, libxslt, database connectors).

System Health Check and Snapshotting

Before proceeding with any in-place upgrade, ensure the RHEL 8 system is in a stable and healthy state. This includes:

• Verifying filesystem integrity: fsck -n /dev/your_root_partition (run on unmounted partitions if possible, or use read-only checks).
• Checking system logs for recurring errors: journalctl -p err -b.
• Ensuring all critical services are running and accessible.
• Performing a full system backup or snapshot. For virtualized environments, this is a critical step. For bare-metal, consider tools like rsync to a separate backup location or LVM snapshots if applicable.

For LVM snapshots, the process would look something like this:

# Identify your root logical volume (e.g., /dev/mapper/vg_system-lv_root)
sudo lvdisplay

# Create a snapshot (e.g., 10GB snapshot of lv_root)
sudo lvcreate -L 10G -s -n root_snapshot /dev/vg_system/lv_root

# Mount the snapshot for backup (optional, but good for verification)
sudo mkdir /mnt/snapshot
sudo mount /dev/vg_system/root_snapshot /mnt/snapshot
# Perform backup from /mnt/snapshot
sudo umount /mnt/snapshot
sudo rmdir /mnt/snapshot

# Remove the snapshot after successful upgrade and verification
sudo lvremove /dev/vg_system/root_snapshot

In-Place Upgrade Procedure: Leapp Framework

RHEL 9 supports in-place upgrades from RHEL 8 using the leapp framework. This is the recommended and most streamlined approach for production systems.

First, ensure your RHEL 8 system is fully updated:

sudo dnf update -y
sudo reboot

Next, install the leapp utility and its RHEL 9 data package:

sudo dnf install -y leapp leapp-repository
sudo reboot # Reboot is often required after installing leapp

Now, run the pre-upgrade assessment. This is crucial for identifying potential issues before the actual upgrade begins. leapp will generate a report detailing any blockers or risks.

sudo leapp preupgrade

Review the generated report, typically located at /var/log/leapp/leapp-report.txt and /var/log/leapp/leapp-report.json. Pay close attention to sections related to Perl, specific modules, or any deprecated packages. Address any identified issues. Common issues might involve unsupported Python versions (though less critical for Perl apps, it can affect system tools), deprecated kernel modules, or specific library versions.

If the pre-upgrade assessment passes without critical blockers, you can proceed with the upgrade. Ensure your system is connected to Red Hat Subscription Management and has access to the RHEL 9 repositories.

sudo leapp upgrade

The system will reboot into a special upgrade environment. The actual upgrade process will then commence. This can take a significant amount of time depending on the system’s complexity and disk speed. Do not interrupt this process.

Post-Upgrade Verification and Dancer2 Application Testing

After the upgrade completes and the system reboots into RHEL 9, the first step is to verify the system’s integrity and basic functionality.

Check the system version:

cat /etc/redhat-release

Verify that all essential services are running:

sudo systemctl status --all

Crucially, test your Perl Dancer2 applications. This involves more than just a simple `curl` to the endpoint. For high-throughput applications, a comprehensive testing suite is essential.

Perl Module Compatibility Check

RHEL 9 ships with newer versions of core Perl modules. While leapp attempts to handle module transitions, manual verification is recommended. The most common breaking changes for Dancer2 apps often stem from updates to modules like Plack, HTTP::Server::PSGI, or underlying JSON/XML parsers.

Re-installing your application’s dependencies using cpanm in the RHEL 9 environment can help identify missing or incompatible modules. It’s advisable to do this in a staging or development environment first.

# Navigate to your application's project directory
cd /path/to/your/dancer2/app

# If using a local cpanm, ensure it's up-to-date
# curl -L https://cpanmin.us | perl - --self-update

# Re-install dependencies (assuming you have a cpanfile or similar)
# If you have a cpanfile:
cpanm --installdeps .

# If you have a list of modules in a file (e.g., modules.txt):
# cpanm --installdeps -n modules.txt

# Or manually install critical modules and check for errors:
cpanm Plack HTTP::Server::PSGI JSON::XS YAML::XS

Monitor the output of cpanm closely for any compilation errors or warnings. These often indicate missing system development libraries (e.g., perl-devel, gcc, specific library headers like openssl-devel).

Application Functional and Performance Testing

Execute your existing test suite for the Dancer2 application. This should include:

• Unit tests
• Integration tests
• End-to-end tests

Beyond functional correctness, performance is critical for high-throughput applications. Use load testing tools to simulate production traffic and compare performance metrics (latency, throughput, error rates) against the RHEL 8 baseline.

Tools like wrk, ab (ApacheBench), or k6 can be used for load generation. Monitor application logs and system resource utilization (CPU, memory, network I/O) during these tests.

# Example using wrk
wrk -t4 -c100 -d30s --latency http://your-dancer2-app.local/api/endpoint

If performance regressions are observed, investigate the following:

• Perl interpreter version differences (RHEL 9 might have a newer default Perl).
• Changes in underlying libraries (e.g., OpenSSL, libxml2).
• Network stack tuning parameters.
• Database connection pooling and performance.
• Application-specific caching mechanisms.

Troubleshooting Common Issues

Perl Module Compilation Failures:

If cpanm fails to install a module, it’s usually due to missing development headers or libraries. Use dnf search to find the corresponding `-devel` package. For example, if a module requires OpenSSL headers:

sudo dnf install openssl-devel

Always ensure you have the base development tools installed:

sudo dnf groupinstall "Development Tools"
sudo dnf install perl-devel

Dancer2 Application Startup Errors:

Review the application logs carefully. Errors might indicate:

• Incorrect configuration file paths (due to system changes).
• Missing environment variables.
• Issues with the PSGI server (e.g., Starman, Plackup) not being found or configured correctly.
• Changes in how Perl modules are loaded or accessed.

Ensure your systemd service file for the Dancer2 application is correctly configured for RHEL 9. Paths to executables, user/group permissions, and environment variable loading might need adjustments.

[Unit]
Description=My Dancer2 Application
After=network.target

[Service]
User=appuser
Group=appgroup
WorkingDirectory=/path/to/your/dancer2/app
ExecStart=/usr/bin/perl /path/to/your/dancer2/app/app.pl --port 5000 --host 0.0.0.0
Restart=on-failure
Environment="MY_APP_ENV=production"
# Ensure correct paths for cpanm or installed modules if not in standard @INC
# Environment="PERL5LIB=/path/to/your/vendor/lib"

[Install]
[Install]
WantedBy=multi-user.target

Database Connectivity Issues:

If your Dancer2 app interacts with a database, verify that the database client libraries and Perl DBD modules (e.g., DBD::mysql, DBD::Pg) are compatible with RHEL 9 and the updated database server versions. Re-installing the relevant DBD module is often necessary.

cpanm DBD::mysql

By following this structured approach, focusing on pre-upgrade assessment, leveraging the leapp framework, and performing rigorous post-upgrade testing, you can successfully migrate your high-throughput Perl Dancer2 legacy applications to RHEL 9 while minimizing downtime and ensuring continued stability.