Upgrading Python 3.10 to Python 3.12 on Ubuntu 22.04 LTS and Re-linking Gunicorn virtualenvs Safely

Prerequisites and Initial Assessment

This guide details the process of upgrading Python from version 3.10 to 3.12 on an Ubuntu 22.04 LTS (Jammy Jellyfish) system, with a specific focus on safely re-linking Gunicorn virtual environments. This is a critical operation for maintaining production services, ensuring compatibility with newer libraries, and leveraging performance improvements in Python 3.12. Before proceeding, ensure you have root or sudo privileges and a clear understanding of your current Python deployment, including any custom C extensions or system-level dependencies that might be affected.

Verify your current Python 3.10 installation and its location. On Ubuntu 22.04, Python 3.10 is typically installed as the system default. We will install Python 3.12 alongside it, rather than replacing the system’s default Python, to minimize disruption to core system services.

which python3
python3 --version
ls -l /usr/bin/python3

Note the output of these commands. You’ll likely see `/usr/bin/python3` pointing to a Python 3.10 executable. We will use `update-alternatives` to manage the `python3` command later, but for now, we focus on installing Python 3.12.

Installing Python 3.12 from Source

While Ubuntu’s repositories might eventually include Python 3.12, building from source offers the most control and ensures you get the latest stable release. This process involves downloading the source code, configuring the build, compiling, and installing.

First, install essential build dependencies. These are crucial for compiling Python from source.

sudo apt update
sudo apt install -y build-essential zlib1g-dev libncurses5-dev libgdbm-dev libnss3-dev libssl-dev libreadline-dev libffi-dev libsqlite3-dev wget libbz2-dev pkg-config

Next, download the Python 3.12 source code. It’s recommended to use `wget` to fetch the tarball directly from the official Python website.

cd /tmp
wget https://www.python.org/ftp/python/3.12.4/Python-3.12.4.tgz
tar -xf Python-3.12.4.tgz
cd Python-3.12.4

Now, configure the build. The --enable-optimizations flag enables profile-guided optimizations (PGO) and Link Time Optimization (LTO), which can significantly improve runtime performance. The --with-ensurepip=install ensures that `pip` is installed with Python.

./configure --enable-optimizations --with-ensurepip=install --prefix=/usr/local

Compile Python. Using make -j $(nproc) will utilize all available CPU cores for faster compilation. This step can take a considerable amount of time.

make -j $(nproc)
sudo make altinstall

The make altinstall command is crucial. It installs Python executables with version-specific suffixes (e.g., python3.12, pip3.12) and avoids overwriting the default python3 binary, which is essential for system stability.

Verify the installation of Python 3.12.

/usr/local/bin/python3.12 --version
/usr/local/bin/pip3.12 --version

Configuring `update-alternatives` for Python 3

To manage which Python version is invoked by the generic python3 command, we use the update-alternatives system. This allows us to switch between installed Python versions gracefully.

First, add both Python 3.10 and Python 3.12 to the alternatives system. We assign priorities; a higher number means higher priority. We’ll give Python 3.10 a higher priority initially to maintain the current system default.

sudo update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.10 1
sudo update-alternatives --install /usr/bin/python3 python3 /usr/local/bin/python3.12 2

Now, you can interactively choose the default Python 3 version.

sudo update-alternatives --config python3

This command will present a menu where you can select the desired Python version by entering its corresponding number. For now, keep Python 3.10 as the default. We will only switch to 3.12 after verifying our virtual environments.

Re-linking Gunicorn Virtual Environments

This is the most critical part of the upgrade. Applications running via Gunicorn are typically deployed within isolated virtual environments. These environments are linked to a specific Python interpreter. When you upgrade Python, these links need to be updated to point to the new interpreter, or the virtual environments themselves need to be recreated with the new Python version.

Identify your Gunicorn virtual environments. They are often located in directories like /opt/venvs/ or within application deployment directories. For each virtual environment, you’ll find a bin/python and bin/pip executable. We need to ensure these point to the correct Python 3.12 interpreter.

The safest approach is to recreate the virtual environments using Python 3.12. This ensures that all packages are installed against the new interpreter and that any C extensions are compiled correctly.

Let’s assume you have a virtual environment named my_app_venv located at /opt/venvs/my_app_venv.

# Navigate to the directory containing your virtual environments
cd /opt/venvs/

# Deactivate any active virtual environment (if applicable)
deactivate

# Backup the old virtual environment (optional but recommended)
mv my_app_venv my_app_venv_backup_$(date +%Y%m%d_%H%M%S)

# Create a new virtual environment using Python 3.12
/usr/local/bin/python3.12 -m venv my_app_venv

# Activate the new virtual environment
source my_app_venv/bin/activate

# Reinstall dependencies from requirements.txt
pip install -r /path/to/your/application/requirements.txt

# Verify the Python interpreter within the venv
which python
python --version

# Deactivate the venv after verification
deactivate

Repeat this process for all your Gunicorn virtual environments. If you don’t have a requirements.txt, you’ll need to manually reinstall packages using pip install package_name.

Updating Gunicorn Service Files

If your Gunicorn services are managed by systemd, you need to update their service files to reflect the new virtual environment path and potentially the Python interpreter path if you’re not relying on the venv’s `bin/python` directly (though using the venv’s interpreter is standard practice).

Locate your Gunicorn systemd service file. It might be in /etc/systemd/system/my_app.service.

[Unit]
Description=Gunicorn instance to serve my_app
After=network.target

[Service]
User=my_app_user
Group=my_app_group
WorkingDirectory=/opt/my_app
Environment="PATH=/opt/venvs/my_app_venv/bin:$PATH"
ExecStart=/opt/venvs/my_app_venv/bin/gunicorn --workers 3 --bind unix:/run/my_app.sock my_app.wsgi:application

[Install]
WantedBy=multi-user.target

In the example above, the crucial line is ExecStart, which explicitly calls the gunicorn executable from the newly created virtual environment. The Environment="PATH=..." line ensures that the system can find the correct executables within the venv. If your ExecStart line directly calls /usr/bin/python3 and then gunicorn, you would need to ensure that /usr/bin/python3 now points to Python 3.12 (after switching alternatives) or update it to use the full path to the venv’s Python interpreter.

After modifying the service file, reload the systemd daemon and restart your Gunicorn service:

sudo systemctl daemon-reload
sudo systemctl restart my_app.service

Switching the System Default Python and Final Verification

Once you have confirmed that all your virtual environments and Gunicorn services are functioning correctly with Python 3.12, you can safely switch the system’s default python3 command to point to Python 3.12.

sudo update-alternatives --config python3

Select the entry corresponding to /usr/local/bin/python3.12. After making the switch, verify the change:

python3 --version

You should now see Python 3.12.x. It’s also prudent to check that system tools that might rely on the default python3 are still functional. Ubuntu 22.04 is generally robust in its use of specific Python versions for system tasks, often using /usr/bin/python3.10 or even /usr/bin/python2.7 for its own scripts. However, always monitor system logs (e.g., journalctl -xe) for any new errors after this change.

Finally, test your applications thoroughly. Ensure all functionalities are working as expected, and check Gunicorn logs for any new warnings or errors.

Troubleshooting Common Issues

• Package Installation Errors: If pip install fails within a recreated virtual environment, it often indicates a missing build dependency for a specific package. Review the error messages carefully and install the necessary development headers (e.g., libpq-dev for PostgreSQL drivers, libjpeg-dev for Pillow).
• Gunicorn Not Starting: Check systemctl status my_app.service and journalctl -u my_app.service for detailed error messages. Common causes include incorrect paths in the service file, permission issues, or the application failing to import due to incompatible libraries.
• Application Errors: If your application runs but exhibits unexpected behavior, it might be due to subtle incompatibilities between libraries and Python 3.12, or missing dependencies. Re-running pip check within the activated virtual environment can help identify broken dependencies.
• System Tool Failures: If core system utilities break after switching python3, revert the update-alternatives change immediately and investigate which system component is strictly dependent on Python 3.10. You might need to adjust the service file for that specific tool or use a virtual environment for your applications and leave the system python3 pointing to 3.10.