Installing and Configuring Supervisor on openSUSE Leap 15.5 to Daemonize Long-Running Python Queue Workers

Prerequisites and Initial Setup

This guide assumes you have a fresh installation of openSUSE Leap 15.5 with root or sudo privileges. We’ll be installing Supervisor, a process control system that allows you to monitor and control a number of processes on UNIX-like operating systems. This is particularly useful for daemonizing long-running applications like Python-based message queue workers.

First, ensure your system is up-to-date:

sudo zypper refresh
sudo zypper update -y

Next, we need to install Supervisor. openSUSE Leap’s repositories typically include a recent version of Supervisor.

sudo zypper install -y supervisor python3-pip

We also install python3-pip as it’s a common dependency for Python applications and might be needed for installing your queue worker dependencies.

Configuring Supervisor

Supervisor’s main configuration file is typically located at /etc/supervisord.conf. However, it’s best practice to manage individual program configurations in separate files within the /etc/supervisor.d/ directory. This keeps your main configuration clean and makes it easier to manage multiple services.

Let’s create a configuration file for our hypothetical Python queue worker. Assume your Python application is located at /opt/my_queue_worker/worker.py and requires a virtual environment located at /opt/my_queue_worker/venv/.

Create a new configuration file, for example, /etc/supervisor.d/my_queue_worker.conf:

[program:my_queue_worker]
command=/opt/my_queue_worker/venv/bin/python /opt/my_queue_worker/worker.py
directory=/opt/my_queue_worker/
user=myuser
autostart=true
autorestart=true
stderr_logfile=/var/log/supervisor/my_queue_worker_err.log
stdout_logfile=/var/log/supervisor/my_queue_worker_out.log
stopsignal=INT

Let’s break down this configuration:

• [program:my_queue_worker]: Defines a program named my_queue_worker. The name before the colon is the group name (if not specified, it defaults to program).
• command: The full path to the executable that Supervisor will run. Here, we specify the Python interpreter within the virtual environment and the path to your worker script.
• directory: The working directory for the command. This is crucial for applications that rely on relative paths or need to access files in their own directory.
• user: The user under which the program will run. It’s highly recommended to run services as a non-root user for security. Ensure this user has the necessary permissions to access the application files and write to log directories.
• autostart=true: Ensures the program starts automatically when Supervisor starts.
• autorestart=true: Configures Supervisor to automatically restart the program if it exits unexpectedly.
• stderr_logfile and stdout_logfile: Specifies the paths for standard error and standard output logs. Supervisor will create these files if they don’t exist.
• stopsignal=INT: Specifies the signal to send to the process when stopping it. INT (SIGINT) is often preferred for Python applications as it allows them to perform graceful shutdowns.

Before proceeding, ensure the log directory exists and the specified user has write permissions:

sudo mkdir -p /var/log/supervisor
sudo chown myuser:myuser /var/log/supervisor

Replace myuser with the actual user you intend to run the worker as.

Enabling and Managing Supervisor

Once your configuration file is in place, you need to tell Supervisor to read it. First, reload the Supervisor configuration to pick up the new program definition.

sudo supervisorctl reread
sudo supervisorctl update

reread tells Supervisor to re-scan its configuration directories for new or changed files. update then applies these changes, starting any new programs that are configured to autostart.

You can now check the status of your worker:

sudo supervisorctl status

This command should show your my_queue_worker as RUNNING. If it’s not, you can start it manually:

sudo supervisorctl start my_queue_worker

To stop the worker:

sudo supervisorctl stop my_queue_worker

To restart the worker:

sudo supervisorctl restart my_queue_worker

To view the logs for your worker:

sudo supervisorctl tail -f my_queue_worker

This command will show the last few lines of the combined stdout and stderr logs and then follow the log file in real-time, similar to tail -f.

Systemd Integration

Supervisor itself needs to be managed as a service. openSUSE Leap uses systemd. Supervisor is typically installed with a systemd service unit file. You can enable and start the Supervisor daemon using systemd commands.

sudo systemctl enable supervisor
sudo systemctl start supervisor
sudo systemctl status supervisor

This ensures that Supervisor starts automatically on boot and that you can manage it using standard systemd commands. If you make changes to the main /etc/supervisord.conf file, you might need to restart the Supervisor service:

sudo systemctl restart supervisor

Advanced Configuration and Best Practices

For production environments, consider these advanced configurations:

• Log Rotation: Supervisor’s default log handling can lead to very large log files. Integrate logrotate to manage these. Create a file like /etc/logrotate.d/supervisor with content similar to this:

/var/log/supervisor/*.log {
    daily
    rotate 7
    compress
    delaycompress
    missingok
    notifempty
    create 0640 root adm
}

• Environment Variables: If your Python worker requires specific environment variables, you can define them within the Supervisor program configuration using the environment directive:

[program:my_queue_worker]
command=/opt/my_queue_worker/venv/bin/python /opt/my_queue_worker/worker.py
directory=/opt/my_queue_worker/
user=myuser
autostart=true
autorestart=true
stderr_logfile=/var/log/supervisor/my_queue_worker_err.log
stdout_logfile=/var/log/supervisor/my_queue_worker_out.log
stopsignal=INT
environment=PYTHONUNBUFFERED="1",DATABASE_URL="postgresql://user:pass@host:port/db"

PYTHONUNBUFFERED="1" is particularly useful for Python applications to ensure that output is written to logs immediately rather than being buffered.

• Process Management: For high-availability scenarios, consider running multiple instances of your worker. Supervisor supports process groups. You can define a program like this:

[program:my_queue_worker_pool]
command=/opt/my_queue_worker/venv/bin/python /opt/my_queue_worker/worker.py
directory=/opt/my_queue_worker/
user=myuser
autostart=true
autorestart=true
numprocs=4
process_name=%(program_name)s_%(process_num)02d
stopsignal=INT
stdout_logfile=/var/log/supervisor/my_queue_worker_pool_%(process_num)02d.log
stderr_logfile=/var/log/supervisor/my_queue_worker_pool_%(process_num)02d.log

In this example, numprocs=4 will start four instances of the worker. process_name=%(program_name)s_%(process_num)02d will name them my_queue_worker_pool_00, my_queue_worker_pool_01, and so on. Each process will have its own log file.

• Graceful Shutdowns: Ensure your Python application is designed to handle SIGINT (or SIGTERM) signals gracefully. This typically involves catching the signal and performing cleanup operations (e.g., finishing current tasks, closing database connections) before exiting.

By following these steps, you can reliably daemonize your long-running Python queue workers on openSUSE Leap 15.5 using Supervisor, ensuring they start automatically, restart on failure, and are managed effectively.