Installing and Setting up a Local Composer Mirror Server on RHEL 9 to accelerate internal developer deployment loops

Prerequisites and Initial Setup

This guide details the installation and configuration of a local Composer mirror server on Red Hat Enterprise Linux 9 (RHEL 9). The primary objective is to significantly accelerate internal developer deployment loops by reducing external network latency and dependency on public repositories. This setup is crucial for enterprises with strict security policies or those operating in environments with limited internet connectivity.

Before proceeding, ensure you have a RHEL 9 system with root or sudo privileges. The system should have sufficient disk space to store the Composer packages, which can grow considerably over time. A minimum of 50GB is recommended, with provisions for future expansion.

Installing Necessary Software Packages

We’ll need a web server to serve the Composer repository. Nginx is a performant and widely adopted choice. Additionally, `rsync` will be used for efficient synchronization of packages.

Install Nginx and `rsync` using `dnf`:

sudo dnf update -y
sudo dnf install -y nginx rsync

Start and enable the Nginx service to ensure it runs on boot:

sudo systemctl start nginx
sudo systemctl enable nginx

Configuring the Composer Mirror Directory

Create a dedicated directory for the Composer mirror. This directory will house the synchronized package data. We’ll use `/var/www/html/composer-mirror` as the base path, which is a common location for web server content.

sudo mkdir -p /var/www/html/composer-mirror/packages
sudo chown -R nginx:nginx /var/www/html/composer-mirror
sudo chmod -R 755 /var/www/html/composer-mirror

Setting up Nginx as a Web Server for the Mirror

Configure Nginx to serve the Composer mirror directory. Create a new Nginx configuration file for this purpose.

# /etc/nginx/conf.d/composer-mirror.conf
server {
    listen 80;
    server_name composer-mirror.your-domain.com; # Replace with your actual domain or IP

    root /var/www/html/composer-mirror;
    index index.html index.htm;

    location / {
        try_files $uri $uri/ =404;
    }

    # Optional: Add basic authentication for security if needed
    # auth_basic "Composer Mirror Access";
    # auth_basic_user_file /etc/nginx/.htpasswd;
}

If you choose to implement basic authentication, you’ll need to create the `.htpasswd` file:

sudo yum install -y httpd-tools # For htpasswd command
sudo htpasswd -c /etc/nginx/.htpasswd your_username # Replace your_username

Test the Nginx configuration and reload the service:

sudo nginx -t
sudo systemctl reload nginx

Automating Package Synchronization with `rsync`

The core of the mirror functionality relies on regularly synchronizing packages from a public Composer repository. We’ll use `rsync` for this. A common strategy is to mirror `https://packagist.org`. A cron job will be used to automate this process.

First, create a script that performs the `rsync` operation. This script will download the necessary metadata and package files.

# /usr/local/bin/sync-composer-mirror.sh
#!/bin/bash

# Configuration
SOURCE_REPO="https://packagist.org"
MIRROR_DIR="/var/www/html/composer-mirror"
LOG_FILE="/var/log/composer-mirror-sync.log"
TIMESTAMP=$(date +"%Y-%m-%d %H:%M:%S")

echo "[$TIMESTAMP] Starting Composer mirror synchronization..." >> "$LOG_FILE"

# Synchronize the main packagist.org metadata
# --delete: remove files from destination that are not in source
# --exclude: exclude specific directories or files if needed
rsync -avz --delete \
    "$SOURCE_REPO/p/\"" \
    "$MIRROR_DIR/packages/" \
    >> "$LOG_FILE" 2>&1

# Synchronize the packages themselves. This is the most time-consuming part.
# You might want to adjust the rsync options based on your needs.
# For example, --bwlimit can be used to limit bandwidth usage.
rsync -avz --delete \
    "$SOURCE_REPO/packages.json" \
    "$MIRROR_DIR/" \
    >> "$LOG_FILE" 2>&1

# You might also want to sync specific package types if you have a lot of them
# For example, to sync all packages starting with 'a':
# rsync -avz --delete "$SOURCE_REPO/a/" "$MIRROR_DIR/packages/a/" >> "$LOG_FILE" 2>&1

echo "[$TIMESTAMP] Composer mirror synchronization finished." >> "$LOG_FILE"
echo "----------------------------------------" >> "$LOG_FILE"

exit 0

Make the script executable:

sudo chmod +x /usr/local/bin/sync-composer-mirror.sh

Scheduling the Synchronization with Cron

Set up a cron job to run the synchronization script at regular intervals. A daily or twice-daily sync is usually sufficient, but this can be adjusted based on how frequently your upstream repositories are updated and your internal development needs.

Edit the root user’s crontab:

sudo crontab -e

Add the following line to run the script every day at 3:00 AM:

0 3 * * * /usr/local/bin/sync-composer-mirror.sh

Save and exit the crontab editor. The cron job will now execute the script automatically.

Configuring Developer Environments to Use the Mirror

Developers can now configure their local Composer installations or CI/CD pipelines to use the internal mirror. This is done by modifying the Composer configuration.

For a single project, you can use the command line:

composer config repositories.local-mirror '{
    "type": "composer",
    "url": "http://composer-mirror.your-domain.com/packages/"
}'

This command adds a new repository configuration to the project’s `composer.json` file. The `repositories.local-mirror` entry specifies the type and URL of your internal mirror.

To configure this globally for a user, you can edit their Composer configuration file (`~/.composer/config.json`):

{
    "config": {
        "repositories": {
            "local-mirror": {
                "type": "composer",
                "url": "http://composer-mirror.your-domain.com/packages/"
            }
        }
    }
}

Alternatively, and often preferred for enterprise-wide adoption, you can set the `COMPOSER_MIRROR_PATH` environment variable. Composer will automatically prepend this path to its repository list.

export COMPOSER_MIRROR_PATH="http://composer-mirror.your-domain.com/packages/"

This environment variable should be set in the shell profiles of developers (e.g., `.bashrc`, `.zshrc`) or within your CI/CD system’s environment variables.

Advanced Considerations and Maintenance

Disk Space Management: The `rsync –delete` option is crucial for preventing the mirror from consuming excessive disk space. Monitor disk usage regularly. You might consider implementing log rotation for the sync logs.

Bandwidth Throttling: If the initial sync or subsequent updates consume too much bandwidth, consider using `rsync`’s `–bwlimit` option in the sync script to cap the transfer rate.

Security: If your internal mirror is accessible from the public internet, ensure it’s secured. Basic authentication is a good first step. For more robust security, consider HTTPS with a valid certificate and potentially IP-based access controls in Nginx.

Mirroring Multiple Repositories: For complex environments, you might need to mirror other Composer repositories besides packagist.org. This can be achieved by adding more `rsync` commands to the sync script, targeting different source URLs and potentially different subdirectories within your mirror.

Initial Sync Time: The first synchronization can take a significant amount of time, depending on your network connection and the size of packagist.org. It’s advisable to run this initial sync during off-peak hours.

Composer’s `packages.json` Handling: Composer relies heavily on the `packages.json` file for metadata. Ensure that this file is correctly synchronized. The `rsync` command in the script handles this.

Testing the Mirror: After setup, test thoroughly by running `composer install` or `composer update` on a project configured to use the mirror. Verify that dependencies are downloaded quickly and without errors. You can also check the Nginx access logs to confirm requests are being served from your mirror server.