Zero-Downtime Blue-Green Deployment Pipelines for Laravel Applications on DigitalOcean

Establishing the Blue-Green Infrastructure on DigitalOcean

The core of a zero-downtime blue-green deployment strategy hinges on maintaining two identical, isolated production environments. For Laravel applications on DigitalOcean, this translates to two distinct sets of Droplets, load balancers, and database configurations. We’ll leverage DigitalOcean’s Load Balancers to abstract the public-facing IP and manage traffic routing between the ‘blue’ (current production) and ‘green’ (staging/new production) environments.

Our infrastructure will consist of:

• Two identical sets of Laravel application Droplets (e.g., 2x web servers, 1x worker server per environment).
• Two identical database instances (e.g., DigitalOcean Managed Databases or self-hosted replicas). Crucially, both environments must point to the *same* database instance to avoid data divergence. This is a common point of failure if not handled correctly. We will use read replicas for the ‘green’ environment during testing and then switch the primary connection during the cutover.
• One DigitalOcean Load Balancer configured to point to the ‘blue’ environment initially.

Database Strategy: The Single Source of Truth

Data consistency is paramount. During a blue-green deployment, both environments must operate against a single, shared database. This means the ‘green’ environment will connect to the *same* production database as the ‘blue’ environment. To mitigate risks during the ‘green’ environment’s testing phase, we’ll configure the ‘green’ application servers to connect to a read-only replica of the production database. This prevents accidental data modifications in the ‘green’ environment that could impact live users. The actual cutover involves a brief period where the ‘green’ environment is switched to write to the primary database, followed by the load balancer redirection.

Automating Deployments with CI/CD and Bash Scripting

A robust CI/CD pipeline is essential. We’ll use a tool like GitLab CI, GitHub Actions, or Jenkins. For this example, we’ll focus on the deployment script itself, assuming your CI/CD runner has SSH access to your Droplets and the necessary permissions.

The deployment script will handle:

• Cloning the latest stable branch into the ‘green’ environment’s deployment directory.
• Running Composer dependencies.
• Running database migrations (carefully, as explained below).
• Clearing caches.
• Performing health checks.

The Deployment Script (`deploy_green.sh`)

This script assumes you have SSH keys set up for your CI/CD runner to access the ‘green’ environment’s web servers. It also assumes a consistent directory structure (e.g., `/var/www/myapp/green`).

#!/bin/bash

# --- Configuration ---
GREEN_WEB_SERVERS=("[email protected]" "[email protected]")
APP_DIR="/var/www/myapp/green"
GIT_BRANCH="main" # Or your production branch
COMPOSER_OPTIONS="--no-dev --optimize-autoloader --no-interaction"
HEALTH_CHECK_URL="https://green.yourdomain.com/health" # A dedicated health check endpoint in Laravel

# --- Script Logic ---
echo "--- Starting Blue-Green Deployment to Green Environment ---"

# 1. Update Code
echo "Updating code on green servers..."
for server in "${GREEN_WEB_SERVERS[@]}"; do
    echo "  Deploying to $server..."
    ssh "$server" "
        cd $APP_DIR || exit 1
        git fetch origin
        git checkout $GIT_BRANCH
        git pull origin $GIT_BRANCH
        echo 'Code updated.'
    "
done

# 2. Install Dependencies and Optimize
echo "Installing Composer dependencies and optimizing..."
for server in "${GREEN_WEB_SERVERS[@]}"; do
    echo "  Running composer on $server..."
    ssh "$server" "
        cd $APP_DIR || exit 1
        composer install $COMPOSER_OPTIONS
        echo 'Composer dependencies installed.'
    "
done

# 3. Database Migrations (CRITICAL: Read-Only during testing, Write during cutover)
# This script assumes migrations are handled separately or are safe to run against a read replica
# during the testing phase. The actual cutover will involve switching the DB connection.
# For this script, we'll assume migrations are applied *after* the cutover or are idempotent.
# A more advanced strategy might involve a separate migration runner.
echo "Skipping direct migrations in this script. Ensure migrations are handled safely."
echo "Consider a separate migration runner or applying them after the cutover."

# 4. Clear Caches
echo "Clearing application caches..."
for server in "${GREEN_WEB_SERVERS[@]}"; do
    echo "  Clearing cache on $server..."
    ssh "$server" "
        cd $APP_DIR || exit 1
        php artisan cache:clear
        php artisan config:clear
        php artisan route:clear
        php artisan view:clear
        echo 'Caches cleared.'
    "
done

# 5. Permissions (if necessary)
# echo "Setting file permissions..."
# for server in "${GREEN_WEB_SERVERS[@]}"; do
#     echo "  Setting permissions on $server..."
#     ssh "$server" "
#         cd $APP_DIR || exit 1
#         # Example: chown -R www-data:www-data storage bootstrap/cache
#         echo 'Permissions set.'
#     "
# done

# 6. Health Check
echo "Performing health check on the green environment..."
HEALTH_CHECK_RESPONSE=$(curl -s -o /dev/null -w "%{http_code}" "$HEALTH_CHECK_URL")

if [ "$HEALTH_CHECK_RESPONSE" = "200" ]; then
    echo "Health check passed ($HEALTH_CHECK_URL returned $HEALTH_CHECK_RESPONSE)."
    echo "--- Green Environment Deployment Complete ---"
    echo "Ready for traffic cutover."
else
    echo "Health check failed ($HEALTH_CHECK_URL returned $HEALTH_CHECK_RESPONSE)."
    echo "--- Green Environment Deployment FAILED ---"
    exit 1
fi

exit 0

Configuring the Load Balancer for Traffic Switching

DigitalOcean Load Balancers are the linchpin for the traffic switch. We’ll configure it to initially point to the ‘blue’ Droplets. The cutover process involves reconfiguring the load balancer’s target pool.

Initial Load Balancer Setup

When you create a DigitalOcean Load Balancer, you’ll specify a target pool. Initially, this pool will contain the IP addresses of your ‘blue’ environment’s web servers. You’ll also configure health checks. These health checks should target your application’s health endpoint (e.g., `/health` in Laravel, which should return a 200 OK status).

The Cutover Process: A Step-by-Step Manual or Scripted Action

The cutover is the most critical phase. It should be as brief as possible to minimize any potential disruption, though ideally, zero.

1. Prepare Green Environment: Run the `deploy_green.sh` script (or your CI/CD equivalent) to deploy the new code to the ‘green’ Droplets.
2. Database Switch (if applicable): If your ‘green’ environment was using a read replica, this is where you’d reconfigure its application to write to the primary database. This might involve updating an `.env` file and restarting PHP-FPM/web servers on the ‘green’ Droplets.
3. Health Check Green: Perform a manual `curl` or use a monitoring tool to ensure the ‘green’ environment is healthy and responding correctly to requests.
4. Update Load Balancer Target Pool: This is the actual traffic switch.
   • Using the DigitalOcean API or Control Panel, remove the ‘blue’ Droplets from the load balancer’s target pool.
   • Add the ‘green’ Droplets to the load balancer’s target pool.
   The load balancer will gradually start sending new traffic to the ‘green’ environment. Existing connections to the ‘blue’ environment will continue until they complete.
5. Monitor: Closely monitor application logs, error rates, and performance metrics for both environments.
6. Rollback (if necessary): If issues arise, the rollback is as simple as reversing step 4: remove ‘green’ Droplets from the LB pool and add ‘blue’ Droplets back.

Automating the Cutover with the DigitalOcean API

For true automation, the load balancer switch can be triggered via the DigitalOcean API. You’ll need an API token and a tool like `curl` or a scripting language (Python is excellent for this).

First, identify your Load Balancer ID and the existing Droplet IDs for your blue and green environments.

Example: Python script to switch Load Balancer targets

import requests
import os

# --- Configuration ---
DO_API_TOKEN = os.environ.get("DO_API_TOKEN") # Store securely, e.g., in CI/CD secrets
LOAD_BALANCER_ID = "YOUR_LOAD_BALANCER_ID"
BLUE_DROPLET_IDS = ["DROPLET_ID_BLUE_1", "DROPLET_ID_BLUE_2"] # IDs of current production Droplets
GREEN_DROPLET_IDS = ["DROPLET_ID_GREEN_1", "DROPLET_ID_GREEN_2"] # IDs of new production Droplets
DO_API_URL = "https://api.digitalocean.com/v2/loadbalancers/"

if not DO_API_TOKEN:
    print("Error: DO_API_TOKEN environment variable not set.")
    exit(1)

headers = {
    "Authorization": f"Bearer {DO_API_TOKEN}",
    "Content-Type": "application/json"
}

def get_load_balancer_config(lb_id):
    """Fetches the current configuration of a load balancer."""
    url = f"{DO_API_URL}{lb_id}"
    response = requests.get(url, headers=headers)
    response.raise_for_status()
    return response.json()

def update_load_balancer_targets(lb_id, new_droplet_ids):
    """Updates the target Droplets for a load balancer."""
    url = f"{DO_API_URL}{lb_id}"
    
    # Fetch current config to preserve other settings (like health checks, ports)
    current_config = get_load_balancer_config(lb_id)
    
    # Assuming there's only one target pool for simplicity. Adjust if you have multiple.
    # We need to preserve the existing health check and port configuration.
    target_pool = current_config['load_balancer']['forwarding_rules'][0] # Adjust index if needed
    
    # Construct the new forwarding rule with the new droplet IDs
    new_forwarding_rule = {
        "entry_protocol": target_pool['entry_protocol'],
        "entry_port": target_pool['entry_port'],
        "target_protocol": target_pool['target_protocol'],
        "target_port": target_pool['target_port'],
        "tls_termination": target_pool.get('tls_termination'), # Include if present
        "health_check": target_pool['health_check'],
        "sticky_sessions": target_pool.get('sticky_sessions') # Include if present
    }
    
    # Add the new droplet IDs to the forwarding rule.
    # The API expects a list of Droplet IDs for the 'droplets' key within the forwarding rule.
    # However, the API documentation implies you update the *entire* load balancer object.
    # A safer approach is to update the 'droplets' field within the *first* forwarding rule.
    
    # Let's re-fetch the LB config to ensure we have the latest structure.
    lb_data = get_load_balancer_config(lb_id)
    
    # Modify the first forwarding rule's droplet list
    if lb_data['load_balancer']['forwarding_rules']:
        lb_data['load_balancer']['forwarding_rules'][0]['droplets'] = new_droplet_ids
    else:
        print("Error: No forwarding rules found for the load balancer.")
        return

    # The update payload should be the entire load balancer object with modifications
    update_payload = {"load_balancer": lb_data['load_balancer']}

    response = requests.put(url, headers=headers, json=update_payload)
    response.raise_for_status()
    print(f"Load balancer {lb_id} updated successfully to target Droplets: {new_droplet_ids}")
    return response.json()

# --- Main Execution ---
if __name__ == "__main__":
    print("Starting load balancer switch...")
    
    # In a real scenario, you'd first verify the green environment is healthy.
    # For this example, we assume the deploy_green.sh script already did that.
    
    try:
        # Switch to green environment
        print(f"Switching load balancer {LOAD_BALANCER_ID} to green Droplets...")
        update_load_balancer_targets(LOAD_BALANCER_ID, GREEN_DROPLET_IDS)
        
        # Optional: After a successful switch and monitoring, you might want to
        # update the load balancer to remove the blue Droplets entirely if they
        # are no longer needed or to prevent accidental traffic.
        # However, the PUT request above *replaces* the droplet list, so the blue ones
        # are implicitly removed. If you had multiple forwarding rules or complex setups,
        # you'd need to be more granular.
        
        print("Load balancer switch complete.")
        
    except requests.exceptions.RequestException as e:
        print(f"An error occurred: {e}")
        if e.response is not None:
            print(f"Response status code: {e.response.status_code}")
            print(f"Response body: {e.response.text}")
        exit(1)
    except Exception as e:
        print(f"An unexpected error occurred: {e}")
        exit(1)

Handling Database Migrations During Cutover

Database migrations are a common source of downtime or errors during deployments. The blue-green strategy helps, but migrations themselves need careful consideration.

The Challenge

If a migration modifies the database schema in a way that is incompatible with the *previous* version of the application code, deploying that migration *before* switching traffic can cause errors. Conversely, if you switch traffic *before* applying the migration, the new code will fail.

Solutions

• Idempotent Migrations: Design your migrations to be safe to run multiple times or to be compatible with both old and new code versions. This is the ideal but often difficult to achieve for complex changes.
• Multi-Step Deployments:
  1. Deploy code that is compatible with the *current* schema.
  2. Run migrations that add new columns or tables (non-breaking changes).
  3. Deploy code that uses the new schema elements.
  4. Run migrations that remove old columns or tables (breaking changes).
  5. Deploy code that relies on the removal of old schema elements.
  This approach requires careful orchestration and can be complex to manage.
• Separate Migration Runner: Have a dedicated service or script that applies migrations. This runner would be triggered *after* the traffic has been switched to the ‘green’ environment. The ‘green’ application servers would initially connect to a read-only replica, and only the migration runner would have write access to the primary database. Once migrations are complete, the ‘green’ application servers can be switched to write mode.
• Database Schema Versioning Tools: Tools like Phinx or Flyway offer more advanced migration management capabilities that can help in complex scenarios.

For a typical Laravel application, the most practical approach for blue-green is often to ensure migrations are backward-compatible (e.g., adding columns before using them) and then run them as part of the ‘green’ deployment process, or immediately after the traffic switch, ensuring the ‘green’ environment is configured to write to the primary database.

Rollback Strategy: Reverting to Blue

The beauty of blue-green is the simplicity of rollback. If the ‘green’ environment exhibits issues after the cutover, the rollback is a swift reversal of the traffic switch.

• Identify the Issue: Quickly diagnose the problem in the ‘green’ environment.
• Reverse Load Balancer Switch: Using the DigitalOcean API (or Control Panel), remove the ‘green’ Droplets from the load balancer’s target pool and add the ‘blue’ Droplets back.
• Analyze and Fix: Investigate the failure in the ‘green’ environment offline. Once fixed, it can be redeployed and become the ‘blue’ environment for the *next* deployment cycle.
• Database Considerations: If the issue involved data corruption in the ‘green’ environment (which should be prevented by read-only replicas during testing), you might need to restore the database from a backup before attempting a rollback or redeployment.

Monitoring and Health Checks: The Safety Net

Comprehensive monitoring is non-negotiable. Before, during, and after the cutover, you must have visibility.

• Application Health Endpoint: A dedicated `/health` route in your Laravel app that checks database connectivity, cache status, and other critical dependencies. This is what the load balancer will use.
• Real-time Logging: Centralized logging (e.g., ELK stack, Logtail, Papertrail) for both ‘blue’ and ‘green’ environments.
• Performance Metrics: Monitor CPU, memory, network I/O, and application response times (e.g., Prometheus/Grafana, Datadog, New Relic).
• Error Tracking: Tools like Sentry or Bugsnag to catch exceptions immediately.

During the cutover, pay close attention to the load balancer’s health check status and any immediate spikes in error rates or latency.