Vengala Vinay

Having 9+ Years of Experience in Software Development

Home » Zero-Downtime Blue-Green Deployment Pipelines for C Applications on DigitalOcean

Zero-Downtime Blue-Green Deployment Pipelines for C Applications on DigitalOcean

Understanding the Blue-Green Deployment Pattern

The blue-green deployment strategy is a technique for releasing software that minimizes downtime and risk. It involves maintaining two identical production environments, referred to as “Blue” and “Green.” At any given time, one environment (e.g., Blue) is running the current production version of the application, while the other (Green) is idle. To deploy a new version, we provision the Green environment with the new code, test it thoroughly, and then switch the router (e.g., load balancer or DNS) to direct all incoming traffic to the Green environment. The Blue environment, now idle, can be used for rollback if issues arise or updated to the next version in a subsequent deployment.

Setting Up DigitalOcean Droplets for Blue and Green Environments

For this setup, we’ll leverage DigitalOcean Droplets. We’ll need at least four Droplets: two for the “Blue” environment, two for the “Green” environment, and two for a load balancer. For simplicity and demonstration, we’ll use Nginx as our load balancer. In a production scenario, you might opt for HAProxy or DigitalOcean’s managed Load Balancer.

First, let’s create the Droplets. We’ll assume a basic Ubuntu 22.04 LTS image. We’ll name them descriptively:

  • blue-app-1
  • blue-app-2
  • green-app-1
  • green-app-2
  • lb-blue
  • lb-green

The blue-app-* and green-app-* Droplets will host our C application instances. The lb-* Droplets will run Nginx, acting as reverse proxies for their respective environments. This provides a basic level of high availability within each environment.

Configuring Nginx as a Load Balancer

We’ll configure Nginx on the lb-blue and lb-green Droplets. The key is to have separate Nginx configurations that point to the application servers within their respective environments. We’ll use a single public IP for our service and switch the DNS A record to point to either lb-blue or lb-green during the cutover.

On the lb-blue Droplet, create an Nginx configuration file (e.g., /etc/nginx/sites-available/blue-app):

Replace <BLUE_APP_IP_1> and <BLUE_APP_IP_2> with the private IP addresses of your blue-app-1 and blue-app-2 Droplets.

server {
    listen 80;
    server_name your_domain.com; # Or the public IP of lb-blue

    location / {
        proxy_pass http://<BLUE_APP_IP_1>:8080; # Assuming your C app listens on port 8080
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

On the lb-green Droplet, create a similar configuration (e.g., /etc/nginx/sites-available/green-app):

Replace <GREEN_APP_IP_1> and <GREEN_APP_IP_2> with the private IP addresses of your green-app-1 and green-app-2 Droplets.

server {
    listen 80;
    server_name your_domain.com; # Or the public IP of lb-green

    location / {
        proxy_pass http://<GREEN_APP_IP_1>:8080; # Assuming your C app listens on port 8080
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

After creating these files, enable the sites and restart Nginx:

sudo ln -s /etc/nginx/sites-available/blue-app /etc/nginx/sites-enabled/
sudo ln -s /etc/nginx/sites-available/green-app /etc/nginx/sites-enabled/
sudo nginx -t
sudo systemctl restart nginx

You’ll need to ensure your firewall rules (e.g., ufw) allow traffic on port 80 for both the load balancer and application Droplets.

Deploying the C Application

Let’s assume your C application is a simple HTTP server that listens on port 8080. We’ll need a deployment script that can be executed on the target application Droplets.

Create a deployment script (e.g., deploy.sh) on your build server or CI/CD runner. This script will handle fetching the new code, compiling it, and starting the application.

#!/bin/bash

APP_DIR="/opt/my_c_app"
GIT_REPO="[email protected]:your_org/your_c_app.git"
BRANCH="main" # Or your deployment branch
APP_PORT="8080"

# Ensure the application directory exists
sudo mkdir -p $APP_DIR
sudo chown -R $(whoami):$(whoami) $APP_DIR # Adjust ownership as needed for your user

# Navigate to the application directory
cd $APP_DIR

# Fetch the latest code
git fetch origin
git checkout $BRANCH
git pull origin $BRANCH

# Compile the application
# This is a placeholder. Adjust your build process accordingly.
# For example, if you use CMake:
# cmake .
# make
# If you have a simple Makefile:
make

# Stop the existing application if it's running
# This assumes you have a systemd service or a simple process manager.
# For simplicity, we'll use pkill here, but a robust solution would use systemd.
echo "Stopping existing application..."
sudo pkill -f "my_c_app" || true # Ignore errors if not running

# Start the new application
echo "Starting new application on port $APP_PORT..."
# Assuming your compiled executable is named 'my_c_app'
nohup ./my_c_app --port $APP_PORT > /var/log/my_c_app.log 2>&1 &

echo "Deployment complete."

You’ll need to ensure that SSH keys are set up for your deployment user to access the application Droplets and that the necessary build tools (compiler, make, etc.) are installed on them. For a production system, managing application processes with systemd is highly recommended over nohup and pkill.

Automating the Deployment Pipeline

A CI/CD tool like GitLab CI, GitHub Actions, or Jenkins is crucial for automating this process. The pipeline will typically involve these stages:

  • Build: Compile the C application.
  • Test: Run unit and integration tests.
  • Deploy to Green: Execute the deploy.sh script on all green-app-* Droplets.
  • Smoke Test Green: Perform basic health checks against the Green environment.
  • Cutover: Switch DNS or load balancer configuration.
  • Monitor: Observe the Green environment for errors.
  • Rollback (if necessary): Switch back to the Blue environment.

Let’s outline a conceptual GitHub Actions workflow (.github/workflows/deploy.yml):

name: Blue-Green Deployment

on:
  push:
    branches:
      - main # Trigger deployment on push to main branch

jobs:
  build_and_test:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v3

      - name: Build C Application
        run: |
          # Your build commands here (e.g., make)
          make

      - name: Run Tests
        run: |
          # Your test commands here
          make test

  deploy_to_green:
    needs: build_and_test
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v3

      - name: Configure SSH
        uses: webfactory/[email protected]
        with:
          ssh-private-key: ${{ secrets.SSH_PRIVATE_KEY }}

      - name: Add known hosts
        run: |
          ssh-keyscan your_green_app_ip_1 >> ~/.ssh/known_hosts
          ssh-keyscan your_green_app_ip_2 >> ~/.ssh/known_hosts
          # Add IPs for all green app droplets

      - name: Deploy to Green App Servers
        run: |
          ssh user@your_green_app_ip_1 "bash -s" < deploy.sh
          ssh user@your_green_app_ip_2 "bash -s" < deploy.sh
          # Repeat for all green app droplets

  smoke_test_green:
    needs: deploy_to_green
    runs-on: ubuntu-latest
    steps:
      - name: Run Smoke Tests
        run: |
          # Use curl or a dedicated testing tool to hit the Green environment's public IP
          # Example: curl -f http://your_green_lb_public_ip/health
          echo "Running smoke tests against Green environment..."
          # Add your actual smoke test commands here

  cutover:
    needs: smoke_test_green
    runs-on: ubuntu-latest
    steps:
      - name: Update DNS to Green LB
        run: |
          # This step requires integration with your DNS provider's API
          # Example using a hypothetical CLI tool:
          # your_dns_cli update-record your_domain.com A your_green_lb_public_ip --ttl 300
          echo "Switching DNS to Green Load Balancer..."
          # Placeholder for actual DNS update command

  monitor_and_rollback:
    needs: cutover
    runs-on: ubuntu-latest
    steps:
      - name: Monitor Green Environment
        run: |
          # Implement monitoring checks here. If errors are detected, trigger rollback.
          echo "Monitoring Green environment..."
          # Example: sleep 600 && curl -f http://your_domain.com/health || echo "Error detected, initiating rollback." && ./rollback.sh

      - name: Rollback to Blue (if needed)
        if: failure() # This step runs only if the previous step fails
        run: |
          echo "Initiating rollback to Blue environment..."
          # This would involve updating DNS back to the Blue LB's public IP
          # your_dns_cli update-record your_domain.com A your_blue_lb_public_ip --ttl 300
          # Placeholder for actual DNS update command
          # You might also want to redeploy the Blue environment to the previous stable version

You'll need to configure the SSH_PRIVATE_KEY secret in your GitHub repository settings. The deploy.sh script should be committed to your repository. The DNS update step requires integrating with your DNS provider's API (e.g., Cloudflare, AWS Route 53, or DigitalOcean's API).

Performing the Cutover

The cutover is the critical step where traffic is redirected. In our setup, this involves changing the DNS A record for your_domain.com to point to the public IP address of the lb-green Droplet. This change will propagate according to the TTL (Time To Live) set for the DNS record.

Manual Cutover Steps (if not fully automated):

  • Ensure the Green environment is fully tested and healthy.
  • Update the DNS A record for your application's domain to point to the public IP of the lb-green Droplet.
  • Monitor application logs and metrics closely for any anomalies.
  • If issues arise, immediately revert the DNS A record to point back to the public IP of the lb-blue Droplet.

Rollback Strategy

The rollback strategy is inherent in the blue-green pattern. If the new deployment on the Green environment exhibits problems:

  • Immediate Revert: The quickest rollback is to simply switch the DNS A record back to the public IP of the lb-blue Droplet. The Blue environment, running the previous stable version, will immediately resume serving traffic.
  • Rollback Deployment: If the issue is severe or requires a code fix, you can redeploy the previous known-good version to the Blue environment (or the Green environment if you've already switched traffic) and then perform a cutover.

It's crucial to have a mechanism to quickly and reliably revert DNS changes. Automating this step within your CI/CD pipeline is highly recommended.

Advanced Considerations

Database Migrations: Handling database schema changes requires careful planning. A common approach is to use backward-compatible migrations. Ensure your new application version can run with the old schema, and the old application version can run with the new schema during the transition. This often involves a multi-step deployment process for the database.

Stateful Applications: For applications that maintain session state or persistent data, managing this during a blue-green deployment can be complex. Strategies include using external session stores (like Redis), ensuring sticky sessions are configured correctly on the load balancer (though this can complicate rollback), or designing the application to be stateless.

Zero-Downtime Database Updates: This is often the hardest part. Consider techniques like:

  • Expand/Contract Pattern: Add new columns/tables in a way that doesn't break the old application, deploy the new application, then remove old columns/tables.
  • Feature Flags: Use feature flags to enable new database interactions gradually.

Health Checks: Implement robust health check endpoints in your C application. The load balancer (or an external monitoring service) should periodically poll these endpoints to ensure application instances are healthy. If an instance fails its health check, it should be removed from the load balancing pool.

Configuration Management: Ensure that configuration files are managed consistently across both environments. Tools like Ansible, Chef, or Puppet can help maintain parity.

Testing in Production (Canary Releases): While blue-green aims for full cutover, you can adapt it for canary releases by initially directing a small percentage of traffic to the Green environment and gradually increasing it as confidence grows.