Vengala Vinay

Having 9+ Years of Experience in Software Development

Home » Zero-Downtime Blue-Green Deployment Pipelines for Python Applications on OVH

Zero-Downtime Blue-Green Deployment Pipelines for Python Applications on OVH

Understanding the Blue-Green Deployment Paradigm

Blue-Green deployment is a strategy for releasing software that minimizes downtime and risk. It involves maintaining two identical production environments, “Blue” and “Green.” At any given time, one environment (e.g., Blue) is serving live production traffic, while the other (Green) is idle. To deploy a new version, we deploy the new code to the idle environment (Green), test it thoroughly, and then switch the router (load balancer) to direct all incoming traffic to the Green environment. The Blue environment, now idle, can be used for further testing, rollback, or updated to the new version for the next deployment cycle.

OVH Infrastructure for Blue-Green Deployments

For this setup on OVH, we’ll leverage several key services:

  • Public Cloud Instances (VMs): To host our Python applications. We’ll need at least two sets of identical instances.
  • Load Balancer (HAProxy): To manage traffic routing between the Blue and Green environments. OVH’s Public Cloud Load Balancer service is ideal.
  • Object Storage (S3-compatible): For storing build artifacts and potentially configuration files.
  • SSH/SCP: For deploying code to the instances.
  • CI/CD Orchestrator: A tool like GitLab CI, GitHub Actions, or Jenkins to automate the build, test, and deployment process. For this example, we’ll assume a GitLab CI setup.

Setting Up the Environments

We need two distinct sets of identical infrastructure. For simplicity, let’s assume each environment (Blue and Green) consists of a single OVH Public Cloud instance running our Python web application (e.g., Flask or Django) behind a load balancer.

Instance Configuration (Example: Ubuntu 22.04)

Each instance will need Python, pip, and a web server (like Gunicorn) to serve the application. We’ll also need to ensure consistent configuration across all instances.

A common approach is to use a configuration management tool (Ansible, Chef, Puppet) or a Dockerfile for consistent setup. Here’s a simplified `Dockerfile` for a Python web app:

# Dockerfile
FROM python:3.9-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

# Expose the port Gunicorn will run on
EXPOSE 8000

# Command to run the application with Gunicorn
# Replace 'your_module:app' with your actual WSGI application entry point
CMD ["gunicorn", "--bind", "0.0.0.0:8000", "your_module:app"]

You would then build this image and deploy it to your instances. For a non-containerized approach, a shell script to set up the environment would be used.

Load Balancer Configuration (HAProxy)

We’ll use OVH’s Public Cloud Load Balancer. The core of the blue-green strategy lies in its configuration. We’ll define two backend server groups, one for Blue and one for Green, and a frontend that directs traffic to one of these groups.

Let’s assume:

  • Blue instances are on IP addresses: 192.168.1.10, 192.168.1.11 (if using multiple instances per environment)
  • Green instances are on IP addresses: 192.168.1.20, 192.168.1.21
  • The Load Balancer’s public IP is 203.0.113.1
  • Our application runs on port 8000 on the instances.

The HAProxy configuration (which you’d apply via the OVH API or control panel) would look something like this:

# HAProxy Configuration Snippet
frontend http_frontend
    bind *:80
    mode http
    default_backend blue_backend

backend blue_backend
    mode http
    balance roundrobin
    option httpchk GET /healthz  # Health check endpoint
    server blue_server_1 192.168.1.10:8000 check
    server blue_server_2 192.168.1.11:8000 check

backend green_backend
    mode http
    balance roundrobin
    option httpchk GET /healthz  # Health check endpoint
    server green_server_1 192.168.1.20:8000 check
    server green_server_2 192.168.1.21:8000 check

Initially, default_backend blue_backend directs all traffic to the Blue environment. The green_backend is inactive.

Automating the Deployment Pipeline (GitLab CI Example)

We’ll define a CI/CD pipeline that handles building the application, deploying it to the idle environment, testing, and performing the switch.

Pipeline Stages

  • Build: Create application artifacts (e.g., Docker image, zipped code).
  • Deploy to Green: Deploy the new version to the Green environment instances.
  • Test Green: Run integration and smoke tests against the Green environment.
  • Promote to Production: Switch the load balancer to direct traffic to the Green environment.
  • Deploy to Blue (Optional): Update the Blue environment to the new version for the next cycle.

GitLab CI Configuration (`.gitlab-ci.yml`)

This example assumes you are using Docker images and have SSH access configured for your GitLab Runner to manage the load balancer. You’ll need to store sensitive information (like SSH keys, OVH API credentials) in GitLab CI/CD variables.

variables:
  APP_NAME: my-python-app
  DOCKER_REGISTRY: registry.gitlab.com/your-group/your-project
  GREEN_SERVERS: "192.168.1.20:8000,192.168.1.21:8000"
  BLUE_SERVERS: "192.168.1.10:8000,192.168.1.11:8000"
  OVH_LOAD_BALANCER_IP: "203.0.113.1" # Public IP of your HAProxy LB
  OVH_API_ENDPOINT: "https://api.us. OVHcloud.com/1.0" # Or your region's endpoint
  OVH_APP_KEY: $OVH_APP_KEY
  OVH_APP_SECRET: $OVH_APP_SECRET
  OVH_CONSUMER_KEY: $OVH_CONSUMER_KEY
  SSH_PRIVATE_KEY: $SSH_PRIVATE_KEY
  SSH_USER: deployer
  SSH_HOST_FOR_LB_CONFIG: 192.168.1.5 # An instance where you can run LB config commands

stages:
  - build
  - deploy_green
  - test_green
  - promote
  - deploy_blue # Optional: Prepare Blue for next cycle

.docker_build: &docker_build
  image: docker:latest
  services:
    - docker:dind
  before_script:
    - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" "$CI_REGISTRY"
  script:
    - docker build -t "$DOCKER_REGISTRY/$APP_NAME:$CI_COMMIT_SHA" .
    - docker push "$DOCKER_REGISTRY/$APP_NAME:$CI_COMMIT_SHA"
  tags:
    - docker

build_app:
  <<: *docker_build
  stage: build

deploy_to_green:
  image: alpine:latest
  stage: deploy_green
  before_script:
    - apk add --no-cache openssh-client rsync
    - eval $(ssh-agent -s)
    - echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -
    - mkdir -p ~/.ssh
    - chmod 700 ~/.ssh
    - echo "StrictHostKeyChecking no" >> ~/.ssh/config
  script:
    - echo "Deploying to Green environment..."
    # Example using rsync to deploy a script that pulls and runs the Docker image
    # In a real scenario, you might use Ansible or a custom script to manage Docker on the target VMs
    - ssh [email protected] "docker pull $DOCKER_REGISTRY/$APP_NAME:$CI_COMMIT_SHA && docker stop $(docker ps -q --filter name=green-app) || true && docker rm $(docker ps -q -a --filter name=green-app) || true && docker run -d --name green-app -p 8000:8000 $DOCKER_REGISTRY/$APP_NAME:$CI_COMMIT_SHA"
    - ssh [email protected] "docker pull $DOCKER_REGISTRY/$APP_NAME:$CI_COMMIT_SHA && docker stop $(docker ps -q --filter name=green-app) || true && docker rm $(docker ps -q -a --name=green-app) || true && docker run -d --name green-app -p 8000:8000 $DOCKER_REGISTRY/$APP_NAME:$CI_COMMIT_SHA"
  tags:
    - shell

test_green_environment:
  image: curlimages/curl:latest
  stage: test_green
  script:
    - echo "Running smoke tests on Green environment..."
    # Replace with actual test commands, e.g., curl against a health check endpoint
    - curl --fail http://192.168.1.20:8000/healthz || exit 1
    - curl --fail http://192.168.1.21:8000/healthz || exit 1
    # Add more comprehensive integration tests here
  tags:
    - shell

promote_to_production:
  image: alpine:latest
  stage: promote
  before_script:
    - apk add --no-cache openssh-client
    - eval $(ssh-agent -s)
    - echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -
    - mkdir -p ~/.ssh
    - chmod 700 ~/.ssh
    - echo "StrictHostKeyChecking no" >> ~/.ssh/config
  script:
    - echo "Switching traffic to Green environment..."
    # This script needs to dynamically update the HAProxy configuration.
    # A robust solution would involve using the OVH API to update the Load Balancer's backend pool.
    # For simplicity, this example assumes direct SSH access to a machine running HAProxy or a management script.
    - ssh $SSH_USER@$SSH_HOST_FOR_LB_CONFIG "
        echo 'Updating HAProxy config to point to Green...'
        # Example: Using sed to modify a local HAProxy config file and reload
        # In a real OVH setup, you'd use the OVH API to modify the LB service.
        sed -i 's/default_backend blue_backend/default_backend green_backend/' /etc/haproxy/haproxy.cfg
        systemctl reload haproxy
      "
    - echo "Traffic switched to Green."
  when: manual # Requires manual approval to switch traffic
  tags:
    - shell

deploy_to_blue:
  image: alpine:latest
  stage: deploy_blue
  before_script:
    - apk add --no-cache openssh-client rsync
    - eval $(ssh-agent -s)
    - echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -
    - mkdir -p ~/.ssh
    - chmod 700 ~/.ssh
    - echo "StrictHostKeyChecking no" >> ~/.ssh/config
  script:
    - echo "Deploying to Blue environment for next cycle..."
    - ssh [email protected] "docker pull $DOCKER_REGISTRY/$APP_NAME:$CI_COMMIT_SHA && docker stop $(docker ps -q --filter name=blue-app) || true && docker rm $(docker ps -q -a --filter name=blue-app) || true && docker run -d --name blue-app -p 8000:8000 $DOCKER_REGISTRY/$APP_NAME:$CI_COMMIT_SHA"
    - ssh [email protected] "docker pull $DOCKER_REGISTRY/$APP_NAME:$CI_COMMIT_SHA && docker stop $(docker ps -q --filter name=blue-app) || true && docker rm $(docker ps -q -a --filter name=blue-app) || true && docker run -d --name blue-app -p 8000:8000 $DOCKER_REGISTRY/$APP_NAME:$CI_COMMIT_SHA"
  tags:
    - shell
  # This stage might run automatically after promotion, or be manual.
  # Consider if you want Blue to be updated immediately or after a period of stability.
  when: on_success # Or 'manual' if you prefer explicit control

Performing the Switch and Rollback

The critical step is the traffic switch. In the `promote_to_production` stage, we need to reconfigure the load balancer. The provided YAML uses a simplified SSH approach to modify a local HAProxy config. In a production OVH environment, you would interact with the OVH Public Cloud Load Balancer API to change the backend pool associated with the frontend.

OVH API Interaction Example (Conceptual Python):

import requests
import json

# Assume you have OVH API credentials and LB ID
OVH_API_URL = "https://api.us. OVHcloud.com/1.0" # Adjust region
LB_ID = "your-load-balancer-id"
FRONTEND_ID = "your-frontend-id" # The ID of the frontend listening on port 80

# Get current frontend configuration
response = requests.get(f"{OVH_API_URL}/cloud/loadbalancer/{LB_ID}/frontend/{FRONTEND_ID}", headers={"X-Auth-Token": "your-ovh-token"})
frontend_config = response.json()

# Find the backend ID for 'green_backend' (you'd need to know this or fetch it)
GREEN_BACKEND_ID = "your-green-backend-id"

# Update the frontend to point to the green backend
update_payload = {
    "defaultBackendId": GREEN_BACKEND_ID
}

response = requests.put(f"{OVH_API_URL}/cloud/loadbalancer/{LB_ID}/frontend/{FRONTEND_ID}",
                        headers={"X-Auth-Token": "your-ovh-token", "Content-Type": "application/json"},
                        data=json.dumps(update_payload))

if response.status_code == 200:
    print("Successfully switched traffic to Green environment.")
else:
    print(f"Error switching traffic: {response.status_code} - {response.text}")
    # Trigger rollback procedure

Rollback Strategy

If the tests on the Green environment fail, or if issues are detected after the switch, a rollback is straightforward. You simply reconfigure the load balancer to point back to the Blue environment (which is still running the previous stable version).

Using the OVH API example, this would involve another `PUT` request to set defaultBackendId back to the ID of the blue_backend.

Advanced Considerations

Database Migrations

Database schema changes are a common challenge. A blue-green deployment requires a backward-compatible schema change. The new application version must be able to run with the old schema, and the old application version must be able to run with the new schema during the transition. This often involves a multi-step deployment:

  1. Deploy a version of the application that can handle both old and new schema structures (if applicable).
  2. Run database migration scripts to update the schema.
  3. Deploy the new application version that relies on the new schema.
  4. Switch traffic.

Alternatively, use database versioning tools like Alembic (for SQLAlchemy) or Django’s migration system, ensuring migrations are applied carefully before or during the deployment phase.

Stateful Applications

For applications with persistent state (e.g., user sessions stored on disk, local caches), blue-green deployments become more complex. Strategies include:

  • Using a shared, external state store (e.g., Redis, Memcached) for sessions.
  • Ensuring state is replicated or can be migrated between environments.
  • Allowing a brief period of downtime or degraded performance during the switch if state cannot be seamlessly transferred.

Canary Releases within Blue-Green

For even greater safety, you can combine blue-green with canary releases. After deploying to the Green environment and performing initial tests, you could route a small percentage of traffic (e.g., 1%, 5%) to Green while the majority still goes to Blue. Monitor closely, and if all is well, gradually increase the traffic to Green until it reaches 100%, at which point the switch is complete.

This requires a more sophisticated load balancer capable of weighted routing, which OVH’s HAProxy-based service supports.

Conclusion

Implementing zero-downtime blue-green deployments on OVH requires careful planning of your infrastructure, robust automation via a CI/CD pipeline, and a clear strategy for handling database changes and potential rollbacks. By leveraging OVH’s Public Cloud Load Balancer and automating the deployment and switching process, you can achieve highly available and reliable releases for your Python applications.