Scaling Ruby on Google Cloud to Handle 50,000+ Concurrent Requests

Architectural Foundation: Load Balancing and Auto-Scaling on Google Cloud

Achieving 50,000+ concurrent requests for a Ruby application on Google Cloud Platform (GCP) necessitates a robust, horizontally scalable architecture. The core components are Google Cloud Load Balancing and Compute Engine’s Managed Instance Groups (MIGs) with auto-scaling. This setup ensures high availability, fault tolerance, and the ability to dynamically adjust resources based on demand.

We’ll leverage a Global External HTTP(S) Load Balancer. This provides a single, global IP address for your application, distributing traffic across multiple regions for optimal performance and resilience. The backend will consist of MIGs, each containing a fleet of Compute Engine instances running your Ruby application.

Configuring Google Cloud Load Balancer

The setup involves several key GCP resources:

Backend Service: Defines how the load balancer distributes traffic to backends.
Health Check: Crucial for determining instance health and removing unhealthy instances from rotation.
URL Map: Routes incoming requests to different backend services based on host and path.
Target Proxy: Terminates client connections (HTTP or HTTPS).
Forwarding Rule: Directs traffic from the load balancer’s IP address to the target proxy.

For a Ruby application, a simple HTTP health check on a dedicated endpoint (e.g., /health) is often sufficient. This endpoint should return a 200 OK status code if the application is responsive.

Example Health Check Configuration (gcloud CLI)

gcloud compute health-checks create http ruby-app-health-check \
    --request-path=/health \
    --port=8080 \
    --check-interval=5s \
    --timeout=5s \
    --unhealthy-threshold=2 \
    --healthy-threshold=2

Example Backend Service Configuration (gcloud CLI)

gcloud compute backend-services create ruby-app-backend-service \
    --protocol=HTTP \
    --port-name=http \
    --health-checks=ruby-app-health-check \
    --global

Example URL Map and Target Proxy (gcloud CLI)

# Create a default URL map
gcloud compute url-maps create ruby-app-url-map \
    --default-service ruby-app-backend-service

# Create an HTTP target proxy
gcloud compute target-http-proxies create ruby-app-http-proxy \
    --url-map=ruby-app-url-map

# Create a global forwarding rule
gcloud compute forwarding-rules create ruby-app-forwarding-rule \
    --address=YOUR_STATIC_IP_ADDRESS \
    --target-http-proxy=ruby-app-http-proxy \
    --ports=80 \
    --global

Note: Replace YOUR_STATIC_IP_ADDRESS with a reserved static IP address for your load balancer. For HTTPS, you would use a Target HTTPS Proxy and configure SSL certificates.

Managed Instance Groups and Auto-scaling

Managed Instance Groups (MIGs) are the workhorses for scaling. They maintain a specified number of VM instances and can automatically scale up or down based on defined metrics.

Instance Template

First, define an instance template. This specifies the machine type, boot disk image, startup script (to install dependencies and start your Ruby app), and any necessary metadata.

The startup script is critical. It should handle:

Installing Ruby, Bundler, and any system dependencies.
Cloning your application code from a repository (e.g., Git).
Running bundle install.
Configuring the application (e.g., database connections, environment variables).
Starting your Ruby web server (e.g., Puma, Unicorn).

Ensure your Ruby application server is configured to listen on 0.0.0.0 and the port specified in your health check (e.g., 8080).

Example Instance Template (gcloud CLI)

gcloud compute instance-templates create ruby-app-template \
    --machine-type=e2-medium \
    --image-family=ubuntu-2004-lts \
    --image-project=ubuntu-os-cloud \
    --metadata=startup-script='#! /bin/bash
        sudo apt-get update -y
        sudo apt-get install -y ruby-full build-essential git
        sudo gem install bundler
        cd /opt/my_ruby_app
        git pull origin main
        bundle install --without development test
        # Assuming Puma is used and configured to run as a service
        # Example: systemctl start puma
    ' \
    --scopes=cloud-platform \
    --tags=http-server,https-server

Note: Adjust machine-type, image-family, and the startup script content to match your application’s specific requirements. For production, consider using a custom image for faster boot times and more control over the environment.

Creating the Managed Instance Group

Now, create the MIG using the instance template. We’ll configure auto-scaling based on CPU utilization.

Example MIG and Auto-scaling Configuration (gcloud CLI)

gcloud compute instance-groups managed create ruby-app-mig \
    --template=ruby-app-template \
    --size=2 \
    --zone=us-central1-a \
    --target-size=2 \
    --min-num-replicas=2 \
    --max-num-replicas=50 \
    --cool-down-period=60 \
    --autoscaling-policy=cpu-utilization=0.6,min-num-replicas=2,max-num-replicas=50,cool-down-period=60

# Add the MIG to the backend service
gcloud compute backend-services add-backend ruby-app-backend-service \
    --instance-group=ruby-app-mig \
    --instance-group-zone=us-central1-a \
    --global

Explanation:

--size=2: Initial number of instances.
--min-num-replicas=2, --max-num-replicas=50: Define the scaling boundaries.
--autoscaling-policy=cpu-utilization=0.6: Scale up when average CPU utilization across the group exceeds 60%.
--cool-down-period=60: Wait 60 seconds after a scaling event before another can occur, preventing rapid fluctuations.

Optimizing Ruby Performance for High Concurrency

Even with a scalable infrastructure, the Ruby application itself must be performant. For high concurrency, consider the following:

Web Server Choice and Configuration

Puma is a popular choice for Ruby on Rails applications due to its multi-threaded and multi-process capabilities. Unicorn is another option, primarily multi-process. For 50,000+ concurrent requests, a hybrid approach (multi-process with multiple threads per process) is often optimal.

Example Puma Configuration (`config/puma.rb`)

# config/puma.rb

# Change to the directory of your application
directory '/opt/my_ruby_app'

# Set the environment
environment ENV.fetch('RAILS_ENV') { 'production' }

# Number of threads to use per worker process
# This is a critical tuning parameter. Start with a value slightly higher
# than the number of CPU cores available to your instance, but monitor
# memory usage. For e2-medium (2 vCPUs), 4-8 threads might be a good start.
threads 4, 8

# Number of worker processes.
# For multi-core machines, this is often set to the number of cores.
# For high concurrency, consider setting this to a higher number if your
# application is I/O bound and can benefit from more concurrent requests
# being processed by different workers.
workers ENV.fetch('WEB_CONCURRENCY') { 4 }.to_i

# Bind to TCP port 8080 and use a Unix socket for communication between workers
# (if using multi-process) or for the master process to communicate with workers.
# For GCP Load Balancer, we bind to 0.0.0.0:8080.
bind "tcp://0.0.0.0:8080"

# If using Unix sockets for inter-process communication (e.g., with Unicorn or
# if Puma is managed by a process manager that uses sockets), uncomment below.
# For direct LB integration, binding to TCP is more common.
# pidfile 'tmp/pids/puma.pid'
# state_path 'tmp/pids/puma.state'

# Activate the master process.
activate_control_ திரெட்

# Preload the application before starting the workers.
# This significantly speeds up the startup time of new workers.
preload_app!

# Logging
stdout_redirect '/var/log/puma.stdout.log', '/var/log/puma.stderr.log', true

# Allow workers to restart gracefully.
restart_dir 'tmp/restart.txt'

# On-worker-boot callback.
# This is called after workers are forked.
on_worker_boot do
  # Worker specific setup for Rails.
  # Example: If you need to re-establish database connections.
  if defined?(ActiveRecord::Base)
    ActiveRecord::Base.connection_pool.disconnect!
  end
end

# Before restarting workers, gracefully shut down the old ones.
before_fork do
  # Worker specific setup for Rails.
  # Example: If you need to re-establish database connections.
  if defined?(ActiveRecord::Base)
    ActiveRecord::Base.connection_pool.disconnect!
  end
end

# Allow Puma to be restarted by `rails restart` command.
plugin :tmp_restart

Tuning threads and workers: This is the most critical part. Start with a ratio of threads to CPU cores that allows for I/O wait. For example, on a 4-core machine, 4 workers with 8 threads each (total 32 threads) might be a starting point if your app is heavily I/O bound. Monitor CPU, memory, and request latency closely. If memory becomes an issue, reduce threads or workers. If CPU is maxed out and latency is high, you might need more workers or larger instances.

Database Connection Pooling

Your database connection pool size must be carefully managed. If your web server is configured with 32 threads, and each thread can potentially grab a connection, your pool size needs to accommodate this. However, an excessively large pool can exhaust database resources.

Example Rails Database Configuration (`config/database.yml`)

production:
  adapter: postgresql
  encoding: unicode
  database: my_app_production
  pool: 25  # Adjust this based on your Puma workers/threads and database capacity
  username: my_app_user
  password: <%= ENV['DATABASE_PASSWORD'] %>
  host: <%= ENV['DATABASE_HOST'] %>
  port: 5432

Tuning pool: The general rule of thumb is pool = (workers * threads) / max_concurrent_requests_per_thread. However, it’s more practical to set it to a value slightly higher than your maximum expected concurrent requests per worker process, and then monitor your database’s connection count. For 50,000+ concurrent requests, you’ll likely be using Cloud SQL or a similar managed database service, which has its own scaling limits.

Caching Strategies

Aggressive caching is paramount. Implement:

Fragment Caching: Cache parts of your views.
Page Caching: For static or rarely changing pages.
Low-Level Caching: Cache expensive computations or data fetches using Redis or Memcached.
HTTP Caching: Leverage `Cache-Control` and `ETag` headers.

Background Jobs

Offload any non-critical, time-consuming tasks (e.g., sending emails, processing images, generating reports) to background job processors like Sidekiq or Delayed Job. These should run on separate worker instances, potentially also managed by MIGs, to avoid blocking your web front-end.

Monitoring and Performance Tuning

Continuous monitoring is essential for identifying bottlenecks and optimizing performance. Utilize GCP’s built-in tools and integrate application-level monitoring.

Google Cloud Monitoring (Stackdriver)

Configure custom metrics and alerts for:

Load Balancer Latency: Track end-to-end request times.
Compute Engine CPU Utilization: Monitor MIG instance load.
Compute Engine Network Throughput: Identify network bottlenecks.
Managed Instance Group Health: Ensure instances are healthy and scaling correctly.
Custom Application Metrics: Track request rates, error rates, and response times from within your Ruby application.

Application Performance Monitoring (APM)

Tools like New Relic, Datadog, or Scout APM provide deep insights into your Ruby application’s performance. They can pinpoint slow database queries, inefficient code paths, and external service call latencies.

Logging

Centralize your application logs using Cloud Logging. Ensure your Ruby application logs are structured (e.g., JSON format) for easier analysis. Monitor logs for:

High error rates.
Slow request processing times (if logged).
Database query performance.

Advanced Considerations

Database Scaling

For 50,000+ concurrent requests, your database will likely become the bottleneck. Consider:

Cloud SQL Read Replicas: Offload read traffic from your primary instance.
Database Sharding: If write contention is high, consider sharding your database. This is a complex undertaking.
Connection Pooling at the Application Level: As discussed, ensure your pool is sized correctly.
Optimized Queries: Profile and optimize your most frequent and slowest SQL queries.

Caching Layer

Deploying a dedicated Redis or Memcached cluster (e.g., Memorystore on GCP) for your caching layer is highly recommended. This offloads read operations from your database and provides faster data retrieval.

CDN Integration

For static assets (CSS, JavaScript, images), use a Content Delivery Network (CDN) like Cloud CDN. This reduces load on your Compute Engine instances and improves client-side load times.

Deployment Strategy

Implement a zero-downtime deployment strategy. This typically involves:

Deploying new code to a subset of instances.
Performing health checks.
Gradually shifting traffic to the new version.
Rolling back if issues are detected.
Tools like Cloud Deploy or custom CI/CD pipelines can automate this.

By combining a robust GCP infrastructure with a finely tuned Ruby application, you can effectively scale to handle tens of thousands of concurrent requests.