Vengala Vinay

Having 9+ Years of Experience in Software Development

Home » Server Monitoring Best Practices: Keeping Your Ruby App and DynamoDB Clusters Alive on Google Cloud

Server Monitoring Best Practices: Keeping Your Ruby App and DynamoDB Clusters Alive on Google Cloud

Proactive Monitoring for Ruby on Rails & DynamoDB on Google Cloud

Maintaining high availability and optimal performance for a Ruby on Rails application backed by Amazon DynamoDB, deployed on Google Cloud Platform (GCP), requires a multi-layered monitoring strategy. This isn’t just about reacting to alerts; it’s about building a robust system that anticipates issues before they impact users. We’ll focus on key metrics, essential tools, and actionable configurations for both the application layer and the data store.

Application Performance Monitoring (APM) with Skylight

For Ruby on Rails, a dedicated APM tool is non-negotiable. Skylight offers excellent insights into request latency, database query performance, and background job execution. Integrating it into your GCP deployment is straightforward.

Skylight Configuration and Key Metrics

Ensure your Gemfile includes the Skylight gem:

gem 'skylight'

After running bundle install, configure your application with your API key. This is typically done in an initializer file (e.g., config/initializers/skylight.rb).

# config/initializers/skylight.rb
Skylight.core.start(
  'YOUR_SKYLIGHT_API_KEY',
  'your-app-name', # e.g., 'my-production-rails-app'
  environment: Rails.env # e.g., 'production'
)

Key metrics to monitor within Skylight:

  • Request Throughput: Requests per minute/second. Sudden drops or spikes can indicate issues.
  • Average Request Time: Overall latency. Monitor trends and outliers.
  • Database Query Time: Crucial for identifying slow queries. Pay attention to queries exceeding 100ms.
  • External Service Calls: Latency and error rates for API integrations.
  • Background Job Performance: Queue depth, job execution time, and failure rates for Sidekiq/Resque.
  • Memory Usage: Track memory consumption per process.
  • CPU Utilization: High CPU can indicate inefficient code or insufficient resources.

Leveraging Google Cloud Monitoring (formerly Stackdriver)

Google Cloud Monitoring is your central hub for infrastructure and application metrics within GCP. It integrates seamlessly with Compute Engine, Kubernetes Engine, and other GCP services.

Configuring Cloud Monitoring Agents

For Compute Engine instances running your Rails app, ensure the Cloud Monitoring agent is installed and configured. This is often pre-installed on GCP images. If not, you can install it manually.

# On a Debian/Ubuntu instance
curl -sSO https://dl.google.com/cloudagents/add-monitoring-agent-repo.sh
sudo bash add-monitoring-agent-repo.sh --also-install

# On a RHEL/CentOS instance
curl -sSO https://dl.google.com/cloudagents/add-monitoring-agent-repo.sh
sudo bash add-monitoring-agent-repo.sh --also-install

Verify the agent status:

sudo systemctl status google-cloud-monitoring-agent

Key GCP Metrics for Rails Instances

Within Cloud Monitoring, focus on these metrics for your Compute Engine instances (or GKE nodes):

  • CPU Utilization: compute.googleapis.com/instance/cpu/utilization. Set alerts for sustained high usage (e.g., > 80% for 5 minutes).
  • Memory Usage: compute.googleapis.com/instance/memory/utilization. Requires the Ops Agent. Monitor for high consumption.
  • Disk I/O: compute.googleapis.com/instance/disk/read_bytes_count and write_bytes_count. High I/O can bottleneck performance.
  • Network Traffic: compute.googleapis.com/instance/network/received_bytes_count and sent_bytes_count. Monitor for unusual spikes or drops.
  • Process Count: If you’re running multiple Rails processes (e.g., via Puma), monitor the number of active processes.

Monitoring DynamoDB Performance and Health

DynamoDB, while managed, still requires careful monitoring to ensure optimal performance and cost-efficiency. AWS CloudWatch is the primary tool here, but we can ingest these metrics into GCP Monitoring for a unified view.

Key DynamoDB Metrics in CloudWatch

Focus on these critical DynamoDB metrics:

  • Consumed Read/Write Capacity Units: ConsumedReadCapacityUnits and ConsumedWriteCapacityUnits. Essential for understanding throughput and potential throttling.
  • Throttled Requests: ReadThrottleEvents and WriteThrottleEvents. Any throttling indicates you’re hitting provisioned capacity limits and need to scale up or optimize queries.
  • System Errors: SystemErrors. Monitor for any server-side errors.
  • Latency: SuccessfulRequestLatency (average, p90, p99). High latency directly impacts your application’s responsiveness.
  • Item Count: ItemCount. Useful for understanding table size and growth.
  • Table Size: TableSizeBytes. Monitor for unexpected growth.

Ingesting CloudWatch Metrics into GCP Monitoring

To achieve a single pane of glass, you can use the CloudWatch agent to export metrics to GCP Monitoring. This involves setting up a CloudWatch agent on an EC2 instance (or a VM in GCP that can access AWS APIs) and configuring it to send metrics to GCP.

First, install the CloudWatch agent on a designated VM. Refer to AWS documentation for the latest installation instructions.

Next, configure the agent’s amazon-cloudwatch-agent.json file. You’ll need to specify both the metrics to collect from DynamoDB and the output destination to GCP Monitoring.

[
  {
    "metrics": {
      "namespace": "AWS/DynamoDB",
      "metrics_collected": {
        "table_metrics": {
          "table_name_filter": {
            "wildcard": "my-dynamodb-table-*"
          },
          "metrics_append_dimensions": [
            "TableName"
          ],
          "metrics": [
            "ConsumedReadCapacityUnits",
            "ConsumedWriteCapacityUnits",
            "ReadThrottleEvents",
            "WriteThrottleEvents",
            "SuccessfulRequestLatency"
          ]
        }
      }
    }
  }
]

You’ll also need to configure the agent to send these metrics to GCP. This typically involves setting up GCP credentials for the agent and specifying the GCP project and metric endpoint. The exact configuration details can be complex and depend on your specific setup, often involving custom agent configurations or third-party tools like Fluentd or Logstash for metric forwarding.

Alerting Strategies and Thresholds

Effective alerting is crucial. Avoid alert fatigue by setting meaningful thresholds and using appropriate notification channels.

Application Alerts (Skylight & GCP)

  • High Request Latency: Alert when average request time exceeds 500ms for 5 minutes.
  • High Error Rate: Alert when the application error rate (5xx responses) exceeds 1% of total requests over 10 minutes.
  • Slow Database Queries: Alert when the average time for a specific critical query exceeds 200ms for 5 minutes.
  • Background Job Failures: Alert when more than 5% of jobs in a critical queue fail within an hour.
  • Resource Saturation: Alert when CPU utilization on any Rails instance consistently exceeds 80% for 15 minutes.

DynamoDB Alerts (GCP Monitoring/CloudWatch)

  • Throttled Requests: Alert immediately if ReadThrottleEvents or WriteThrottleEvents are greater than 0 for any table. This is a critical indicator of performance degradation.
  • High Latency: Alert when the 95th percentile of SuccessfulRequestLatency exceeds 150ms for 5 minutes.
  • Consumed Capacity Approaching Limit: For provisioned tables, alert when ConsumedReadCapacityUnits or ConsumedWriteCapacityUnits consistently exceed 80% of provisioned capacity for 10 minutes.
  • System Errors: Alert on any increase in SystemErrors.

Log Aggregation and Analysis

Beyond metrics, centralized logging is vital for debugging and root cause analysis. GCP’s Cloud Logging (formerly Stackdriver Logging) is the natural choice for applications running on GCP.

Configuring Log Collection

Ensure your Rails application logs to standard output (stdout) and standard error (stderr) when running in containers or on Compute Engine. The Cloud Logging agent automatically collects these logs.

For specific log files (e.g., Puma logs, Sidekiq logs), you might need to configure the Ops Agent (the successor to the Cloud Monitoring agent) to tail these files and send them to Cloud Logging.

# Example Ops Agent configuration snippet for log collection
logging:
  receivers:
    - type: files
      name: rails-app-logs
      record_log_line: true
      include_paths:
        - /var/log/rails/production.log
      log_name: rails-production
  processors:
    - type: parsing_ளர்
      name: parse-rails-log
      log_format: json # If your app logs in JSON
      # Or use regex for non-JSON logs
  forwarders:
    - type: google_cloud
      name: google-cloud-logging
      # ... other configurations ...

DynamoDB Log Analysis

DynamoDB itself doesn’t generate application-level logs in the same way. However, enabling DynamoDB Debug Logging (if necessary for deep diagnostics) and integrating it with CloudWatch Logs is possible. More commonly, you’ll analyze the application’s logs for interactions with DynamoDB, correlating them with the metrics collected.

Health Checks and Synthetic Monitoring

Proactive health checks and synthetic monitoring simulate user interactions to catch issues before users do.

Application Health Endpoints

Implement a simple health check endpoint in your Rails application (e.g., /health) that checks the status of critical dependencies like the database connection and any external services.

# config/routes.rb
get '/health', to: 'health#show'

# app/controllers/health_controller.rb
class HealthController < ApplicationController
  def show
    # Check database connection
    begin
      ActiveRecord::Base.connection.execute('SELECT 1')
      db_status = 'OK'
    rescue => e
      db_status = "Error: #{e.message}"
    end

    # Add checks for other critical services (e.g., Redis, external APIs)

    if db_status == 'OK' # && other_services_ok
      render json: { status: 'OK', database: db_status }, status: :ok
    else
      render json: { status: 'ERROR', database: db_status }, status: :internal_server_error
    end
  end
end

GCP Load Balancer Health Checks

Configure your GCP HTTP(S) Load Balancer to use this health check endpoint. This ensures that unhealthy instances are automatically removed from the load balancing pool.

# Example gcloud command to create a health check
gcloud compute health-checks create http /health \
  --port 8080 \
  --request-path=/health \
  --check-interval=5s \
  --timeout=5s \
  --unhealthy-threshold=2 \
  --healthy-threshold=2 \
  --global

Synthetic Monitoring with Cloud Monitoring Uptime Checks

Use GCP Cloud Monitoring’s Uptime Checks to periodically ping your application’s public endpoints (including the health check) from various global locations. This provides an external perspective on availability.

Conclusion

A comprehensive monitoring strategy for your Ruby on Rails application and DynamoDB cluster on GCP involves integrating APM, infrastructure metrics, database-specific metrics, centralized logging, and proactive health checks. By focusing on key metrics, configuring appropriate alerts, and leveraging tools like Skylight and Google Cloud Monitoring, you can build a resilient and performant system that minimizes downtime and ensures a positive user experience.