Server Monitoring Best Practices: Keeping Your Perl App and DynamoDB Clusters Alive on Google Cloud

Proactive Perl Application Health Checks on Google Cloud

Maintaining the health of a Perl application, especially one serving critical functions, requires more than just basic process monitoring. We need to implement deep health checks that validate application-level functionality. On Google Cloud, this often involves leveraging Compute Engine instances and potentially Kubernetes Engine (GKE). A common pattern is to expose a dedicated health check endpoint within the Perl application itself.

Consider a Perl application using the Mojolicious web framework. We can add a simple route to expose an HTTP endpoint that performs internal checks.

Perl Mojolicious Health Check Endpoint

package MyApp::Controller::Health;
use Mojo::Base 'Mojolicious::Controller';

sub check {
    my $self = shift;

    # Simulate a database connection check (replace with actual DB logic)
    my $db_ok = 1; # Assume true for this example
    eval {
        # Example: Connect to a hypothetical database
        # my $dbh = DBI->connect("dbi:Pg:dbname=mydb;host=localhost", "user", "pass");
        # if (!$dbh) { die "DB connection failed: " . DBI::errstr; }
        # $dbh->ping;
        # $dbh->disconnect;
    };
    if ($@) {
        $db_ok = 0;
        $self->log->error("Database health check failed: $@");
    }

    # Simulate an external service check
    my $external_service_ok = 1;
    eval {
        # Example: Make a simple HTTP request to an external dependency
        # my $ua = LWP::UserAgent->new;
        # my $response = $ua->get('http://external.service.com/health');
        # if (!$response->is_success) { die "External service unhealthy: " . $response->status_line; }
    };
    if ($@) {
        $external_service_ok = 0;
        $self->log->error("External service health check failed: $@");
    }

    if ($db_ok && $external_service_ok) {
        $self->render(text => 'OK', status => 200);
    } else {
        $self->render(text => 'Unhealthy', status => 503);
    }
}

1;

In your Mojolicious application’s routes file (e.g., myapp/routes/home.pm):

package MyApp::Routes::Home;
use Mojo::Base 'Mojolicious::Routes';

sub define_routes {
    my $r = shift;

    # Basic route
    $r->get('/')->to('welcome#hello');

    # Health check route
    $r->get('/health')->to('health#check');
}

1;

This endpoint, when accessed via HTTP, will perform internal checks and return a 200 OK if everything is healthy, or a 503 Service Unavailable if any critical component fails. This is crucial for load balancers and orchestrators.

Configuring Google Cloud Load Balancer Health Checks

Google Cloud Load Balancing (both HTTP(S) Load Balancing and Network Load Balancing) can be configured to use these application-level health checks. For HTTP(S) Load Balancing, you’ll define a Health Check resource that points to your application’s health endpoint.

Here’s how you’d configure it using `gcloud` CLI:

gcloud compute health-checks create http perl-app-health-check \
    --request-path="/health" \
    --port=8080 \
    --check-interval=5s \
    --timeout=5s \
    --unhealthy-threshold=2 \
    --healthy-threshold=2 \
    --description="Health check for Perl application instances"

Explanation:

• --request-path="/health": Specifies the URL path the load balancer will request.
• --port=8080: The port your Perl application is listening on. Adjust this as necessary.
• --check-interval=5s: How often to perform the check.
• --timeout=5s: How long to wait for a response.
• --unhealthy-threshold=2: Number of consecutive failures to mark an instance unhealthy.
• --healthy-threshold=2: Number of consecutive successes to mark an instance healthy.

This health check resource then needs to be associated with your backend service. If you’re using a managed instance group (MIG), you’d attach this health check to the MIG’s instance template or directly to the backend service that the MIG is part of.

Monitoring DynamoDB with Cloud Monitoring and Custom Metrics

DynamoDB, being a managed service, offers robust built-in metrics through AWS CloudWatch. However, for a unified view within Google Cloud’s ecosystem, especially if you’re migrating or have a hybrid setup, you’ll want to ingest these metrics into Google Cloud Monitoring (formerly Stackdriver). Alternatively, if your Perl application interacts with DynamoDB, you can emit custom metrics from your application.

Ingesting CloudWatch Metrics to Google Cloud Monitoring

The most straightforward way to get AWS CloudWatch metrics into Google Cloud Monitoring is by using the Cloud Monitoring agent with its CloudWatch integration. This requires setting up IAM roles and permissions correctly on the AWS side and configuring the agent on a Google Cloud VM that has access to your AWS account (e.g., via IAM roles for EC2 or by using AWS access keys securely).

First, ensure the Cloud Monitoring agent is installed on a VM. Then, configure its collection settings. You’ll need to create a configuration file, typically in /etc/google-cloud-monitoring-agent/config.d/.

logging:
  # Log CloudWatch metrics to Google Cloud Logging
  # This is optional but useful for debugging
  forward_to_cloud_logging: true

metrics:
  # Collect metrics from AWS CloudWatch
  cloudwatch:
    # Replace with your AWS region
    aws_region: "us-east-1"
    # List of DynamoDB tables to monitor
    tables:
      - table_name: "your-dynamodb-table-name-1"
        # List of metrics to collect for each table
        metrics:
          - "ConsumedReadCapacityUnits"
          - "ConsumedWriteCapacityUnits"
          - "ProvisionedReadCapacityUnits"
          - "ProvisionedWriteCapacityUnits"
          - "ThrottledRequests"
          - "SuccessfulRequestLatency"
      - table_name: "your-dynamodb-table-name-2"
        metrics:
          - "ConsumedReadCapacityUnits"
          - "ConsumedWriteCapacityUnits"
          - "ThrottledRequests"
          - "SuccessfulRequestLatency"
    # Optional: Specify AWS credentials if not using IAM roles for EC2
    # credentials:
    #   access_key_id: "YOUR_ACCESS_KEY_ID"
    #   secret_access_key: "YOUR_SECRET_ACCESS_KEY"

After placing this configuration file (e.g., /etc/google-cloud-monitoring-agent/config.d/cloudwatch.yaml), restart the agent:

sudo systemctl restart google-cloud-monitoring-agent

Once ingested, these metrics will appear in Google Cloud Monitoring under the “CloudWatch” metric source, allowing you to create dashboards and alerting policies alongside your GCP resources.

Emitting Custom Perl Metrics for DynamoDB Operations

For more granular insights or to correlate DynamoDB operations directly with your application’s performance, emitting custom metrics from your Perl application is highly effective. We can use the Google Cloud Monitoring client libraries for Perl, or more commonly, push metrics via the Cloud Monitoring API or OpenTelemetry.

Here’s an example using a hypothetical Perl client that interacts with DynamoDB and pushes custom metrics. This example assumes you have a way to authenticate and send metrics to the Cloud Monitoring API (e.g., using a service account key).

package MyApp::DynamoDBMonitor;
use strict;
use warnings;
use Google::Cloud::Monitoring::V3;
use Google::Cloud::Monitoring::V3::MetricServiceClient;
use Google::Cloud::Monitoring::V3::MetricDescriptor;
use Google::Cloud::Monitoring::V3::Metric;
use Google::Cloud::Monitoring::V3::Point;
use Google::Cloud::Monitoring::V3::TimeInterval;
use Google::Cloud::Monitoring::V3::TypedValue;
use Google::Cloud::Monitoring::V3::ListTimeSeriesRequest;
use Time::HiRes qw(time);

sub new {
    my ($class, $project_id, $service_account_file) = @_;
    my $self = bless {}, $class;

    $self->{project_id} = $project_id;
    $self->{client} = Google::Cloud::Monitoring::V3::MetricServiceClient->new(
        project_id => $project_id,
        # If using a service account file:
        # json_key_file => $service_account_file,
    );

    # Ensure metric descriptors exist (or create them)
    $self->ensure_metric_descriptor('custom.googleapis.com/dynamodb/operation_latency');
    $self->ensure_metric_descriptor('custom.googleapis.com/dynamodb/operation_count');
    $self->ensure_metric_descriptor('custom.googleapis.com/dynamodb/throttled_operations');

    return $self;
}

sub ensure_metric_descriptor {
    my ($self, $metric_type) = @_;
    # In a real scenario, you'd check if it exists first and only create if not.
    # For simplicity, we'll just attempt to create.
    # A more robust implementation would use get_metric_descriptor and handle errors.

    my $descriptor = Google::Cloud::Monitoring::V3::MetricDescriptor->new({
        type       => $metric_type,
        metricKind => 'DELTA' eq $metric_type =~ /count|throttled/i ? 'DELTA' : 'GAUGE',
        valueType  => 'INT64' eq $metric_type =~ /count|throttled/i ? 'INT64' : 'DOUBLE',
        description => "Custom metric for DynamoDB $metric_type",
        labels => [
            { key => 'operation', description => 'Type of DynamoDB operation (e.g., GetItem, PutItem)' },
            { key => 'table_name', description => 'Name of the DynamoDB table' },
        ],
    });

    eval {
        $self->{client}->create_metric_descriptor(name => "projects/$self->{project_id}", metricDescriptor => $descriptor);
        $self->log("Created metric descriptor: $metric_type");
    };
    if ($@) {
        # Ignore "already exists" errors, log others
        if ($@ !~ /already exists/i) {
            $self->log("Error creating metric descriptor $metric_type: $@");
        }
    }
}

sub record_operation {
    my ($self, $operation_type, $table_name, $duration_ms, $is_throttled) = @_;

    my $now = time();

    # Record latency
    if (defined $duration_ms) {
        my $latency_metric = Google::Cloud::Monitoring::V3::Metric->new({
            type => 'custom.googleapis.com/dynamodb/operation_latency',
            labels => {
                operation => $operation_type,
                table_name => $table_name,
            },
        });
        my $latency_value = Google::Cloud::Monitoring::V3::TypedValue->new({ doubleValue => $duration_ms });
        my $latency_point = Google::Cloud::Monitoring::V3::Point->new({
            interval => Google::Cloud::Monitoring::V3::TimeInterval->new({
                endTime => { seconds => int($now), nanos => int(($now - int($now)) * 1e9) },
            }),
            value => $latency_value,
        });
        $self->push_metric($latency_metric, $latency_point);
    }

    # Record operation count
    my $count_metric = Google::Cloud::Monitoring::V3::Metric->new({
        type => 'custom.googleapis.com/dynamodb/operation_count',
        labels => {
            operation => $operation_type,
            table_name => $table_name,
        },
    });
    my $count_value = Google::Cloud::Monitoring::V3::TypedValue->new({ int64Value => 1 });
    my $count_point = Google::Cloud::Monitoring::V3::Point->new({
        interval => Google::Cloud::Monitoring::V3::TimeInterval->new({
            endTime => { seconds => int($now), nanos => int(($now - int($now)) * 1e9) },
        }),
        value => $count_value,
    });
    $self->push_metric($count_metric, $count_point);

    # Record throttled operations
    if ($is_throttled) {
        my $throttled_metric = Google::Cloud::Monitoring::V3::Metric->new({
            type => 'custom.googleapis.com/dynamodb/throttled_operations',
            labels => {
                operation => $operation_type,
                table_name => $table_name,
            },
        });
        my $throttled_value = Google::Cloud::Monitoring::V3::TypedValue->new({ int64Value => 1 });
        my $throttled_point = Google::Cloud::Monitoring::V3::Point->new({
            interval => Google::Cloud::Monitoring::V3::TimeInterval->new({
                endTime => { seconds => int($now), nanos => int(($now - int($now)) * 1e9) },
            }),
            value => $throttled_value,
        });
        $self->push_metric($throttled_metric, $throttled_point);
    }
}

sub push_metric {
    my ($self, $metric, $point) = @_;

    my $time_series = Google::Cloud::Monitoring::V3::TimeSeries->new({
        metric => $metric,
        points => [$point],
    });

    my $request = Google::Cloud::Monitoring::V3::ListTimeSeriesRequest->new({
        name => "projects/$self->{project_id}",
        filter => qq(metric.type = "${metric->{type}}" AND metric.labels.operation = "${metric->{labels}{operation}}" AND metric.labels.table_name = "${metric->{labels}{table_name}}"),
        interval => {
            endTime => { seconds => int(time()), nanos => int((time() - int(time())) * 1e9) },
            # Look back a short period to potentially find existing time series for aggregation
            startTime => { seconds => int(time() - 60), nanos => 0 },
        },
        aggregation => {
            alignmentPeriod => { seconds => 60 }, # Aggregate over 60 seconds
            perSeriesAligner => 'ALIGN_SUM', # Sum up points within the alignment period
        },
    });

    my $existing_series = $self->{client}->list_time_series(request => $request);

    if ($existing_series && @{$existing_series}) {
        # If an existing time series is found for this metric and labels,
        # we can potentially update it or aggregate. For simplicity here,
        # we'll just push a new point. A more advanced approach would
        # aggregate the new point into the existing time series if the
        # alignment period hasn't passed.
        # For DELTA metrics, it's often better to push individual points and let GCP aggregate.
        # For GAUGE metrics, you might want to update the latest value.
        # This example pushes a new point, suitable for DELTA metrics like counts.
    }

    eval {
        $self->{client}->create_time_series(name => "projects/$self->{project_id}", timeSeries => [$time_series]);
        $self->log("Pushed metric: $metric->{type} for table $metric->{labels}{table_name}");
    };
    if ($@) {
        $self->log("Error pushing metric: $@");
    }
}

sub log {
    my ($self, $message) = @_;
    print STDERR "[", scalar(localtime), "] $message\n";
}

1;

To use this, you would instantiate it and call record_operation after each DynamoDB interaction:

use MyApp::DynamoDBMonitor;

my $project_id = 'your-gcp-project-id';
my $service_account_file = '/path/to/your/service-account-key.json'; # Optional

my $monitor = MyApp::DynamoDBMonitor->new($project_id, $service_account_file);

# Example usage within your application logic:
my $start_time = time();
my $operation_type = 'GetItem';
my $table_name = 'users';
my $is_throttled = 0;

# ... perform DynamoDB GetItem operation ...
# For example, using AWS SDK for Perl (Paws)
# my $result = $dynamodb_client->get_item({
#     table_name => $table_name,
#     key => { id => { S => 'user123' } },
# });
# if ($result->{error}) {
#     if ($result->{error}->{code} == 400 && $result->{error}->{message} =~ /ProvisionedThroughputExceededException/) {
#         $is_throttled = 1;
#     }
#     # Handle other errors
# }

my $end_time = time();
my $duration_ms = ($end_time - $start_time) * 1000;

$monitor->record_operation($operation_type, $table_name, $duration_ms, $is_throttled);

This approach provides fine-grained visibility into your application’s interaction with DynamoDB, enabling you to set up alerts for high latency, excessive throttling, or unusual operation counts directly within Google Cloud Monitoring.

Alerting Strategies for Production Systems

Effective alerting is the cornerstone of proactive system management. For both your Perl application and DynamoDB, we need to define alert policies that are actionable and minimize alert fatigue.

Perl Application Alerts

Leveraging the health check endpoint, we can configure alerts in Google Cloud Monitoring:

• Metric: loadbalancing.googleapis.com/https/backend_service/request_count (or equivalent for your LB type)
• Filter: `backend_service_name=”your-backend-service-name” AND response_code_class=”CLASS_5XX”` (or specifically for 503s if your health check returns that on failure)
• Condition: Alert when the rate of 5xx errors exceeds a threshold (e.g., > 0 for 5 minutes).
• Notification Channels: PagerDuty, Slack, email.

Additionally, monitor application logs for critical errors. Configure log-based metrics for specific error patterns (e.g., “Database connection error”, “External service timeout”) and set alerts on those metrics.

DynamoDB Alerts (via CloudWatch/GCP Monitoring)

Using the ingested CloudWatch metrics or custom metrics:

• Metric: cloudwatch.amazonaws.com/DynamoDB/ConsumedReadCapacityUnits (or custom.googleapis.com/dynamodb/operation_count with operation=”Read”)
• Condition: Alert if ConsumedReadCapacityUnits is consistently close to or exceeding ProvisionedReadCapacityUnits (if using provisioned capacity). A common alert is for ThrottledRequests > 0 for a sustained period.
• Metric: cloudwatch.amazonaws.com/DynamoDB/SuccessfulRequestLatency (or custom.googleapis.com/dynamodb/operation_latency)
• Condition: Alert if the 95th or 99th percentile latency exceeds a defined threshold (e.g., > 500ms).
• Metric: custom.googleapis.com/dynamodb/throttled_operations
• Condition: Alert if any throttled operations are detected for a sustained period.

For custom metrics, ensure your alert policies are configured to aggregate correctly (e.g., sum of throttled operations over a period, average latency). The key is to set thresholds that indicate a genuine problem requiring immediate attention, rather than transient spikes.