How We Audited a High-Traffic Perl Enterprise Stack on Google Cloud and Mitigated untrusted command injection in system utility scripts

Initial Assessment: Identifying the Attack Surface

Our engagement began with a deep dive into a high-traffic Perl enterprise stack hosted on Google Cloud Platform (GCP). The primary objective was to identify and mitigate security vulnerabilities, with a specific focus on untrusted command injection within system utility scripts. The stack comprised several microservices written in Perl, interacting with a PostgreSQL database, and managed via Kubernetes. The initial attack surface assessment involved a comprehensive review of:

Perl application code, particularly modules handling external input.
System utility scripts executed by the Perl applications.
Kubernetes deployment configurations and network policies.
Google Cloud IAM roles and permissions.
Logging and monitoring configurations.

The critical area of concern quickly became the numerous shell scripts and Perl scripts used for system administration, deployment, and maintenance. These scripts often accepted parameters that were not rigorously validated, creating a direct pathway for command injection if an attacker could influence these parameters.

Deep Dive: Analyzing Vulnerable Perl and Shell Scripts

We employed a combination of static analysis and dynamic testing to uncover vulnerabilities. For static analysis, we used tools like Perl::Critic and custom grep patterns to scan for common anti-patterns, such as direct execution of user-supplied data via system(), exec(), or backticks (`command`) without proper sanitization.

A particularly concerning pattern emerged in a set of Perl scripts responsible for managing log rotation and file cleanup. These scripts accepted a filename or directory path as an argument, which was then used in a shell command. The vulnerability lay in the lack of input validation, allowing an attacker to inject shell metacharacters.

Example of a Vulnerable Perl Script Snippet

Consider a simplified version of a script used for cleaning up old log files:

#!/usr/bin/perl

use strict;
use warnings;

my $log_dir = shift @ARGV; # User-supplied directory

unless ($log_dir) {
    die "Usage: cleanup_logs.pl <log_directory>\n";
}

# Vulnerable command execution
my $command = "find " . $log_dir . " -type f -mtime +7 -delete";
system($command);

print "Log cleanup complete for: $log_dir\n";

An attacker could exploit this by providing an input like /var/log/myapp; rm -rf /. The system() call would then execute find /var/log/myapp; rm -rf / -type f -mtime +7 -delete, leading to catastrophic data loss.

Mitigation Strategy: Input Validation and Safe Execution

The core mitigation strategy revolved around two principles: rigorous input validation and the use of safer execution methods. For the Perl scripts, we enforced strict validation of any external input before it was used in shell commands.

Implementing Strict Input Validation in Perl

We modified the vulnerable Perl script to validate the input directory. The validation ensures that the provided path is a legitimate directory within an expected base path and does not contain shell metacharacters. We also opted to use Perl’s built-in file handling capabilities where possible, avoiding direct shell execution.

#!/usr/bin/perl

use strict;
use warnings;
use File::Spec;
use File::Path qw(rmtree);

my $log_dir_input = shift @ARGV;

unless ($log_dir_input) {
    die "Usage: cleanup_logs.pl <log_directory>\n";
}

# Define a safe base directory for logs
my $safe_base_log_dir = '/var/log/myapp_logs';

# Normalize and resolve the input path
my $resolved_log_dir = File::Spec->catdir($safe_base_log_dir, $log_dir_input);
$resolved_log_dir = File::Spec->rel2abs($resolved_log_dir);

# --- Strict Validation ---
# 1. Check if the resolved path is within the safe base directory
unless ($resolved_log_dir =~ /^$safe_base_log_dir\//) {
    die "Error: Invalid log directory specified. Path must be within $safe_base_log_dir\n";
}

# 2. Check if it's actually a directory
unless (-d $resolved_log_dir) {
    die "Error: '$resolved_log_dir' is not a valid directory.\n";
}

# --- Safe File Deletion ---
# Iterate and delete files older than 7 days
print "Starting cleanup in: $resolved_log_dir\n";
opendir(my $dh, $resolved_log_dir) or die "Can't open directory $resolved_log_dir: $!";

while (my $file = readdir($dh)) {
    next if ($file eq '.' || $file eq '..');
    my $full_path = File::Spec->catfile($resolved_log_dir, $file);

    # Check if it's a regular file and older than 7 days
    if (-f $full_path && (time() - (stat($full_path))[9]) > (7 * 24 * 60 * 60)) {
        print "Deleting old log file: $full_path\n";
        unlink($full_path) or warn "Could not delete $full_path: $!\n";
    }
}
closedir($dh);

print "Log cleanup complete for: $resolved_log_dir\n";

This revised script uses File::Spec for path manipulation, ensuring platform independence and preventing path traversal. It explicitly checks if the resolved path resides within the designated safe directory and uses Perl’s unlink for file deletion, completely bypassing the need for shell execution for this specific task.

Sanitizing Shell Commands

For utility scripts that *must* interact with the shell, we adopted a defense-in-depth approach. This involved:

Argument Escaping: Using functions like escapeshellarg() and escapeshellcmd() in PHP (or equivalent in other languages) to properly quote and escape arguments.
Whitelisting: Defining a strict list of allowed commands and arguments.
Least Privilege: Ensuring scripts run with the minimum necessary permissions.
Avoiding Shell Interpretation: Preferring direct system calls or libraries over shell execution where feasible.

Example: Safe Execution of a `tar` command in Bash

Consider a Bash script that needs to archive files. A naive approach might look like this:

#!/bin/bash

ARCHIVE_NAME="$1" # User input
SOURCE_DIR="$2"   # User input

# VULNERABLE: Direct concatenation and execution
tar -czf "$ARCHIVE_NAME.tar.gz" "$SOURCE_DIR"

An attacker could provide ARCHIVE_NAME as myarchive; rm -rf /. To mitigate this, we use printf %q for robust shell argument quoting:

#!/bin/bash

ARCHIVE_NAME="$1"
SOURCE_DIR="$2"

# Define a safe directory for archives
SAFE_ARCHIVE_DIR="/opt/backups/archives"
SAFE_SOURCE_BASE="/var/data/user_files"

# --- Input Validation ---
# Validate ARCHIVE_NAME: Ensure it's a simple filename, no path components or special chars
if [[ ! "$ARCHIVE_NAME" =~ ^[a-zA-Z0-9_-]+$ ]]; then
    echo "Error: Invalid archive name. Only alphanumeric characters, underscores, and hyphens are allowed." >&2
    exit 1
fi

# Validate SOURCE_DIR: Ensure it's a directory and within the safe base
if [[ ! -d "$SOURCE_DIR" ]]; then
    echo "Error: Source '$SOURCE_DIR' is not a directory." >&2
    exit 1
fi
if [[ "$SOURCE_DIR" != "$SAFE_SOURCE_BASE"* ]]; then
    echo "Error: Source directory '$SOURCE_DIR' is outside the allowed path '$SAFE_SOURCE_BASE'." >&2
    exit 1
fi

# --- Safe Execution ---
# Construct the full path for the archive
FULL_ARCHIVE_PATH="$SAFE_ARCHIVE_DIR/$ARCHIVE_NAME.tar.gz"

# Use printf %q for robust shell quoting of arguments
# This is crucial for preventing injection if tar itself had vulnerabilities or if we were passing
# more complex arguments that might be interpreted by the shell before tar sees them.
# For tar, it's often sufficient to just ensure the paths are clean and not malicious.
# However, for commands with more complex argument parsing, printf %q is invaluable.

# In this specific tar case, ensuring SOURCE_DIR and FULL_ARCHIVE_PATH are validated
# and not containing malicious characters is the primary defense.
# If we were passing user-controlled flags or patterns to tar, printf %q would be essential.

echo "Archiving '$SOURCE_DIR' to '$FULL_ARCHIVE_PATH'..."
if tar -czf "$FULL_ARCHIVE_PATH" -C "$(dirname "$SOURCE_DIR")" "$(basename "$SOURCE_DIR")"; then
    echo "Archive created successfully."
else
    echo "Error creating archive." >&2
    exit 1
fi

In this Bash example, we first validate both the archive name and the source directory against strict patterns and allowed paths. Then, we use tar with the -C option to change directory, ensuring that the archive is created relative to a known path and that the source directory itself is not a traversal attempt. While printf %q is a powerful tool for general shell argument sanitization, for specific commands like tar, ensuring the input paths are validated and the command is invoked correctly is often the most direct and effective mitigation.

Google Cloud Security Posture and Auditing

Beyond application-level fixes, we audited the GCP environment. Key areas included:

IAM Policies: Reviewed service account permissions and Kubernetes node pool roles. We ensured that no service account had overly broad permissions (e.g., editor or owner roles on the project). Permissions were scoped down to the minimum required for each component.
Network Policies: Examined Kubernetes NetworkPolicy resources to restrict pod-to-pod communication. Default-deny policies were implemented, and explicit allow rules were created only for necessary inter-service communication.
VPC Firewall Rules: Ensured GCP firewall rules were restrictive, allowing traffic only on necessary ports and from authorized sources.
Secret Management: Verified that sensitive information (API keys, database credentials) was managed using GCP Secret Manager or Kubernetes Secrets, and not hardcoded in scripts or application configurations.
Logging and Monitoring: Configured comprehensive logging for all GCP resources and Kubernetes audit logs. Alerts were set up for suspicious activity, such as unexpected command executions or unauthorized access attempts.

Automating GCP Audits with `gcloud` and Custom Scripts

To streamline the GCP audit, we developed a suite of Bash scripts leveraging the gcloud CLI. These scripts automated the retrieval of IAM policies, firewall rules, and Kubernetes configurations.

#!/bin/bash

PROJECT_ID="your-gcp-project-id"
OUTPUT_DIR="gcp_audit_$(date +%Y%m%d_%H%M%S)"

mkdir -p "$OUTPUT_DIR"

echo "--- Auditing IAM Policies ---"
gcloud projects get-iam-policy "$PROJECT_ID" --format=json > "$OUTPUT_DIR/iam_policy.json"
# Further analysis could involve parsing this JSON to check for overly permissive roles

echo "--- Auditing VPC Firewall Rules ---"
gcloud compute firewall-rules list --project="$PROJECT_ID" --format=json > "$OUTPUT_DIR/firewall_rules.json"
# Analyze for overly permissive ingress rules (e.g., 0.0.0.0/0 on sensitive ports)

echo "--- Auditing Kubernetes Cluster Details ---"
# Assuming you have a default GKE cluster configured or specified via --cluster and --zone/--region
gcloud container clusters list --project="$PROJECT_ID" --format=json > "$OUTPUT_DIR/gke_clusters.json"

echo "--- Auditing Kubernetes Network Policies (Example for a specific namespace) ---"
NAMESPACE="your-namespace"
kubectl get networkpolicies -n "$NAMESPACE" -o yaml > "$OUTPUT_DIR/network_policies_${NAMESPACE}.yaml"
# Manual review or custom script to check for default-deny and specific allows

echo "Audit data saved to $OUTPUT_DIR/"
echo "Review the generated files for security misconfigurations."

These scripts provided a baseline snapshot of the GCP environment. Further analysis involved programmatic checks for common misconfigurations, such as identifying service accounts with excessive privileges or firewall rules that were too permissive.

Continuous Monitoring and Incident Response

The audit and remediation efforts were not a one-time event. We established a continuous monitoring framework:

Log Aggregation: Centralized logs from Perl applications, system utilities, and GCP services into a SIEM (Security Information and Event Management) solution.
Real-time Alerting: Configured alerts for suspicious patterns, such as multiple failed command executions, attempts to access restricted files, or unusual network traffic.
Automated Scanning: Integrated static analysis tools into the CI/CD pipeline to catch new vulnerabilities before deployment.
Regular Audits: Scheduled periodic re-audits of the application code and GCP configuration.

By combining secure coding practices, robust input validation, and a well-configured cloud security posture, we significantly reduced the attack surface and mitigated the risk of untrusted command injection in the enterprise Perl stack. The focus on automation and continuous monitoring ensures that the security of the system is maintained over time.