Vengala Vinay

Having 9+ Years of Experience in Software Development

Home » How We Audited a High-Traffic Perl Enterprise Stack on AWS and Mitigated untrusted command injection in system utility scripts

How We Audited a High-Traffic Perl Enterprise Stack on AWS and Mitigated untrusted command injection in system utility scripts

Initial Triage: Identifying the Attack Vector

Our engagement began with a critical alert from our AWS GuardDuty service, flagging suspicious outbound network activity originating from a fleet of EC2 instances running a high-traffic Perl enterprise application. The activity pointed towards potential command injection vulnerabilities within system utility scripts that were being executed by the application. The primary concern was the ability of an unauthenticated or low-privileged attacker to execute arbitrary commands on the underlying operating system, leading to data exfiltration, service disruption, or further lateral movement within the AWS environment.

The application stack was a complex, multi-tiered Perl monolith deployed across numerous EC2 instances, leveraging RDS for its database and S3 for object storage. The critical observation was that several Perl scripts, intended for internal system management and data processing, were directly invoking shell commands. This is a common pattern in legacy Perl applications, often implemented using backticks (“ ` “) or the `system()` and `qx()` functions without proper sanitization of user-supplied or externally influenced input.

Deep Dive into Perl Script Vulnerabilities

We initiated a systematic audit of the Perl codebase, focusing on scripts that interacted with the operating system. The primary targets were functions that constructed shell commands dynamically. A common anti-pattern we identified was the direct concatenation of user input into command strings. For instance, a script responsible for processing uploaded files might construct a command like this:

my $filename = param('file_upload'); # Assume 'param' retrieves user input
my $command = "mv /tmp/$filename /data/uploads/";
system($command);

In this simplified example, if an attacker could control the value of $filename, they could inject shell metacharacters. For example, submitting "my_file.txt; rm -rf /" as the filename would result in the execution of mv /tmp/my_file.txt; rm -rf / /data/uploads/, leading to catastrophic data loss.

Another prevalent vulnerability pattern involved the use of qx() (equivalent to backticks) for capturing command output, which suffers from the same injection risks:

my $user_id = param('user');
my $user_info_command = "id -u " . $user_id;
my $output = qx($user_info_command);
# Process $output

An attacker could provide a malicious $user_id like "123; cat /etc/passwd", leading to the execution of id -u 123; cat /etc/passwd, exposing sensitive system information.

Mitigation Strategy: Secure Command Execution

The core principle for mitigation is to avoid constructing shell commands with unsanitized input. When shell interaction is unavoidable, it must be done with extreme caution. We adopted a multi-pronged approach:

1. Prefer Perl Modules Over Shell Commands

The first and most effective mitigation is to replace shell command invocations with equivalent Perl modules. For file operations, the standard File::Copy module is a direct and safe replacement for mv:

use File::Copy;

my $source_file = "/tmp/" . param('file_upload');
my $destination_path = "/data/uploads/";

# Basic validation: ensure the filename doesn't contain path traversal or shell metacharacters
# A more robust solution would involve a strict whitelist of allowed characters.
if (param('file_upload') =~ m{^[a-zA-Z0-9_\-\.]+$}) {
    if (!-d $destination_path) {
        mkdir $destination_path or die "Cannot create directory $destination_path: $!";
    }
    if (copy($source_file, $destination_path . basename(param('file_upload')))) {
        unlink $source_file; # Clean up original if copy succeeds
        # Success
    } else {
        # Handle copy error
    }
} else {
    # Handle invalid filename
}

Similarly, for user information, the getpwuid() function from the passwd module can retrieve user details without invoking external processes.

2. Input Validation and Sanitization

When direct shell execution is absolutely necessary (e.g., for legacy tools or specific system utilities not easily replicated in Perl), rigorous input validation and sanitization are paramount. This involves:

  • Whitelisting: Define a strict set of allowed characters or patterns for any input that will be part of a command. Reject any input that deviates from this whitelist.
  • Escaping: For arguments that are not part of the command itself but are passed to it, use functions like escapeshellarg() and escapeshellcmd(). It’s crucial to understand the difference:
    • escapeshellarg(): Safely quotes a string so it can be passed as a single argument to a shell command. It adds single quotes around the string and escapes any existing single quotes within the string.
    • escapeshellcmd(): Escapes any characters in a string that might be used to terminate a command or inject new commands. This is less safe than escapeshellarg() and should be used with extreme caution, typically only when the input is *part* of the command itself (e.g., a command name), not an argument.

Applying escapeshellarg() to our earlier example:

use File::Basename;

my $filename = param('file_upload');
my $safe_filename = basename( $filename ); # Use basename to prevent directory traversal

# Further validation: ensure the filename is safe and doesn't contain malicious characters
# A regex for allowed characters is highly recommended.
if ($safe_filename =~ m{^[a-zA-Z0-9_\-\.]+$}) {
    my $command = "mv " . escapeshellarg("/tmp/" . $safe_filename) . " " . escapeshellarg("/data/uploads/");
    system($command);
} else {
    # Handle invalid filename
}

Note the use of basename() in conjunction with escapeshellarg(). basename() strips any directory components from the filename, preventing path traversal attacks (e.g., ../../etc/passwd). Then, escapeshellarg() ensures that even if the filename somehow contained shell metacharacters, they would be treated as literal characters by the shell.

3. Principle of Least Privilege

The application’s execution context (the IAM role assigned to the EC2 instances) was reviewed. We ensured that the application’s service account had only the minimum necessary AWS permissions. Furthermore, on the EC2 instances themselves, the user running the Perl application was restricted to only the commands and file system paths it absolutely needed. This involved careful configuration of sudoers files and file permissions.

4. Runtime Monitoring and Alerting

To detect and respond to any residual or newly introduced vulnerabilities, we enhanced our monitoring. This included:

  • AWS CloudTrail: Enabled for all relevant services to log API calls.
  • EC2 Instance System Logs: Configured to capture shell command execution (e.g., via auditd or by enabling shell history logging for the application user, though this can be noisy).
  • GuardDuty: Continued to monitor for suspicious network traffic and potential exploitation patterns.
  • Custom Application Logging: Instrumented the Perl application to log all instances of system(), qx(), and backtick usage, along with the constructed commands and their arguments. This provided invaluable visibility during the remediation phase and for ongoing auditing.
sub safe_system {
    my ($command) = @_;
    my $log_command = $command; # Log the raw command before sanitization for debugging

    # Basic sanitization for demonstration. Real-world would be more robust.
    # This example assumes $command is already a safe string or uses escapeshellarg internally.
    # If $command is constructed from parts, each part must be escaped.

    print STDERR "Executing command: $log_command\n"; # Log for audit
    # In a real scenario, send this to a centralized logging system (e.g., CloudWatch Logs)

    return system($command);
}

# Usage:
# my $filename = param('file_upload');
# my $safe_filename = basename($filename);
# if ($safe_filename =~ m{^[a-zA-Z0-9_\-\.]+$}) {
#     my $cmd_str = "mv " . escapeshellarg("/tmp/" . $safe_filename) . " " . escapeshellarg("/data/uploads/");
#     safe_system($cmd_str);
# }

Deployment and Verification

The remediation involved a phased rollout. First, we identified all vulnerable scripts and prioritized them based on the sensitivity of the data they handled and the likelihood of exploitation. We then refactored the code, replacing shell calls with Perl modules where possible, and applying escapeshellarg() and strict validation where shell execution remained necessary. Each change was accompanied by unit tests and integration tests to ensure functionality was preserved and security was enhanced.

Post-deployment verification included:

  • Manual penetration testing targeting the previously vulnerable endpoints.
  • Automated security scanning of the codebase for common injection patterns.
  • Monitoring of application logs and system audit trails for any signs of attempted exploitation or unexpected command execution.

This comprehensive approach, combining code refactoring, robust input validation, principle of least privilege, and enhanced monitoring, successfully mitigated the untrusted command injection risks within the Perl enterprise stack on AWS, significantly improving its security posture.