How We Audited a High-Traffic Perl Enterprise Stack on OVH and Mitigated untrusted command injection in system utility scripts

Initial Triage: Identifying the Attack Vector

Our engagement began with a critical alert from an internal monitoring system indicating anomalous outbound network traffic originating from a legacy Perl application server hosted on OVH. The traffic patterns suggested a potential command and control (C2) communication channel. The immediate suspicion fell on a common vulnerability: untrusted command injection within system utility scripts that the Perl application leveraged.

The architecture involved a Perl CGI application serving a high-traffic e-commerce platform, interacting with a MySQL backend, and relying on several shell scripts for tasks like log rotation, data processing, and external API calls. The OVH environment provided dedicated servers with a mix of managed and self-managed components.

Deep Dive into Perl Application Code and System Interactions

The first step was a thorough code audit of the Perl application. We focused on any instances where external input (from HTTP requests, configuration files, or database entries) was used to construct or execute shell commands. The `system()`, `exec()`, backticks (` “ `), and `qx()` functions in Perl are notorious for this if not handled with extreme care.

A common pattern we look for is:

# Potentially vulnerable code snippet
my $user_id = param('user_id'); # Assuming 'param' gets user input
my $command = "process_user_data.sh --id " . $user_id;
system($command);

In this example, if a malicious user provides an input like `123; rm -rf /`, the executed command becomes `process_user_data.sh –id 123; rm -rf /`, leading to catastrophic data loss. We systematically reviewed all such invocations.

Auditing System Utility Scripts

Beyond the Perl application itself, the system utility scripts were equally critical. These scripts, often written in Bash, were frequently called by the Perl application and could also be indirectly exploited if they accepted user-controlled parameters without proper sanitization.

We identified a script, `rotate_logs.sh`, which was responsible for archiving and compressing log files. It accepted a date argument to specify which logs to process. The original script looked something like this:

#!/bin/bash

LOG_DIR="/var/log/myapp"
DATE_TO_PROCESS="$1"

if [ -z "$DATE_TO_PROCESS" ]; then
  echo "Error: Date argument is required."
  exit 1
fi

echo "Processing logs for date: $DATE_TO_PROCESS"

# Vulnerable command execution
find "$LOG_DIR" -name "*.$DATE_TO_PROCESS.log" -exec gzip {} \;
find "$LOG_DIR" -name "*.$DATE_TO_PROCESS.log.gz" -exec mv {} "$LOG_DIR/archive/" \;

An attacker could exploit this by passing a crafted date string. For instance, if the Perl application called this script with `rotate_logs.sh “2023-10-27; touch /tmp/pwned”`, the `find` command would execute `gzip` on legitimate files, and then the `touch` command would be executed, creating a file in `/tmp`. In a more sophisticated attack, this could be used to download and execute arbitrary code.

Exploitation Scenario and Evidence Gathering

The anomalous outbound traffic was traced to a process spawned by the Perl application. Using `strace` on the running Perl process (carefully, in a controlled manner to avoid impacting production further), we observed system calls related to `popen()` and `system()`, confirming our suspicion of command execution. The specific commands being executed, however, were obfuscated.

We then focused on the `rotate_logs.sh` script. By manually crafting an input that mimicked what an attacker might send, we were able to reproduce the behavior. For example, executing:

/opt/myapp/scripts/rotate_logs.sh "2023-10-27; echo 'VULNERABLE' >> /tmp/exploit_test.txt"

This command successfully created the `/tmp/exploit_test.txt` file with the content “VULNERABLE”, confirming the injection vulnerability. The outbound traffic was likely the result of a more advanced payload, such as a reverse shell or data exfiltration, executed via a similar injection technique.

Mitigation Strategy: Input Validation and Parameterization

The core of the mitigation strategy involved two primary approaches: strict input validation and, where possible, avoiding direct command execution by using safer alternatives.

Fortifying the Perl Application

For any Perl code that needed to interact with the shell, we implemented robust sanitization. The `IPC::Cmd` module is a safer alternative to direct `system()` calls as it provides better control and argument escaping. If `IPC::Cmd` was not an option, we enforced strict validation of input parameters.

use IPC::Cmd qw(run);
use CGI qw(:all);

my $q = CGI->new;
my $user_id = $q->param('user_id');

# Validate user_id to ensure it's purely numeric
unless ($user_id =~ /^\d+$/) {
    die "Invalid user ID format.";
}

# Use IPC::Cmd for safer execution
my ($success, $err, $stdout, $stderr) = run(
    command => 'process_user_data.sh',
    arguments => ['--id', $user_id],
    verbose => 0,
);

if (!$success) {
    error_log("Error processing user $user_id: $stderr");
    # Handle error appropriately
}

The key here is the regex `^\d+$` which ensures that `$user_id` consists solely of digits. This prevents any shell metacharacters from being interpreted.

Securing System Utility Scripts

The `rotate_logs.sh` script was refactored to be more resilient. The primary change was to validate the date argument rigorously and to avoid passing it directly into `find`’s `-exec` or `mv` commands without proper quoting and validation.

#!/bin/bash

LOG_DIR="/var/log/myapp"
DATE_TO_PROCESS="$1"

# Validate DATE_TO_PROCESS format (e.g., YYYY-MM-DD)
if ! [[ "$DATE_TO_PROCESS" =~ ^[0-9]{4}-[0-9]{2}-[0-9]{2}$ ]]; then
  echo "Error: Invalid date format. Expected YYYY-MM-DD."
  exit 1
fi

echo "Processing logs for date: $DATE_TO_PROCESS"

# Safely construct filenames and use them
LOG_PATTERN="${LOG_DIR}/*.$DATE_TO_PROCESS.log"
ARCHIVE_DIR="${LOG_DIR}/archive"

# Ensure archive directory exists
mkdir -p "$ARCHIVE_DIR"

# Use find with proper quoting and avoid direct command injection in -exec
# The {} is automatically quoted by find's -exec
find "$LOG_DIR" -maxdepth 1 -type f -name "*.$DATE_TO_PROCESS.log" -print0 | while IFS= read -r -d $'\0' file; do
  echo "Gzipping: $file"
  gzip "$file"
done

find "$LOG_DIR" -maxdepth 1 -type f -name "*.$DATE_TO_PROCESS.log.gz" -print0 | while IFS= read -r -d $'\0' file; do
  echo "Archiving: $file"
  mv "$file" "$ARCHIVE_DIR/"
done

In this revised script:

A regular expression `^[0-9]{4}-[0-9]{2}-[0-9]{2}$` is used to strictly validate the date format.
Filenames are constructed using variables and then used with `find -print0` and `while read -d $’\0’` loop for safe processing of filenames that might contain spaces or special characters.
The `gzip` and `mv` commands are called with quoted arguments, preventing shell interpretation of any characters within the filenames themselves.

Post-Mitigation Monitoring and Verification

Following the code and script remediation, a critical phase was enhanced monitoring. We implemented stricter firewall rules on the OVH instance to limit outbound connections to only essential services and ports. Intrusion Detection Systems (IDS) were configured to flag any attempts to execute shell commands from the Perl application or its child processes that did not match expected patterns.

We also deployed additional logging to capture all executed commands by the application’s user, along with their arguments. This provided an audit trail and an immediate alert mechanism for any suspicious activity. Regular vulnerability scans, specifically targeting command injection flaws, were scheduled.

The success of the mitigation was verified by attempting to re-exploit the identified vulnerabilities using the same payloads that were previously successful. All attempts were blocked by the input validation or resulted in logged errors without any unintended command execution.

Long-Term Strategy: Modernization and Principle of Least Privilege

While the immediate vulnerabilities were addressed, this incident highlighted the risks associated with maintaining legacy systems and extensive use of shell scripting for application logic. The long-term strategy involves:

Gradual modernization of the Perl application, potentially migrating critical components to more modern languages with better built-in security features and libraries.
Reducing reliance on external shell scripts by re-implementing their functionality within the application using safer, language-native constructs.
Strict enforcement of the Principle of Least Privilege: ensuring the application and its utility scripts run under dedicated, unprivileged user accounts with minimal necessary permissions. This limits the blast radius of any future compromise.
Regular security training for development teams focusing on secure coding practices, particularly concerning input handling and external process execution.

This case study underscores the persistent threat of command injection in enterprise environments, especially those with legacy codebases. Proactive auditing, rigorous input validation, and a commitment to secure coding practices are paramount in protecting high-traffic systems.