How We Audited a High-Traffic Ruby Enterprise Stack on OVH and Mitigated Insecure Deserialization in legacy session handling

Initial Stack Assessment and OVH Environment Reconnaissance

Our engagement began with a deep dive into the existing Ruby on Rails enterprise stack, hosted on OVH’s infrastructure. The primary concern was a recent security audit flagging potential vulnerabilities in legacy session handling. The stack comprised several Rails applications, a PostgreSQL database cluster, Redis for caching and session storage, and Nginx acting as a reverse proxy and load balancer. The OVH environment, while robust, presented specific considerations regarding network segmentation, firewall rules, and access control mechanisms that needed to be thoroughly understood before any intrusive testing.

The first step involved mapping the network topology and identifying all exposed services. We utilized Nmap for initial port scanning and service enumeration across the relevant OVH IP ranges. This was followed by a more targeted reconnaissance of the Nginx configuration to understand request routing, SSL termination, and any pre-authentication or rate-limiting measures in place.

Deep Dive into Legacy Session Handling and Insecure Deserialization Vectors

The core of the security concern lay in the application’s session management. Older versions of Rails, and indeed many custom implementations, relied on serializing session data (often hashes or objects) into a format that could be stored and later retrieved. Common serialization formats include YAML, JSON, or even Ruby’s native `Marshal` module. The critical vulnerability arises when deserialization occurs without proper validation of the input data. An attacker can craft malicious serialized objects that, upon deserialization, execute arbitrary code on the server.

We focused our audit on the specific points where session data was read from storage (Redis, in this case) and deserialized. The application used a custom middleware for session handling, bypassing the default Rails session store. This custom middleware was the primary target.

Identifying the Serialization Format

The first crucial step was to determine the serialization format used. By inspecting the data stored in Redis, we could infer the format. In this particular legacy stack, the session data was being serialized using Ruby’s `Marshal` module. This is a particularly dangerous choice for session data as it allows for the serialization of arbitrary Ruby objects, including those with `initialize` or other methods that can be triggered during deserialization.

Crafting a Malicious Payload (Proof of Concept)

To demonstrate the vulnerability, we needed to craft a malicious Ruby object that, when serialized and then deserialized by the vulnerable application, would execute a command. A common technique involves leveraging Ruby’s `yaml` or `Marshal` deserialization to instantiate an object that, during its initialization or a subsequent method call, executes system commands. For `Marshal`, this often involves creating a custom class that overrides methods like `_load` or `initialize`.

Consider a simplified, illustrative example of a malicious Ruby object that could be used for `Marshal` deserialization. Note: This is for educational purposes and should NEVER be run in a production environment without extreme caution and isolation.

Example: Malicious Ruby Object for Marshal Deserialization

# This is a *highly simplified* and *illustrative* example.
# Real-world exploits are often more complex and evasive.

require 'yaml' # Although we're demonstrating Marshal, YAML is another common vector.

class Exploit
  def initialize(cmd = nil)
    @cmd = cmd || 'id >&2' # Default to 'id' for demonstration
  end

  def _load(yaml_string)
    # In a real Marshal exploit, this method might not be directly called,
    # but the object's initialization or other methods would be triggered.
    # For Marshal, the object's state is serialized. When deserialized,
    # if the object has specific methods that are called implicitly or explicitly,
    # they can be exploited.
    # A more direct Marshal exploit might involve a class that has a method
    # like 'perform_action' which is then called after deserialization.
    # For simplicity, let's simulate a command execution trigger.

    # This is a conceptual representation. Actual Marshal exploits
    # often leverage specific Ruby internals or gem vulnerabilities.
    # A common pattern is to have a method that is called post-deserialization.
    # For instance, if the application code does `session_data.some_method`,
    # and `some_method` is defined in our Exploit class, it could be triggered.

    # A more direct approach for Marshal:
    # If the application deserializes and then calls a method on the object,
    # we can define that method to execute our command.
    # Example: If app code is `session_object.process_data`, and we define
    # `process_data` in our Exploit class.

    # For this example, let's assume a hypothetical `run_command` method
    # is called on the deserialized object.
    run_command(@cmd)
  end

  def run_command(command)
    # This is the part that would execute on the server.
    # In a real exploit, this would be carefully crafted to be stealthy.
    system(command)
  end
end

# To generate the malicious payload for Marshal:
# payload_object = Exploit.new("touch /tmp/pwned_by_antigravity")
# malicious_data = Marshal.dump(payload_object)
# puts malicious_data

The key here is that `Marshal.dump` serializes the object’s state. When `Marshal.load` is called on this data, it reconstructs the `Exploit` object. If the application code then proceeds to call a method on this deserialized object (e.g., `session_data.process_data`), and our `Exploit` class has a `process_data` method defined, that method will be executed, leading to arbitrary code execution.

Simulating the Attack Vector

We simulated this by first injecting a malicious session cookie into the application. This involved crafting a `Marshal.dump` payload containing an instance of our `Exploit` class, targeting a command like `whoami` or `id`. This payload was then Base64 encoded (as is common practice for cookies) and set as the session cookie.

# On an attacker's machine, to generate the payload:
RUBY_CODE="require 'yaml'; class Exploit; def initialize(cmd); @cmd = cmd; end; def _load(s); system(@cmd); end; end; payload = Exploit.new('id >&2'); puts Base64.strict_encode64(Marshal.dump(payload))"
echo -n "$RUBY_CODE" | ruby -e 'eval ARGV[0]'

The output of this command would be a Base64 encoded string. This string was then used to set the application’s session cookie. Upon the next request, the application’s session middleware would read this cookie, decode it, and attempt to `Marshal.load` it. If the application’s code subsequently called a method on the deserialized object (e.g., `session_object.some_method`), our `_load` method (or another triggered method) would execute the `id` command on the server, and its output would be sent to standard error, which in a web context might be logged or even reflected in the response depending on error handling.

Mitigation Strategies: From Vulnerability to Secure Session Handling

The immediate and most effective mitigation was to eliminate the use of `Marshal` for session serialization. This required a multi-pronged approach:

1. Migrating to a Secure Serialization Format

The recommended approach is to use a format that does not allow for arbitrary code execution. JSON is a widely adopted and safe choice for serializing simple data structures like hashes and arrays. Most modern web frameworks have robust JSON serialization capabilities.

We refactored the custom session middleware to use `JSON` for serialization and deserialization. This involved ensuring that only primitive data types (strings, numbers, booleans, arrays, hashes) were stored in the session. Any complex Ruby objects that were previously being serialized needed to be converted to a JSON-compatible representation before storage.

# Example of refactored session handling (conceptual)

# Original (Vulnerable)
# session_data = Marshal.load(Base64.decode64(cookie_value))

# New (Secure)
require 'json'

# When storing:
session_hash = { user_id: 123, preferences: { theme: 'dark' } }
serialized_session = Base64.strict_encode64(JSON.generate(session_hash))
# Set serialized_session as cookie

# When retrieving:
encoded_cookie = request.cookies['session_id']
decoded_cookie = Base64.strict_decode64(encoded_cookie)
session_data = JSON.parse(decoded_cookie) # This is now a Hash, not an arbitrary object.

# Ensure data types are as expected
unless session_data.is_a?(Hash) && session_data['user_id'].is_a?(Integer)
  # Handle invalid session data - potentially clear it or log an alert
  session_data = {}
end

2. Implementing Input Validation and Sanitization

Even with a secure serialization format like JSON, it’s crucial to validate the structure and types of the deserialized data. The application should not blindly trust the session data. We added explicit checks to ensure that the deserialized session hash contained the expected keys and that their values were of the correct data types. Any deviation would result in the session being considered invalid and potentially cleared.

3. Upgrading Dependencies and Frameworks

While not directly addressing the `Marshal` issue, a general security best practice is to keep all dependencies, including the Ruby on Rails framework itself, up-to-date. Newer versions often include security patches and deprecate or remove insecure functionalities. We initiated a process to identify and upgrade all outdated gems and the Rails framework to their latest stable, supported versions.

4. Leveraging Secure Session Stores

For high-traffic applications, storing session data directly in cookies can become inefficient and a security risk (even with secure serialization). We evaluated and recommended migrating to a dedicated session store like Redis or Memcached, configured with appropriate authentication and TLS encryption. This offloads session management from the application server and allows for more robust security controls.

5. Network and Infrastructure Hardening (OVH Specific)

In parallel with application-level fixes, we reviewed the OVH infrastructure configuration. This included:

Firewall Rules: Ensuring that only necessary ports were open to the public internet and that internal services (like Redis) were not directly accessible from the outside.
Access Control: Implementing strict SSH access controls, using key-based authentication, and limiting sudo privileges.
Redis Security: Configuring Redis with a strong password (`requirepass`), disabling the `FLUSHALL` command in production, and binding it to specific internal IP addresses.
Nginx Configuration: Reviewing Nginx logs for suspicious activity, ensuring proper SSL/TLS configurations, and implementing rate limiting where appropriate.

# Example Nginx configuration snippet for Redis access control
# This assumes Redis is running on localhost:6379 and Nginx is on the same server.
# In a multi-server setup, this would be an internal IP.

# In your application's Nginx config or a separate conf.d file:
# This is NOT for direct Redis access, but for application routing.
# The application code itself would connect to Redis.

# For application servers connecting to Redis:
# Ensure your application's Redis client is configured to use authentication
# and potentially TLS if Redis is exposed internally over a network.

# Example of Redis configuration in redis.conf:
# bind 127.0.0.1 # Or the specific internal IP of the Redis server
# protected-mode yes
# requirepass your_very_strong_redis_password
# tls-port 0 # Disable TLS if not using it, or configure properly

Post-Mitigation Validation and Ongoing Monitoring

Following the implementation of these changes, a comprehensive re-audit was performed. This included automated vulnerability scanning, manual penetration testing focused on session hijacking and deserialization vulnerabilities, and code reviews of the modified session handling logic. We also established enhanced logging and monitoring for:

Unusual session cookie patterns.
Deserialization errors or unexpected data structures in session data.
High rates of invalid session attempts.
System command execution attempts (if any could be inferred from logs).

By migrating away from insecure deserialization and implementing robust validation and infrastructure hardening, we successfully mitigated the identified risks, significantly improving the security posture of the high-traffic Ruby enterprise stack hosted on OVH.