Vengala Vinay

Having 9+ Years of Experience in Software Development

Home » How We Audited a High-Traffic Ruby Enterprise Stack on OVH and Mitigated Server-Side Request Forgery (SSRF) in webhook parsers

How We Audited a High-Traffic Ruby Enterprise Stack on OVH and Mitigated Server-Side Request Forgery (SSRF) in webhook parsers

Initial Stack Assessment and OVH Environment Reconnaissance

Our engagement began with a deep dive into the existing Ruby enterprise stack hosted on OVH. The primary objective was to identify potential security vulnerabilities, with a specific focus on Server-Side Request Forgery (SSRF) vectors, particularly within webhook processing modules. The OVH environment, while robust, presented its own set of considerations regarding network segmentation, firewall rules, and internal service discovery. We started by mapping out the core components: the Ruby on Rails application servers, the PostgreSQL database cluster, Redis for caching and job queuing, and the load balancing layer, likely Nginx or HAProxy. Understanding the network topology within OVH was critical; specifically, we needed to know which internal IP ranges were accessible from the application servers and what services were exposed on those ranges.

Our initial reconnaissance involved analyzing deployed services and their configurations. This included:

  • Application Server Configuration: Examining nginx.conf (or HAProxy config), database.yml, redis.yml, and any custom middleware or Rack applications.
  • Network Access Control: Reviewing OVH firewall rules (iptables on hosts or OVH’s network firewall interface) to understand ingress and egress restrictions.
  • Service Discovery: Identifying how services communicate with each other. Are they using hardcoded IPs, DNS, or a service registry?
  • Webhook Ingress Points: Pinpointing all external endpoints designed to receive webhook payloads.

A key aspect of this phase was to understand the trust boundaries. What internal resources were the application servers expected to interact with? Were these resources sensitive, and what authentication mechanisms were in place?

Deep Dive into Webhook Parsers and SSRF Attack Surface

The most promising area for SSRF vulnerabilities often lies in components that fetch external resources based on user-controlled input. In this stack, webhook parsers were prime candidates. These modules typically receive a URL or a payload containing a URL from an external service, and then the application server makes an HTTP request to that URL to fetch additional data or trigger an action. If not properly validated, an attacker could manipulate these URLs to make the server request arbitrary internal resources.

We focused on code patterns that involved making HTTP requests. Common Ruby libraries for this include Net::HTTP, HTTParty, Faraday, and Open-URI. The critical vulnerability arises when the target URL is constructed or influenced by untrusted input without sufficient sanitization or validation.

Consider a simplified, vulnerable example:

Vulnerable Code Snippet (Ruby on Rails)

Imagine a controller action that processes an incoming webhook from a third-party service. The webhook payload might contain a URL to fetch additional metadata.

# app/controllers/webhooks_controller.rb
class WebhooksController < ApplicationController
  def process_event
    event_data = JSON.parse(request.body.read)
    metadata_url = event_data['metadata_url']

    # Vulnerable: Directly uses user-provided URL without validation
    begin
      response = HTTParty.get(metadata_url)
      process_metadata(response.parsed_response)
      render json: { status: 'success' }, status: :ok
    rescue StandardError => e
      render json: { status: 'error', message: e.message }, status: :internal_server_error
    end
  end

  private

  def process_metadata(data)
    # ... logic to process metadata ...
  end
end

In this scenario, an attacker could send a webhook payload with metadata_url set to an internal IP address or a local hostname, such as http://127.0.0.1:9200/_status (if Elasticsearch is running locally) or http://169.254.169.254/latest/meta-data/iam/security-credentials/role-name (if on a cloud provider with instance metadata services). The application server would then make the request, potentially leaking sensitive information or allowing further exploitation.

Mitigation Strategies: Validation, Sanitization, and Network Controls

Addressing SSRF requires a multi-layered approach. We implemented several key strategies:

1. Strict URL Validation and Whitelisting

The most effective defense is to only allow requests to known, trusted domains or IP addresses. If the webhook parser is expected to fetch data from a specific set of external services, these should be explicitly whitelisted.

# app/controllers/webhooks_controller.rb
class WebhooksController < ApplicationController
  ALLOWED_METADATA_HOSTS = %w[
    api.thirdparty.com
    internal-api.example.com
  ].freeze

  def process_event
    event_data = JSON.parse(request.body.read)
    metadata_url_str = event_data['metadata_url']

    unless metadata_url_str.present?
      render json: { status: 'error', message: 'metadata_url is required' }, status: :bad_request
      return
    end

    begin
      uri = URI.parse(metadata_url_str)

      # 1. Scheme validation
      unless %w[http https].include?(uri.scheme)
        render json: { status: 'error', message: 'Invalid URL scheme' }, status: :bad_request
        return
      end

      # 2. Host validation against whitelist
      unless ALLOWED_METADATA_HOSTS.include?(uri.host)
        render json: { status: 'error', message: 'Disallowed host' }, status: :bad_request
        return
      end

      # 3. Prevent access to localhost and private IP ranges
      if uri.host == 'localhost' || is_private_ip?(uri.host)
        render json: { status: 'error', message: 'Access to private IPs/localhost denied' }, status: :bad_request
        return
      end

      # If all checks pass, proceed with the request
      response = HTTParty.get(uri.to_s)
      process_metadata(response.parsed_response)
      render json: { status: 'success' }, status: :ok

    rescue URI::InvalidURIError
      render json: { status: 'error', message: 'Invalid URL format' }, status: :bad_request
    rescue StandardError => e
      render json: { status: 'error', message: e.message }, status: :internal_server_error
    end
  end

  private

  def process_metadata(data)
    # ... logic to process metadata ...
  end

  # Helper to check for private IP addresses (simplified)
  def is_private_ip?(host)
    # This is a basic check; a more robust solution might use IPAddr library
    # or a comprehensive regex for private ranges (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16)
    # and loopback (127.0.0.0/8).
    # For OVH, consider internal OVH IP ranges as well if applicable.
    # Example: Check if it's a private IP according to RFC 1918
    ip = IPAddress(host) rescue nil
    return false unless ip
    ip.private? || ip.loopback?
  rescue NameError # IPAddress gem not loaded
    # Fallback to basic string checks if IPAddress gem is not available
    # This is less reliable and should be avoided in production.
    host.start_with?('10.') ||
    host.start_with?('192.168.') ||
    (host.start_with?('172.') && host.split('.')[1].to_i >= 16 && host.split('.')[1].to_i <= 31) ||
    host == 'localhost' || host == '127.0.0.1'
  end
end

This revised code performs several crucial checks:

  • Scheme Validation: Ensures only http or https schemes are allowed.
  • Host Whitelisting: Verifies that the requested host is in a predefined list of trusted domains.
  • Private IP/Loopback Prevention: Explicitly blocks requests to localhost and private IP address ranges. The is_private_ip? helper function (ideally using a gem like ipaddress) is vital here.

2. Network-Level Controls (OVH Specific)

Beyond application-level fixes, we leveraged OVH’s network infrastructure. This involved:

  • Outbound Firewall Rules: Configuring iptables on the application servers or using OVH’s network firewall service to restrict outbound connections. Only allow connections to specific external IP addresses or ranges that are known to be legitimate endpoints for webhook data retrieval. Deny all other outbound traffic by default.
  • Internal Network Segmentation: Ensuring that application servers cannot directly reach sensitive internal services (like databases, internal APIs, or management interfaces) unless absolutely necessary. If such access is required, it should be via a tightly controlled proxy or API gateway.
  • DNS Resolution Control: In some advanced scenarios, manipulating DNS resolution on the application servers could prevent them from resolving internal hostnames to their private IP addresses.

For example, a restrictive iptables rule on the application servers might look like this:

# Allow established connections
sudo iptables -A OUTPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

# Allow DNS queries (UDP/TCP port 53) to specific DNS servers
sudo iptables -A OUTPUT -p udp --dport 53 -d 8.8.8.8 -j ACCEPT
sudo iptables -A OUTPUT -p tcp --dport 53 -d 8.8.8.8 -j ACCEPT
# Add OVH's DNS servers if applicable

# Allow HTTP/HTTPS to whitelisted external IPs/ranges
# Example: Allow access to 203.0.113.0/24
sudo iptables -A OUTPUT -p tcp -d 203.0.113.0/24 --dport 80 -j ACCEPT
sudo iptables -A OUTPUT -p tcp -d 203.0.113.0/24 --dport 443 -j ACCEPT

# Deny all other outbound traffic by default
sudo iptables -P OUTPUT DROP

Note: This is a highly restrictive example. In a real-world scenario, you would need to carefully identify all legitimate outbound destinations and ports required by the application, including any necessary communication with OVH services, monitoring agents, or external APIs.

3. Input Sanitization and Canonicalization

While whitelisting is preferred, robust sanitization is a necessary fallback. This involves:

  • URL Canonicalization: Normalizing URLs before validation. This includes resolving relative paths, handling different encodings, and removing unnecessary characters.
  • IP Address Parsing: Using libraries to correctly parse and validate IP addresses, distinguishing between IPv4 and IPv6, and identifying private/reserved ranges.
  • DNS Rebinding Protection: Implementing checks to prevent DNS rebinding attacks, where an attacker controls a DNS server that initially resolves to a public IP but later resolves to a private IP.

Post-Mitigation Testing and Verification

After implementing the code changes and network configurations, a rigorous testing phase was essential. This involved:

  • Penetration Testing: Simulating various SSRF attack vectors against the webhook endpoints. This included attempts to access localhost, internal OVH IP ranges, and cloud provider metadata endpoints (even if not applicable in this specific OVH setup, it’s good practice).
  • Fuzzing: Using automated tools to send malformed or unexpected URLs to the webhook parser to uncover edge cases missed during manual testing.
  • Log Analysis: Monitoring application and firewall logs for any suspicious outbound connection attempts that were blocked. This helps validate the effectiveness of the implemented controls.
  • Code Review: A final, thorough code review of the modified webhook processing logic to ensure no new vulnerabilities were introduced and that the existing ones were fully addressed.

We specifically tested scenarios like:

  • metadata_url: http://127.0.0.1:8080/internal_api
  • metadata_url: http://192.168.1.1/admin
  • metadata_url: http://localhost:5432/users
  • metadata_url: http://169.254.169.254/latest/meta-data/ (if applicable to the hosting environment)
  • metadata_url: http://attacker-controlled-domain.com/redirect?to=http://10.0.0.5/sensitive_data (testing for chained SSRF or redirection vulnerabilities)

Successful mitigation was confirmed when all such attempts resulted in a clear denial (e.g., a 400 Bad Request or 403 Forbidden response) without the application server making any outbound connection to the disallowed target.