Vengala Vinay

Having 9+ Years of Experience in Software Development

Home » How We Audited a High-Traffic Ruby Enterprise Stack on Linode and Mitigated Server-Side Request Forgery (SSRF) in webhook parsers

How We Audited a High-Traffic Ruby Enterprise Stack on Linode and Mitigated Server-Side Request Forgery (SSRF) in webhook parsers

Initial Audit Scope and Methodology

Our engagement focused on a high-traffic Ruby on Rails enterprise application hosted on Linode. The primary objective was to identify and mitigate security vulnerabilities, with a specific emphasis on Server-Side Request Forgery (SSRF) within webhook processing. Our methodology involved a multi-pronged approach: static code analysis, dynamic security testing, infrastructure review, and log analysis.

The stack comprised:

  • Ruby on Rails (v6.x)
  • PostgreSQL
  • Redis
  • Nginx (as reverse proxy)
  • Sidekiq (for background jobs)
  • Linode infrastructure (compute instances, managed databases, object storage)

Static Code Analysis: Identifying SSRF Vectors

We began with a deep dive into the codebase, specifically targeting areas that handle external HTTP requests, particularly those triggered by incoming webhook payloads. The goal was to pinpoint functions or libraries that could be manipulated to make arbitrary requests to internal or external resources.

Key areas of focus included:

  • Webhook parsing controllers and services.
  • Any code that dynamically constructs URLs based on user-supplied input.
  • HTTP client libraries (e.g., `Net::HTTP`, `HTTParty`, `Faraday`).
  • Background job processors that might interact with external services based on webhook data.

A common SSRF pattern involves taking a URL or hostname from an incoming request and then using it in a subsequent outgoing request without proper validation. For instance, a webhook might contain a `callback_url` parameter. If this parameter is directly used in a `Net::HTTP.get` call without sanitization, an attacker could provide an internal IP address or a loopback address.

Example Vulnerable Code Snippet (Ruby)

# app/controllers/webhooks_controller.rb
class WebhooksController << ApplicationController
  def process_payload
    payload = JSON.parse(request.body.read)
    callback_url = payload['callback_url']

    # VULNERABLE: Directly using user-supplied URL for an outgoing request
    response = Net::HTTP.get(URI.parse(callback_url))

    # ... process response ...
    render json: { status: 'success' }
  rescue JSON::ParserError
    render json: { status: 'error', message: 'Invalid JSON' }, status: :bad_request
  rescue URI::InvalidURIError
    render json: { status: 'error', message: 'Invalid callback URL' }, status: :bad_request
  end
end

In this simplified example, if an attacker sends a payload like:

{
  "data": "some_data",
  "callback_url": "http://169.254.169.254/latest/meta-data/"
}

The application would attempt to fetch metadata from the Linode instance metadata service (or cloud provider’s equivalent), potentially leaking sensitive information. The `URI::InvalidURIError` rescue is insufficient as it only catches malformed URIs, not valid but malicious ones.

Dynamic Security Testing: Exploitation and Verification

Following static analysis, we moved to dynamic testing to confirm the identified vulnerabilities and explore potential exploitation paths. This involved crafting malicious webhook payloads and observing the application’s behavior.

Tools used:

  • `curl` for sending crafted HTTP requests.
  • Burp Suite or OWASP ZAP for intercepting and modifying requests.
  • A dedicated listener service (e.g., a simple Python Flask app or a ngrok tunnel) to receive outbound requests from the target application.

Simulating SSRF Exploitation

We tested various scenarios:

  • Accessing internal network IPs (e.g., `http://10.0.0.1`, `http://192.168.1.1`).
  • Accessing the Linode instance metadata service (if applicable and exposed internally).
  • Attempting to resolve internal DNS names.
  • Using different protocols (e.g., `file://`, `gopher://` – though less common in Ruby’s standard libraries without specific gems).

For the vulnerable code snippet above, we would send a request using `curl` to the application’s webhook endpoint:

curl -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "data": "sensitive_info",
    "callback_url": "http://169.254.169.254/v1/instance/metadata"
  }' \
  https://your-app.com/webhooks/process

If the application successfully made a request to `http://169.254.169.254/v1/instance/metadata` and returned the metadata (or an error indicating it tried to connect), the SSRF vulnerability would be confirmed.

Infrastructure and Configuration Review

While the primary focus was application code, we also reviewed the Linode infrastructure configuration for any misconfigurations that could exacerbate SSRF risks or provide alternative attack vectors.

Nginx Configuration

We examined the Nginx configuration to ensure it wasn’t inadvertently exposing internal services or allowing direct access to sensitive endpoints. For instance, ensuring Nginx was configured to only proxy to the application’s internal IP and port, and not allowing direct access to other internal services.

# /etc/nginx/sites-available/your-app
server {
    listen 80;
    server_name your-app.com;
    client_max_body_size 100M; # Important for webhook payloads

    location / {
        proxy_pass http://127.0.0.1:3000; # Assuming Rails app runs on port 3000
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    # Ensure no direct access to internal services is allowed if Nginx were to proxy them
    # Example: If you had an internal admin panel on port 8080, you'd want to proxy it
    # and not allow direct access.
    # location /admin_internal {
    #     proxy_pass http://127.0.0.1:8080;
    #     # ... other proxy settings ...
    # }
}

We also looked for any `proxy_redirect` directives that might be misconfigured, although this is less common for SSRF exploitation directly.

Linode Network Security

Linode’s firewall rules (or Security Groups if using a cloud provider with that abstraction) were reviewed. The principle of least privilege dictates that outbound connections from application servers should be restricted to only necessary destinations. While Linode’s default firewall might be permissive, custom rules should ideally limit outbound traffic to known external APIs and prevent access to private IP ranges.

Example of a restrictive outbound firewall rule (conceptual, actual implementation varies):

# Hypothetical Linode Firewall Rule
# Allow outbound HTTP/HTTPS to specific external IPs/ranges
# Deny outbound to all private IP ranges (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16)
# Deny outbound to loopback (127.0.0.1)
# Deny outbound to metadata service IP (e.g., 169.254.169.254)

It’s crucial to understand that network-level controls are a defense-in-depth measure. Application-level validation remains the primary defense against SSRF.

Mitigation Strategies: Secure Webhook Parsing

The most effective way to mitigate SSRF in webhook parsers is to validate and sanitize all user-controlled input that is used to construct outgoing requests. This involves a combination of allowlisting and denylisting, along with careful URL construction.

1. Strict Allowlisting of Domains/IPs

The most robust approach is to only permit requests to a predefined, hardcoded list of trusted domains or IP addresses. If the webhook parser needs to send data to a specific external service, that service’s domain should be the only one allowed.

# app/services/webhook_processor.rb
require 'net/http'
require 'uri'

class WebhookProcessor
  ALLOWED_CALLBACK_HOSTS = %w(
    api.trusted-partner.com
    notifications.example.com
  ).freeze

  def process(payload)
    callback_url_str = payload['callback_url']
    return unless callback_url_str

    begin
      uri = URI.parse(callback_url_str)

      # 1. Validate Scheme
      unless uri.scheme == 'http' || uri.scheme == 'https'
        Rails.logger.warn("Invalid scheme for callback URL: #{callback_url_str}")
        return
      end

      # 2. Validate Host against allowlist
      unless ALLOWED_CALLBACK_HOSTS.include?(uri.host)
        Rails.logger.warn("Disallowed host for callback URL: #{uri.host} from #{callback_url_str}")
        return
      end

      # 3. Prevent IP Address Resolution (optional but good practice)
      # This requires a more sophisticated check, potentially using a gem or custom logic
      # to ensure it's not an IP address that resolves to a disallowed host.
      # For simplicity here, we rely on the host allowlist.

      # 4. Construct and make the request safely
      http = Net::HTTP.new(uri.host, uri.port)
      http.use_ssl = (uri.scheme == 'https')
      request = Net::HTTP::Post.new(uri.request_uri)
      request['Content-Type'] = 'application/json'
      request.body = { data: payload['data'] }.to_json

      response = http.request(request)
      Rails.logger.info("Callback to #{callback_url_str} successful: #{response.code}")

    rescue URI::InvalidURIError => e
      Rails.logger.error("Invalid URI for callback: #{callback_url_str} - #{e.message}")
    rescue SocketError => e
      Rails.logger.error("Socket error for callback: #{callback_url_str} - #{e.message}")
    rescue StandardError => e
      Rails.logger.error("Error processing callback to #{callback_url_str}: #{e.message}")
    end
  end
end

2. Denylisting of Internal/Private IPs and Reserved Ranges

If a strict allowlist isn’t feasible (e.g., the webhook needs to call arbitrary external services, which is generally discouraged), a denylist can be used as a secondary measure. This involves blocking requests to known internal IP ranges and loopback addresses.

# app/services/webhook_processor.rb (continued)
require 'ipaddr'

class WebhookProcessor
  # ... (previous code) ...

  def process(payload)
    callback_url_str = payload['callback_url']
    return unless callback_url_str

    begin
      uri = URI.parse(callback_url_str)

      # ... (scheme validation) ...

      # Denylist check for IP addresses
      if uri.host && IPAddr.valid?(uri.host)
        ip_addr = IPAddr.new(uri.host)
        if ip_addr.private? || ip_addr.loopback? || ip_addr.linklocal? || ip_addr.localhost?
          Rails.logger.warn("Attempted to connect to private/loopback IP: #{uri.host} from #{callback_url_str}")
          return
        end
      end

      # If not an IP, check against a denylist of internal hostnames (less common, requires DNS resolution)
      # For simplicity, we focus on IP denylisting here.

      # ... (rest of the request logic) ...

    rescue URI::InvalidURIError => e
      # ...
    rescue SocketError => e
      # ...
    rescue IPAddr::Error => e
      Rails.logger.warn("Invalid IP address format for callback host: #{uri.host} - #{e.message}")
    rescue StandardError => e
      # ...
    end
  end
end

Note: The `IPAddr` gem in Ruby is excellent for this. `ip_addr.private?` covers RFC 1918 ranges. `loopback?` covers `127.0.0.0/8`. `linklocal?` covers `169.254.0.0/16`. `localhost?` is for `127.0.0.1`.

3. Using a Dedicated HTTP Client Gem with Security Features

Gems like `Faraday` offer more advanced configuration options, including middleware for request validation and security. While not a silver bullet, they can simplify the implementation of security checks.

# Gemfile
# gem 'faraday'
# gem 'faraday_middleware'

# app/services/faraday_webhook_client.rb
require 'faraday'
require 'faraday_middleware'
require 'ipaddr'

class FaradayWebhookClient
  ALLOWED_CALLBACK_HOSTS = %w(api.trusted-partner.com notifications.example.com).freeze

  def initialize
    @connection = Faraday.new do |conn|
      conn.request :json # Encode request body as JSON
      conn.response :logger, Rails.logger # Log requests
      conn.adapter Faraday.default_adapter # Use the default adapter
      conn.options.timeout = 5 # Set a reasonable timeout
      conn.options.open_timeout = 2 # Set a reasonable open timeout
    end

    # Add custom middleware for validation
    @connection.use :request, Faraday::Request::CallbackValidator, allowed_hosts: ALLOWED_CALLBACK_HOSTS
  end

  def send_data(url_str, data)
    uri = URI.parse(url_str)

    # Perform initial validation before passing to Faraday
    unless uri.scheme == 'http' || uri.scheme == 'https'
      Rails.logger.warn("Invalid scheme for callback URL: #{url_str}")
      return false
    end

    # The Faraday::Request::CallbackValidator middleware will handle host/IP checks

    begin
      response = @connection.post do |req|
        req.url url_str
        req.headers['Content-Type'] = 'application/json'
        req.body = data.to_json
      end
      Rails.logger.info("Callback to #{url_str} successful: #{response.status}")
      true
    rescue Faraday::ConnectionFailed => e
      Rails.logger.error("Faraday connection failed for #{url_str}: #{e.message}")
      false
    rescue Faraday::TimeoutError => e
      Rails.logger.error("Faraday timeout for #{url_str}: #{e.message}")
      false
    rescue Faraday::ClientError => e
      Rails.logger.error("Faraday client error for #{url_str}: #{e.message}")
      false
    rescue Faraday::ServerError => e
      Rails.logger.error("Faraday server error for #{url_str}: #{e.message}")
      false
    rescue URI::InvalidURIError => e
      Rails.logger.error("Invalid URI for callback: #{url_str} - #{e.message}")
      false
    rescue StandardError => e
      Rails.logger.error("Unexpected error sending callback to #{url_str}: #{e.message}")
      false
    end
  end
end

# Custom Faraday Middleware
module Faraday
  class Request::CallbackValidator < Middleware
    def initialize(app, options = {})
      super(app)
      @allowed_hosts = options[:allowed_hosts] || []
    end

    def call(env)
      uri = URI.parse(env[:url].to_s)

      # Validate Host against allowlist
      unless @allowed_hosts.include?(uri.host)
        # Check if it's an IP address and if it's disallowed
        if IPAddr.valid?(uri.host)
          ip_addr = IPAddr.new(uri.host)
          if ip_addr.private? || ip_addr.loopback? || ip_addr.linklocal? || ip_addr.localhost?
            raise Faraday::ClientError, "Disallowed private/loopback IP address: #{uri.host}"
          end
          # If it's a public IP, we might still want to check against a list of known bad IPs
          # or rely on DNS resolution if the host is not an IP.
        else
          # If it's not an IP and not in the allowlist, it's disallowed.
          raise Faraday::ClientError, "Disallowed host: #{uri.host}"
        end
      end

      @app.call(env)
    end
  end
end

This approach encapsulates the validation logic within a reusable middleware, making it cleaner and easier to apply across different parts of the application that might make external calls.

Logging and Monitoring for Anomalies

Effective logging and monitoring are critical for detecting and responding to attempted SSRF attacks. All logs related to outgoing HTTP requests, especially those originating from webhook processing, should be captured and analyzed.

Key Log Data to Capture

  • Timestamp of the request.
  • Source IP address of the original webhook request.
  • The constructed URL being requested.
  • The resolved IP address (if available).
  • The outcome of the request (success, failure, error type).
  • Any validation failures (e.g., disallowed host, private IP).

On Linode, this typically involves configuring your application to log to stdout/stderr and using a log aggregation tool (like Fluentd, Logstash, or a cloud-native solution) to collect these logs from your instances. These logs can then be sent to a centralized logging platform (e.g., Elasticsearch, Splunk, Datadog) for analysis and alerting.

Alerting on Suspicious Activity

Set up alerts for:

  • Repeated validation failures for callback URLs.
  • Attempts to connect to private IP ranges or loopback addresses.
  • Unexpected outbound connection attempts to external IPs not in the allowlist.
  • High volume of errors related to external HTTP requests.

By combining robust code-level validation with comprehensive logging and proactive alerting, the risk of successful SSRF exploitation can be significantly reduced.