Vengala Vinay

Having 9+ Years of Experience in Software Development

Home » High-Throughput Caching Strategies: Scaling MongoDB for PHP Application APIs

High-Throughput Caching Strategies: Scaling MongoDB for PHP Application APIs

Leveraging Redis for MongoDB API Caching: A Deep Dive

When scaling MongoDB-backed PHP APIs for high throughput, a robust caching strategy is paramount. Direct database queries, especially for frequently accessed, read-heavy datasets, can quickly become a bottleneck. This document outlines advanced caching techniques using Redis, focusing on practical implementation patterns for PHP applications.

Cache Invalidation Strategies: The Core Challenge

The primary challenge in any caching system is maintaining data consistency. Stale data served from the cache erodes user trust and can lead to application errors. For MongoDB APIs, common invalidation patterns include Time-To-Live (TTL) and explicit invalidation triggered by write operations.

Time-To-Live (TTL) Based Caching

TTL is the simplest approach. Data is stored in Redis with an expiration time. After the TTL expires, the data is automatically removed, forcing a fresh fetch from MongoDB on the next request. This is suitable for data that can tolerate a small degree of staleness.

PHP Implementation with Predis/Redis Client

We’ll use the popular predis/predis library for interacting with Redis. Ensure it’s installed via Composer:

composer require predis/predis

Here’s a typical pattern for fetching data with TTL:

<?php
require 'vendor/autoload.php';

use Predis\Client;

// Configuration
$redisConfig = [
    'scheme' => 'tcp',
    'host' => '127.0.0.1',
    'port' => 6379,
];
$mongoDbConfig = [
    'dsn' => 'mongodb://localhost:27017',
    'database' => 'my_app_db',
    'collection' => 'products',
];
$cacheTtlSeconds = 300; // 5 minutes

// Initialize Redis client
try {
    $redis = new Client($redisConfig);
    $redis->connect();
} catch (Exception $e) {
    die("Could not connect to Redis: " . $e->getMessage());
}

// Initialize MongoDB client (using mongodm/mongodm for example)
// In a real app, this would be a more robust setup
$mongoClient = new \MongoDB\Client($mongoDbConfig['dsn']);
$collection = $mongoClient->selectDatabase($mongoDbConfig['database'])->selectCollection($mongoDbConfig['collection']);

function getProductById(string $productId, Client $redis, array $mongoConfig, int $ttl): ?array
{
    $cacheKey = "product:{$productId}";

    // 1. Try to fetch from Redis cache
    $cachedProduct = $redis->get($cacheKey);

    if ($cachedProduct) {
        // Cache hit
        return json_decode($cachedProduct, true);
    }

    // 2. Cache miss: Fetch from MongoDB
    // Replace with your actual MongoDB query logic
    $product = $collection->findOne(['_id' => new MongoDB\BSON\ObjectId($productId)]);

    if (!$product) {
        return null; // Product not found
    }

    $productArray = $product->getArrayCopy(); // Convert BSON document to array

    // 3. Store in Redis cache with TTL
    $redis->setex($cacheKey, $ttl, json_encode($productArray));

    return $productArray;
}

// Example usage:
$productId = '60d5ec49f8f1b3a2b4c8d9e1'; // Replace with a valid MongoDB ObjectId
$productData = getProductById($productId, $redis, $mongoDbConfig, $cacheTtlSeconds);

if ($productData) {
    header('Content-Type: application/json');
    echo json_encode($productData);
} else {
    http_response_code(404);
    echo json_encode(['error' => 'Product not found']);
}
?>

Explicit Invalidation with Write Operations

For critical data where even slight staleness is unacceptable, explicit invalidation is preferred. This involves removing cache entries whenever the underlying data in MongoDB is modified (created, updated, or deleted).

Triggering Invalidation from API Endpoints

This pattern requires integrating cache invalidation logic directly into your API’s write operations (POST, PUT, DELETE). The challenge here is ensuring the invalidation happens reliably, even in the face of application errors.

<?php
require 'vendor/autoload.php';

use Predis\Client;
use MongoDB\Client as MongoClient;
use MongoDB\BSON\ObjectId;

// ... (Redis and MongoDB client initialization as above) ...

function updateProduct(string $productId, array $updateData, Client $redis, MongoClient $mongoClient, string $dbName, string $collectionName): bool
{
    $cacheKey = "product:{$productId}";
    $filter = ['_id' => new ObjectId($productId)];

    // 1. Update the document in MongoDB
    $result = $mongoClient->selectDatabase($dbName)->selectCollection($collectionName)->updateOne($filter, ['$set' => $updateData]);

    if ($result->getModifiedCount() === 1) {
        // 2. Invalidate the cache entry if the update was successful
        $redis->del($cacheKey);
        return true;
    }

    return false;
}

function deleteProduct(string $productId, Client $redis, MongoClient $mongoClient, string $dbName, string $collectionName): bool
{
    $cacheKey = "product:{$productId}";
    $filter = ['_id' => new ObjectId($productId)];

    // 1. Delete the document from MongoDB
    $result = $mongoClient->selectDatabase($dbName)->selectCollection($collectionName)->deleteOne($filter);

    if ($result->getDeletedCount() === 1) {
        // 2. Invalidate the cache entry if the deletion was successful
        $redis->del($cacheKey);
        return true;
    }

    return false;
}

// Example usage for update:
$productIdToUpdate = '60d5ec49f8f1b3a2b4c8d9e1';
$updatePayload = ['price' => 19.99, 'stock' => 50];
if (updateProduct($productIdToUpdate, $updatePayload, $redis, $mongoClient, $mongoDbConfig['database'], $mongoDbConfig['collection'])) {
    echo "Product {$productIdToUpdate} updated and cache invalidated.\n";
} else {
    echo "Failed to update product {$productIdToUpdate}.\n";
}

// Example usage for delete:
$productIdToDelete = '60d5ec49f8f1b3a2b4c8d9e2';
if (deleteProduct($productIdToDelete, $redis, $mongoClient, $mongoDbConfig['database'], $mongoDbConfig['collection'])) {
    echo "Product {$productIdToDelete} deleted and cache invalidated.\n";
} else {
    echo "Failed to delete product {$productIdToDelete}.\n";
}
?>

Advanced Caching Patterns

Cache Stampede Prevention (Thundering Herd)

A cache stampede occurs when a popular cached item expires, and thousands of concurrent requests simultaneously miss the cache, all attempting to regenerate the same data from the database. This can overwhelm the database. Redis offers solutions like:

  • Locking: Implement a distributed lock (e.g., using Redis SETNX) to ensure only one process regenerates the data. Other processes wait for the lock to be released.
  • Stale-While-Revalidate: Serve the stale (expired) data immediately while a background process regenerates it. Once regenerated, update the cache.

Implementing Locking with Redis SETNX

This pattern ensures only one request fetches data from MongoDB and populates the cache. Other concurrent requests will wait briefly and retry, or return a “please try again later” response.

<?php
// ... (Redis and MongoDB client initialization) ...

function getProductByIdWithLock(string $productId, Client $redis, MongoClient $mongoClient, array $mongoConfig, int $ttl, int $lockTimeout = 5): ?array
{
    $cacheKey = "product:{$productId}";
    $lockKey = "lock:product:{$productId}";
    $lockTtl = 10; // Lock will expire after 10 seconds to prevent deadlocks

    // 1. Try to fetch from Redis cache
    $cachedProduct = $redis->get($cacheKey);
    if ($cachedProduct) {
        return json_decode($cachedProduct, true);
    }

    // 2. Attempt to acquire a lock
    // SETNX: Set if Not Exists. Returns 1 if the key was set, 0 otherwise.
    if ($redis->setnx($lockKey, time())) {
        // Lock acquired
        $redis->expire($lockKey, $lockTtl); // Set an expiration for the lock

        try {
            // 3. Cache miss: Fetch from MongoDB
            $product = $mongoClient->selectDatabase($mongoConfig['database'])->selectCollection($mongoConfig['collection'])->findOne(['_id' => new ObjectId($productId)]);

            if (!$product) {
                $redis->del($lockKey); // Release lock if not found
                return null;
            }

            $productArray = $product->getArrayCopy();

            // 4. Store in Redis cache with TTL
            $redis->setex($cacheKey, $ttl, json_encode($productArray));

            // 5. Release the lock
            $redis->del($lockKey);

            return $productArray;

        } catch (Exception $e) {
            // Ensure lock is released even if MongoDB operation fails
            $redis->del($lockKey);
            // Log the error: error_log("MongoDB operation failed: " . $e->getMessage());
            return null; // Or re-throw exception
        }
    } else {
        // Lock not acquired, another process is regenerating the cache
        // Wait briefly and retry, or return a specific response
        usleep(200000); // Wait 200ms
        return getProductByIdWithLock($productId, $redis, $mongoClient, $mongoConfig, $ttl, $lockTimeout); // Recursive retry
    }
}

// Example usage:
$productId = '60d5ec49f8f1b3a2b4c8d9e1';
$productData = getProductByIdWithLock($productId, $redis, $mongoClient, $mongoDbConfig, $cacheTtlSeconds);

if ($productData) {
    header('Content-Type: application/json');
    echo json_encode($productData);
} else {
    http_response_code(404);
    echo json_encode(['error' => 'Product not found or cache regeneration in progress']);
}
?>

Cache Partitioning and Sharding

As your dataset grows and traffic increases, a single Redis instance might become a bottleneck. Consider:

  • Redis Cluster: For horizontal scaling of Redis itself, distributing keys across multiple nodes.
  • Application-Level Sharding: Implement logic in your PHP application to route cache keys to different Redis instances or databases based on a sharding key (e.g., user ID, product category).

Application-Level Sharding Example

This example demonstrates routing cache keys based on the product ID’s hash.

<?php
// ... (Redis client initialization for multiple instances) ...

$redisInstances = [
    'instance1' => new Client(['host' => 'redis1.example.com', 'port' => 6379]),
    'instance2' => new Client(['host' => 'redis2.example.com', 'port' => 6379]),
    // ... more instances
];

function getRedisInstanceForKey(string $key, array $instances): Client
{
    $instanceKeys = array_keys($instances);
    $numInstances = count($instanceKeys);
    // Simple hash-based sharding
    $hash = crc32($key);
    $instanceIndex = $hash % $numInstances;
    return $instances[$instanceKeys[$instanceIndex]];
}

function getProductByIdSharded(string $productId, array $redisInstances, MongoClient $mongoClient, array $mongoConfig, int $ttl): ?array
{
    $cacheKey = "product:{$productId}";
    $redis = getRedisInstanceForKey($cacheKey, $redisInstances);

    // Attempt to get from the assigned Redis instance
    $cachedProduct = $redis->get($cacheKey);

    if ($cachedProduct) {
        return json_decode($cachedProduct, true);
    }

    // Cache miss: Fetch from MongoDB
    $product = $mongoClient->selectDatabase($mongoConfig['database'])->selectCollection($mongoConfig['collection'])->findOne(['_id' => new ObjectId($productId)]);

    if (!$product) {
        return null;
    }

    $productArray = $product->getArrayCopy();

    // Store in the assigned Redis instance with TTL
    $redis->setex($cacheKey, $ttl, json_encode($productArray));

    return $productArray;
}

// Example usage:
$productId = '60d5ec49f8f1b3a2b4c8d9e1';
$productData = getProductByIdSharded($productId, $redisInstances, $mongoClient, $mongoDbConfig, $cacheTtlSeconds);

// ... (rest of the response handling) ...
?>

Monitoring and Performance Tuning

Effective caching requires continuous monitoring. Key metrics to track include:

  • Cache Hit Rate: The percentage of requests served from the cache. Aim for >90% for read-heavy APIs.
  • Latency: Measure the time taken for cache lookups and database fetches.
  • Redis Memory Usage: Monitor memory consumption to avoid Redis evictions or OOM errors.
  • CPU Usage: Track CPU load on both Redis and application servers.

Tools like Redis’s built-in `INFO` command, Prometheus with Redis Exporter, and application performance monitoring (APM) solutions are invaluable for this.

# Example using redis-cli to get basic stats
redis-cli INFO memory
redis-cli INFO stats

Tuning involves adjusting TTLs, optimizing Redis data structures (e.g., using Hashes for complex objects), and potentially scaling Redis infrastructure.