Vengala Vinay

Having 9+ Years of Experience in Software Development

Home » Step-by-Step: Diagnosing Out of Memory (OOM) Killer terminating PHP-FPM pool workers on Linode Servers

Step-by-Step: Diagnosing Out of Memory (OOM) Killer terminating PHP-FPM pool workers on Linode Servers

Identifying the OOM Killer’s Handiwork

When your PHP-FPM pool workers are being unceremoniously terminated, the first place to look is the system logs. The Linux Out-of-Memory (OOM) Killer is designed to reclaim memory by terminating processes when the system is critically low on RAM. Its actions are typically logged in syslog, which on most modern Linux distributions, means checking /var/log/syslog or /var/log/messages. You’re looking for messages containing “Out of memory” or “killed process”.

A typical OOM killer log entry will look something like this:

Oct 26 10:30:01 linode-server kernel: Out of memory: Kill process 12345 (php-fpm) score 987 or sacrifice child
Oct 26 10:30:01 linode-server kernel: Killed process 12345, UID 33, total-vm:123456kB, anon-rss:67890kB, file-rss:0kB
Oct 26 10:30:01 linode-server kernel: oom_reaper: reaped memory: 67890kB

The key pieces of information here are the process ID (PID) of the killed process (12345 in this example), the process name (php-fpm), and the memory statistics (total-vm and anon-rss) at the time of termination. The score indicates how “killable” the process was deemed by the OOM killer.

Analyzing PHP-FPM Configuration and Usage

Once you’ve confirmed the OOM killer is involved, the next step is to scrutinize your PHP-FPM configuration and how it’s being utilized. The most common culprits are overly aggressive process management settings or memory-hungry PHP scripts.

Tuning PHP-FPM Process Manager Settings

PHP-FPM’s process manager settings, found in your PHP-FPM pool configuration file (e.g., /etc/php/8.1/fpm/pool.d/www.conf), directly control how many worker processes are spawned and how they are managed. The relevant directives are:

  • pm.max_children: The maximum number of child processes that will be spawned at the same time.
  • pm.start_servers: The number of child processes to start when the FPM master process is started.
  • pm.min_spare_servers: The desired minimum number of idle supervisor processes.
  • pm.max_spare_servers: The desired maximum number of idle supervisor processes.
  • pm.process_idle_timeout: The number of seconds after which a child process will be killed if it is idle.

A common mistake is setting pm.max_children too high for the available RAM. Each PHP-FPM worker process consumes memory, not just for the PHP interpreter itself, but also for the application code, loaded extensions, and the data it’s processing. If you have 100 workers and each consumes 50MB of RAM, that’s already 5GB dedicated to PHP-FPM workers, not including the OS, web server, database, and other services.

To diagnose, you can temporarily reduce pm.max_children and monitor system memory usage. For example, if you have 2GB of RAM and were previously running with pm.max_children = 100, try reducing it to pm.max_children = 20 or even lower, and observe if the OOM killer stops terminating processes.

; Example: /etc/php/8.1/fpm/pool.d/www.conf
[www]
user = www-data
group = www-data
listen = /run/php/php8.1-fpm.sock
listen.owner = www-data
listen.group = www-data
listen.mode = 0660
pm = dynamic
pm.max_children = 20       ; Reduced from a potentially higher value
pm.min_spare_servers = 5
pm.max_spare_servers = 10
pm.start_servers = 2
pm.process_idle_timeout = 10s
request_terminate_timeout = 60s
php_admin_value[memory_limit] = 128M

After modifying the configuration, remember to reload or restart PHP-FPM:

sudo systemctl reload php8.1-fpm
# or
sudo systemctl restart php8.1-fpm

Investigating Memory-Hungry PHP Scripts

Even with conservative PHP-FPM settings, a single PHP script that consumes excessive memory can trigger an OOM event, especially if it’s executed by multiple concurrent workers. This is often due to:

  • Loading large datasets into memory (e.g., reading entire files, large database result sets).
  • Complex recursive functions without proper termination conditions.
  • Memory leaks in custom code or third-party libraries.
  • Inefficient image manipulation or data processing.

To pinpoint such scripts, you can leverage PHP’s built-in memory profiling capabilities or use external tools.

Using memory_get_usage() and memory_get_peak_usage()

You can instrument your PHP code to track memory consumption. Add calls to memory_get_usage() (current memory usage) and memory_get_peak_usage() (peak memory usage since script start) at critical points in your application, especially within loops or before and after memory-intensive operations. Log these values.

<?php
// ... your code ...

$start_memory = memory_get_usage();
$start_peak_memory = memory_get_peak_usage();

// Perform a memory-intensive operation
$large_data = file_get_contents('very_large_file.json');
$data_array = json_decode($large_data, true);
unset($large_data); // Free up memory if no longer needed

$end_memory = memory_get_usage();
$end_peak_memory = memory_get_peak_usage();

error_log(sprintf(
    "Memory usage: Start=%d, End=%d, Peak=%d. Operation consumed %d bytes (peak).",
    $start_memory,
    $end_memory,
    $end_peak_memory,
    $end_peak_memory - $start_peak_memory
));

// ... rest of your code ...
?>

Analyze your PHP error logs (e.g., /var/log/php/php8.1-fpm.log) for these logged memory statistics. Scripts showing consistently high peak memory usage are prime candidates for optimization.

Leveraging Xdebug’s Profiler

For more in-depth analysis, Xdebug’s profiler can generate detailed call graphs showing function call times and memory usage. Configure Xdebug to generate cachegrind files and then use a tool like KCachegrind (Linux/Windows) or Webgrind (web-based) to visualize the data.

; Example: /etc/php/8.1/fpm/conf.d/xdebug.ini
[xdebug]
zend_extension=xdebug.so
xdebug.mode=profile
xdebug.output_dir=/tmp/xdebug_profiling
xdebug.profiler_enable_trigger=1
xdebug.profiler_trigger_value="XDEBUG_PROFILE"
xdebug.collect_memory_garbage=1

With this configuration, you can trigger profiling for a specific request by adding a GET or POST parameter (e.g., ?XDEBUG_PROFILE=1) or a cookie. The profiler output will be saved in /tmp/xdebug_profiling. Analyze these files to identify functions or code paths that consume the most memory.

System-Level Memory Monitoring and Tuning

Beyond PHP-FPM configuration, understanding your server’s overall memory landscape is crucial. Linode servers, like any VPS, have finite resources.

Monitoring Overall System Memory

Use standard Linux tools to monitor memory usage. top, htop, and free -h are your primary allies.

# Real-time monitoring with htop (install if not present: sudo apt install htop)
htop

# Quick overview of memory usage
free -h

# Monitor memory usage over time (e.g., every 5 seconds)
watch -n 5 free -h

Pay attention to the “available” memory reported by free. When this drops critically low, the OOM killer becomes active. Also, observe the memory usage of the php-fpm processes themselves. If individual workers are consistently consuming large amounts of RAM (e.g., hundreds of MB each), it points to inefficient PHP code.

Swappiness and Swap Usage

If your server has swap space configured, the kernel’s swappiness parameter influences how aggressively it uses swap. A high swappiness value means the kernel will try to move inactive memory pages to swap sooner, which can sometimes prevent OOM events but can also lead to performance degradation due to slow disk I/O.

# Check current swappiness value
cat /proc/sys/vm/swappiness

# Temporarily set swappiness (e.g., to 10 for less aggressive swapping)
sudo sysctl vm.swappiness=10

# To make it permanent, edit /etc/sysctl.conf
# Add or modify the line:
# vm.swappiness = 10
# Then apply:
sudo sysctl -p

While increasing swappiness might seem like a solution, it’s often a band-aid. If your server is constantly swapping heavily, it’s a strong indicator that you need more RAM or need to optimize your application’s memory footprint. Monitor swap usage with free -h or vmstat.

Kernel OOM Killer Tuning (Use with Extreme Caution)

The OOM killer’s behavior can be influenced by the oom_score_adj value for individual processes. This value ranges from -1000 (never kill) to +1000 (always kill first). By default, PHP-FPM workers have a moderate score. You can adjust this, but it’s generally **not recommended** as a primary solution. It can mask underlying memory issues and lead to system instability if critical processes are protected from the OOM killer.

To check the current score for a PHP-FPM process (replace 12345 with the actual PID):

cat /proc//oom_score_adj

To adjust it (e.g., make it less likely to be killed):

# Example: Reduce the likelihood of killing a specific PHP-FPM worker
sudo sh -c 'echo -500 > /proc//oom_score_adj'

This change is temporary and will be lost on process restart. Making it persistent requires more complex systemd unit file modifications or init scripts, which are beyond the scope of a quick fix and highlight the risks involved. Focus on optimizing your application and PHP-FPM configuration first.

Preventative Measures and Best Practices

Proactive measures are always better than reactive firefighting. Implement these practices to minimize the chances of hitting OOM conditions:

  • Resource Monitoring: Set up continuous monitoring (e.g., using Prometheus/Grafana, Datadog, or Linode’s built-in tools) to track RAM usage, swap usage, and PHP-FPM worker counts. Set alerts for high memory utilization.
  • Code Reviews: Regularly review PHP code for potential memory leaks or inefficient memory usage patterns.
  • Load Testing: Simulate expected traffic loads to identify memory bottlenecks before they impact production.
  • PHP Version Updates: Newer PHP versions often include performance improvements and better memory management. Keep your PHP version updated.
  • Optimize Dependencies: Ensure all libraries and frameworks are up-to-date and efficiently coded.
  • Consider Process Manager Tuning: Use the ondemand process manager if your traffic is highly variable and you want to save memory during idle periods, though this can introduce slight latency on initial requests.

By systematically diagnosing the OOM killer’s actions, analyzing your PHP-FPM and application configurations, and implementing robust monitoring, you can effectively resolve and prevent PHP-FPM worker terminations due to out-of-memory conditions on your Linode servers.