Resolving Out of Memory (OOM) Killer terminating PHP-FPM pool workers Under Peak Event Traffic on Linode

4. This spawning continues until `pm.max_children` is reached.

Tuning Recommendations:

`pm.max_children`: As calculated above, based on available memory and per-process usage.
`pm.start_servers`: Set this to a value that can handle moderate baseline traffic. A good starting point is `pm.max_children / 2` or `pm.max_children / 3`. For `max_children = 60`, `start_servers = 20` might be reasonable.
`pm.min_spare_servers`: Keep this low, e.g., 2-5. You don’t want too many idle processes consuming memory unnecessarily.
`pm.max_spare_servers`: This should be set to accommodate short bursts of traffic without immediately spawning new processes. A value like 5-10 is often sufficient. If `max_spare_servers` is too high, you might end up with many idle workers consuming memory. If it’s too low, new processes will be spawned more aggressively.
`pm.process_idle_timeout`: Controls how long an idle process waits before being killed. A shorter timeout (e.g., 10s) can free up memory faster during lulls, but too short might cause excessive respawning.

Consider the trade-off: higher `max_children` allows more concurrent requests but consumes more RAM. Lower `max_children` conserves RAM but can lead to request queuing and slower response times during peaks.

Investigating Application-Level Memory Leaks

Sometimes, the issue isn’t just the number of processes but how much memory *each* process consumes over time. This points to memory leaks within your PHP application code or its dependencies (e.g., Composer packages).

1. Enable Profiling:

Use tools like Xdebug or Blackfire.io to profile your application under load. Look for functions or code paths that consistently consume large amounts of memory or show increasing memory usage over multiple requests.

// Example using memory_get_usage() and memory_get_peak_usage()
// Place these strategically in your code or use a profiler

$start_memory = memory_get_usage();
$start_peak_memory = memory_get_peak_usage();

// ... your code ...

$end_memory = memory_get_usage();
$end_peak_memory = memory_get_peak_usage();

error_log(sprintf("Memory usage: Start=%d, End=%d, Peak=%d",
    $start_memory, $end_memory, $end_peak_memory));

2. Analyze Dependencies:

Third-party libraries can also be memory hogs. If you suspect a specific library, try disabling it temporarily or using an older/newer version to see if the memory usage changes. Composer’s `composer show –tree` can help visualize dependencies.

3. Database Query Optimization:

Large datasets fetched from the database and loaded entirely into PHP arrays are a common source of high memory consumption. Use techniques like:

Fetching data in batches (pagination).
Using database cursors or streaming results where possible.
Selecting only necessary columns.
Optimizing SQL queries to reduce the amount of data processed.

System-Level Memory Management and Swappiness

While not directly related to PHP-FPM configuration, understanding your system’s memory management is crucial. Linode instances, like any VPS, have finite RAM. If RAM is exhausted, the system *might* use swap space, but this is significantly slower and can lead to performance degradation. The OOM killer is typically invoked *before* swap is heavily utilized, or if swap is also exhausted.

1. Check Swap Usage:

swapon --show
free -h

If your system is actively swapping heavily, it’s a strong indicator of insufficient RAM for the workload, and the OOM killer is a likely consequence.

2. Adjust Swappiness:

The `swappiness` parameter controls how aggressively the kernel swaps memory pages. A value of 0 means the kernel will avoid swapping as much as possible, while 100 means it will swap aggressively. For performance-sensitive applications where OOM is a concern, you might want to reduce swappiness.

# Check current swappiness
cat /proc/sys/vm/swappiness

# Temporarily set swappiness (e.g., to 10)
sudo sysctl vm.swappiness=10

# Make it permanent by editing /etc/sysctl.conf
# Add or modify the line:
# vm.swappiness = 10

Reducing swappiness can help prevent the system from swapping out critical processes when memory pressure is high, potentially delaying or preventing the OOM killer’s intervention, but it also means you’ll hit OOM faster if RAM is truly exhausted.

Monitoring and Alerting Strategy

Proactive monitoring is key to preventing OOM events. Implement a robust monitoring solution that tracks:

System Memory Usage: Total, used, free, available.
PHP-FPM Process Count: Number of active and idle workers.
Individual PHP-FPM Process Memory: Average and peak `RES` usage.
Swap Usage.
Load Average.
Nginx/Apache Request Rate and Error Rate.

Tools like Prometheus with Node Exporter and PHP-FPM Exporter, Datadog, New Relic, or even basic `sar` and `vmstat` combined with custom scripts can provide this visibility. Set up alerts for:

High memory utilization (e.g., > 85-90% available memory).
High PHP-FPM worker count approaching `pm.max_children`.
Sudden spikes in PHP-FPM process memory.
Kernel OOM killer messages detected in logs.

By combining log analysis, careful configuration tuning, application profiling, and comprehensive monitoring, you can effectively diagnose and resolve OOM Killer issues impacting your PHP-FPM pool workers during peak traffic.

3. If idle workers (up to `pm.max_spare_servers`) are available, they are used. If not, new workers are spawned.

4. This spawning continues until `pm.max_children` is reached.

Tuning Recommendations:

`pm.max_children`: As calculated above, based on available memory and per-process usage.
`pm.start_servers`: Set this to a value that can handle moderate baseline traffic. A good starting point is `pm.max_children / 2` or `pm.max_children / 3`. For `max_children = 60`, `start_servers = 20` might be reasonable.
`pm.min_spare_servers`: Keep this low, e.g., 2-5. You don’t want too many idle processes consuming memory unnecessarily.
`pm.max_spare_servers`: This should be set to accommodate short bursts of traffic without immediately spawning new processes. A value like 5-10 is often sufficient. If `max_spare_servers` is too high, you might end up with many idle workers consuming memory. If it’s too low, new processes will be spawned more aggressively.
`pm.process_idle_timeout`: Controls how long an idle process waits before being killed. A shorter timeout (e.g., 10s) can free up memory faster during lulls, but too short might cause excessive respawning.

Investigating Application-Level Memory Leaks

1. Enable Profiling:

// Example using memory_get_usage() and memory_get_peak_usage()
// Place these strategically in your code or use a profiler

$start_memory = memory_get_usage();
$start_peak_memory = memory_get_peak_usage();

// ... your code ...

$end_memory = memory_get_usage();
$end_peak_memory = memory_get_peak_usage();

error_log(sprintf("Memory usage: Start=%d, End=%d, Peak=%d",
    $start_memory, $end_memory, $end_peak_memory));

2. Analyze Dependencies:

3. Database Query Optimization:

Large datasets fetched from the database and loaded entirely into PHP arrays are a common source of high memory consumption. Use techniques like:

Fetching data in batches (pagination).
Using database cursors or streaming results where possible.
Selecting only necessary columns.
Optimizing SQL queries to reduce the amount of data processed.

System-Level Memory Management and Swappiness

1. Check Swap Usage:

swapon --show
free -h

If your system is actively swapping heavily, it’s a strong indicator of insufficient RAM for the workload, and the OOM killer is a likely consequence.

2. Adjust Swappiness:

# Check current swappiness
cat /proc/sys/vm/swappiness

# Temporarily set swappiness (e.g., to 10)
sudo sysctl vm.swappiness=10

# Make it permanent by editing /etc/sysctl.conf
# Add or modify the line:
# vm.swappiness = 10

Monitoring and Alerting Strategy

Proactive monitoring is key to preventing OOM events. Implement a robust monitoring solution that tracks:

System Memory Usage: Total, used, free, available.
PHP-FPM Process Count: Number of active and idle workers.
Individual PHP-FPM Process Memory: Average and peak `RES` usage.
Swap Usage.
Load Average.
Nginx/Apache Request Rate and Error Rate.

Tools like Prometheus with Node Exporter and PHP-FPM Exporter, Datadog, New Relic, or even basic `sar` and `vmstat` combined with custom scripts can provide this visibility. Set up alerts for:

High memory utilization (e.g., > 85-90% available memory).
High PHP-FPM worker count approaching `pm.max_children`.
Sudden spikes in PHP-FPM process memory.
Kernel OOM killer messages detected in logs.

3. If idle workers (up to `pm.max_spare_servers`) are available, they are used. If not, new workers are spawned.

4. This spawning continues until `pm.max_children` is reached.

Tuning Recommendations:

`pm.max_children`: As calculated above, based on available memory and per-process usage.
`pm.start_servers`: Set this to a value that can handle moderate baseline traffic. A good starting point is `pm.max_children / 2` or `pm.max_children / 3`. For `max_children = 60`, `start_servers = 20` might be reasonable.
`pm.min_spare_servers`: Keep this low, e.g., 2-5. You don’t want too many idle processes consuming memory unnecessarily.
`pm.max_spare_servers`: This should be set to accommodate short bursts of traffic without immediately spawning new processes. A value like 5-10 is often sufficient. If `max_spare_servers` is too high, you might end up with many idle workers consuming memory. If it’s too low, new processes will be spawned more aggressively.
`pm.process_idle_timeout`: Controls how long an idle process waits before being killed. A shorter timeout (e.g., 10s) can free up memory faster during lulls, but too short might cause excessive respawning.

Investigating Application-Level Memory Leaks

1. Enable Profiling:

// Example using memory_get_usage() and memory_get_peak_usage()
// Place these strategically in your code or use a profiler

$start_memory = memory_get_usage();
$start_peak_memory = memory_get_peak_usage();

// ... your code ...

$end_memory = memory_get_usage();
$end_peak_memory = memory_get_peak_usage();

error_log(sprintf("Memory usage: Start=%d, End=%d, Peak=%d",
    $start_memory, $end_memory, $end_peak_memory));

2. Analyze Dependencies:

3. Database Query Optimization:

Large datasets fetched from the database and loaded entirely into PHP arrays are a common source of high memory consumption. Use techniques like:

Fetching data in batches (pagination).
Using database cursors or streaming results where possible.
Selecting only necessary columns.
Optimizing SQL queries to reduce the amount of data processed.

System-Level Memory Management and Swappiness

1. Check Swap Usage:

swapon --show
free -h

If your system is actively swapping heavily, it’s a strong indicator of insufficient RAM for the workload, and the OOM killer is a likely consequence.

2. Adjust Swappiness:

# Check current swappiness
cat /proc/sys/vm/swappiness

# Temporarily set swappiness (e.g., to 10)
sudo sysctl vm.swappiness=10

# Make it permanent by editing /etc/sysctl.conf
# Add or modify the line:
# vm.swappiness = 10

Monitoring and Alerting Strategy

Proactive monitoring is key to preventing OOM events. Implement a robust monitoring solution that tracks:

System Memory Usage: Total, used, free, available.
PHP-FPM Process Count: Number of active and idle workers.
Individual PHP-FPM Process Memory: Average and peak `RES` usage.
Swap Usage.
Load Average.
Nginx/Apache Request Rate and Error Rate.

Tools like Prometheus with Node Exporter and PHP-FPM Exporter, Datadog, New Relic, or even basic `sar` and `vmstat` combined with custom scripts can provide this visibility. Set up alerts for:

High memory utilization (e.g., > 85-90% available memory).
High PHP-FPM worker count approaching `pm.max_children`.
Sudden spikes in PHP-FPM process memory.
Kernel OOM killer messages detected in logs.

2. Traffic increases. Requests start queuing. PHP-FPM needs more workers.

3. If idle workers (up to `pm.max_spare_servers`) are available, they are used. If not, new workers are spawned.

4. This spawning continues until `pm.max_children` is reached.

Tuning Recommendations:

`pm.max_children`: As calculated above, based on available memory and per-process usage.
`pm.start_servers`: Set this to a value that can handle moderate baseline traffic. A good starting point is `pm.max_children / 2` or `pm.max_children / 3`. For `max_children = 60`, `start_servers = 20` might be reasonable.
`pm.min_spare_servers`: Keep this low, e.g., 2-5. You don’t want too many idle processes consuming memory unnecessarily.
`pm.max_spare_servers`: This should be set to accommodate short bursts of traffic without immediately spawning new processes. A value like 5-10 is often sufficient. If `max_spare_servers` is too high, you might end up with many idle workers consuming memory. If it’s too low, new processes will be spawned more aggressively.
`pm.process_idle_timeout`: Controls how long an idle process waits before being killed. A shorter timeout (e.g., 10s) can free up memory faster during lulls, but too short might cause excessive respawning.

Investigating Application-Level Memory Leaks

1. Enable Profiling:

// Example using memory_get_usage() and memory_get_peak_usage()
// Place these strategically in your code or use a profiler

$start_memory = memory_get_usage();
$start_peak_memory = memory_get_peak_usage();

// ... your code ...

$end_memory = memory_get_usage();
$end_peak_memory = memory_get_peak_usage();

error_log(sprintf("Memory usage: Start=%d, End=%d, Peak=%d",
    $start_memory, $end_memory, $end_peak_memory));

2. Analyze Dependencies:

3. Database Query Optimization:

Large datasets fetched from the database and loaded entirely into PHP arrays are a common source of high memory consumption. Use techniques like:

Fetching data in batches (pagination).
Using database cursors or streaming results where possible.
Selecting only necessary columns.
Optimizing SQL queries to reduce the amount of data processed.

System-Level Memory Management and Swappiness

1. Check Swap Usage:

swapon --show
free -h

If your system is actively swapping heavily, it’s a strong indicator of insufficient RAM for the workload, and the OOM killer is a likely consequence.

2. Adjust Swappiness:

# Check current swappiness
cat /proc/sys/vm/swappiness

# Temporarily set swappiness (e.g., to 10)
sudo sysctl vm.swappiness=10

# Make it permanent by editing /etc/sysctl.conf
# Add or modify the line:
# vm.swappiness = 10

Monitoring and Alerting Strategy

Proactive monitoring is key to preventing OOM events. Implement a robust monitoring solution that tracks:

System Memory Usage: Total, used, free, available.
PHP-FPM Process Count: Number of active and idle workers.
Individual PHP-FPM Process Memory: Average and peak `RES` usage.
Swap Usage.
Load Average.
Nginx/Apache Request Rate and Error Rate.

Tools like Prometheus with Node Exporter and PHP-FPM Exporter, Datadog, New Relic, or even basic `sar` and `vmstat` combined with custom scripts can provide this visibility. Set up alerts for:

High memory utilization (e.g., > 85-90% available memory).
High PHP-FPM worker count approaching `pm.max_children`.
Sudden spikes in PHP-FPM process memory.
Kernel OOM killer messages detected in logs.

2. Traffic increases. Requests start queuing. PHP-FPM needs more workers.

3. If idle workers (up to `pm.max_spare_servers`) are available, they are used. If not, new workers are spawned.

4. This spawning continues until `pm.max_children` is reached.

Tuning Recommendations:

`pm.max_children`: As calculated above, based on available memory and per-process usage.
`pm.start_servers`: Set this to a value that can handle moderate baseline traffic. A good starting point is `pm.max_children / 2` or `pm.max_children / 3`. For `max_children = 60`, `start_servers = 20` might be reasonable.
`pm.min_spare_servers`: Keep this low, e.g., 2-5. You don’t want too many idle processes consuming memory unnecessarily.
`pm.max_spare_servers`: This should be set to accommodate short bursts of traffic without immediately spawning new processes. A value like 5-10 is often sufficient. If `max_spare_servers` is too high, you might end up with many idle workers consuming memory. If it’s too low, new processes will be spawned more aggressively.
`pm.process_idle_timeout`: Controls how long an idle process waits before being killed. A shorter timeout (e.g., 10s) can free up memory faster during lulls, but too short might cause excessive respawning.

Investigating Application-Level Memory Leaks

1. Enable Profiling:

// Example using memory_get_usage() and memory_get_peak_usage()
// Place these strategically in your code or use a profiler

$start_memory = memory_get_usage();
$start_peak_memory = memory_get_peak_usage();

// ... your code ...

$end_memory = memory_get_usage();
$end_peak_memory = memory_get_peak_usage();

error_log(sprintf("Memory usage: Start=%d, End=%d, Peak=%d",
    $start_memory, $end_memory, $end_peak_memory));

2. Analyze Dependencies:

3. Database Query Optimization:

Large datasets fetched from the database and loaded entirely into PHP arrays are a common source of high memory consumption. Use techniques like:

Fetching data in batches (pagination).
Using database cursors or streaming results where possible.
Selecting only necessary columns.
Optimizing SQL queries to reduce the amount of data processed.

System-Level Memory Management and Swappiness

1. Check Swap Usage:

swapon --show
free -h

If your system is actively swapping heavily, it’s a strong indicator of insufficient RAM for the workload, and the OOM killer is a likely consequence.

2. Adjust Swappiness:

# Check current swappiness
cat /proc/sys/vm/swappiness

# Temporarily set swappiness (e.g., to 10)
sudo sysctl vm.swappiness=10

# Make it permanent by editing /etc/sysctl.conf
# Add or modify the line:
# vm.swappiness = 10

Monitoring and Alerting Strategy

Proactive monitoring is key to preventing OOM events. Implement a robust monitoring solution that tracks:

System Memory Usage: Total, used, free, available.
PHP-FPM Process Count: Number of active and idle workers.
Individual PHP-FPM Process Memory: Average and peak `RES` usage.
Swap Usage.
Load Average.
Nginx/Apache Request Rate and Error Rate.

Tools like Prometheus with Node Exporter and PHP-FPM Exporter, Datadog, New Relic, or even basic `sar` and `vmstat` combined with custom scripts can provide this visibility. Set up alerts for:

High memory utilization (e.g., > 85-90% available memory).
High PHP-FPM worker count approaching `pm.max_children`.
Sudden spikes in PHP-FPM process memory.
Kernel OOM killer messages detected in logs.

1. Current state: 10 active workers, 5 idle workers. Total PHP-FPM processes = 15.

2. Traffic increases. Requests start queuing. PHP-FPM needs more workers.

3. If idle workers (up to `pm.max_spare_servers`) are available, they are used. If not, new workers are spawned.

4. This spawning continues until `pm.max_children` is reached.

Tuning Recommendations:

`pm.max_children`: As calculated above, based on available memory and per-process usage.
`pm.start_servers`: Set this to a value that can handle moderate baseline traffic. A good starting point is `pm.max_children / 2` or `pm.max_children / 3`. For `max_children = 60`, `start_servers = 20` might be reasonable.
`pm.min_spare_servers`: Keep this low, e.g., 2-5. You don’t want too many idle processes consuming memory unnecessarily.
`pm.max_spare_servers`: This should be set to accommodate short bursts of traffic without immediately spawning new processes. A value like 5-10 is often sufficient. If `max_spare_servers` is too high, you might end up with many idle workers consuming memory. If it’s too low, new processes will be spawned more aggressively.
`pm.process_idle_timeout`: Controls how long an idle process waits before being killed. A shorter timeout (e.g., 10s) can free up memory faster during lulls, but too short might cause excessive respawning.

Investigating Application-Level Memory Leaks

1. Enable Profiling:

// Example using memory_get_usage() and memory_get_peak_usage()
// Place these strategically in your code or use a profiler

$start_memory = memory_get_usage();
$start_peak_memory = memory_get_peak_usage();

// ... your code ...

$end_memory = memory_get_usage();
$end_peak_memory = memory_get_peak_usage();

error_log(sprintf("Memory usage: Start=%d, End=%d, Peak=%d",
    $start_memory, $end_memory, $end_peak_memory));

2. Analyze Dependencies:

3. Database Query Optimization:

Large datasets fetched from the database and loaded entirely into PHP arrays are a common source of high memory consumption. Use techniques like:

Fetching data in batches (pagination).
Using database cursors or streaming results where possible.
Selecting only necessary columns.
Optimizing SQL queries to reduce the amount of data processed.

System-Level Memory Management and Swappiness

1. Check Swap Usage:

swapon --show
free -h

If your system is actively swapping heavily, it’s a strong indicator of insufficient RAM for the workload, and the OOM killer is a likely consequence.

2. Adjust Swappiness:

# Check current swappiness
cat /proc/sys/vm/swappiness

# Temporarily set swappiness (e.g., to 10)
sudo sysctl vm.swappiness=10

# Make it permanent by editing /etc/sysctl.conf
# Add or modify the line:
# vm.swappiness = 10

Monitoring and Alerting Strategy

Proactive monitoring is key to preventing OOM events. Implement a robust monitoring solution that tracks:

System Memory Usage: Total, used, free, available.
PHP-FPM Process Count: Number of active and idle workers.
Individual PHP-FPM Process Memory: Average and peak `RES` usage.
Swap Usage.
Load Average.
Nginx/Apache Request Rate and Error Rate.

Tools like Prometheus with Node Exporter and PHP-FPM Exporter, Datadog, New Relic, or even basic `sar` and `vmstat` combined with custom scripts can provide this visibility. Set up alerts for:

High memory utilization (e.g., > 85-90% available memory).
High PHP-FPM worker count approaching `pm.max_children`.
Sudden spikes in PHP-FPM process memory.
Kernel OOM killer messages detected in logs.

1. Current state: 10 active workers, 5 idle workers. Total PHP-FPM processes = 15.

2. Traffic increases. Requests start queuing. PHP-FPM needs more workers.

3. If idle workers (up to `pm.max_spare_servers`) are available, they are used. If not, new workers are spawned.

4. This spawning continues until `pm.max_children` is reached.

Tuning Recommendations:

`pm.max_children`: As calculated above, based on available memory and per-process usage.
`pm.start_servers`: Set this to a value that can handle moderate baseline traffic. A good starting point is `pm.max_children / 2` or `pm.max_children / 3`. For `max_children = 60`, `start_servers = 20` might be reasonable.
`pm.min_spare_servers`: Keep this low, e.g., 2-5. You don’t want too many idle processes consuming memory unnecessarily.
`pm.max_spare_servers`: This should be set to accommodate short bursts of traffic without immediately spawning new processes. A value like 5-10 is often sufficient. If `max_spare_servers` is too high, you might end up with many idle workers consuming memory. If it’s too low, new processes will be spawned more aggressively.
`pm.process_idle_timeout`: Controls how long an idle process waits before being killed. A shorter timeout (e.g., 10s) can free up memory faster during lulls, but too short might cause excessive respawning.

Investigating Application-Level Memory Leaks

1. Enable Profiling:

// Example using memory_get_usage() and memory_get_peak_usage()
// Place these strategically in your code or use a profiler

$start_memory = memory_get_usage();
$start_peak_memory = memory_get_peak_usage();

// ... your code ...

$end_memory = memory_get_usage();
$end_peak_memory = memory_get_peak_usage();

error_log(sprintf("Memory usage: Start=%d, End=%d, Peak=%d",
    $start_memory, $end_memory, $end_peak_memory));

2. Analyze Dependencies:

3. Database Query Optimization:

Large datasets fetched from the database and loaded entirely into PHP arrays are a common source of high memory consumption. Use techniques like:

Fetching data in batches (pagination).
Using database cursors or streaming results where possible.
Selecting only necessary columns.
Optimizing SQL queries to reduce the amount of data processed.

System-Level Memory Management and Swappiness

1. Check Swap Usage:

swapon --show
free -h

If your system is actively swapping heavily, it’s a strong indicator of insufficient RAM for the workload, and the OOM killer is a likely consequence.

2. Adjust Swappiness:

# Check current swappiness
cat /proc/sys/vm/swappiness

# Temporarily set swappiness (e.g., to 10)
sudo sysctl vm.swappiness=10

# Make it permanent by editing /etc/sysctl.conf
# Add or modify the line:
# vm.swappiness = 10

Monitoring and Alerting Strategy

Proactive monitoring is key to preventing OOM events. Implement a robust monitoring solution that tracks:

System Memory Usage: Total, used, free, available.
PHP-FPM Process Count: Number of active and idle workers.
Individual PHP-FPM Process Memory: Average and peak `RES` usage.
Swap Usage.
Load Average.
Nginx/Apache Request Rate and Error Rate.

Tools like Prometheus with Node Exporter and PHP-FPM Exporter, Datadog, New Relic, or even basic `sar` and `vmstat` combined with custom scripts can provide this visibility. Set up alerts for:

High memory utilization (e.g., > 85-90% available memory).
High PHP-FPM worker count approaching `pm.max_children`.
Sudden spikes in PHP-FPM process memory.
Kernel OOM killer messages detected in logs.

Identifying the OOM Killer’s Handiwork

The first and most critical step in resolving OOM Killer events is to confirm it’s indeed the culprit. During peak traffic, system logs become your primary forensic tool. We’re looking for specific kernel messages indicating a process was terminated due to memory exhaustion. On most Linux systems, these messages are logged via `syslog` or `journald`.

A common indicator is a message like:

Oct 26 10:30:01 linode1 kernel: [12345.678901] Out of memory: Kill process 9876 (php-fpm) score 1234,
Oct 26 10:30:01 linode1 kernel: [12345.678901]   total-vm:12345678kB, anon-rss:8765432kB, file-rss:123456kB, shmem-rss:0kB
Oct 26 10:30:01 linode1 kernel: [12345.678901] Out of memory: Memory cgroup out of memory: Kill process 9876 (php-fpm) score 1234 or sacrifice child
Oct 26 10:30:01 linode1 kernel: [12345.678901] Killed process 9876 (php-fpm) total-vm:12345678kB, anon-rss:8765432kB, file-rss:123456kB, shmem-rss:0kB

The key elements here are “Out of memory: Kill process” and the process name, which in our case is `php-fpm`. The `score` indicates the OOM killer’s heuristic for choosing which process to terminate – higher scores mean a more likely candidate. The memory breakdown (total-vm, anon-rss, file-rss) provides crucial clues about the memory usage pattern of the killed process.

To quickly check recent OOM events, you can use `journalctl` (if your system uses systemd):

sudo journalctl -k | grep -i "out of memory\|killed process\|oom-killer"

If your system uses `rsyslog`, you’ll likely find these messages in `/var/log/syslog`, `/var/log/messages`, or a dedicated kernel log file. A `grep` command on these files will serve the same purpose.

Analyzing PHP-FPM Pool Configuration

Once confirmed, the next step is to scrutinize your PHP-FPM pool configuration. The `pm` (process manager) settings are paramount. For high-traffic scenarios, `pm = dynamic` or `pm = ondemand` are often preferred over `pm = static` to conserve resources when idle, but they can also lead to rapid worker spawning and potential OOM issues if not tuned correctly.

Let’s examine a typical `www.conf` file, usually located at `/etc/php/[version]/fpm/pool.d/www.conf` or similar:

[global]
error_log = /var/log/php-fpm/error.log
pid = /run/php/php7.4-fpm.pid

[www]
user = www-data
group = www-data
listen = /run/php/php7.4-fpm.sock
listen.owner = www-data
listen.group = www-data
listen.mode = 0660

; Process Manager Settings
pm = dynamic
pm.max_children = 50
pm.start_servers = 5
pm.min_spare_servers = 2
pm.max_spare_servers = 10
pm.max_requests = 500
pm.process_idle_timeout = 10s

; Resource Limits (often overlooked)
; php_admin_value[memory_limit] = 128M
; php_admin_flag[display_errors] = off

Key parameters to focus on:

pm.max_children: This is the absolute maximum number of child processes that will be spawned. If your application, under peak load, requires more than this number of workers to handle requests concurrently, you’ll hit this limit. When `pm` is `dynamic` or `ondemand`, PHP-FPM will try to spawn up to this number.
pm.start_servers: The number of child processes started when the master process is started.
pm.min_spare_servers: The desired minimum number of idle supervisor processes.
pm.max_spare_servers: The desired maximum number of idle supervisor processes. If there are more idle processes than this, they will be killed.
pm.max_requests: The number of requests each child process should execute before respawning. This helps prevent memory leaks in long-running scripts.
php_admin_value[memory_limit]: The maximum amount of memory a single PHP script can consume. This is crucial. If individual scripts are memory-hungry, even with few children, you can exhaust system RAM.

The OOM killer is a system-level defense. PHP-FPM’s configuration dictates how many processes it *wants* to run, and each of those processes consumes memory. When the sum of memory used by PHP-FPM workers, the web server (Nginx/Apache), the database, and the operating system itself exceeds available RAM, the OOM killer steps in.

Tuning `pm.max_children` and Memory Limits

The most direct approach is to increase `pm.max_children` or decrease `memory_limit`. However, this is often a reactive measure. A more strategic approach involves understanding your actual memory footprint.

1. Calculate Available Memory:

free -h

Note the `available` memory. This is what’s truly free for applications. Remember that the OS and other services (Nginx, MySQL, etc.) also consume memory. A good rule of thumb is to leave at least 10-20% of total RAM free for the OS and caching.

2. Estimate Per-Process Memory Usage:

This is the trickiest part. You need to find the *average* and *peak* memory usage of a single PHP-FPM worker process under load. You can use tools like `htop` or `top` and filter for `php-fpm` processes. Observe the `RES` (Resident Set Size) or `VIRT` (Virtual Memory Size) column. `RES` is a better indicator of actual physical RAM usage.

# While under moderate load, observe memory usage
htop -p $(pgrep php-fpm | paste -sd,)

# Or using top
top -p $(pgrep php-fpm | paste -sd,) -b -n 1 | grep php-fpm

Let’s say you observe that a typical PHP-FPM worker process consumes an average of 50MB of RAM (`RES`).

3. Determine `pm.max_children` Target:

If your server has 4GB of RAM (4096MB) and you want to reserve 1GB (1024MB) for the OS and other services, you have 3072MB available for PHP-FPM. If each worker uses 50MB, then:

3072MB / 50MB/process = 61.44 processes

This suggests setting `pm.max_children` to around 60. However, this is a *static* calculation. During peak traffic, memory usage can spike due to:

Inefficient code or memory leaks in your application.
Large data processing (e.g., generating reports, processing images).
Database queries returning large result sets that are loaded into PHP memory.
External API calls returning large payloads.
Caching mechanisms that load significant data into memory.

Therefore, it’s often wise to set `pm.max_children` slightly lower than the theoretical maximum and monitor closely. You might start with 40-50 and gradually increase if needed, always observing system memory.

4. Adjust `memory_limit` in `php.ini` or Pool Configuration:

If individual scripts are the culprits, you might need to increase `memory_limit`. However, be cautious. A high `memory_limit` can allow a single buggy script to consume excessive resources, potentially leading to OOM even with a low `pm.max_children`. It’s better to optimize scripts than to blindly increase `memory_limit`. You can set this globally in `php.ini` or per pool in `www.conf` using `php_admin_value[memory_limit] = 256M`.

Optimizing PHP-FPM `pm` Settings for Dynamic Loads

For dynamic `pm` settings, the interplay between `pm.max_children`, `pm.start_servers`, `pm.min_spare_servers`, and `pm.max_spare_servers` is critical for handling traffic spikes without overwhelming the system.

Scenario: Traffic Spike

1. Current state: 10 active workers, 5 idle workers. Total PHP-FPM processes = 15.

2. Traffic increases. Requests start queuing. PHP-FPM needs more workers.

3. If idle workers (up to `pm.max_spare_servers`) are available, they are used. If not, new workers are spawned.

4. This spawning continues until `pm.max_children` is reached.

Tuning Recommendations:

`pm.max_children`: As calculated above, based on available memory and per-process usage.
`pm.start_servers`: Set this to a value that can handle moderate baseline traffic. A good starting point is `pm.max_children / 2` or `pm.max_children / 3`. For `max_children = 60`, `start_servers = 20` might be reasonable.
`pm.min_spare_servers`: Keep this low, e.g., 2-5. You don’t want too many idle processes consuming memory unnecessarily.
`pm.max_spare_servers`: This should be set to accommodate short bursts of traffic without immediately spawning new processes. A value like 5-10 is often sufficient. If `max_spare_servers` is too high, you might end up with many idle workers consuming memory. If it’s too low, new processes will be spawned more aggressively.
`pm.process_idle_timeout`: Controls how long an idle process waits before being killed. A shorter timeout (e.g., 10s) can free up memory faster during lulls, but too short might cause excessive respawning.

Investigating Application-Level Memory Leaks

1. Enable Profiling:

// Example using memory_get_usage() and memory_get_peak_usage()
// Place these strategically in your code or use a profiler

$start_memory = memory_get_usage();
$start_peak_memory = memory_get_peak_usage();

// ... your code ...

$end_memory = memory_get_usage();
$end_peak_memory = memory_get_peak_usage();

error_log(sprintf("Memory usage: Start=%d, End=%d, Peak=%d",
    $start_memory, $end_memory, $end_peak_memory));

2. Analyze Dependencies:

3. Database Query Optimization:

Large datasets fetched from the database and loaded entirely into PHP arrays are a common source of high memory consumption. Use techniques like:

Fetching data in batches (pagination).
Using database cursors or streaming results where possible.
Selecting only necessary columns.
Optimizing SQL queries to reduce the amount of data processed.

System-Level Memory Management and Swappiness

1. Check Swap Usage:

swapon --show
free -h

If your system is actively swapping heavily, it’s a strong indicator of insufficient RAM for the workload, and the OOM killer is a likely consequence.

2. Adjust Swappiness:

# Check current swappiness
cat /proc/sys/vm/swappiness

# Temporarily set swappiness (e.g., to 10)
sudo sysctl vm.swappiness=10

# Make it permanent by editing /etc/sysctl.conf
# Add or modify the line:
# vm.swappiness = 10

Monitoring and Alerting Strategy

Proactive monitoring is key to preventing OOM events. Implement a robust monitoring solution that tracks:

System Memory Usage: Total, used, free, available.
PHP-FPM Process Count: Number of active and idle workers.
Individual PHP-FPM Process Memory: Average and peak `RES` usage.
Swap Usage.
Load Average.
Nginx/Apache Request Rate and Error Rate.

Tools like Prometheus with Node Exporter and PHP-FPM Exporter, Datadog, New Relic, or even basic `sar` and `vmstat` combined with custom scripts can provide this visibility. Set up alerts for:

High memory utilization (e.g., > 85-90% available memory).
High PHP-FPM worker count approaching `pm.max_children`.
Sudden spikes in PHP-FPM process memory.
Kernel OOM killer messages detected in logs.

Resolving Out of Memory (OOM) Killer terminating PHP-FPM pool workers Under Peak Event Traffic on Linode

Investigating Application-Level Memory Leaks

System-Level Memory Management and Swappiness

Monitoring and Alerting Strategy

Investigating Application-Level Memory Leaks

System-Level Memory Management and Swappiness

Monitoring and Alerting Strategy

Investigating Application-Level Memory Leaks

System-Level Memory Management and Swappiness

Monitoring and Alerting Strategy

Investigating Application-Level Memory Leaks

System-Level Memory Management and Swappiness

Monitoring and Alerting Strategy

Investigating Application-Level Memory Leaks

System-Level Memory Management and Swappiness

Monitoring and Alerting Strategy

Investigating Application-Level Memory Leaks

System-Level Memory Management and Swappiness

Monitoring and Alerting Strategy

Investigating Application-Level Memory Leaks

System-Level Memory Management and Swappiness

Monitoring and Alerting Strategy

Identifying the OOM Killer’s Handiwork

Analyzing PHP-FPM Pool Configuration

Tuning `pm.max_children` and Memory Limits

Optimizing PHP-FPM `pm` Settings for Dynamic Loads

Investigating Application-Level Memory Leaks

System-Level Memory Management and Swappiness

Monitoring and Alerting Strategy

Recent Posts

Top Categories

Our Products

Our Services