systemd OOM Killed, Failed, High CPU & Permission Denied: Complete Troubleshooting Guide

Understanding systemd OOM and Service Failures

systemd places every service inside a cgroup (control group), giving the kernel granular visibility into per-service resource consumption. When a service's anonymous memory exceeds its configured MemoryMax= ceiling, or when host-level memory pressure becomes critical, one of two OOM killers fires:

1. Kernel OOM killer — activated only after a memory allocation fails. It scores all processes and terminates the highest-scoring one. Reactions are abrupt and can cause data corruption if the target holds open file descriptors or database locks.
2. systemd-oomd (systemd ≥ v246) — a userspace daemon that monitors PSI (Pressure Stall Information) metrics and proactively kills cgroups before the kernel must act. Its kills are cleaner but can be over-aggressive with default thresholds.

Exact Error Messages to Look For

In journalctl, kernel OOM events produce:

kernel: Out of memory: Killed process 14321 (java) total-vm:8192000kB, anon-rss:6291456kB, file-rss:0kB
systemd[1]: myapp.service: Main process exited, code=killed, status=9/KILL
systemd[1]: myapp.service: Failed with result 'oom-kill'.
systemd[1]: Failed to start My Application Service.

systemd-oomd kills look like:

systemd-oomd[812]: Killed /system.slice/myapp.service due to memory pressure for /system.slice/myapp.service being 70.12% > 60.00% for > 20s with reclaim activity
systemd[1]: myapp.service: Consumed 14.231s CPU time.
systemd[1]: myapp.service: Failed with result 'oom-kill'.

Permission denied service failures:

myapp[14400]: open /var/lib/myapp/data/records.db: permission denied
systemd[1]: myapp.service: Control process exited, code=exited, status=1/FAILURE
systemd[1]: myapp.service: Failed with result 'exit-code'.

Missing executable:

systemd[1]: myapp.service: Executable not found
systemd[1]: myapp.service: Failed at step EXEC spawning /usr/local/bin/myapp: No such file or directory
systemd[1]: Failed to start My Application Service.

Step 1: First-Pass Diagnosis

Always read the journal before touching any configuration:

# Full status with recent log lines
systemctl status myapp.service

# Last 200 lines of service journal
journalctl -u myapp.service -n 200 --no-pager

# Kernel ring buffer for OOM events since last boot
journalctl -k -b | grep -iE 'out of memory|oom|killed process'

# All errors and above since last boot
journalctl -b -p err..emerg --no-pager

# systemd-oomd kills in the last hour
journalctl -u systemd-oomd --since '1 hour ago'

# All failed units on the system
systemctl list-units --state=failed

Step 2: Diagnose and Fix OOM Kills

Identify current memory usage per cgroup:

# Real-time cgroup resource view
systemd-cgtop -d 1 -n 5

# Current limit and usage for a specific service
systemctl show myapp.service --property=MemoryMax,MemoryHigh,MemoryCurrent

# Kernel-level cgroup memory counters
cat /sys/fs/cgroup/system.slice/myapp.service/memory.current
cat /sys/fs/cgroup/system.slice/myapp.service/memory.max

Increase the memory ceiling with a drop-in override (preferred over editing the unit file directly):

systemctl edit myapp.service

In the editor, add:

[Service]
# Hard ceiling — processes OOM-killed if exceeded
MemoryMax=4G
# Soft ceiling — kernel reclaims memory aggressively above this value
MemoryHigh=3G
# Continue running after an OOM kill (default is 'stop')
OOMPolicy=continue

Apply the change:

systemctl daemon-reload
systemctl restart myapp.service

Protect a critical service from the kernel OOM killer:

The kernel assigns each process an oom_score. Lower scores survive longer. -1000 means never kill:

[Service]
# Range: -1000 (immune) to 1000 (kill first). Use -900 for critical services.
OOMScoreAdjust=-900

Tune systemd-oomd to reduce false-positive kills:

Edit /etc/systemd/oomd.conf:

[OOM]
# Intervene only when swap is 90%+ consumed
SwapUsedLimit=90%
# Memory pressure percentage threshold before considering a kill
DefaultMemoryPressureLimit=80%
# Sustained pressure window before a kill is issued
DefaultMemoryPressureDurationSec=30s

Restart oomd: systemctl restart systemd-oomd

Add swap space to extend available memory:

free -h && swapon --show
fallocate -l 4G /swapfile
chmod 600 /swapfile
mkswap /swapfile
swapon /swapfile
echo '/swapfile none swap sw 0 0' >> /etc/fstab

Step 3: Fix Permission Denied Errors

Permission denied in systemd services originates from three distinct enforcement layers. You must check all three.

Layer 1 — POSIX filesystem permissions:

# Determine which user the service runs as
systemctl show myapp.service --property=User,Group

# Trace the full path for permission bits
namei -l /var/lib/myapp/data

# Correct ownership
chown -R myapp:myapp /var/lib/myapp/
chmod 750 /var/lib/myapp/

Layer 2 — systemd sandboxing directives:

Modern hardened unit files mount the filesystem read-only, silently overriding POSIX permissions:

systemctl show myapp.service | grep -E 'ProtectSystem|ProtectHome|ReadOnly|PrivateTmp|InaccessiblePaths|NoNewPrivileges'

If ProtectSystem=strict or ProtectSystem=full is set, all paths are read-only by default. Fix with a drop-in:

[Service]
ProtectSystem=strict
# Explicitly carve out writable paths
ReadWritePaths=/var/lib/myapp /run/myapp

Note: PrivateTmp=yes creates an isolated /tmp namespace, so paths your code hardcodes under /tmp will not be the same directory the host sees.

Layer 3 — SELinux or AppArmor MAC policies:

For SELinux (RHEL, Fedora, CentOS Stream):

# Find AVC denials from the last 5 minutes
ausearch -m avc -ts recent 2>/dev/null | tail -30
# Also visible in the journal
journalctl | grep 'avc: denied' | tail -20
# Restore the correct file context
restorecon -Rv /var/lib/myapp/
# If restorecon does not resolve it, create a targeted policy module
audit2allow -a -M myapp_local
semodule -i myapp_local.pp

For AppArmor (Ubuntu, Debian):

aa-status
journalctl | grep 'apparmor="DENIED"' | tail -20
# Put profile in complain mode to stop blocking without disabling logging
aa-complain /etc/apparmor.d/usr.bin.myapp

Step 4: Investigate and Throttle High CPU

Identify the consuming service and process:

# Live cgroup CPU view, 1-second refresh
systemd-cgtop -d 1

# Kernel cgroup CPU accounting
cat /sys/fs/cgroup/system.slice/myapp.service/cpu.stat

Detect restart-loop CPU amplification — a service crashing and restarting hundreds of times per hour will saturate CPU even if the process itself is lightweight:

# Count start/stop events in the last 10 minutes
journalctl -u myapp.service --since '10 minutes ago' | grep -cE 'Started|Stopped|Failed'

# Check cumulative restart count
systemctl show myapp.service --property=NRestarts,ActiveEnterTimestamp

Apply CPU quota and restart rate limiting in a drop-in:

[Service]
# Hard CPU quota: 50% of one logical core
CPUQuota=50%
# Scheduling weight relative to other services (default 100)
CPUWeight=50

# Stop restarting after 3 failures within any 5-minute window
Restart=on-failure
RestartSec=30s
StartLimitIntervalSec=300
StartLimitBurst=3

Step 5: Capture and Analyze Core Dumps

By default, systemd routes core dumps through systemd-coredump:

# List recorded dumps
coredumpctl list

# Detail for the most recent crash
coredumpctl info

# Extract the core file for analysis
coredumpctl dump -o /tmp/myapp.core -- /usr/bin/myapp

# Open in GDB
gdb /usr/bin/myapp /tmp/myapp.core
(gdb) bt full
(gdb) info registers

If no dumps appear, check /etc/systemd/coredump.conf:

[Coredump]
Storage=external
Compress=yes
ProcessSizeMax=8G
ExternalSizeMax=8G
KeepFree=1G

Step 6: Fix Services That Will Not Start

If systemctl start myapp.service immediately fails or stalls:

# Validate the unit file syntax before applying any changes
systemd-analyze verify /etc/systemd/system/myapp.service

# Test the ExecStart command manually under the service user
sudo -u myapp /usr/bin/myapp --config /etc/myapp/config.yaml

# Check ordering dependencies and identify circular requirements
systemctl list-dependencies myapp.service
systemd-analyze critical-chain myapp.service

# Trace startup timing
systemd-analyze blame | head -20

If the service times out before reporting ready, it likely starts slowly or uses the wrong Type=:

[Service]
# Default is 90s; increase for slow-starting JVM or Python apps
TimeoutStartSec=300
# Use Type=simple unless the process explicitly calls sd_notify()
Type=simple

After every unit file change, reload and verify:

systemctl daemon-reload
systemctl restart myapp.service
systemctl status myapp.service
journalctl -u myapp.service -n 30 --no-pager