Troubleshooting 'systemctl failed' and 'permission denied' Errors in Linux
Resolve 'systemctl failed' and 'permission denied' errors. Learn how to read systemd logs, diagnose failing services, and fix systemctl not working.
- A 'systemctl failed' state indicates that the service process exited with an error code, crashed, or timed out during execution.
- The 'permission denied' error occurs when executing systemctl commands without root privileges (sudo) or proper Polkit configurations.
- Always diagnose failures by inspecting the exact service status and querying the journald logs using 'journalctl -xeu <service>'.
| Command | When to Use | Output Context | Risk |
|---|---|---|---|
| systemctl --failed | Initial diagnosis to identify broken services | High-level list of failed units | Low |
| systemctl status <unit> | Checking the state of a specific service | Active state, PID, exit code, and last 10 log lines | Low |
| journalctl -xeu <unit> | Deep log analysis for a failed service | Full error logs with systemd catalog explanations | Low |
| systemctl daemon-reload | After modifying a .service configuration file | None (reloads systemd manager configuration) | Medium |
Understanding the Error
When managing Linux servers, encountering a systemctl failed state is a rite of passage for System Administrators, DevOps engineers, and SREs. The systemctl command is the primary control interface for systemd, the init system and service manager adopted by almost all major Linux distributions (Ubuntu, Debian, RHEL, CentOS, Arch, and SUSE). When systemctl is not working or returns restrictive errors like systemctl permission denied, it directly impacts application availability, infrastructure automation, and overall system stability.
The warning 'systemctl failed' is fundamentally a generic indicator. It simply means that systemd attempted to start, stop, or manage a service (often referred to as a 'unit'), but the underlying process exited with a non-zero error code, timed out, or crashed unexpectedly. To effectively troubleshoot and resolve these issues, engineers must bridge the gap between this high-level symptom and the granular root cause hiding within the process execution environment.
Common Error Messages and Variations
You might see several variations of failures depending on what exactly broke within the systemd ecosystem:
- Job for [service] failed because the control process exited with error code. (The most common service crash).
- Failed to restart [service]: Access denied / systemctl permission denied. (An authorization and privilege issue).
- Failed to connect to bus: Host is down / No such file or directory. (Often seen in WSL, Docker containers, or chroot environments where the systemd daemon is not running or systemctl is not working natively).
- Warning: The unit file, source configuration file or drop-ins of [service] changed on disk. Run 'systemctl daemon-reload' to reload units. (A configuration drift warning).
Diagnosing 'systemctl failed log' Entries
The most critical step in resolving systemd issues is checking the logs. Unlike older SysVinit systems where logs were scattered unpredictably across /var/log, systemd uses the tightly integrated journald daemon to collect standard output (stdout) and standard error (stderr) from all managed services in a centralized, binary format.
When a service fails, systemctl typically prompts you to check the logs using journalctl -xe or systemctl status <service>.service. Let's break down the professional diagnostic workflow.
Step 1: Identify the Failing Services
Often, you might not know exactly which service failed, especially during a complex system boot sequence or after a massive infrastructure update. Use the following command to list all units currently registered in a failed state:
systemctl --failed
This will output a formatted table showing the UNIT name, LOAD state, ACTIVE state, SUB state, and a brief DESCRIPTION. Identify the exact name of the service (e.g., nginx.service, mysql.service, kubelet.service, or a custom application unit) that requires intervention.
Step 2: Deep Dive into the Service Status
Once you have pinpointed the unit name, query its current operational status:
systemctl status [your-service-name].service
The output will provide the 'Active:' state (which will likely highlight 'failed (Result: exit-code)' in red text). Pay close attention to the 'Main PID' (Process ID) and the specific exit code returned by the kernel. The status output also conveniently prints the last 10 lines of the service's log directly from the journal, which often contains the exact runtime error, missing file path, or syntax mistake that caused the crash.
Step 3: Query the Journal
If the status output isn't informative enough (e.g., the service crashed silently or the error happened further back in the execution timeline), you need to query the full systemctl failed log. Use journalctl restricted to the specific unit:
journalctl -u [your-service-name].service --no-pager | tail -n 50
For an expanded view with system-level context—which is incredibly helpful if the Linux Out-Of-Memory (OOM) killer terminated your service, or if there are cascading dependency failures—use:
journalctl -xeu [your-service-name].service
The -x flag adds supplementary explanatory text from the systemd catalog, the -e flag jumps immediately to the end of the pager (the most recent logs), and the -u flag strictly filters the output by the unit name.
Resolving 'systemctl permission denied'
The systemctl permission denied error almost always stems from a lack of appropriate OS-level privileges or misconfigured PolicyKit (polkit) rules.
By default, mutating state in systemd (starting, stopping, enabling, disabling services, or reloading the daemon) requires root privileges because these actions affect the global system state. If you execute systemctl restart nginx as a standard, unprivileged user, you will be met with:
Failed to restart nginx.service: Interactive authentication required.
or
Failed to restart nginx.service: Access denied
Fixing Permission Issues:
- Use sudo: The simplest, most standard fix for administrators is to elevate privileges by prepending the command with
sudo:sudo systemctl restart [service]. - Leverage User Units: If you are running an application strictly for your own user account (like a personal background worker, a local development database, or a user-specific node script), you should run it as a systemd user service. Place the
.servicefile in~/.config/systemd/user/and run commands with the--userflag:systemctl --user start [service]. This executes entirely within your user namespace, requires no root privileges, and completely avoids the permission denied error. - Configure Polkit Rules: If you need a specific non-root user (e.g., a CI/CD pipeline runner or a junior developer) to manage a specific system service without granting them blanket
sudoaccess, you can write a targeted Polkit rule. Create a JavaScript-based rule file in/etc/polkit-1/rules.d/10-manage-service.rulesallowing the specific user or group to bypass the authentication prompt exclusively fororg.freedesktop.systemd1.manage-unitsactions on that exact service.
When 'systemctl not working' Entirely
Sometimes, systemctl itself is completely unresponsive, hangs indefinitely, or returns catastrophic infrastructure-level errors like 'Failed to connect to bus'. This means the systemctl client tool cannot communicate with the core systemd daemon over the D-Bus IPC mechanism.
Common Causes for this Total Failure:
- Running in an unprivileged container (Docker/Podman): By default, Docker containers do not run systemd as PID 1 to adhere to the single-process-per-container philosophy. If you attempt to run
systemctlinside a standard Ubuntu or Debian container image, it will fail to connect to the bus. Fix: Use alternative, lightweight process managers likesupervisord,s6-overlay, or if absolutely necessary, run the container in privileged mode mapping the cgroups and setting/sbin/initas the entrypoint (Note: this is strongly discouraged for secure production environments). - WSL1 or legacy WSL2 instances: The Windows Subsystem for Linux (WSL) did not historically support systemd out of the box. Fix: Ensure you are running a modern WSL2 kernel and have explicitly enabled systemd support by creating or editing the
/etc/wsl.conffile to includesystemd=trueunder the[boot]section, followed by restarting the WSL instance from Windows. - Chroot and Recovery environments: If you are operating inside a
chrootjail (e.g., during bare-metal system recovery via a Live USB), the systemd daemon isn't actively running within that restricted scope. Fix: You cannot natively usesystemctlinside a basic chroot. Instead, you must manage service symlinks manually within/etc/systemd/system/or use the--root=/path/to/chrootparameter if the host system'ssystemctlbinary is compatible with the chroot target.
Fixing the Underlying Service Failure
Once you have successfully bypassed the permission roadblocks and extracted the relevant error logs, you must fix the actual application or configuration that is causing the service to fail. Common reasons a service enters the failed state include:
- Syntax errors in the configuration file: If a web server like Nginx or Apache has a misplaced semicolon or typo in its config, the pre-flight check executed by systemd will fail, and systemctl will report a failure. Always run the application's native configuration test (e.g.,
sudo nginx -torsudo apachectl configtest) before restarting the service. - Missing dependencies or unmounted filesystems: The service might require a network interface to be up, a database port to be open, or an external NFS volume to be mounted that simply isn't ready yet. Use
Requires=,Wants=, andAfter=directives in the[Unit]section of the.servicefile to rigorously enforce boot ordering and dependency trees. - Incorrect Permissions on ExecStart: The binary specified in the
ExecStart=directive must be physically executable by the user specified in theUser=directive. Verify file ownership and permissions usingls -l /path/to/binaryand ensure the user has read/execute rights recursively down the directory tree. - OOM (Out of Memory) Kills: Check the kernel ring buffer using
dmesg -T | grep -i oomto see if the Linux kernel terminated your process because the system exhausted all available physical RAM and swap space.
The Importance of Daemon-Reload
Whenever you manually modify a .service unit file located in /etc/systemd/system/ or /usr/lib/systemd/system/, systemd will not automatically or immediately recognize those changes. The daemon caches unit files in memory for performance. If you try to restart the service after making an edit, you will receive a persistent warning. You must instruct systemd to flush its cache and re-parse all unit files by running:
sudo systemctl daemon-reload
After successfully reloading the daemon, you can safely execute sudo systemctl restart [service] and rigorously verify its recovery and stable operation with systemctl status [service].
Frequently Asked Questions
#!/bin/bash
# Diagnostic script for checking and analyzing failed systemd services
echo "[+] Checking for failed systemd services..."
# List failed services, extracting just the unit names
FAILED_SERVICES=$(systemctl --failed --plain --no-legend | awk '{print $1}')
if [ -z "$FAILED_SERVICES" ]; then
echo "[+] No failed services found. System is healthy."
exit 0
fi
echo "[!] Failed services detected:"
echo "$FAILED_SERVICES"
echo "----------------------------------------"
# Loop through each failed service and extract the last 15 log lines
for service in $FAILED_SERVICES; do
echo "[>] Fetching journalctl logs for $service:"
# Require sudo or root for system-level journal access
sudo journalctl -u "$service" --no-pager | tail -n 15
echo "----------------------------------------"
echo "[>] Service Status for $service:"
sudo systemctl status "$service" --no-pager
echo "========================================"
done
echo "[i] Troubleshooting complete. Review the logs above for exit codes or syntax errors."Error Medic Editorial & SRE Team
Written by senior Site Reliability Engineers and Linux administrators with over a decade of experience managing large-scale, highly available infrastructure and debugging complex systemd ecosystems.