HAProxy 'Connection Refused': Complete Troubleshooting Guide (2024)

Understanding 'Connection Refused' in HAProxy

When HAProxy attempts to forward a client request to a backend server and receives a TCP RST (reset) packet — or no response at all on a non-routable path — it logs the error and returns a 503 Service Unavailable to the client. The raw kernel error is ECONNREFUSED (errno 111 on Linux). You will see it in HAProxy logs as:

Server myapp/web1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 1ms

or during an active request:

backend myapp has no server available!

Before touching any configuration, you must distinguish between three distinct failure modes:

1. HAProxy itself is not listening — the client cannot even reach the proxy frontend
2. HAProxy is up but cannot reach the backend — the proxy is healthy but all upstream servers report DOWN
3. Intermittent connection refused — some requests succeed; others fail due to flapping, overload, or connection pool exhaustion

Each mode has a different diagnosis path.

Step 1: Confirm HAProxy Is Running and Listening

The very first command to run is:

systemctl status haproxy

If the unit shows failed or inactive, read the journal:

journalctl -u haproxy -n 50 --no-pager

Common reasons HAProxy fails to start and logs Address already in use or cannot bind socket:

• Another process holds the bind port (e.g., Nginx also trying to bind :443)
• The configured IP address does not exist on any local interface yet (race condition at boot)
• SELinux or AppArmor blocking the bind syscall

Check what is already on the port:

ss -tlnp | grep ':80\|:443\|:8080'

If HAProxy is running, verify it is actually listening on the expected frontend port:

ss -tlnp | grep haproxy
# Expected output example:
# LISTEN  0  128  0.0.0.0:80  0.0.0.0:*  users:(("haproxy",pid=12345,fd=5))

If the port is absent from the output, the bind directive in haproxy.cfg is wrong. Open the config and inspect the frontend section:

frontend http_in
    bind *:80          # OK — wildcard
    bind 10.0.1.5:443  # WRONG if 10.0.1.5 doesn't exist locally

Always validate the config before reloading:

haproxy -c -f /etc/haproxy/haproxy.cfg
# Should print: Configuration file is valid

Step 2: Inspect the HAProxy Stats and Logs

If HAProxy is running, enable or read the stats socket to see real-time backend state:

# Via netcat / socat (add to haproxy.cfg if not present: stats socket /run/haproxy/admin.sock)
echo 'show servers state' | socat stdio /run/haproxy/admin.sock

The output columns include server name, current state (UP/DOWN/MAINT), and the last health-check result. A DOWN state with L4CON (Layer 4 connection) is exactly ECONNREFUSED.

For log-based diagnosis, ensure HAProxy logging is enabled in haproxy.cfg:

global
    log /dev/log local0
    log /dev/log local1 notice

defaults
    log     global
    option  httplog
    option  dontlognull

Then tail the log in real time while reproducing the error:

tail -f /var/log/haproxy.log | grep -E 'DOWN|refused|503|backend'

Step 3: Test Backend Connectivity Directly from the HAProxy Host

SSH onto the HAProxy node and attempt to connect to each backend server manually:

# TCP connectivity test (no HTTP)
curl -sv --connect-timeout 5 http://10.0.1.10:8080/health

# Raw TCP using bash /dev/tcp
timeout 3 bash -c '</dev/tcp/10.0.1.10/8080 && echo open || echo refused'

# netcat
nc -zv 10.0.1.10 8080

If these commands also show Connection refused, the problem is definitively on the backend side — not in HAProxy's configuration. Proceed to Step 4.

If curl succeeds but HAProxy still reports DOWN, the issue is in the health-check configuration (wrong path, wrong port, SSL mismatch). Compare the option httpchk path against what the backend actually exposes:

backend myapp
    option httpchk GET /healthz
    server web1 10.0.1.10:8080 check

If the app only exposes /health (not /healthz), HAProxy receives a 404 which it may interpret as a failure.

Step 4: Fix a Down Backend

If the backend service is down:

# Check the service on the backend node
ssh 10.0.1.10 'systemctl status myapp'

# Restart it
ssh 10.0.1.10 'systemctl restart myapp'

# Confirm it is listening
ssh 10.0.1.10 'ss -tlnp | grep 8080'

Once the backend is listening, HAProxy health checks will promote it back to UP within one check interval (default inter 2s). To force an immediate check:

echo 'set server myapp/web1 state ready' | socat stdio /run/haproxy/admin.sock

Step 5: Fix Firewall Rules

If the backend is listening but unreachable from the HAProxy host, inspect iptables:

# On the HAProxy host — check OUTPUT chain
iptables -L OUTPUT -n -v | grep -E 'DROP|REJECT'

# On the backend host — check INPUT chain
iptables -L INPUT -n -v | grep -E 'DROP|REJECT'

# Trace a specific packet (requires iptables-legacy or nft)
iptables -I INPUT 1 -p tcp --dport 8080 -j LOG --log-prefix "DEBUG-8080: "
# Then check: dmesg | grep DEBUG-8080

To allow traffic from HAProxy to backend on Ubuntu/Debian with ufw:

# On the backend node — allow HAProxy's IP
ufw allow from 10.0.1.5 to any port 8080

For cloud environments (AWS Security Groups, GCP Firewall Rules, Azure NSG), ensure the HAProxy node's private IP or security group is whitelisted as an ingress source on the backend's port.

Step 6: Addressing HAProxy Slow or Timeout Symptoms

If connections are not refused outright but HAProxy is slow, check:

Too many open files / file descriptor exhaustion:

cat /proc/$(pidof haproxy)/limits | grep 'open files'
# If max is 1024, increase it:
ulimit -n 65536  # in systemd unit: LimitNOFILE=65536

Connection queue buildup:

echo 'show info' | socat stdio /run/haproxy/admin.sock | grep -E 'MaxConn|CurrConns|Uptime'

If CurrConns approaches MaxConn, increase maxconn in global and in the frontend/backend blocks.

DNS resolution delays for backend hostnames:

HAProxy resolves backend hostnames at startup by default. Use resolvers with resolve-prefer ipv4 and a short hold valid TTL, or switch to IP addresses in the server directives.

Preventing Recurrence

1. Enable active health checks with check inter 2s rise 2 fall 3 on every server directive
2. Set option redispatch so HAProxy retries a refused connection on a different backend server rather than failing immediately
3. Configure retries 3 in the defaults section
4. Use errorfile 503 /etc/haproxy/errors/503.http to present a user-friendly error page
5. Export metrics via the Prometheus exporter (haproxy_exporter) and alert on haproxy_server_status{state="DOWN"} before users notice