Why does HAProxy say 'Layer4 connection problem, info: Connection refused'?

This indicates HAProxy attempted a TCP connection to the backend server's IP and port, but the destination operating system explicitly rejected it. This usually means the backend application is not running, is bound to the wrong IP (like localhost), or a local firewall is rejecting the packet.

How do I fix HTTP 503 Service Unavailable in HAProxy?

A 503 means HAProxy has marked all backend servers as DOWN. Check the HAProxy stats page or logs to see why the health checks are failing. Verify the backend services are running, the network path is open, and that the health check URL (if configured) returns a 200 OK status.

HAProxy works locally but external clients get Connection Refused. Why?

This happens when HAProxy is bound to localhost (`bind 127.0.0.1:80` or `bind 127.0.0.1:443`) in its frontend configuration. To accept external traffic, change the bind directive to `bind *:80` or `bind 0.0.0.0:80` to listen on all network interfaces.

Can SELinux cause connection refused in HAProxy?

Yes. On SELinux-enforcing systems (like CentOS/RHEL), HAProxy is restricted by default from connecting to non-standard HTTP ports (e.g., connecting to a backend on port 8080). You can resolve this by running `setsebool -P haproxy_connect_any 1`.

Why is HAProxy randomly dropping connections during high load?

Intermittent connection failures under load usually point to resource exhaustion. The most common culprits are hitting the `maxconn` limit in HAProxy, or ephemeral port exhaustion on the HAProxy server. Tuning `sysctl` for local port ranges and enabling `tcp_tw_reuse` can mitigate port exhaustion.

Troubleshooting HAProxy 'Connection Refused' Errors: A Complete Guide

Diagnostic Approaches Compared
Method	When to Use	Time	Risk
Netcat (nc) / Telnet	Initial validation of Layer 4 TCP connectivity from HAProxy to backend.	1 min	Low
HAProxy Stats Page	Checking real-time health check statuses, connection limits, and server states.	2 mins	Low
TCPDump / Wireshark	Deep packet inspection when connections are inexplicably dropped or reset (RST).	15 mins	Medium (can impact I/O under heavy load)
SELinux Audit Logs	When running on RHEL/CentOS and HAProxy cannot connect to non-standard ports.	5 mins	Low

Understanding 'Connection Refused' in HAProxy

When you encounter a 'Connection Refused' error—often materializing as an HTTP 502 Bad Gateway, an HTTP 503 Service Unavailable, or a raw TCP RST (Reset) packet—it signifies an immediate failure to establish a TCP connection. In an architecture utilizing HAProxy, this error can occur at two distinct boundaries: the 'Frontend' (between the client and HAProxy) and the 'Backend' (between HAProxy and your upstream servers). Diagnosing this requires isolating which side of the proxy is failing.

Often, this issue is accompanied by user reports that 'HAProxy is not working' or 'HAProxy is slow'. While slowness is usually related to latency, timeouts, or resource exhaustion (like maxconn limits), a hard 'connection refused' is almost always a binary state: the target port is closed, the process is dead, or a firewall is actively rejecting the traffic.

Diagnosing the Scope of the Problem

Before executing commands that alter system state, we must determine the exact location of the failure.

1. The Frontend Failure (Client to HAProxy)

If a client attempts to connect to your service (e.g., via curl https://api.example.com) and receives an immediate curl: (7) Failed to connect to api.example.com port 443: Connection refused, the issue lies at the HAProxy entry point. This means HAProxy itself is either not running, has crashed, or is not binding to the expected port.

To verify this on the HAProxy server, check the listening sockets: sudo ss -tulpn | grep haproxy

If HAProxy is not listed, check its service status and system logs to determine why it failed to start: sudo systemctl status haproxy sudo journalctl -xeu haproxy.service

Common frontend binding issues include another service (like Nginx or Apache) already occupying port 80 or 443, or insufficient privileges (HAProxy must start as root to bind to ports below 1024).

2. The Backend Failure (HAProxy to Upstream)

If the client receives an HTTP 503 error containing a message like No server is available to handle this request, the HAProxy frontend is successfully accepting connections, but it cannot forward them to the backend pool.

Looking into the HAProxy logs (typically located at /var/log/haproxy.log), you will likely find entries resembling: haproxy[12345]: backend backend_api has no server available! haproxy[12345]: Server backend_api/node1 is DOWN, reason: Layer4 connection problem, info: "Connection refused" at step 1 of tcp-check

This confirms HAProxy is attempting a TCP handshake (sending a SYN packet), but the upstream server is responding with a RST, ACK packet, actively refusing the connection.

Step-by-Step Backend Troubleshooting

Once you have isolated the issue to the backend, follow these structured steps to identify the root cause.

Step 1: Verify the Backend Service is Running and Bound Correctly

Log into the backend server (node1 in our example). The most frequent cause of a refused connection is that the application service (Node.js, Python/Gunicorn, Java/Tomcat) has crashed or was never started.

Check the service status: sudo systemctl status my-application

If the service is running, you must verify the IP address it is bound to. Developers often accidentally bind services to localhost (127.0.0.1). If a service listens only on localhost, it will refuse external connections from HAProxy.

Check the listening interfaces: sudo ss -tulpn | grep <application_port>

Look for 0.0.0.0:<port> or the specific internal IP address (e.g., 10.0.0.5:<port>). If it shows 127.0.0.1:<port>, you must reconfigure your application to listen on all interfaces or the specific network interface HAProxy targets.

Step 2: Validate Network Connectivity from HAProxy

Return to the HAProxy server. We must bypass HAProxy entirely and attempt a raw TCP connection to the backend to eliminate HAProxy configuration as the variable.

Use netcat (nc) or telnet to test the connection: nc -vz 10.0.0.5 8080

If the output is nc: connect to 10.0.0.5 port 8080 (tcp) failed: Connection refused, you have confirmed the issue is at the network layer or the operating system layer of the backend server.

Step 3: Investigate Firewalls and Network Security Groups

If the service is listening on 0.0.0.0 but nc fails from HAProxy, a firewall is likely blocking or rejecting the traffic. Unlike 'dropped' packets (which result in timeouts), 'connection refused' means the firewall rule is explicitly set to REJECT the packet, which sends a TCP RST back to the sender.

On the backend server, check local firewall rules:

For Ubuntu/Debian using UFW: sudo ufw status
For RHEL/CentOS using Firewalld: sudo firewall-cmd --list-all
For raw iptables: sudo iptables -L -n -v | grep 8080

Ensure there is an explicit ALLOW rule for the HAProxy IP address to access the backend port.

Additionally, if your infrastructure is hosted on AWS, GCP, or Azure, verify the cloud provider's Network Security Groups (NSGs) or Security Group rules. Ensure the HAProxy instance's security group is allowed ingress to the backend instance's security group on the required port.

Step 4: SELinux and AppArmor Policies

On Red Hat Enterprise Linux (RHEL), CentOS, or Fedora, SELinux is enabled by default. SELinux strictly controls which ports applications can bind to and which external connections they can make.

If HAProxy is attempting to connect to a non-standard port (e.g., 8080, 8443) and returning 503s, SELinux on the HAProxy server itself might be blocking the outbound connection.

Check the SELinux audit logs on the HAProxy server: sudo tail /var/log/audit/audit.log | grep haproxy | grep denied

If you see denials, you must instruct SELinux to allow HAProxy to make arbitrary outbound network connections. This is done by toggling an SELinux boolean: sudo setsebool -P haproxy_connect_any 1

Step 5: HAProxy Health Check Configuration

Sometimes the backend service is perfectly healthy, but HAProxy misinterprets its status due to a misconfigured health check (option httpchk or check). If a health check fails, HAProxy marks the server as DOWN. Any subsequent client requests routed to that backend will immediately fail, often manifesting as a 503 Service Unavailable.

Review your haproxy.cfg backend definition:

backend app_cluster
    balance roundrobin
    option httpchk GET /health HTTP/1.1\r\nHost:\ api.example.com
    server node1 10.0.0.5:8080 check inter 2000 rise 2 fall 3

If the application's /health endpoint requires authentication, has moved, or is returning a 4xx/5xx HTTP status code, HAProxy will sever the connection. Manually test the exact health check URL from the HAProxy server using curl -v http://10.0.0.5:8080/health -H "Host: api.example.com" to ensure it returns an HTTP 200 OK.

Step 6: Resource Exhaustion (The 'HAProxy Slow' Phenomenon)

If 'connection refused' errors occur intermittently, especially during high traffic periods, you are likely experiencing resource exhaustion.

Max Connections (maxconn): Both HAProxy and the backend servers have limits on concurrent connections. If HAProxy reaches its global maxconn limit, it will stop accepting new client connections. If a backend server reaches its maxconn parameter, HAProxy will queue requests. If the queue fills up, connections are dropped.
Ephemeral Port Exhaustion: When HAProxy connects to a backend, it uses a local ephemeral port. If you have massive throughput without HTTP keep-alives, HAProxy will rapidly burn through the available local ports (usually 32,768 to 60,999). When no ports are left, connect() system calls fail.

To diagnose port exhaustion, run netstat -an | grep TIME_WAIT | wc -l. If this number is extremely high (e.g., > 25,000), you need to tune your sysctl parameters on the HAProxy server to expand the port range and recycle connections faster:

sudo sysctl -w net.ipv4.ip_local_port_range="1024 65535" sudo sysctl -w net.ipv4.tcp_tw_reuse=1

Advanced Debugging with TCPDump

When logs and standard commands fail to reveal the issue, packet capture is the ultimate source of truth. Use tcpdump on the HAProxy server to monitor traffic between the load balancer and the backend.

sudo tcpdump -i any -nn -S -vvv host 10.0.0.5 and port 8080

Analyze the TCP flags. A healthy handshake looks like:

HAProxy -> Backend: Flags [S] (SYN)
Backend -> HAProxy: Flags [S.] (SYN-ACK)
HAProxy -> Backend: Flags [.] (ACK)

If you see: HAProxy -> Backend: Flags [S] Backend -> HAProxy: Flags [R.] (RST-ACK) This irrefutably proves the backend OS is actively rejecting the connection (e.g., closed port or iptables REJECT rule).

If you see repeated Flags [S] with no response at all, the packets are being silently dropped by a firewall (iptables DROP rule or Cloud Security Group).

Conclusion

Resolving HAProxy connection refused errors demands a systematic, layer-by-layer approach. By isolating the failure to either the frontend or backend, utilizing raw socket tools like netcat, inspecting local firewalls and SELinux, and scrutinizing health check logic, you can quickly restore highly available traffic flow. Remember that in distributed systems, the load balancer is often the messenger; the root cause frequently lies in backend configuration or network topology.