Fixing HAProxy 503 Service Unavailable, 502 Bad Gateway & 504 Timeout Errors

Understanding the Error

When running HAProxy as a load balancer or reverse proxy, encountering HTTP 5xx errors like 503 Service Unavailable, 502 Bad Gateway, or 504 Gateway Timeout indicates a breakdown in communication between HAProxy and your backend servers. You might also encounter lower-level TCP issues such as Connection refused or Connection reset by peer.

HAProxy sits between your clients and your backend applications. When a client makes a request, HAProxy attempts to forward it to an available backend server. If no servers are available, or if the connection fails, HAProxy must return an error to the client.

Common Error Definitions:

• 503 Service Unavailable: HAProxy has no healthy backend servers available to handle the request. This happens if all servers fail their health checks or if the maxconn limit is reached without a queue.
• 502 Bad Gateway: HAProxy successfully connected to the backend, but the backend returned an invalid or incomplete response, or closed the connection prematurely.
• 504 Gateway Timeout: HAProxy established a connection, but the backend failed to respond within the timeout server limit configured in HAProxy.
• Connection Refused: The backend server's operating system actively rejected the connection attempt (usually because the application process is down or listening on the wrong IP/port).
• Connection Reset: The connection was established, but abruptly dropped by the backend or an intermediary firewall (TCP RST packet received).

Step 1: Diagnose the Root Cause

Before making changes, you need to pinpoint exactly why HAProxy is failing to route traffic successfully.

1. Review the HAProxy Stats Page

If you have the HAProxy stats page enabled, this is your first stop. Look at the Status column for your backend servers. Are they marked UP, DOWN, or MAINT? If they are DOWN, HAProxy will return 503s because it believes the servers are unavailable.

2. Analyze HAProxy Logs

HAProxy logs provide detailed termination state codes that tell you exactly where the connection failed. Look for lines like this:

Jul 25 10:00:00 haproxy[1234]: 192.168.1.10:54321 [25/Jul/2026:10:00:00.000] frontend~ backend/server1 0/0/0/-1/0 503 212 - - sC-- 1/1/0/0/0 0/0 "GET / HTTP/1.1"

Pay close attention to the termination state flags (e.g., sC--):

• sC--: The connection to the server failed.
• sH--: The server aborted the connection or timeout occurred waiting for the server to send HTTP headers (often leads to 502).
• cD--: The client aborted the connection.

3. Verify Backend Application Health

Log into the backend server itself. Is the application running? Use ss -tulnp or netstat -tulnp to verify the application is listening on the expected port and IP address (not just 127.0.0.1 if HAProxy is on a different machine).

Step 2: Implement Fixes

Depending on your diagnosis, apply the appropriate fixes below.

Scenario A: Fixing 503 Service Unavailable (Failing Health Checks)

If HAProxy marks servers as DOWN, verify your health check configuration. Often, a misconfigured option httpchk expects a 200 OK response, but the backend is returning a 301 Redirect or 401 Unauthorized for the root path (/).

Solution: Update the health check to expect the correct status code or point it to a dedicated health endpoint.

backend web_servers
    mode http
    # Expect 200 or 301/302 for the health check
    http-check expect status 200,301,302
    # Or point to a dedicated endpoint
    option httpchk GET /healthz
    server web1 10.0.0.11:80 check fall 3 rise 2

Scenario B: Fixing 504 Gateway Timeout

If your backend is processing a heavy request (like generating a large report), HAProxy might give up before the backend finishes.

Solution: Increase the timeout server value in your defaults or backend section. Ensure timeout client is also adjusted if necessary.

defaults
    timeout connect 5000ms
    timeout client  50000ms
    # Increase from 50s to 120s for slow backend processing
    timeout server  120000ms

Scenario C: Fixing Connection Refused / Connection Reset

If you see Connection refused, it's a network-level issue.

1. Check Firewalls: Ensure iptables, firewalld, or cloud security groups allow traffic from the HAProxy IP to the backend IP on the specific port.
2. Check Bind Addresses: The backend application must bind to 0.0.0.0 or its specific LAN IP, not 127.0.0.1.

If you see Connection reset (502 Bad Gateway), the backend application might be crashing during the request, or there's an MTU mismatch on the network. Check the backend application error logs (e.g., Nginx, Tomcat, Node.js) for crashes or out-of-memory errors.

Step 3: Advanced Network Troubleshooting

If logs and basic checks don't reveal the issue, use tcpdump on the HAProxy machine to observe the raw traffic.

Run a packet capture filtering by the backend server's IP. Look for the three-way handshake (SYN, SYN-ACK, ACK).

• If you see HAProxy send a SYN and get a RST back immediately, the backend application is down or refusing connections.
• If you see HAProxy send a SYN and get no response, a firewall is dropping the packets.

By systematically verifying the backend state, analyzing HAProxy logs for termination codes, and tuning health checks and timeouts, you can resolve the majority of HAProxy 5xx and connection errors.