Error Medic

Memcached Connection Refused: Complete Troubleshooting Guide (with Fix Commands)

Fix memcached connection refused, crashes, out-of-memory, and slow response errors. Step-by-step diagnosis commands and proven fixes for Linux.

Last updated:
Last verified:
1,929 words
Key Takeaways
  • Connection refused usually means memcached is not running, bound to the wrong interface, or blocked by a firewall — check with 'systemctl status memcached' and 'ss -tlnp | grep 11211'
  • Out-of-memory evictions cause cache misses that look like slowness; set -m to at least 256MB and monitor with 'memcached-tool localhost:11211 stats'
  • A crashed memcached process often leaves no socket listener; restart with 'systemctl restart memcached' then inspect /var/log/syslog for OOM killer entries
  • Slow memcached is almost always a network or eviction issue, not a CPU issue — rule out max_connections limits and slab imbalance before tuning
Fix Approaches Compared
MethodWhen to UseTimeRisk
systemctl restart memcachedProcess is dead or unresponsive< 1 minLow — cache is cold after restart
Bind address fix (-l 0.0.0.0)Client on different host gets ECONNREFUSED< 5 minMedium — exposes port; add firewall rule
Increase memory limit (-m)Stats show high eviction rate or OOM kills5–10 minLow — requires restart
Raise max connections (-c)Logs show 'Too many open files' or dropped clients5 minLow
Firewall rule (ufw/iptables)Port 11211 reachable locally but not remotely5 minMedium — ensure IP allowlisting
Slab tuning (-f factor)Persistent memory waste shown in slab stats30+ minMedium — requires restart and testing
Upgrade memcachedKnown crash bug in installed version30+ minLow with staging test first

Understanding Memcached Connection Refused

When a client application throws Connection refused (ECONNREFUSED) against memcached, the TCP handshake to port 11211 never completes. This happens at the OS level before any memcached protocol exchange, which means the diagnosis is almost always infrastructure — not application code.

The five root causes, in order of frequency:

  1. Memcached process is not running — killed by OOM killer, init system, or crash
  2. Wrong bind address — memcached started with -l 127.0.0.1 but client connects from another host
  3. Firewall blocking port 11211 — iptables/ufw DROP rule or missing ACCEPT rule
  4. Wrong port in client config — client points to 11212 or a custom port
  5. Socket backlog exhaustion — under extreme load the listen queue fills and new connections are refused

Step 1: Confirm the Process Is Running

systemctl status memcached

If the unit is inactive (dead) or failed, check the journal:

journalctl -u memcached -n 50 --no-pager

Look for lines like:

  • Out of memory: Kill process <pid> (memcached) — Linux OOM killer
  • signal 11 or segfault — memcached crash (usually a bug or corrupted binary)
  • Too many open files — file descriptor limit hit

If it is not running, start it and verify the port is open:

systemctl start memcached
ss -tlnp | grep 11211

Step 2: Check the Bind Address

By default many distribution packages ship memcached bound to 127.0.0.1 only. If your application server is on a different host, you will receive ECONNREFUSED even though memcached is running fine.

Inspect the running process arguments:

ps aux | grep memcached

You may see: /usr/bin/memcached -m 64 -p 11211 -u memcache -l 127.0.0.1

To allow remote connections, edit /etc/memcached.conf (Debian/Ubuntu) or /etc/sysconfig/memcached (RHEL/CentOS) and change:

-l 127.0.0.1

to

-l 0.0.0.0

Or bind to a specific private IP: -l 10.0.1.5

Then restart: systemctl restart memcached

Security note: Never expose memcached to the public internet. Restrict access with a firewall rule immediately after changing the bind address.


Step 3: Check Firewall Rules

UFW (Ubuntu):

ufw status
ufw allow from 10.0.1.0/24 to any port 11211

iptables:

iptables -L INPUT -n -v | grep 11211
iptables -A INPUT -s 10.0.1.0/24 -p tcp --dport 11211 -j ACCEPT

Test connectivity from client host:

nc -zv memcached-host 11211
# or
telnet memcached-host 11211

A successful connection shows Connected to memcached-host. You can then type stats and press Enter to see server statistics.


Step 4: Diagnose Memcached Out of Memory

Memcached uses a slab allocator and evicts least-recently-used items when full. High eviction rates cause cache misses that surface as application slowness or unexpected DB load spikes — not necessarily a crash.

Check eviction stats:

memcached-tool localhost:11211 stats | grep -E 'evictions|bytes|limit_maxbytes|curr_items'

Key fields:

  • evictions — non-zero and growing means you are out of memory
  • bytes — current memory in use
  • limit_maxbytes — configured max (e.g., 67108864 = 64 MB)
  • curr_items — number of keys currently stored

To increase the memory limit, edit config and set -m 512 (512 MB) or higher, then restart.

For slab imbalance (memory allocated to wrong-sized slabs):

memcached-tool localhost:11211 display

Look for slabs with evicted > 0 next to slabs with large free_chunks. This indicates your -f growth factor needs tuning for your key size distribution.


Step 5: Diagnose Memcached Slow Response

Slow memcached (>1ms for get operations on a LAN) is almost always one of:

  • Network latency — verify with ping and check for packet loss
  • High eviction churn — cache thrashing from insufficient memory (see Step 4)
  • Connection pool exhaustion — application opening new connections per request
  • Too many connections — check curr_connections vs -c limit
# Check current connections
memcached-tool localhost:11211 stats | grep curr_connections

# Check total connections ever accepted (should not be growing linearly per request)
memcached-tool localhost:11211 stats | grep total_connections

If curr_connections is close to the -c value (default 1024), increase it:

-c 4096

Also ensure your application uses a persistent connection pool — opening a new TCP connection for each cache operation adds 0.5–2ms RTT overhead and will eventually exhaust the connection limit.


Step 6: Verify After Every Change

After any config change and restart, run a full smoke test:

# Set a test key
echo -e 'set testkey 0 60 5\r\nhello\r\n' | nc localhost 11211

# Retrieve it
echo -e 'get testkey\r\n' | nc localhost 11211

# Should return:
# VALUE testkey 0 5
# hello
# END

Then tail your application logs for 2–5 minutes to confirm the connection refused errors have stopped.


Handling Repeated Crashes (memcached not working after restart)

If memcached restarts successfully but crashes again within minutes:

  1. Check OOM killer: dmesg | grep -i 'killed process' — if memcached appears here, the system has less RAM than the -m value plus OS overhead. Reduce -m or add RAM.
  2. Check ulimit: cat /proc/$(pgrep memcached)/limits | grep 'open files' — if Max open files is 1024 and you have high connection counts, add LimitNOFILE=65536 to /etc/systemd/system/memcached.service.d/override.conf.
  3. Check for known bugs: Run memcached -V and compare against the memcached changelog for crash-related fixes in newer versions.

Frequently Asked Questions

bash
#!/usr/bin/env bash
# memcached-diagnose.sh — full diagnostic for connection/performance issues
# Usage: bash memcached-diagnose.sh [host] [port]

HOST="${1:-localhost}"
PORT="${2:-11211}"

echo "=== 1. Process status ==="
systemctl status memcached --no-pager 2>/dev/null || service memcached status 2>/dev/null || ps aux | grep memcached | grep -v grep

echo ""
echo "=== 2. Listening address and port ==="
ss -tlnp | grep "${PORT}" || netstat -tlnp 2>/dev/null | grep "${PORT}"

echo ""
echo "=== 3. TCP connectivity test ==="
nc -zv -w3 "${HOST}" "${PORT}" 2>&1 && echo "CONNECTED OK" || echo "CONNECTION FAILED"

echo ""
echo "=== 4. Firewall rules for port ${PORT} ==="
iptables -L INPUT -n -v 2>/dev/null | grep "${PORT}" || echo "(no iptables rules matching port or iptables not available)"
ufw status 2>/dev/null | grep "${PORT}" || true

echo ""
echo "=== 5. Key stats (requires connection) ==="
if nc -zw1 "${HOST}" "${PORT}" 2>/dev/null; then
  echo -e 'stats\r\n' | nc -w2 "${HOST}" "${PORT}" | grep -E 'uptime|curr_connections|total_connections|evictions|bytes |limit_maxbytes|curr_items|cmd_get|cmd_set|get_hits|get_misses'
else
  echo "Cannot connect to ${HOST}:${PORT} — skipping stats"
fi

echo ""
echo "=== 6. OOM killer evidence ==="
dmesg --time-format iso 2>/dev/null | grep -i 'killed\|oom' | tail -10 || dmesg | grep -i 'killed\|oom' | tail -10

echo ""
echo "=== 7. File descriptor limits ==="
MCPID=$(pgrep -x memcached 2>/dev/null)
if [ -n "${MCPID}" ]; then
  echo "PID: ${MCPID}"
  cat "/proc/${MCPID}/limits" 2>/dev/null | grep -E 'open files|processes'
else
  echo "memcached not running"
fi

echo ""
echo "=== 8. Memory and swap ==="
free -h
vmstat 1 3

echo ""
echo "=== 9. Recent journal errors ==="
journalctl -u memcached -n 30 --no-pager 2>/dev/null || tail -50 /var/log/syslog 2>/dev/null | grep -i memcached

echo ""
echo "=== 10. Quick smoke test (set + get) ==="
if nc -zw1 "${HOST}" "${PORT}" 2>/dev/null; then
  SET_RESP=$(printf 'set diagtest 0 30 2\r\nok\r\n' | nc -w2 "${HOST}" "${PORT}")
  GET_RESP=$(printf 'get diagtest\r\n' | nc -w2 "${HOST}" "${PORT}")
  echo "SET response: ${SET_RESP}"
  echo "GET response: ${GET_RESP}"
else
  echo "Skipping smoke test — no connection"
fi

echo ""
echo "=== Diagnosis complete ==="
E

Error Medic Editorial

The Error Medic Editorial team consists of senior SREs and DevOps engineers with experience operating high-traffic Linux infrastructure. We write evidence-based troubleshooting guides grounded in real incident postmortems and production debugging sessions.

Sources

Related Guides