HAProxy in LXC Proxmox Closes Connections: Problem Solution

Problem with Immediate HAProxy Connection Closure in Proxmox LXC Containers

The problem with immediate connection closure to HAProxy in Proxmox LXC containers is usually related to network configuration issues, backend settings, or container limitations. The “Connection closed by foreign host” message indicates that HAProxy is accepting the connection but immediately closing it without properly processing the traffic.

Contents

• Basic Causes of the Problem
• LXC Network Configuration Issues
• HAProxy Backend Issues
• Security Settings and Firewall
• Step-by-Step Solution
• HAProxy Timeout Optimization
• Configuration Recommendations

Basic Causes of the Problem

Immediate connection closure to HAProxy in LXC containers can be caused by several main factors:

1. Backend availability issues: As shown in your logs, backends are marked as unavailable due to Layer4 timeout
2. Incorrect LXC container network configuration: Containers may have network stack limitations
3. Firewall settings: Issues with iptables or other firewalls
4. Container resource limitations: CPU, memory, or network limits
5. Port conflicts: Ports may be used by other processes

Important: The “Connection closed by foreign host” message usually means that HAProxy successfully accepts the connection but closes it when trying to forward traffic to a backend that is unavailable.

LXC Network Configuration Issues

LXC containers in Proxmox have specific network configuration features that can cause similar problems:

1. Network Configuration Modes

In Proxmox, LXC containers can use different network configuration modes:

• Bridge mode: Container connects to the host’s network bridge
• VETH pair: Virtual interface pair
• MACVLAN: Direct connection to the physical interface

Problem: In bridge mode, issues with ARP cache and routing can occur, especially for intercommunication between VMs and LXC containers.

2. Container Network Limitations

LXC containers have network stack limitations by default:

# Check container network limitations
cat /proc/1/cgroup | grep cpu
cat /proc/1/cgroup | grep memory

3. Container Network Configuration

Ensure the container’s network configuration is correct:

# Check network configuration in the container
ip addr show
ip route show
ss -lntp | grep haproxy

HAProxy Backend Issues

Warnings about unavailable backends are the primary cause of connection closure problems:

1. Layer4 timeout errors

As seen in your logs:

Server k8s-api-back/lb1 is DOWN, reason: Layer4 timeout
backend 'k8s-api-back' has no server available!

This means HAProxy cannot establish TCP connections to backend servers.

2. Possible causes of Layer4 timeout:

• Incorrect IP addresses or ports for backends
• Firewall blocking connections from HAProxy to backends
• Network unavailability of backends
• Network or server overload

3. Backend Availability Check

# Check backend availability from the HAProxy container
telnet <backend_ip> <backend_port>
nc -zv <backend_ip> <backend_port>

Security Settings and Firewall

1. Proxmox Firewall

Check firewall settings on the Proxmox host:

# Check iptables rules
iptables -L -n -v

2. Firewall in LXC Container

Ensure there are no blocking rules in the container:

# Check rules in the container (if using ufw or firewalld)
ufw status
firewall-cmd --list-all

3. SELinux or AppArmor

If SELinux or AppArmor is enabled, they may block connections:

# Check SELinux status
sestatus

Step-by-Step Solution

Step 1: Check Basic Network Configuration

1. Ensure the container has the correct IP address:

ip addr show

2. Check port availability:

ss -lntp | grep haproxy

3. Check availability from the Proxmox host:

telnet <container_ip> 6443

Step 2: Check HAProxy Configuration

Check the HAProxy configuration file:

cat /etc/haproxy/haproxy.cfg

Ensure that:

• All listening ports are specified correctly
• Backends have correct IP addresses and ports
• There are no syntax errors

Step 3: Check Backend Availability

Check availability of all backends from the HAProxy container:

# Check each backend
for backend in $(grep "server" /etc/haproxy/haproxy.cfg | awk '{print $2}'); do
    echo "Testing $backend"
    echo $backend | sed 's/:/ /' | xargs -I {} sh -c 'nc -zv {} || echo "Failed: {}"'
done

Step 4: Check Container Network Limitations

1. Check network stack limitations:

cat /proc/1/cgroup

2. Increase network limitations if necessary:

# In the container configuration file
lxc.cgroup.memory.limit_in_bytes = 4G
lxc.cgroup.cpu.shares = 1024

Step 5: Check Firewall and Network Rules

1. Temporarily stop the firewall for testing:

systemctl stop ufw  # or other firewall

2. Check if this resolves the issue.

HAProxy Timeout Optimization

Setting the correct timeouts can help resolve connection closure issues:

1. Configure Timeouts in HAProxy Configuration

global
    timeout client 300s
    timeout server 300s
    timeout connect 5s
    timeout http-request 5s

defaults
    timeout client 300s
    timeout server 300s
    timeout connect 5s
    timeout http-request 5s

2. Configure Backend Checks

backend k8s-api-back
    server lb1 <backend_ip>:<port> check inter 30s fall 3 rise 2
    server lb2 <backend_ip>:<port> check inter 30s fall 3 rise 2 backup

3. Disable Checks for Testing

For diagnostics, you can temporarily disable checks:

backend k8s-api-back
    server lb1 <backend_ip>:<port> no-check

Configuration Recommendations

1. Optimal LXC Configuration for HAProxy

# Recommended settings for LXC container with HAProxy
lxc.cgroup.memory.limit_in_bytes = 2G
lxc.cgroup.cpu.shares = 512
lxc.cap.drop =
lxc.cap.keep = NET_BIND_SERVICE
lxc.uts.name = haproxy-container

2. Monitoring and Logging

Enable detailed HAProxy logging:

global
    log /dev/log local0
    log /dev/log local1 notice
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin
    stats timeout 30s

defaults
    log global
    option httplog
    option dontlognull
    retries 3

3. Automatic Backend Availability Check

Create a script to check backend availability:

#!/bin/bash
# check_backends.sh

BACKENDS=("backend1:8080" "backend2:8080" "backend3:8080")

for backend in "${BACKENDS[@]}"; do
    host=$(echo $backend | cut -d: -f1)
    port=$(echo $backend | cut -d: -f2)
    
    if nc -z -w5 $host $port; then
        echo "✓ $backend is available"
    else
        echo "✗ $backend is unavailable"
        # Restart HAProxy or send notification
    fi
done

4. Use Health Check

Add health check to monitor backend status:

backend k8s-api-back
    option httpchk GET /health
    http-check expect status 200
    server lb1 <backend_ip>:<port> check inter 10s fall 3 rise 2

Conclusion

The problem with HAProxy connection closure in Proxmox LXC containers is usually resolved by:

1. Checking network configuration of the container and backend availability
2. Optimizing timeouts in the HAProxy configuration
3. Checking firewall and network limitations of the container
4. Setting up proper health checks for backends

The main cause of the problem is backend unavailability, which causes HAProxy to close connections immediately after establishing them. It’s recommended to start by checking the availability of all backends from the HAProxy container, then gradually configure timeouts and checks.

To prevent similar issues in the future:

• Regularly monitor backend status
• Use automatic failover to backup servers
• Configure appropriate timeouts based on application type
• Enable detailed logging for quick diagnostics

Sources

1. Proxmox and HAProxy – janfla.slask.pl
2. HAProxy: backend ‘app’ has no server available - Docker Community Forums
3. Layer4 “Connection refused” with haproxy - Stack Overflow
4. haproxy Server XXXXX is DOWN, reason: Layer4 timeout - Stack Overflow
5. Backend servers are down due to Layer4 connection problem - HAProxy community
6. haProxy in LXC | Proxmox Support Forum
7. VMs cannot access services hosted on LXC Containers (Connection Timed Out) - Proxmox Support Forum