Question

I have haproxy setup with 2 backends: be1 and be2

I’m using ACL to route based on the path.

When be2 begins to develop a queue, the requests to be1 are negatively affected — what normally takes 100ms takes 2-3 seconds (just like what happens to the requests going to be2).

Is there a way to allow be2 to queue up without affecting performance on be1?

At peak, I was serving about 2000 req/s.

    global
    log 127.0.0.1   local0
    log 127.0.0.1   local1 notice
    #log loghost    local0 info
    maxconn 2000
    #chroot /usr/share/haproxy
    user haproxy
    group haproxy
    daemon
    #debug
    #quiet
    ulimit-n 65535
    stats socket /var/run/haproxy.sock
    nopoll

defaults
    log global
    mode    http
    option  httplog
    option  dontlognull
    retries 3
    option redispatch
    maxconn 2000
    contimeout  5000
    clitimeout  50000
    srvtimeout  50000

frontend http_in *:80
    option httpclose
    option forwardfor
    acl vt path_beg /route/1
    use_backend be2 if vt
    default_backend be1

backend be1
    balance leastconn
    option httpchk HEAD /redirect/are_you_alive HTTP/1.0
    server 01-2C2P9HI x:80 check inter 3000 rise 2 fall 3 maxconn 500

backend be2
    balance leastconn
    option httpchk HEAD /redirect/are_you_alive HTTP/1.0
    server 01-3TPDP27 x:80 check inter 3000 rise 2 fall 3 maxconn 250
    server 01-3CR0FKC x:80 check inter 3000 rise 2 fall 3 maxconn 250
    server 01-3E9CVMP x:80 check inter 3000 rise 2 fall 3 maxconn 250
    server 01-211LQMA x:80 check inter 3000 rise 2 fall 3 maxconn 250
    server 01-3H974V3 x:80 check inter 3000 rise 2 fall 3 maxconn 250
    server 01-13UCFVO x:80 check inter 3000 rise 2 fall 3 maxconn 250
    server 01-0HPIGGT x:80 check inter 3000 rise 2 fall 3 maxconn 250
    server 01-2LFP88F x:80 check inter 3000 rise 2 fall 3 maxconn 250
    server 01-1TIQBDH x:80 check inter 3000 rise 2 fall 3 maxconn 250
    server 01-2GG2LBB x:80 check inter 3000 rise 2 fall 3 maxconn 250
    server 01-1H5231E x:80 check inter 3000 rise 2 fall 3 maxconn 250
    server 01-0KIOVID x:80 check inter 3000 rise 2 fall 3 maxconn 250

listen stats 0.0.0.0:7474       #Listen on all IP's on port 9000
    mode http
    balance
    timeout client 5000
    timeout connect 4000
    timeout server 30000
    #This is the virtual URL to access the stats page
    stats uri /haproxy_stats        
    #Authentication realm. This can be set to anything. Escape space characters with a backslash.
    stats realm HAProxy\ Statistics 
    #The user/pass you want to use. Change this password!
    stats auth ge:test123
    #This allows you to take down and bring up back end servers.
    #This will produce an error on older versions of HAProxy.
    stats admin if TRUE

Not sure how I didn’t notice this yesterday, but seeing that maxconn is set to 2000… so that is likely one of my issues?

Answer 1

There are two different maxconn settings. One for frontends and other for backends. The setting for frontends limit the incoming connections, so even though your backend is available it would not get the request as it is queued on the frontend side. Once requests goes through frontend the backend queuing takes place. Frontends are affected by maxconn setting in “default” section, so I would increase that to 4000 for example, as backend should be able to handle it.

Please not that maxconn does not limit requests per second, but simultaneous connections. You might be having some HTTP keep-alive requests active that could limit the available throughput a lot.