Sunday, November 22, 2026
TR
LinuxOpen Source

Setting Up an Active/Passive Load Balancer with HAProxy and Keepalived

Setting Up an Active/Passive Load Balancer with HAProxy and Keepalived
Contents
  1. Ubuntu 24.04 LTS (Noble Numbat) — July 2026
  2. 1. Topology and Background Information
  3. Versions to be used
  4. Prerequisites
  5. 2. System Preparation (On Both Servers)
  6. 2.1 Updates and basic packages
  7. 2.2 Hostname and /etc/hosts
  8. 2.3 Identify the network interface name
  9. 2.4 Kernel parameters (CRITICAL)
  10. 2.5 File descriptor limits
  11. 3. HAProxy 3.4 LTS Installation (On Both Servers)
  12. 3.1 Certificate directory (if you will use HTTPS termination)
  13. 4. HAProxy Configuration
  14. 4.1 /etc/haproxy/haproxy.cfg
  15. 4.2 Validating the configuration and starting the service
  16. 4.3 Logging
  17. 5. Keepalived Installation and Configuration
  18. 5.1 Health check script (on both servers)
  19. 5.2 State change notification script (optional, recommended)
  20. 5.3 lb01 (MASTER) — /etc/keepalived/keepalived.conf
  21. 5.4 lb02 (BACKUP) — /etc/keepalived/keepalived.conf
  22. 5.5 Parameters that MUST be the same on both nodes
  23. 5.6 Starting the service
  24. 6. Firewall (UFW) Settings
  25. 7. Verification and Test Scenarios
  26. 7.1 Check that the VIP is on the right node
  27. 7.2 Watch the Keepalived logs
  28. 7.3 Observe the VRRP traffic
  29. 7.4 Service access test
  30. 7.5 Failover test — stopping the HAProxy service
  31. 7.6 Failover test — shutting the node down completely
  32. 7.7 Uninterrupted access test
  33. 8. Preemption (Take-Back) Behavior
  34. 9. Troubleshooting
  35. 10. Operational Notes
  36. 10.1 Configuration synchronization
  37. 10.2 Maintenance procedure (zero downtime)
  38. 10.3 Certificate renewal
  39. 10.4 Monitoring recommendations
  40. 10.5 Updating

Ubuntu 24.04 LTS (Noble Numbat) — July 2026


1. Topology and Background Information

RoleHostnameIP AddressKeepalived StatePriority
Primary LBlb0110.10.34.97MASTER150
Secondary LBlb0210.10.34.98BACKUP100
VIP (Virtual IP)lb-vip10.10.34.99Floating address—

How it works:

                        Clients
                            │
                            ▼
                   VIP: 10.10.34.99
                            │
              ┌─────────────┴─────────────┐
              │                           │
      ┌───────▼────────┐          ┌───────▼────────┐
      │  lb01 (.97)    │  VRRP    │  lb02 (.98)    │
      │  MASTER        │◄────────►│  BACKUP        │
      │  HAProxy (running)        │  HAProxy (running)
      └───────┬────────┘          └───────┬────────┘
              └─────────────┬─────────────┘
                            ▼
                   Backend sunucular

Important: The HAProxy service runs continuously on both servers. What is active/passive is not HAProxy, but which machine the VIP is on. Keepalived monitors its peer with the VRRP protocol (IP protocol no: 112); if the MASTER goes down, the VIP moves to the BACKUP within seconds and is announced to the switch with a gratuitous ARP.

Versions to be used

ComponentVersionSource
Operating systemUbuntu 24.04.x LTS—
HAProxy3.4.x LTSppa:vbernat/haproxy-3.4
Keepalived2.2.8 (distribution repository)Ubuntu Noble main

Note: Ubuntu 24.04's own repository contains HAProxy 2.8.x. For a long-lived installation, we are installing the LTS branch 3.4 from the PPA. If you do not have PPA access or your organization's policy does not allow it, the repository version (2.8) is also supported until 2028 and is compatible with the configurations here.

Prerequisites

  • Root or sudo privileges on both VMs
  • The two servers must be in the same L2 broadcast domain (required for VRRP to work)
  • The address 10.10.34.99 must be outside the DHCP pool and must not be in use on another device
  • If you use VMware/Proxmox, the MAC Address Changes and Forged Transmits permissions must be enabled on the vSwitch/port-group — otherwise the VIP will not move

2. System Preparation (On Both Servers)

2.1 Updates and basic packages

sudo apt update && sudo apt -y full-upgrade
sudo apt -y install software-properties-common curl vim net-tools \
    chrony psmisc iproute2 socat

Time synchronization is critical for log correlation in HA clusters:

sudo timedatectl set-timezone Europe/Istanbul
sudo systemctl enable --now chrony
timedatectl status

2.2 Hostname and /etc/hosts

On lb01:

sudo hostnamectl set-hostname lb01

On lb02:

sudo hostnamectl set-hostname lb02

On both, add the following to the /etc/hosts file:

10.10.34.97   lb01
10.10.34.98   lb02
10.10.34.99   lb-vip

2.3 Identify the network interface name

ip -br link show

In the output you will see a name such as ens160, ens192 or eth0. This document uses ens192 — replace it with your own interface name.

2.4 Kernel parameters (CRITICAL)

HAProxy must be able to bind to an IP that is not present on the machine (the VIP). Because the VIP does not exist on the passive node, HAProxy will fail at startup without this parameter:

sudo tee /etc/sysctl.d/99-haproxy-ha.conf > /dev/null <<'EOF'
# Var olmayan (VIP) adrese bind izni
net.ipv4.ip_nonlocal_bind = 1
net.ipv6.ip_nonlocal_bind = 1

# Settings for connection load
net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.tcp_max_syn_backlog = 8192
net.core.somaxconn = 8192
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 15
net.core.netdev_max_backlog = 16384
EOF

sudo sysctl --system

Verification:

sysctl net.ipv4.ip_nonlocal_bind

2.5 File descriptor limits

sudo mkdir -p /etc/systemd/system/haproxy.service.d
sudo tee /etc/systemd/system/haproxy.service.d/limits.conf > /dev/null <<'EOF'
[Service]
LimitNOFILE=200000
EOF
sudo systemctl daemon-reload

3. HAProxy 3.4 LTS Installation (On Both Servers)

sudo add-apt-repository --yes ppa:vbernat/haproxy-3.4
sudo apt update
sudo apt -y install haproxy=3.4.\*

If the PPA has not yet been published for 3.4, use ppa:vbernat/haproxy-3.2 (LTS, supported until 2030). You can see which versions are available with apt-cache madison haproxy.

Verify the version and the repository:

haproxy -v
apt-cache madison haproxy

Pin the package to prevent automatic major version jumps (optional but recommended):

sudo tee /etc/apt/preferences.d/haproxy > /dev/null <<'EOF'
Package: haproxy
Pin: version 3.4.*
Pin-Priority: 1001
EOF

3.1 Certificate directory (if you will use HTTPS termination)

sudo mkdir -p /etc/haproxy/certs
sudo chmod 700 /etc/haproxy/certs
sudo chown haproxy:haproxy /etc/haproxy/certs

HAProxy expects the certificate and key combined in a single file:

# fullchain.pem + private.key -> site.pem
sudo bash -c 'cat /yol/fullchain.pem /yol/private.key > /etc/haproxy/certs/site.pem'
sudo chmod 600 /etc/haproxy/certs/site.pem
sudo chown haproxy:haproxy /etc/haproxy/certs/site.pem

4. HAProxy Configuration

First, back up the existing file:

sudo cp /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.orj

4.1 /etc/haproxy/haproxy.cfg

The following content must be exactly the same on both servers (the only difference is the localpeer value in the peers section; we resolve that automatically from the hostname as well).

#---------------------------------------------------------------------
# GLOBAL
#---------------------------------------------------------------------
global
    log         /dev/log local0
    log         /dev/log local1 notice
    chroot      /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin
    stats timeout 30s
    user        haproxy
    group       haproxy
    daemon

    maxconn     40000
    nbthread    4                 # set according to your vCPU count

    # TLS defaults (modern, compatible profile)
    ca-base  /etc/ssl/certs
    crt-base /etc/haproxy/certs
    ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets
    ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305
    ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
    ssl-default-server-options ssl-min-ver TLSv1.2 no-tls-tickets
    tune.ssl.default-dh-param 2048

#---------------------------------------------------------------------
# PEERS - stick-table (session persistence) sync between the two LBs
#---------------------------------------------------------------------
peers lb_cluster
    peer lb01 10.10.34.97:10000
    peer lb02 10.10.34.98:10000

#---------------------------------------------------------------------
# DEFAULTS
#---------------------------------------------------------------------
defaults
    log     global
    mode    http
    option  httplog
    option  dontlognull
    option  forwardfor  except 127.0.0.0/8
    option  http-server-close
    retries 3
    option  redispatch

    timeout connect         5s
    timeout client          60s
    timeout server          60s
    timeout http-request    10s
    timeout http-keep-alive 10s
    timeout queue           30s
    timeout check           5s
    timeout tunnel          1h        # for WebSocket / long-lived connections

    errorfile 400 /etc/haproxy/errors/400.http
    errorfile 403 /etc/haproxy/errors/403.http
    errorfile 408 /etc/haproxy/errors/408.http
    errorfile 500 /etc/haproxy/errors/500.http
    errorfile 502 /etc/haproxy/errors/502.http
    errorfile 503 /etc/haproxy/errors/503.http
    errorfile 504 /etc/haproxy/errors/504.http

#---------------------------------------------------------------------
# STATS INTERFACE - NOT on the VIP, listens on the node's own IP
# This way you can also see the state of the passive node.
#---------------------------------------------------------------------
listen stats
    bind    *:8404
    mode    http
    stats   enable
    stats   uri /stats
    stats   refresh 10s
    stats   show-node
    stats   show-legends
    stats   admin if TRUE
    stats   auth admin:BurayaGucluBirParolaYaz
    no log

#---------------------------------------------------------------------
# FRONTEND - HTTP (80) : redirect to HTTPS
#---------------------------------------------------------------------
frontend fe_http
    bind    10.10.34.99:80
    mode    http

    # Allow Let's Encrypt HTTP-01 validation to pass
    acl is_acme path_beg /.well-known/acme-challenge/
    use_backend be_acme if is_acme

    http-request redirect scheme https code 301 unless { ssl_fc } or is_acme

#---------------------------------------------------------------------
# FRONTEND - HTTPS (443)
#---------------------------------------------------------------------
frontend fe_https
    bind    10.10.34.99:443 ssl crt /etc/haproxy/certs/ alpn h2,http/1.1
    mode    http

    # Security headers
    http-response set-header Strict-Transport-Security "max-age=31536000; includeSubDomains"
    http-response set-header X-Content-Type-Options nosniff
    http-response set-header X-Frame-Options SAMEORIGIN

    # Pass client information to the backend
    http-request set-header X-Forwarded-Proto https
    http-request set-header X-Forwarded-Port %[dst_port]

    # Simple rate limiting (optional) - synced between the two LBs
    stick-table type ip size 200k expire 60s store http_req_rate(10s) peers lb_cluster
    http-request track-sc0 src
    http-request deny deny_status 429 if { sc_http_req_rate(0) gt 300 }

    default_backend be_web

#---------------------------------------------------------------------
# BACKEND - Web sunucular
#---------------------------------------------------------------------
backend be_web
    mode    http
    balance roundrobin

    option  httpchk
    http-check send meth GET uri /health ver HTTP/1.1 hdr Host healthcheck.local
    http-check expect status 200

    # Session persistence (cookie-based) - remove if not needed
    cookie SRVID insert indirect nocache

    default-server inter 3s fall 3 rise 2 maxconn 2000 check

    server web01 10.10.34.11:8080 cookie web01
    server web02 10.10.34.12:8080 cookie web02
    # server web03 10.10.34.13:8080 cookie web03 backup

#---------------------------------------------------------------------
# BACKEND - ACME (if you will use certbot standalone)
#---------------------------------------------------------------------
backend be_acme
    mode    http
    server  acme 127.0.0.1:8888

#---------------------------------------------------------------------
# EXAMPLE: Load balancing in TCP mode (e.g. MSSQL, PostgreSQL, RDP, SMTP)
#---------------------------------------------------------------------
#listen li_mssql
#    bind    10.10.34.99:1433
#    mode    tcp
#    option  tcplog
#    balance leastconn
#    timeout client 30m
#    timeout server 30m
#    default-server inter 5s fall 3 rise 2 check
#    server db01 10.10.34.21:1433
#    server db02 10.10.34.22:1433 backup

4.2 Validating the configuration and starting the service

Always run a syntax check after every change:

sudo haproxy -c -f /etc/haproxy/haproxy.cfg

The output should be Configuration file is valid. Then:

sudo systemctl enable --now haproxy
sudo systemctl status haproxy --no-pager

Seamless reload (existing connections are preserved):

sudo systemctl reload haproxy

Always use reload instead of restart. Because HAProxy 3.x runs in master-worker mode, no connections are lost during a reload.

4.3 Logging

The Ubuntu package automatically creates the /etc/rsyslog.d/49-haproxy.conf file. Verify it:

sudo systemctl restart rsyslog
sudo tail -f /var/log/haproxy.log

If there are no logs, make sure the imuxsock module is active in /etc/rsyslog.conf.


5. Keepalived Installation and Configuration

sudo apt -y install keepalived
keepalived -v

5.1 Health check script (on both servers)

sudo tee /etc/keepalived/check_haproxy.sh > /dev/null <<'EOF'
#!/bin/bash
# Is the HAProxy process alive and is the stats port responding?
# Exit 0 = healthy, non-zero = unhealthy

if ! /bin/pidof haproxy > /dev/null 2>&1; then
    exit 1
fi

# Liveness check via the admin socket
if ! echo "show info" | /usr/bin/socat stdio /run/haproxy/admin.sock > /dev/null 2>&1; then
    exit 1
fi

exit 0
EOF

sudo chown root:root /etc/keepalived/check_haproxy.sh
sudo chmod 0755 /etc/keepalived/check_haproxy.sh

Test the script manually:

sudo /etc/keepalived/check_haproxy.sh; echo "Sonuc: $?"
sudo tee /etc/keepalived/notify.sh > /dev/null <<'EOF'
#!/bin/bash
TYPE=$1
NAME=$2
STATE=$3

/usr/bin/logger -t keepalived-notify "VRRP durumu degisti: ${NAME} -> ${STATE}"

case $STATE in
    "MASTER")
        # VIP moved to this node. Take extra actions here if needed:
        /usr/bin/systemctl reload haproxy
        ;;
    "BACKUP"|"FAULT")
        # VIP left this node
        ;;
esac
exit 0
EOF

sudo chown root:root /etc/keepalived/notify.sh
sudo chmod 0755 /etc/keepalived/notify.sh

5.3 lb01 (MASTER) — /etc/keepalived/keepalived.conf

sudo tee /etc/keepalived/keepalived.conf > /dev/null <<'EOF'
global_defs {
    router_id LB01
    script_user root
    enable_script_security
    vrrp_skip_check_adv_addr
    vrrp_garp_master_refresh 60
    vrrp_garp_master_delay   1
    max_auto_priority        -1
}

vrrp_script chk_haproxy {
    script   "/etc/keepalived/check_haproxy.sh"
    interval 2          # 2 saniyede bir kontrol
    timeout  3
    fall     2          # 2 failed checks -> unhealthy
    rise     2          # 2 successful checks  -> healthy
    weight   -60        # if unhealthy, priority drops by 60 (150 -> 90 < 100)
}

vrrp_instance VI_LB {
    state      MASTER
    interface  ens192              # <<< REPLACE WITH YOUR OWN INTERFACE NAME
    virtual_router_id 51           # between 1-255, unique within the L2 domain
    priority   150
    advert_int 1

    # Unicast VRRP - more reliable than multicast in virtualized/cloud environments
    unicast_src_ip 10.10.34.97
    unicast_peer {
        10.10.34.98
    }

    authentication {
        auth_type PASS
        auth_pass Vrrp2026          # MAXIMUM 8 CHARACTERS (extra characters are truncated)
    }

    virtual_ipaddress {
        10.10.34.99/24 dev ens192 label ens192:vip
    }

    track_script {
        chk_haproxy
    }

    track_interface {
        ens192
    }

    notify /etc/keepalived/notify.sh
}
EOF

sudo chmod 600 /etc/keepalived/keepalived.conf

5.4 lb02 (BACKUP) — /etc/keepalived/keepalived.conf

sudo tee /etc/keepalived/keepalived.conf > /dev/null <<'EOF'
global_defs {
    router_id LB02
    script_user root
    enable_script_security
    vrrp_skip_check_adv_addr
    vrrp_garp_master_refresh 60
    vrrp_garp_master_delay   1
    max_auto_priority        -1
}

vrrp_script chk_haproxy {
    script   "/etc/keepalived/check_haproxy.sh"
    interval 2
    timeout  3
    fall     2
    rise     2
    weight   -60
}

vrrp_instance VI_LB {
    state      BACKUP
    interface  ens192              # <<< REPLACE WITH YOUR OWN INTERFACE NAME
    virtual_router_id 51           # must be the SAME as on the MASTER
    priority   100
    advert_int 1

    unicast_src_ip 10.10.34.98
    unicast_peer {
        10.10.34.97
    }

    authentication {
        auth_type PASS
        auth_pass Vrrp2026          # must be the SAME as on the MASTER
    }

    virtual_ipaddress {
        10.10.34.99/24 dev ens192 label ens192:vip
    }

    track_script {
        chk_haproxy
    }

    track_interface {
        ens192
    }

    notify /etc/keepalived/notify.sh
}
EOF

sudo chmod 600 /etc/keepalived/keepalived.conf

5.5 Parameters that MUST be the same on both nodes

ParameterValueDescription
virtual_router_id51If different, both nodes become MASTER (split-brain)
auth_passVrrp2026If different, VRRP packets are rejected
advert_int1A mismatch generates a warning
virtual_ipaddress10.10.34.99/24Same VIP
Subnet mask/24Must match your network's actual mask

5.6 Starting the service

# First lb01, then lb02
sudo systemctl enable --now keepalived
sudo systemctl status keepalived --no-pager

6. Firewall (UFW) Settings

VRRP uses IP protocol number 112, and UFW does not support it by name. The most practical method is to fully authorize the peer node:

# --- Her iki sunucuda ---
sudo ufw allow 22/tcp comment 'SSH'
sudo ufw allow 80/tcp comment 'HTTP'
sudo ufw allow 443/tcp comment 'HTTPS'
sudo ufw allow from 10.10.34.0/24 to any port 8404 proto tcp comment 'HAProxy stats'

# --- on lb01: full trust in the peer (VRRP + peers port) ---
sudo ufw allow from 10.10.34.98 comment 'VRRP + HAProxy peer - lb02'

# --- on lb02 ---
sudo ufw allow from 10.10.34.97 comment 'VRRP + HAProxy peer - lb01'

sudo ufw --force enable
sudo ufw status numbered

If you prefer to use multicast VRRP (if you remove the unicast_peer blocks), additionally add the following before the COMMIT line of the /etc/ufw/before.rules file:

-A ufw-before-input  -d 224.0.0.18 -p vrrp -j ACCEPT
-A ufw-before-output -d 224.0.0.18 -p vrrp -j ACCEPT

Then sudo ufw reload.

Recommendation: In virtualization environments (VMware, Hyper-V, OpenStack) and in inter-VLAN environments, multicast is frequently filtered. The unicast configuration in this document is more reliable by default.


7. Verification and Test Scenarios

7.1 Check that the VIP is on the right node

On lb01:

ip -br addr show ens192

The output should show both 10.10.34.97/24 and 10.10.34.99/24.

On lb02, the same command should show only 10.10.34.98/24. If the VIP appears on both nodes, there is a split-brain (see Section 9).

7.2 Watch the Keepalived logs

sudo journalctl -u keepalived -f

Expected lines:

  • lb01: Entering MASTER STATE
  • lb02: Entering BACKUP STATE

7.3 Observe the VRRP traffic

sudo tcpdump -i ens192 -n vrrp

You should see an advertisement packet from the MASTER once per second.

7.4 Service access test

From an external machine:

curl -I http://10.10.34.99
curl -Ik https://10.10.34.99

Stats pages (separately for each node):

http://10.10.34.97:8404/stats
http://10.10.34.98:8404/stats

7.5 Failover test — stopping the HAProxy service

On lb01:

sudo systemctl stop haproxy

Expected: the chk_haproxy script fails within 2 checks (about 4 seconds), the priority drops from 150 to 90, and lb02 (100) takes over. Check on lb02:

ip -br addr show ens192   # the VIP should appear here

Rollback:

sudo systemctl start haproxy   # lb01'de

The priority goes back up to 150 and the VIP returns to lb01 (preemption).

7.6 Failover test — shutting the node down completely

sudo systemctl stop keepalived   # veya sunucuyu reboot edin

The VIP should move to lb02 immediately.

7.7 Uninterrupted access test

During the failover, send requests continuously in a terminal:

while true; do
  curl -s -o /dev/null -w "%{http_code} %{time_total}s\n" http://10.10.34.99/
  sleep 0.5
done

An interruption of at most 1-3 seconds at the moment of failover is acceptable.


8. Preemption (Take-Back) Behavior

In the default configuration, when lb01 comes back, it automatically takes the VIP back. This causes a brief additional interruption. If there is a risk of "flapping" (constantly going back and forth), you can make the take-back delayed by adding the following setting on lb01:

    preempt_delay 60      # 60 saniye bekle, sonra devral

If you want to disable take-back entirely, set state BACKUP on both nodes and add the following to lb01:

    state BACKUP
    priority 150
    nopreempt

nopreempt works only together with state BACKUP. If it is used with state MASTER, it is ignored.


9. Troubleshooting

SymptomPossible CauseSolution
VIP is on both nodes (split-brain)VRRP packets are not reaching the peerCheck the firewall/VLAN, tcpdump -i ens192 vrrp, and that virtual_router_id and auth_pass match
HAProxy does not start: cannot bind socketip_nonlocal_bind is offsysctl net.ipv4.ip_nonlocal_bind must be 1
VIP moves but no traffic arrivesThe switch ARP table was not updatedThe vrrp_garp_master_refresh setting, the hypervisor "Forged Transmits" permission
chk_haproxy is always FAULTScript path/permission is wrongThe script must be owned by root and be 0755; when enable_script_security is active, it cannot be in a world-writable directory
VRRP auth warningauth_pass is longer than 8 charactersShorten the password to 8 characters
Backend is always DOWNHealth check path is wrongEdit the http-check line according to your backend's real endpoint
Another VRRP cluster on the same L2 is conflictingvirtual_router_id conflictChoose a different ID (e.g. 52)

Useful commands:

# Keepalived configuration syntax check
sudo keepalived --config-test -f /etc/keepalived/keepalived.conf

# Keepalived detailed state dump
sudo systemctl reload keepalived   # generates /tmp/keepalived.data and .stats
cat /tmp/keepalived.data

# HAProxy live statistics (via socket)
echo "show stat" | sudo socat stdio /run/haproxy/admin.sock | column -s, -t
echo "show info" | sudo socat stdio /run/haproxy/admin.sock
echo "show servers state" | sudo socat stdio /run/haproxy/admin.sock

# Take a backend server out for maintenance (no downtime)
echo "set server be_web/web01 state maint" | sudo socat stdio /run/haproxy/admin.sock
echo "set server be_web/web01 state ready" | sudo socat stdio /run/haproxy/admin.sock

10. Operational Notes

10.1 Configuration synchronization

haproxy.cfg must remain exactly the same on both nodes. Manual copying is a source of errors; use git or rsync:

# lb01'den lb02'ye
sudo rsync -avz --delete /etc/haproxy/ root@10.10.34.98:/etc/haproxy/
ssh root@10.10.34.98 'haproxy -c -f /etc/haproxy/haproxy.cfg && systemctl reload haproxy'

In the long term, I recommend managing it with Ansible — configuration drift is the most common cause of failures in HA clusters.

10.2 Maintenance procedure (zero downtime)

If you are going to perform maintenance on lb01:

# 1. Stop keepalived on lb01 -> VIP moves to lb02
sudo systemctl stop keepalived
# 2. Verify the VIP arrived on lb02
# 3. Perform maintenance on lb01 (package updates, reboot, etc.)
# 4. Restart the services on lb01
sudo systemctl start haproxy && sudo systemctl start keepalived

10.3 Certificate renewal

After updating the .pem file in the /etc/haproxy/certs/ directory:

sudo systemctl reload haproxy

or, without interruption, via the socket:

echo -e "set ssl cert /etc/haproxy/certs/site.pem <<\n$(cat /yeni/site.pem)\n" | sudo socat stdio /run/haproxy/admin.sock
echo "commit ssl cert /etc/haproxy/certs/site.pem" | sudo socat stdio /run/haproxy/admin.sock

10.4 Monitoring recommendations

  • The HAProxy Prometheus exporter is built in; you can publish /metrics by adding it to the frontend: frontend fe_metrics bind *:8405 http-request use-service prometheus-exporter if { path /metrics } no log
  • Critical metrics to monitor: the number of UP servers in the backend, the 5xx rate, queue length, session usage percentage, and which node the VIP is on
  • Set up an alert on the Entering MASTER STATE line in journalctl -u keepalived — it catches unexpected failovers

10.5 Updating

sudo apt update && sudo apt list --upgradable | grep -E 'haproxy|keepalived'
# Update the passive node first, test it, then fail over and move on to the other one

Kerim Koç

Systems and infrastructure engineer. I write about Linux, open source, high availability and security.