Kubernetes - haproxy+keepalived

If you deploy your own k8s and you have multiple control plane nodes (i.e. multiple kube-apiserver), haproxy can be used to load balance the apiservers.

Then the haproxy will be the single point of failure. We can use keepalived to have BACKUP haproxy standing by.

haproxy: load balance HTTP requests to multiple kube-apiservers, and make them HA.
keepalived: keepalived makes haproxy highly available; an implementation of VRRP.

Virtual IP

IP will appear in ip a if the Node is alive, otherwise will disappear (can be simulated by systemctl stop haproxy) and the vip will appear in another configured node.

A VIP is also known as a floating IP addresses.

Keepalived

Load-balancing relies on Linux Virtual Server (IPVS); high-availability is achieved by VRRP protocol. VRRP is a fundamental brick for router failover.

In the event that the master server (haproxy1) fails, keepalived assigns the virtual IP address (e.g. 10.0.0.10/24) to the eth0 interface on the backup server (haproxy2), which becomes the master server.

keepalived leverages the Linux kernel feature of floating IP addresses through Virtual Router Redundancy Protocol (VRRP): the MASTER sends out advertisements at regular intervals; if the MASTER stops sending advertisements, a new MASTER is elected. Usually, the VRRP protocol ensures that only one of the participating instances is running as MASTER. The other instances are in the BACKUP state listening for multicast packets from an instance with a higher priority.

If the BACKUP instance fails to receive VRRP advertisements for a period longer than the KEEPALIVED_ADVERT_INT, the BACKUP instance moves to the MASTER state and it takes the VIP for itself.

To confirms we’re listening for relevant VRRP traffic:

$ ss -l | grep vrrp

# VRRP traffic (IP protocol 112)
$ tcpdump -i vxlan0 'ip proto 112' -n

How to deploy haproxy and keepalived?

haproxy and keepalived can be containerized or non-containerized.

If Not containerized

Can be installed by automation tools like ansible; managed by systemctl:

$ systemctl restart haproxy
$ systemctl enable haproxy
$ systemctl restart keepalived
$ systemctl enable keepalived

Configs may be found in the following locations:

/etc/haproxy/haproxy.cfg
/usr/local/etc/haproxy/haproxy.cfg
/etc/keepalived/keepalived.conf
/usr/local/etc/keepalived/keepalived.conf

If Containerized

haproxy and keepalived can be deployed as static pods (i.e. add yamls to /etc/kubernetes/manifests)

Check keepalived config:

$ kubectl -n kube-system exec kube-keepalived-master -- cat /etc/keepalived/keepalived.conf

Verify if they are running:

$ crictl ps | grep -e apiserver -e haproxy -e keepalived