====== keepalived ====== ===== Links ===== * [[https://www.keepalived.org/changelog.html|changelog]] * https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/load_balancer_administration/ch-keepalived-overview-vsa * [[https://www.digitalocean.com/community/tutorials/how-to-set-up-highly-available-web-servers-with-keepalived-and-floating-ips-on-ubuntu-14-04|Floating IP bei digitalocean]] * http://blog.unicsolution.com/2015/01/kamailio-high-availability-with.html * [[https://github.com/wikimedia/PyBal|Pybal lvs-monitor]] ===== Installation ===== apt install keepalived sudo echo "net.ipv4.ip_nonlocal_bind = 1" >> /etc/sysctl.conf sudo echo "net.ipv6.ip_nonlocal_bind = 1" >> /etc/sysctl.conf sudo sysctl -p [[https://serverfault.com/questions/236626/how-to-bind-a-non-local-ipv6-address|How to bind a non-local IPv6 address?]] ===== Konfiguration ===== ==== prüfen ==== ''keepalived -t -f /etc/keepalived/keepalived.conf -l'' ==== mit gesplitteter Config ==== mkdir /etc/keepalived/conf.d # Configuration File for keepalived # http://www.keepalived.org/doc/configuration_synopsis.html # global_defs { notification_email { root@DOMAIN.TLD } # dont forget "smtp_alert" in your instances! notification_email_from root@DOMAIN.TLD smtp_server localhost smtp_connect_timeout 30 } include /etc/keepalived/conf.d/*.conf ==== Floating IP ==== Ein sehr simples Setup mit IP-failover (MASTER <-> BACKUP). /etc/keepalived/conf.d/$DIENST1.conf vrrp_instance DIENST1 { state MASTER smtp_alert interface eth0 virtual_router_id 51 # unique ID! priority 100 # master -> highest advert_int 1 authentication { # dont use pass unless on 100% secure net, its send in cleartext https://louwrentius.com/configuring-attacking-and-securing-vrrp-on-linux.html # auth_type PASS # much secure: auth_type AH # max. length: 8 chars auth_pass 12345678 } virtual_ipaddress { 1.2.3.4/32 } } BACKUPs (ID kleiner + eindeutig!) vrrp_instance DIENST1 { state BACKUP interface eth0 virtual_router_id 51 # unique ID! priority 99 # lower than master! advert_int 1 authentication { # dont use pass unless on 100% secure net, its send in cleartext https://louwrentius.com/configuring-attacking-and-securing-vrrp-on-linux.html # auth_type PASS # much secure: auth_type AH auth_pass 12345678 } virtual_ipaddress { 1.2.3.4/32 } } ==== Load-Balancer IP an loopback binden ==== ''/etc/network/interfaces'' # The loopback network interface auto lo iface lo inet loopback up ip addr add 1.2.3.4/32 dev lo down ip addr del 1.2.3.4/32 dev lo Dann soll aber nicht die IP via ARP [[http://kb.linuxvirtualserver.org/wiki/Using_arp_announce/arp_ignore_to_disable_ARP|von "lo" aus antworten]] (das macht der load-balancer): 'Datei /etc/sysctl.conf'' ((sysctl -p nicht vergessen)) # ipvs settings for realservers ("cluster nodes"): net.ipv4.conf.all.arp_ignore = 1 net.ipv4.conf.all.arp_announce = 2 net.ipv4.conf.default.arp_ignore = 1 net.ipv4.conf.default.arp_announce = 2 ==== IPv6-Besonderheiten ==== Wenn die "virtual_ipaddress" kein Netzmaske (bzw. /128) hat, dann setzt keepalived die "preferred lifetime" auf 0 (preferred_lft 0) und die Anzeige mit ip a zeigt "deprecated": inet6 0:0:0:0:0:FFFF:0A01:0164/128 scope global deprecated nodad valid_lft forever preferred_lft 0sec Das ist aber schlecht weil die IP dann nicht mehr als "preferred" source-Adresse benutzt wird sobald weitere v6-IPs (ohne preferred_lft 0) hinzugefügt werden. Die Problematik wird auch in einem ähnlichen Kontext [[https://angristan.xyz/fix-ipv6-hetzner-cloud/|bei Hetzner zum Problem]] und hier weiter erläutert: [[http://kennystechtalk.blogspot.com/2015/12/how-long-does-deprecated-ipv6-address.html|How long does a deprecated IPv6 address remain attached to an interface?]]. **Lösung**: Maske mit eintragen (z.B. /64). keepalived unterstützt keine gemixten ipv4 / ipv6-Angaben bei virtual_ipaddress (Grund ist die fehlende vrrp-Protokollunterstützung dafür). **Lösung**: * Entweder in eine eigene vrrp_instance verschieben * oder diese in einen virtual_ipaddress_excluded-Block setzen. Damit sind die Adresse nicht mit im vvrp-Paket aber werden dennoch mit hochgefahren: ... virtual_ipaddress_excluded { 0:0:0:0:0:FFFF:0A01:0164 0:0:0:0:0:FFFF:0A01:0165 } ==== Dienst via Skript prüfen ==== Wenn der Service nicht läuft, macht auch die Cluster-IP keinen Sinn. vrrp_script check_service1_health { script "/etc/keepalived/check_service1_health.sh" interval 5 # check every 5 seconds fall 2 # require 2 failures for KO rise 4 # require 4 successes for OK } vrrp_instance DIENST1 { [...] track_script { check_service1_health } Port Checks: vrrp_script chk_http_port { script " Läuft ein bestimmter Prozess? (hier haproxy) vrrp_script chk_haproxy { script "killall -0 haproxy" # cheaper than pidof interval 2 # check every 2 seconds } Interface UP oder DOWN? vrrp_instance DIENST1 { [...] track_interface { eth0 weight 2 # prio = +2 if UP eth2 weight -2 # prio = -2 if DOWN eth3 # no weight, fault if down } ==== Skripte bei Statusänderung ==== vrrp_instance DIENST1 { [...] notify /usr/local/sbin/notify-keepalived.sh # OR: # notify_master "/etc/keepalived/master-backup.sh MASTER" # notify_backup "/etc/keepalived/master-backup.sh BACKUP" # notify_fault "/etc/keepalived/master-backup.sh FAULT" } #!/bin/bash # $1 = "INSTANCE" or "GROUP" TYPE=$1 # $2 = name of instance or group NAME=$2 # $3 = target state of transition, "MASTER", "BACKUP" or "FAULT" STATE=$3 SERVICENAME=XY case $STATE in "MASTER") /bin/systemctl start $SERVICENAME ;; "BACKUP") /bin/systemctl stop $SERVICENAME ;; "FAULT") /bin/systemctl stop $SERVICENAME exit 0 ;; *) /sbin/logger "$SERVICENAME unknown state" exit 1 ;; esac ==== unicast statt multicast ==== Clouds (wie Amazon AWS) unterstützen kein multicast (und Layer2 steht auch nicht zur Verfügung), Daher muss auf unicast umgestellt werden. Auf dem master: use_vmac # forces VRRP virtual router to use a virtual MAC address as described in RFC (00:00:5e:00:01:07 - last octet is VRID). vmac_xmit_base # forces VRRP to use the physical interface MAC address as source when it sends its own packets (avoid any IP filtering by port security). unicast_src_ip 1.2.3.4 # My IP unicast_peer { 5.6.7.8 # peer IP } Auf dem Backup umgekehrt: use_vmac vmac_xmit_base unicast_src_ip 5.6.7.8 # My IP, optional das device: "dev ens3" unicast_peer { 1.2.3.4 # Peer IP } notify_master /etc/keepalived/master.sh script notify_master (/etc/keepalived/master.sh) #!/bin/bash EIP=9.8.7.6 INSTANCE_ID=i-abcd1234 /usr/local/bin/aws ec2 disassociate-address --public-ip $EIP /usr/local/bin/aws ec2 associate-address --public-ip $EIP --instance-id $INSTANCE_ID Umstellung überprüfen: tshark -f "vrrp"oder wenn auth_type AH benutzt wird:tshark -f "ah" * **use_vmac**: VRRP virtual router nehmen eine virtuelle MAC-Adresse, das beugt ARP-caching-Problemen bei den clients. * **vmac_xmit_base**: VRRP nimmt für seine eigene Paket die physikalische MAC der Netzwerkkarte, beugt Problemen mit Filterung bei port security. **Quellen**: * https://blog.rapid7.com/2014/12/03/keepalived-and-haproxy-in-aws-an-exploratory-guide/ * https://docs.syseleven.de/syseleven-stack/de/howtos/l3-high-availability ===== Fehlerbehebung ===== ==== SECURITY VIOLATION - scripts are being executed but script_security not enabled ==== global_defs { ... enable_script_security } ==== Unable to load ipset library - libipset.so.3: cannot open shared object file: No such file or directory ==== apt install ipset #libipset3