octavia_failover_outage.pdf

Transcript

1. Outage of Octavia-LB/Amphora failover
2. MASTER Amphora Failover MASTER amphora failover 100 90 M B VIP VIP move to BACKUP → About 3s outage. Failover 100 90 M B Sep 15 04:11:20 Keepalived_vrrp[4643]: (735bfba6a1ca499b8cfdb5a3e4135198) Receive advertisement timeout Sep 15 04:11:20 Keepalived_vrrp[4643]: (735bfba6a1ca499b8cfdb5a3e4135198) Entering MASTER STATE Sep 15 04:11:20 Keepalived_vrrp[4643]: (735bfba6a1ca499b8cfdb5a3e4135198) setting VIPs. Sep 15 04:11:20 Keepalived_vrrp[4643]: (735bfba6a1ca499b8cfdb5a3e4135198) setting Virtual Routes Sep 15 04:11:20 Keepalived_vrrp[4643]: (735bfba6a1ca499b8cfdb5a3e4135198) setting Virtual Rules Sep 15 04:11:20 Keepalived_vrrp[4643]: (735bfba6a1ca499b8cfdb5a3e4135198) Sending/queueing gratuitous ARPs on eth1 for 192.168.20.15 Sep 15 04:11:20 Keepalived_vrrp[4643]: Sending gratuitous ARP on eth1 for 192.168.20.15 VIP - keepalived is not gracefully stopped, so the VIP fails over to BACKUP after the 3s VRRP timeout.
3. Load balancer Failover Start Load balancer Failover Start reconstruction at Backup side (Because Master has vip) Delete old Backup(B) After create new Backup(B’), assign B’ to “BACKUP” Amphora Start reconstruction at Master Side Delete old Master(M) -> 0-5 sec down at this time 90 M B VIP 100 90 M B Boot B’ VIP 100 90 90 M B B’ VIP 100 Delete VIP 90 M B’ VIP 100 VIP 90 M B’ VIP VIP 100 100 90 M’ M Boot M’ B’ Delete VIP After create new Master(M’), assign M’ to “MASTER” Amphora Load balancer Failover 100 VIP VIP 100 90 M’ B’ VIP - Old MASTER and new BACKUP cannot communicate each other. - Both hold the VIP, likely causing ARP conflicts. - Traffic shifts to the new BACKUP within 5s, likely due to garp_master_refresh=5.