Failsafe Patroni 3.0

What?! Patroni is the tool for implementing PostgreSQL high-availability and automatic failover, isn't it already failsafe on its own?

If you are an experienced Patroni user, you know that it relies on DCS (Distributed Configuration Store) to keep PostgreSQL cluster information in a consistent way ensuring that there is only one leader at a time. And of course, you also know that primary is demoted if Patroni can’t update the leader lock when DCS (Etcd, Consul, Zookeeper, or Kubernetes API) is not accessible or experiencing temporary problems, which could be very frustrating.

In this talk we will introduce a new Patroni feature – DCS failsafe mode, which is aimed at keeping primary running in case of a DCS failure. We will reveal some ideas behind, share important implementation details, do a live demo, and give guidance on considerations whether the feature should be used in specific environments, or it is better to refrain from it.

Transcript

1. Failsafe Patroni 3.0 Prague PostgreSQL Developer Day Alexander Kukushkin & Polina Bungina 2023 • 02 • 01 Presented by
2. About us • Principal Software Engineer @Microsoft • The Patroni guy • akukushkin@microsoft.com • Twitter: @cyberdemn • Software Engineer @ZalandoTech • polina.bungina@zalando.de • Twitter: @hugh_capet Polina Bungina Alexander Kukushkin
3. Agenda Introduction to Patroni Observer problem Demo 1 DCS dailsafe feature Demo 2 Conclusion
4. 4 • Service-Level Agreement (SLA) • Recovery point objective (RPO) • Recovery time objective (RTO) Do we need it at all?
5. 5 • Cluster state stored in Distributed Configuration Store (DCS) ◦ ZooKeeper ◦ Etcd ◦ Consul ◦ Kubernetes control-plane • Session/TTL to expire data (i.e. leader key) • Atomic CAS operations • Watches for important keys Architecture overview
6. 6 Leader race DCS CREATE “/leader”, “A”, ttl=30, prevExists=False CREATE “/leader”, “B”, ttl=30, prevExists=False Success Fail A B promote
7. 7 Normal operational mode DCS UPDATE “/leader”, “A”, ttl=30 prevValue=”A” Success A B WATCH(“/leader”) primary replica
8. 8 Normal operational mode DCS NOTIFY(“/leader”, expired=True) replica B
9. 9 Normal operational mode DCS B promote primary CREATE “/leader”, “B”, ttl=30, prevExists=False Success
10. 10 DCS can't be accessed UPDATE “/leader”, “A”, ttl=30 prevValue=”A” Fail A B primary replica
11. 11 DCS can't be accessed UPDATE “/leader”, “A”, ttl=30 prevValue=”A” Success A B primary replica
12. 12 Why did update fail? • DCS is down? • Network issues?
13. 13 Network partition UPDATE “/leader”, “A”, ttl=30 prevValue=”A” Fail A primary B replica
14. 14 primary Leader key expired UPDATE “/leader”, “A”, ttl=30 prevValue=”A” Fail B CREATE “/leader”, “B”, ttl=30 prevExists=False Success promote A primary
15. 15 So, to be on the safe side… UPDATE “/leader”, “A”, ttl=30 prevValue=”A” Fail A demote primary B replica
16. 16 So, to be on the safe side… A replica primary B promote CREATE “/leader”, “B”, ttl=30 prevExists=False Success
17. 17 Still not perfect A replica CREATE “/leader”, “B”, ttl=30 prevExists=False Fail B replica
19. 19 • Etcd, Zookeeper - very unlikely (if configured correctly) • Consul - local agent is a SPoF! • Kubernetes control-plane - typical SLA for managed services is 99.95% (4h22m per year) DCS down
20. 20 What if… A B standby primary Do you see DCS? NO
21. 21 Split-brain! A B standby primary Do you see DCS? NO C primary
22. 22 • Continue to run as primary if can see ALL Patroni nodes • Don’t allow “unknown” nodes to become primary! Idea
23. 23 • Patroni clusters are mainly “static”, but nodes can join and leave • If topology changes - write list of of Patroni nodes names to DCS • Nodes outside of this list are “unknown” and not allowed to become primary “Unknown” node?
24. 24 DCS failsafe mode UPDATE “/leader”, “A”, ttl=30 Fail node1 nodeN 1 POST /failsafe 2 [cache primary data for ttl] 3 4 200 OK /failsafe: node1, node2, …, nodeN
25. 25 • Introduce /failsafe key - list of currently presented members in the cluster ◦ Maintained by the leader ◦ Cache its value in Patroni (on all nodes) • Introduce POST /failsafe REST API endpoint ◦ Payload contains information about primary and permanent logical slots ◦ Primary checks response code and demotes if not 200 Implementation details
26. 26 • Replica disqualifies itself from the leader race if not listed in the DCS /failsafe key • Primary executes the failsafe check only with nodes from the failsafe list ◦ Continue as primary if ALL nodes are accessible ◦ Otherwise demote • Replicas call pg_advance_replication_slot() if necessary. Implementation details (continue)
27. 27 $ patronictl edit-config --- +++ @@ -4,3 +4,4 @@ use_pg_rewind: true retry_timeout: 10 ttl: 30 +failsafe_mode: on Apply these changes? [y/N]: y Configuration changed $ etcdctl get /service/batman/failsafe { "postgresql0": "http://127.0.0.1:8008/patroni", "postgresql1": "http://127.0.0.1:8009/patroni" } How to enable failsafe mode $ curl http://127.0.0.1:8008/failsafe { "postgresql0": "http://127.0.0.1:8008/patroni", "postgresql1": "http://127.0.0.1:8009/patroni" }
28. 28 $ curl -s http://127.0.0.1:8008/patroni | jq . { "state": "running", "postmaster_start_time": "2023-01-26 16:11:04.848424+00:00", "role": "master", "server_version": 150001, "xlog": {"location": 67419584}, "timeline": 2, "replication": [ {"usename": "replicator", "application_name": "postgresql1", "client_addr": "127.0.0.1", "state": "streaming", "sync_state": "async", "sync_priority": 0}], "cluster_unlocked": true, "failsafe_mode_is_active": true, "dcs_last_seen": 1674749503, "database_system_identifier": "7192993973708324892", "patroni": {"version": "3.0.0", "scope": "demo"} } Monitoring
30. 30 • When nodes could change their names after “restart” (with old storage) ◦ If ALL nodes are restarted at the same time cluster will not recover automatically Example: • K8s deployment without StatefulSet ◦ Crunchy Postgres Operator (PGO) When not to use it
31. 31 Thank you! Questions?