Iago: High Availability Postgres with etcd (CoreOS Fest 2017)

Chain’s enterprise blockchain networks rely on etcd an Postgres for storage. We use Postgres to store blockchain data, and rely on Postgres’s replication features to ensure that clients can always access their assets and create transactions. I wrote a service that manages Postgres replicas. This service was heavily inspired by Joyent’s Manatee project, but uses different technology; namely, Go and etcd instead of Node and Zookeeper.

In this talk, I’ll introduce blockchains and a couple of different consensus algorithms before covering how we use etcd to coordinate replicas in our high-availability management system. I’ll talk about how we configured both etcd and Postgres, walk through the high-level design as well as some code, demonstrate failover, and show how blockchain services communicate with this system.

Transcript

1. Iago: High-Availability Databases with etcd C O R E O S F E S T J U N E 2 0 1 7 Tess Rinearson, Software Engineer at Chain @_tessr
2. • Blockchains and Network Consensus • Postgres Replication • etcd and Raft • Iago Itself • What happens when things go wrong? • Future Work • Why “Iago” 2
3. What is a blockchain? 3
4. What is a blockchain? 3
5. 4 Block 1 tx tx tx tx tx Block 2 tx tx prev block hash tx tx tx
6. 4 Block 1 tx tx tx tx tx Block 2 tx tx prev block hash tx tx tx tx tx prev block hash tx tx tx
7. 5 block block block bloc
9. 6 block block block block block block block block block block block block block block block block block block
10. 7 http://new.abb.com/enterprise-software/enterprise-software-for-mining
11. 7 out of scope http://new.abb.com/enterprise-software/enterprise-software-for-mining
14. Blockchain Attributes: Pick n 9
15. Blockchain Attributes: Pick n 1.Replicated Ledger 9
16. Blockchain Attributes: Pick n 1.Replicated Ledger 2.Cryptographic Commitment History 9
17. Blockchain Attributes: Pick n 1.Replicated Ledger 2.Cryptographic Commitment History 3.Transactions Authenticated by Public Key Crypto 9
18. Blockchain Attributes: Pick n 1.Replicated Ledger 2.Cryptographic Commitment History 3.Transactions Authenticated by Public Key Crypto 4.Accounting Rules (UTXO model) 9
19. Blockchain Attributes: Pick n 1.Replicated Ledger 2.Cryptographic Commitment History 3.Transactions Authenticated by Public Key Crypto 4.Accounting Rules (UTXO model) 5.Shared Access 9
20. Blockchain Attributes: Pick n 1.Replicated Ledger 2.Cryptographic Commitment History 3.Transactions Authenticated by Public Key Crypto 4.Accounting Rules (UTXO model) 5.Shared Access 6.Proof of Work 9
23. 11 + Chain Protocol Chain Core
26. Infrastructure for Permissioned, Multi-Asset Blockchains 14
27. Federated Consensus • Designed for permissioned blockchains • One node is the generator: • Aggregates transactions and proposes blocks to the signer nodes • Some number of nodes are signers • Verify that block is valid and doesn’t conflict with another block at the same height, and add their own signatures to the block • Network requires m-of-n signers to sign each block (m,n specified by consensus program) • Remaining nodes are participants • Can audit blocks but network doesn’t rely on them 15
28. 16 Generator Signer Signer Signer Participant Participant
29. Federated Consensus • Safety Guarantees: • There is only one valid block at a given height, assuming that no more than (2M-N-1) signers violate protocol • e.g. for a 3-of-4 network, 2 signers would have to maliciously fail for network failure • Protocol supports consensus improvements 17
30. 18 Generator and signer nodes must be robust.
32. 20 A SINGLE NODE:
33. 21 Postgres: How and Why?
34. 22 Synchronous Replication Asynchronous Replication Send Send Ack aka Streaming Replication
35. cored 23 Primary Sync Rep Async Rep cored A B C
36. cored 24 Primary Sync Rep cored A B C
37. 25 Iago
38. _.---.._ _ _.-' \ \ ''-. .' '-,_.-' / / / '''. ( _ o : '._ .-' '-._ \ \- ---] '-.___.-') )..-' (_/ 26
39. etcd • Distributed key-value store built by CoreOS ❤ • Uses Raft consensus algorithm: https://raft.github.io/raft.pdf • Not Byzantine Fault Tolerant • …but that’s okay, because this is internal to a single node 27
40. Raft 28
41. Raft • Consensus: “How do we ensure that multiple nodes apply state changes in the same order?” • Breaks “consensus” into three independent subproblems: • Leader election • Log replication • Safety 29
42. Raft • Consensus: “How do we ensure that multiple nodes apply state changes in the same order?” • Breaks “consensus” into three independent subproblems: • Leader election • Log replication • Safety 29 out of scope
43. Anatomy of an Iago Peer 30 Iago Sitter etcd
44. // RunPeer runs a single Iago peer. It will wait until at least two other // peers are also online before it launches Postgres and starts its healthcheck. // PostgresPort and DatabaseName must be set before calling RunPeer. func RunPeer(ctx context.Context, kapi client.KeysAPI) error { err := createEphemeralNode(ctx, kapi) if err != nil { return err } // refreshEphemeralNode gets called every at increments of // nodeTTL / 2, so the ephemeral node's ttl is refreshed after // half of it has passed. go func() { for range time.Tick(nodeTTL / 2) { log.Println("refreshing ttl") refreshEphemeralNode(ctx, kapi) } }() return runPeer(ctx, kapi) } 31
45. Primary S e Sync Rep S e Async Rep S e Async Rep 2 S e Asy S
46. Cluster State • Primary • Sync rep • Ordered list of async reps • Generation • Initial WAL position 33
47. func runSyncPeerMonitor(ctx context.Context, kapi client.KeysAPI, p *pg.PG, cluster *clusterState) error { primaryWatch, genWatch := watchAll(ctx, kapi, cluster.Primary, cluster.generation) var err error for { select { case primaryChange := <-primaryWatch: if isExpiry(primaryChange) { logChange(primaryChange, "primary has gone offline") p.Promote(ctx) return runPrimaryMonitor(ctx, kapi, p, cluster) } case genChange := <-genWatch: logChange(genChange, "generation changed; restarting") return runPeer(ctx, kapi) case pgErr := <-p.Err: logError(pgErr, "failed the pg healthcheck") stepDownFatal(ctx, kapi, p, "old sync peer failed the pg healthcheck") } } } 34
48. func runSyncPeerMonitor(ctx context.Context, kapi client.KeysAPI, p *pg.PG, cluster *clusterState) error { primaryWatch, genWatch := watchAll(ctx, kapi, cluster.Primary, cluster.generation) var err error for { select { case primaryChange := <-primaryWatch: if isExpiry(primaryChange) { logChange(primaryChange, "primary has gone offline") p.Promote(ctx) return runPrimaryMonitor(ctx, kapi, p, cluster) } case genChange := <-genWatch: logChange(genChange, "generation changed; restarting") return runPeer(ctx, kapi) case pgErr := <-p.Err: logError(pgErr, "failed the pg healthcheck") stepDownFatal(ctx, kapi, p, "old sync peer failed the pg healthcheck") } } } 35
49. func runSyncPeerMonitor(ctx context.Context, kapi client.KeysAPI, p *pg.PG, cluster *clusterState) error { primaryWatch, genWatch := watchAll(ctx, kapi, cluster.Primary, cluster.generation) var err error for { select { case primaryChange := <-primaryWatch: if isExpiry(primaryChange) { logChange(primaryChange, "primary has gone offline") p.Promote(ctx) return runPrimaryMonitor(ctx, kapi, p, cluster) } case genChange := <-genWatch: logChange(genChange, "generation changed; restarting") return runPeer(ctx, kapi) case pgErr := <-p.Err: logError(pgErr, "failed the pg healthcheck") stepDownFatal(ctx, kapi, p, "old sync peer failed the pg healthcheck") } } } 36
50. 37 func runSyncPeerMonitor(ctx context.Context, kapi client.KeysAPI, p *pg.PG, cluster *clusterState) error { primaryWatch, genWatch := watchAll(ctx, kapi, cluster.Primary, cluster.generation) var err error for { select { case primaryChange := <-primaryWatch: if isExpiry(primaryChange) { logChange(primaryChange, "primary has gone offline") p.Promote(ctx) return runPrimaryMonitor(ctx, kapi, p, cluster) } case genChange := <-genWatch: logChange(genChange, "generation changed; restarting") return runPeer(ctx, kapi) case pgErr := <-p.Err: logError(pgErr, "failed the pg healthcheck") stepDownFatal(ctx, kapi, p, "old sync peer failed the pg healthcheck") } } }
51. Iago Using Iago 38 cored cored SQL queries Iago etcd hostname resolution
52. What happens when something goes wrong? 39
53. If the Primary goes offline… 40 cored cored A B C D Primary Sync Rep Async Reps Generation Ephemeral Nodes A B C, D 1 A, B, C, D
54. If the Primary goes offline… 40 cored cored A B C D Primary Sync Rep Async Reps Generation Ephemeral Nodes A B C, D 1 A, B, C, D
55. If the Primary goes offline… 41 cored cored A B C D Primary Sync Rep Async Reps Generation Ephemeral Nodes A B C, D 1 , B, C, D
56. If the Primary goes offline… 41 cored cored A B C D Primary Sync Rep Async Reps Generation Ephemeral Nodes A B C, D 1 , B, C, D
57. If the Primary goes offline… 42 cored cored A B C D Primary Sync Rep Async Reps Generation Ephemeral Nodes A B C, D 1 , B, C, D
58. If the Primary goes offline… 42 cored cored A B C D Primary Sync Rep Async Reps Generation Ephemeral Nodes A B C, D 1 , B, C, D
59. If the Primary goes offline… 42 cored cored A B C D Primary Sync Rep Async Reps Generation Ephemeral Nodes A B C, D 1 , B, C, D Primary Sync Rep Async Reps Generation Ephemeral Nodes B C D 2 , B, C, D
60. If the Primary goes offline… 42 cored cored A B C D Primary Sync Rep Async Reps Generation Ephemeral Nodes A B C, D 1 , B, C, D Primary Sync Rep Async Reps Generation Ephemeral Nodes B C D 2 , B, C, D
61. If the Primary goes offline… 43 cored cored A B C D Primary Sync Rep Async Reps Generation Ephemeral Nodes A B C, D 1 , B, C, D Primary Sync Rep Async Reps Generation Ephemeral Nodes B C D 2 , B, C, D
62. If the Primary goes offline… 43 cored cored A B C D Primary Sync Rep Async Reps Generation Ephemeral Nodes A B C, D 1 , B, C, D Primary Sync Rep Async Reps Generation Ephemeral Nodes B C D 2 , B, C, D
63. If the Primary goes offline… 43 cored cored A B C D Primary Sync Rep Async Reps Generation Ephemeral Nodes A B C, D 1 , B, C, D Primary Sync Rep Async Reps Generation Ephemeral Nodes B C D 2 , B, C, D
64. If the Primary goes offline… 44 cored cored A B C D Primary Sync Rep Async Reps Generation Ephemeral Nodes A B C, D 1 , B, C, D Primary Sync Rep Async Reps Generation Ephemeral Nodes B C D 2 , B, C, D
65. If the Primary goes offline… 44 cored cored A B C D Primary Sync Rep Async Reps Generation Ephemeral Nodes A B C, D 1 , B, C, D Primary Sync Rep Async Reps Generation Ephemeral Nodes B C D 2 , B, C, D
66. If the Primary goes offline… 44 cored cored A B C D Primary Sync Rep Async Reps Generation Ephemeral Nodes A B C, D 1 , B, C, D Primary Sync Rep Async Reps Generation Ephemeral Nodes B C D 2 , B, C, D
67. If the Primary goes offline… 45 cored cored A B C D Primary Sync Rep Async Reps Generation Ephemeral Nodes A B C, D 1 , B, C, D Primary Sync Rep Async Reps Generation Ephemeral Nodes B C D 2 , B, C, D
68. If the Primary goes offline… 46 cored cored A B C D Primary Sync Rep Async Reps Generation Ephemeral Nodes A B C, D 1 , B, C, D Primary Sync Rep Async Reps Generation Ephemeral Nodes B C D 2 B, C, D
69. 47 cored cored A B C D Primary Sync Rep Async Reps Generation Ephemeral Nodes A B C, D 1 A, B, C, D What if the Sync Rep goes offline (but doesn’t know it)?
70. 47 cored cored A B C D Primary Sync Rep Async Reps Generation Ephemeral Nodes A B C, D 1 A, B, C, D What if the Sync Rep goes offline (but doesn’t know it)?
71. 48 cored cored A B C D Primary Sync Rep Async Reps Generation Ephemeral Nodes A B C, D 1 A, , C, D What if the Sync Rep goes offline (but doesn’t know it)?
72. 48 cored cored A B C D Primary Sync Rep Async Reps Generation Ephemeral Nodes A B C, D 1 A, , C, D What if the Sync Rep goes offline (but doesn’t know it)?
73. 49 cored cored A B C D Primary Sync Rep Async Reps Generation Ephemeral Nodes A C D 2 A, , C, D What if the Sync Rep goes offline (but doesn’t know it)?
74. 50 cored cored A B C D Primary Sync Rep Async Reps Generation Ephemeral Nodes A C D 2 A, , C, D What if the Sync Rep goes offline (but doesn’t know it)?
75. 50 cored cored A B C D Primary Sync Rep Async Reps Generation Ephemeral Nodes A C D 2 A, , C, D What if the Sync Rep goes offline (but doesn’t know it)?
76. 50 cored cored A B C D Primary Sync Rep Async Reps Generation Ephemeral Nodes A C D 2 A, , C, D What if the Sync Rep goes offline (but doesn’t know it)?
77. 51 cored cored A B C D Primary Sync Rep Async Reps Generation Ephemeral Nodes A C D 2 A, , C, D What if the Sync Rep goes offline (but doesn’t know it)?
78. 51 cored cored A B C D Primary Sync Rep Async Reps Generation Ephemeral Nodes A C D 2 A, , C, D What if the Sync Rep goes offline (but doesn’t know it)?
79. 52 cored cored A B C D Primary Sync Rep Async Reps Generation Ephemeral Nodes A C D 2 A, B, C, D What if the Sync Rep goes offline (but doesn’t know it)?
80. 52 cored cored A B C D Primary Sync Rep Async Reps Generation Ephemeral Nodes A C D 2 A, B, C, D What if the Sync Rep goes offline (but doesn’t know it)?
81. A Very Hairy Failure Case 53 cored cored A B C D Primary Sync Rep Async Reps Gen Ephemeral Nodes A B C, D 1 A, B, C, D
82. A Very Hairy Failure Case 53 cored cored A B C D Primary Sync Rep Async Reps Gen Ephemeral Nodes A B C, D 1 A, B, C, D tx
83. A Very Hairy Failure Case 53 cored cored A B C D Primary Sync Rep Async Reps Gen Ephemeral Nodes A B C, D 1 A, B, C, D tx tx
84. A Very Hairy Failure Case 53 cored cored A B C D Primary Sync Rep Async Reps Gen Ephemeral Nodes A B C, D 1 A, B, C, D tx tx ack
85. A Very Hairy Failure Case 53 cored cored A B C D Primary Sync Rep Async Reps Gen Ephemeral Nodes A B C, D 1 A, B, C, D tx tx ack ack
86. A Very Hairy Failure Case 53 cored cored A B C D Primary Sync Rep Async Reps Gen Ephemeral Nodes A B C, D 1 A, B, C, D tx tx ack ack X
87. A Very Hairy Failure Case 53 cored cored A B C D Primary Sync Rep Async Reps Gen Ephemeral Nodes A B C, D 1 A, B, C, D tx tx ack ack X
88. A Very Hairy Failure Case 54 cored cored A B C D tx tx ack ack Primary Sync Rep Async Reps Gen Ephemeral Nodes B C D 2 B, C, D
89. A Very Hairy Failure Case 54 cored cored A B C D tx tx ack ack Primary Sync Rep Async Reps Gen Ephemeral Nodes B C D 2 B, C, D
90. Cluster State, revisited • Primary • Sync rep • Ordered list of async reps • Generation • Initial WAL position 55
91. A Very Hairy Failure Case, with the WAL 56 cored cored A
 n B n C n D n Primary Sync Rep Async Reps Gen Ephemeral Nodes WAL Position A B C, D 1 A, B, C, D n Local WAL Position
92. A Very Hairy Failure Case, with the WAL 56 cored cored A
 n B n C n D n Primary Sync Rep Async Reps Gen Ephemeral Nodes WAL Position A B C, D 1 A, B, C, D n tx Local WAL Position
93. A Very Hairy Failure Case, with the WAL 56 cored cored A
 n B n C n D n Primary Sync Rep Async Reps Gen Ephemeral Nodes WAL Position A B C, D 1 A, B, C, D n tx tx Local WAL Position
94. A Very Hairy Failure Case, with the WAL 56 cored cored A
 n B n C n D n Primary Sync Rep Async Reps Gen Ephemeral Nodes WAL Position A B C, D 1 A, B, C, D n tx tx ack Local WAL Position
95. 57 cored cored A
 n+1 B n+1 C n D n Primary Sync Rep Async Reps Gen Ephemeral Nodes WAL Position A B C, D 1 A, B, C, D n tx tx ack Local WAL Position A Very Hairy Failure Case, with the WAL
96. 58 cored cored A
 n+1 B n+1 C n D n Primary Sync Rep Async Reps Gen Ephemeral Nodes WAL Position A B C, D 1 A, B, C, D n tx tx ack Local WAL Position A Very Hairy Failure Case, with the WAL
97. 58 cored cored A
 n+1 B n+1 C n D n Primary Sync Rep Async Reps Gen Ephemeral Nodes WAL Position A B C, D 1 A, B, C, D n tx tx ack Local WAL Position ack A Very Hairy Failure Case, with the WAL
98. 58 cored cored A
 n+1 B n+1 C n D n Primary Sync Rep Async Reps Gen Ephemeral Nodes WAL Position A B C, D 1 A, B, C, D n tx tx ack Local WAL Position ack A Very Hairy Failure Case, with the WAL
99. 59 cored cored A
 n+1 B n+1 C n D n Primary Sync Rep Async Reps Gen Ephemeral Nodes WAL Position A B C, D 1 , B, C, D n tx tx ack Local WAL Position ack A Very Hairy Failure Case, with the WAL
100. 59 cored cored A
 n+1 B n+1 C n D n Primary Sync Rep Async Reps Gen Ephemeral Nodes WAL Position A B C, D 1 , B, C, D n tx tx ack Local WAL Position ack A Very Hairy Failure Case, with the WAL
101. 60 cored cored A
 n+1 B n+1 C n D n Primary Sync Rep Async Reps Gen Ephemeral Nodes WAL Position B C D 2 B, C, D n+1 tx tx ack Local WAL Position ack A Very Hairy Failure Case, with the WAL
102. 61 cored cored A
 n+1 B n+1 C n D n Primary Sync Rep Async Reps Gen Ephemeral Nodes WAL Position B C D 2 B, C, D n+1 tx tx ack Local WAL Position ack A Very Hairy Failure Case, with the WAL
103. 61 cored cored A
 n+1 B n+1 C n D n Primary Sync Rep Async Reps Gen Ephemeral Nodes WAL Position B C D 2 B, C, D n+1 tx tx ack Local WAL Position ack A Very Hairy Failure Case, with the WAL
104. 61 cored cored A
 n+1 B n+1 C n D n Primary Sync Rep Async Reps Gen Ephemeral Nodes WAL Position B C D 2 B, C, D n+1 tx tx ack Local WAL Position ack A Very Hairy Failure Case, with the WAL
105. Future Work 62
106. 63 http://archive.datacenterdynamics.com/focus/archive/2014/03/converged-nyc-time-warner-cables-own-data-center-efficiency-metric
107. Operational Complexity >>> Code Complexity 64
108. cored Iago Iago Iago currently 65 cored cored Iago sitter etcd
109. Iago cored cored Iago Iago without etcd 66 Iago sitter cored raft
110. Iago cored cored Iago Iago without etcd 66 Iago sitter cored raft /create-asset /build-transaction /raft/join /raft/msg
112. Why “Iago”? 68
113. We’re hiring! 69
114. Tess Rinearson tess@chain.com 70 @_tessr