etcd @ Strange Loop 2014

Transcript

1. etcd @coreoslinux
2. About Me CTO/CO-FOUNDER systems engineer @brandonphilips github.com/philips
3. /etc distributed
4. open source software highly available and reliable sequentially consistent watchable exposed via HTTP runtime reconfigurable
5. Data Store API -X GET Get Wait -X PUT Put Create CAS -X DELETE Delete CAD
6. etcd basics clusters and bootstrapping
7. Leader Follower etcd Cluster
8. bootstrapping Candidate
9. GET discovery.etcd.io/new
10. discovery.etcd.io/6eadeac2 6eadeac2d
11. 6eadeac2d/state CREATE
12. CREATE 6eadeac2d/state
13. CREATE 6eadeac2d/state Key Value Index state started 5890 n0 10.0.2.0 5891
14. 6eadeac2d/state Key Value Index state started 5890 n0 10.0.2.1 5891 n1 10.0.2.4 5898 ...
15. bootstrapped Leader Follower
17. 6eadeac2d/state CREATE
19. PUT /6eadeac2d/state?prevExist=false ‘started’
20. 1 /6eadeac2d/state started PUT /6eadeac2d/state?prevExist=false ‘started’ Entry
21. 1 /6eadeac2d/state started PUT /6eadeac2d/state?prevExist=false ‘started’ Index
22. 1 /6eadeac2d/state started PUT /6eadeac2d/state?prevExist=false ‘started’ Key
23. 1 /6eadeac2d/state started PUT /6eadeac2d/state?prevExist=false ‘started’ Value
24. 1 2 3 4 { Log
25. 1 2 3 4 Entries
26. 1 2 3 4 Indexes
27. Sequential Consistency Operations* are atomically executed in the same sequential order on all machines.
28. 1 1 1 2 Pet=dog Pet=cat Pet=cat 1 2 PUT Pet = cat PUT Pet = dog
29. 1 1 1 2 2 1 2 PUT Pet = cat PUT Pet = dog Pet=dog Pet=dog Pet=cat
30. 1 1 1 2 2 2 1 2 PUT Pet = cat PUT Pet = dog Pet=dog Pet=dog Pet=dog
31. Sequential Consistency Real-time
32. 1 1 1 2 GET Pet @ 10:00.0 -> 1[cat]!? GET Pet @ 10:00.0 -> 2[dog] 2
33. 1 1 1 2 2 2 GET Pet @ 10:00.1 -> 1[dog]
34. Sequential Consistency Index Time
35. 1 1 1 2 GET Pet @ 2 -> blocking GET Pet @ 2 -> 2[dog] 2
36. 1 1 1 2 GET Pet @ 2 -> 2[dog] 2 2
37. etcd guarantees that a get at index X will always return the same result. Avoid thinking in terms of real time because with network latency the result is always out-of-date.
38. Quorum GETs GET via Raft
39. 1 1 1 2 2
40. 1 1 1 2 QGET A 2
41. 1 1 1 2 QGET A -> 2[dog] 2 2
42. 1 1 1 2 QGET A -> 2[dog] 2 2 3 3
43. Watchable Changes HTTP Long-poll
44. 1 2 3 > GET asdf?waitIndex=4&wait=true HTTP/1.1 > Accept: */* > < HTTP/1.1 200 OK < Content-Type: application/json < X-Etcd-Index: 3 < X-Raft-Index: 97 < X-Raft-Term: 0 < BLOCK
45. 1 2 3 4 > GET asdf?waitIndex=4&wait=true HTTP/1.1 > Accept: */* > < HTTP/1.1 200 OK < Content-Type: application/json < X-Etcd-Index: 3 < X-Raft-Index: 97 < X-Raft-Term: 0 < {"action":"set","node":{"key":"/asdf","value":"foobar"," modifiedIndex":4,"createdIndex":4}}
46. 1 2 3 4 > GET asdf?waitIndex=4&wait=true HTTP/1.1 > Accept: */* > < HTTP/1.1 200 OK < Content-Type: application/json < X-Etcd-Index: 4 < X-Raft-Index: 516 < X-Raft-Term: 0 < {"action":"set","node":{"key":"/asdf","value":"foobar"," modifiedIndex":4,"createdIndex":4}}
47. Event History History isn’t forever, prepare!
48. Availability In a 2F+1 cluster tolerate F machine failures
49. Available
50. Available
51. Available
52. Unavailable
53. Master Election Fast recovery (5-10*typical RTT) from temporarily unavailable
54. Available Leader Follower
55. Leader Follower Available
56. Leader Follower Temporarily Unavailable
57. Leader Follower Available
58. Durable log files, snapshots and backups
59. Time To Live Keys Usage for Leader Election
60. Idx Key Value Expiration Time 18 sched m3 Sept 18 2:11:30
61. Idx Key Value Expiration Time 18 sched m3 Sept 18 2:11:30 schedlr m3
62. cas(sched, 18, m3) cas(sched, 18, m3) schedlr m3 Idx Key Value Expiration Time 18 sched m3 Sept 18 2:11:30
63. cas(sched, 30, m3) cas(sched, 30, m3) schedlr m3 Idx Key Value Expiration Time 30 sched m3 Sept 18 2:12:50
64. cas(sched, 45, m3) cas(sched, 45, m3) schedlr m3 Idx Key Value Expiration Time 45 sched m3 Sept 18 2:13:30
65. sync(2:13:00) sync(2:13:00) Idx Key Value Expiration Time 45 sched m3 Sept 18 2:13:30
66. sync(2:13:15) sync(2:13:15) Idx Key Value Expiration Time 45 sched m3 Sept 18 2:13:30
67. sync(2:13:30) sync(2:13:30) Idx Key Value Expiration Time 45 sched m3 Sept 18 2:13:30
68. sync(2:13:30) sync(2:13:30) Idx Key Value Expiration Time
69. create(sched, m5) create(sched, m5) Idx Key Value Expiration Time 50 sched m5 Sept 18 2:13:35 schedlr m5
70. Applications locksmith
73. Cluster Wide Reboot Lock 1. Need reboot to reboot? Decrement the semaphore key atomically with etcd. 2. manager.Reboot() and wait... 3. After rebooting increment the semaphore key in etcd atomically.
74. Applications kubernetes and fleet
75. You Scheduler API Scheduler Machine(s)
76. Cluster Work Scheduling 1. Cluster API writes desired work into etcd keyspace. 2. Agents running on individual machines pick up work assigned to them. 3. Agents report where work is running and current status.
77. Applications vulcan, confd, dns and distributed git
78. Raft etcd’s consensus algorithm
79. github.com/coreos/etcd/raft contributed to the raft paper a (now recommended) configuration protocol.
80. n := raft.StartNode(0x01, []int64{0x02, 0x03}, 3, 1)
81. n := raft.StartNode(0x01, []int64{0x02, 0x03}, 3, 1)
82. n := raft.StartNode(0x01, []int64{0x02, 0x03}, 3, 1)
83. func recvRaftRPC( ctx context.Context, m raftpb.Message) { n.Step(ctx, m) }
84. func recvRaftRPC( ctx context.Context, m raftpb.Message) { n.Step(ctx, m) }
85. func recvRaftRPC( ctx context.Context, m raftpb.Message) { n.Step(ctx, m) }
86. func recvRaftRPC( ctx context.Context, m raftpb.Message) { n.Step(ctx, m) }
87. n.Propose(ctx, data)
88. req = Req{ Path = /Pet Val = Dog } n.Propose(ctx, req)
89. for { select { ... case rd := <-s.Node.Ready(): saveToStable(rd.State, rd.Entries) process(rd.CommittedEntries) send(rd.Messages) ... }
90. Mistakes so far...
91. Log files Filesystems truncate and corrupt data. Solutions: • Must use checksumming in the file to ensure sanity • Throwing out broken log files must be handled by the server
92. etcd machine naming Trusted users to manage unique names across the cluster. This went poorly. • Misconfiguration from bugs • Misconfiguration by users • Machine cloning on the cloud Solution: etcd data-dir owns a unique uuid.
93. sync() in the cloud Slow, slow, slow: • User #1 OpenStack on spinning disk: 6s • User #2 AWS EBS backed: 1.5s Solution: • Tune etcd to expect this long latency. • Write batching and handling of behind machines.
94. Towards etcd 1.0 mvcc data store
95. More efficient handling of events Non-blocking snapshots Better read performance when contention is high
96. Towards etcd 1.0 read/write scalability
97. Handle read and watch Eventual consistency w/ time bound Reduce the pressure on the actual cluster Watch is expensive Abusive clients are normal Read-only proxy
98. Fast Promotion Standbys
99. Fast Promotion Leader Follower Standby
100. Fast Promotion Leader Follower Standby
101. Fast Promotion Leader Follower Standby
102. Thanks we’re hiring we like pull requests github.com/coreos/etcd
103. mount etcd-global at /global of etcd-local Mounts
104. Read-write proxy Handle all HTTP requests Reduce CPU load on the cluster HTTP parsing Command Encoding/Decoding Limit the number of incoming connections
105. Scalability Read-write proxy Provide the same garatueen as the actual etcd cluster. Cache invalidation is hard
106. Scalability Core Cluster Clients read/watch write