Russell Brown - Oh, you wanted to increment a counter? Good luck with that!

Slide 1

Slide 1 text

basho "Oh, you wanted to *increment a counter*?! Good luck with that!” [1] Russell Brown [email protected] [1] https://twitter.com/lindsey/status/575006945213485056

Slide 2

Slide 2 text

basho https://twitter.com/lindsey/status/575006945213485056

Slide 3

Slide 3 text

basho Jokes ARE better explained

Slide 4

Slide 4 text

basho https://twitter.com/dizzyd/status/35566447145127936

Slide 5

Slide 5 text

basho Number one feature request 2011 - Counters

Slide 6

Slide 6 text

basho HOW HARD CAN IT BE?

Slide 7

Slide 7 text

No content

Slide 8

Slide 8 text

basho

Slide 9

Slide 9 text

basho

Slide 10

Slide 10 text

basho Leslie Lamport ``A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable''

Slide 11

Slide 11 text

basho https://en.wikipedia.org/wiki/Distributed_computing A distributed system is a software system in which components located on networked computers communicate and coordinate their actions by passing messages. The components interact with each other in order to achieve a common goal.

Slide 12

Slide 12 text

basho Distributed Counters - WHY? • Many many updates - scale/perf • Fault tolerance - machines crash • Latency - local data centres • Containers/“microservices”

Slide 13

Slide 13 text

basho CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP

Slide 14

Slide 14 text

basho CAP • Consistency • Availability • Partition Tolerance

Slide 15

Slide 15 text

Replica A Replica B Replica C Client X Client Y add 1 add 1 NO!!!! :( CP

Slide 16

Slide 16 text

Replica A Replica B Replica C Client X Client Y Add 1 C’ Add 1 AP

Slide 17

Slide 17 text

basho Why Are You Counting? • Money / Votes • Real world resources (seats on a place, rental cars) • Unique IDs • Events (likes, hits, plays)

Slide 18

Slide 18 text

basho “Strong Consistency” • ZooKeeper/HyperDex-Replicant/Riak-Ensemble • ZAB/Chain Replication/Paxos/Raft/VSR • Compare and swap • Starvation • Single data centre

Slide 19

Slide 19 text

basho Not Really Counting - IDs • Ticket Servers - ﬂickr ticket server • Flake - boundary • SnowFlake - Twitter

Slide 20

Slide 20 text

basho Eventually Consistent • Riak/Cassandra etc • CRDTs Counters • Drift/Partial failure • Idempotence • Bounded-Counters - reservations

Slide 21

Slide 21 text

basho Probabilistic • HyperLogLog etc • Bounded error estimates • Trade off space/speed for accuracy • Redis (soon to-be riak) and others

Slide 22

Slide 22 text

basho https://twitter.com/lindsey/status/575006945213485056

Slide 23

Slide 23 text

basho Take Home • Understand your requirements • Know What you are counting and WHY • Understand the trade offs • Make them consciously • Chose the least consistency/accuracy your use case allows