Russell Brown - Oh, you wanted to increment a counter? Good luck with that!

Presented at Hey! #22 on 29th March, 2016.

Transcript

1. basho "Oh, you wanted to *increment a counter*?! Good luck with that!” [1] Russell Brown russelldb@basho.com [1] https://twitter.com/lindsey/status/575006945213485056
2. basho https://twitter.com/lindsey/status/575006945213485056
3. basho Jokes ARE better explained
4. basho https://twitter.com/dizzyd/status/35566447145127936
5. basho Number one feature request 2011 - Counters
6. basho HOW HARD CAN IT BE?
8. basho
9. basho
10. basho Leslie Lamport ``A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable''
11. basho https://en.wikipedia.org/wiki/Distributed_computing A distributed system is a software system in which components located on networked computers communicate and coordinate their actions by passing messages. The components interact with each other in order to achieve a common goal.
12. basho Distributed Counters - WHY? • Many many updates - scale/perf • Fault tolerance - machines crash • Latency - local data centres • Containers/“microservices”
13. basho CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP
14. basho CAP • Consistency • Availability • Partition Tolerance
15. Replica A Replica B Replica C Client X Client Y add 1 add 1 NO!!!! :( CP
16. Replica A Replica B Replica C Client X Client Y Add 1 C’ Add 1 AP
17. basho Why Are You Counting? • Money / Votes • Real world resources (seats on a place, rental cars) • Unique IDs • Events (likes, hits, plays)
18. basho “Strong Consistency” • ZooKeeper/HyperDex-Replicant/Riak-Ensemble • ZAB/Chain Replication/Paxos/Raft/VSR • Compare and swap • Starvation • Single data centre
19. basho Not Really Counting - IDs • Ticket Servers - flickr ticket server • Flake - boundary • SnowFlake - Twitter
20. basho Eventually Consistent • Riak/Cassandra etc • CRDTs Counters • Drift/Partial failure • Idempotence • Bounded-Counters - reservations
21. basho Probabilistic • HyperLogLog etc • Bounded error estimates • Trade off space/speed for accuracy • Redis (soon to-be riak) and others
22. basho https://twitter.com/lindsey/status/575006945213485056
23. basho Take Home • Understand your requirements • Know What you are counting and WHY • Understand the trade offs • Make them consciously • Chose the least consistency/accuracy your use case allows