The Persistence of Data

Slides from my talk at http://www.nycsoftware.org/events/59254322/ in New York City.

Transcript

1. The Persistence of Data Existential Engineering
2. $
3. whoami $
4. whoami $ Name: Tom Santero $ Title: Technical Evangelist Company: Basho Technologies Twitter: @tsantero
5. whoami $ Name: Tom Santero $ ./presentation Title: Technical Evangelist Company: Basho Technologies Twitter: @tsantero
6. existence precedes essence
7. code the way we think
8. BIG DATA
9. unbounded, cumbersome datasets? search and aggregation? complex analysis? visualization? capture?
10. distributed computing
11. access model
12. master slave slave slave slave slave
15. node node node node node
16. Key/Value Datamodel key value metadata
17. blobs in buckets
18. request quorums replicas fallbacks
19. Consistent Hashing
20. Consistent Hashing • 160-bit integer keyspace 0 2160/2 2160/4
21. Consistent Hashing • 160-bit integer keyspace • divided into !xed number of evenly-sized partitions 32 partitions 0 2160/2 2160/4
22. Consistent Hashing • 160-bit integer keyspace • divided into !xed number of evenly-sized partitions • partitions are claimed by nodes in the cluster 32 partitions node 0 node 1 node 2 node 3 0 2160/2 2160/4
23. Consistent Hashing • 160-bit integer keyspace • divided into !xed number of evenly-sized partitions • partitions are claimed by nodes in the cluster • replicas go to the N partitions following the key node 0 node 1 node 2 node 3
24. Consistent Hashing • 160-bit integer keyspace • divided into !xed number of evenly-sized partitions • partitions are claimed by nodes in the cluster • replicas go to the N partitions following the key node 0 node 1 node 2 node 3 hash(“meetups/NYSE”) N=3
25. scalability
26. Add twice as much X to get twice as much Y
27. Add twice as much X to get twice as much Y horizontal scalability
28. Add twice as much X to get twice as much Y horizontal scalability vertical scalability
29. Disaster Scenario
30. Disaster Scenario • node fails X X X X X X X X
31. Disaster Scenario • node fails • requests go to fallback hash(“meetups/NYSE”) X X X X X X X X
32. Disaster Scenario • node fails • requests go to fallback • node comes back hash(“meetups/NYSE”)
33. Disaster Scenario • node fails • requests go to fallback • node comes back • “Hando"” - data returns to recovered node hash(“meetups/NYSE”)
34. Disaster Scenario • node fails • requests go to fallback • node comes back • “Hando"” - data returns to recovered node • normal operations resume hash(“meetups/NYSE”)
36. eventual consistency
37. full-text search secondary indexes map/reduce
38. Go forth and scale.
39. http://github.com/basho/riak Go forth and scale.