The Persistence of Data

Transcript

1. The Persistence of Data Existential Engineering

2. $

3. whoami $

4. whoami $ Name: Tom Santero $ Title: Technical Evangelist Company:

   Basho Technologies Twitter: @tsantero

5. whoami $ Name: Tom Santero $ ./presentation Title: Technical Evangelist

   Company: Basho Technologies Twitter: @tsantero

6. existence precedes essence

7. code the way we think

8. BIG DATA

9. unbounded, cumbersome datasets? search and aggregation? complex analysis? visualization? capture?

10. distributed computing

11. access model

12. master slave slave slave slave slave

13. None
14. None

15. node node node node node

16. Key/Value Datamodel key value metadata

17. blobs in buckets

18. request quorums replicas fallbacks

19. Consistent Hashing

20. Consistent Hashing • 160-bit integer keyspace 0 2160/2 2160/4

21. Consistent Hashing • 160-bit integer keyspace • divided into !xed

    number of evenly-sized partitions 32 partitions 0 2160/2 2160/4

22. Consistent Hashing • 160-bit integer keyspace • divided into !xed

    number of evenly-sized partitions • partitions are claimed by nodes in the cluster 32 partitions node 0 node 1 node 2 node 3 0 2160/2 2160/4

23. Consistent Hashing • 160-bit integer keyspace • divided into !xed

    number of evenly-sized partitions • partitions are claimed by nodes in the cluster • replicas go to the N partitions following the key node 0 node 1 node 2 node 3

24. Consistent Hashing • 160-bit integer keyspace • divided into !xed

    number of evenly-sized partitions • partitions are claimed by nodes in the cluster • replicas go to the N partitions following the key node 0 node 1 node 2 node 3 hash(“meetups/NYSE”) N=3

25. scalability

26. Add twice as much X to get twice as much

    Y

27. Add twice as much X to get twice as much

    Y horizontal scalability

28. Add twice as much X to get twice as much

    Y horizontal scalability vertical scalability

29. Disaster Scenario

30. Disaster Scenario • node fails X X X X X

    X X X

31. Disaster Scenario • node fails • requests go to fallback

    hash(“meetups/NYSE”) X X X X X X X X

32. Disaster Scenario • node fails • requests go to fallback

    • node comes back hash(“meetups/NYSE”)

33. Disaster Scenario • node fails • requests go to fallback

    • node comes back • “Hando"” - data returns to recovered node hash(“meetups/NYSE”)

34. Disaster Scenario • node fails • requests go to fallback

    • node comes back • “Hando"” - data returns to recovered node • normal operations resume hash(“meetups/NYSE”)
35. None

36. eventual consistency

37. full-text search secondary indexes map/reduce

38. Go forth and scale.

39. http://github.com/basho/riak Go forth and scale.