Riak Lunch'n'learn June 2013

A short talk about Riak, Basho & Dynamo, Presented at a Techcube Lunch'n'learn.

Transcript

1. Sam Elliott - Basho
   Techcube - 2013/06
   

   View Slide

2. Riak
   What is Riak?
   Riak is an open source distributed, key-value database.
   Riak is architected for Availability, Fault-Tolerance, Operational
   Simplicity, and Scalability.
   

   View Slide

3. Basho & Riak
   • Basho originally wrote Riak
   • Riak is now Open-Source
   • Developers employed by Basho
   • Basho sell Enterprise Features &
   Support
   

   View Slide

4. Dynamo
   Dynamo: Amazon’s Highly Available Key-value Store
   This paper presents the design and implementation of Dynamo, a
   highly available key-value storage system that some of Amazons
   core services use to provide an always-on experience.
   

   View Slide

5. CAP Theorem
   • Consistency
   • Availability
   • Partition-tolerance
   

   View Slide

6. CAP Theorem
   • Consistency
   • Availability
   • Partition-tolerance
   

   View Slide

7. CAP Theorem
   • Consistency
   • Availability
   • Partition-tolerance
   

   View Slide

8. CAP Theorem
   • Consistency
   • Availability
   • Partition-tolerance
   Eventual Consistency
   A value will eventually see all its updates, once all partition
   repairs are complete.
   

   View Slide

9. Consistent Hashing
   0
   2160
   

   View Slide

10. Consistent Hashing
    0
    2160 hash(<>,<>)
    

    View Slide

11. Consistent Hashing
    0
    2160 hash(<>,<>)
    riak1.example.com
    

    View Slide

12. Consistent Hashing
    0
    2160 hash(<>,<>)
    riak1.example.com
    

    View Slide

13. Consistent Hashing
    0
    2160
    riak1.example.com
    

    View Slide

14. Replicas & Quorums
    0
    2160 hash(<>,<>)
    read(<>, <>)
    

    View Slide

15. Replicas & Quorums
    0
    2160 hash(<>,<>)
    read(<>, <>, N=3)
    

    View Slide

16. Replicas & Quorums
    0
    2160 hash(<>,<>)
    read(<>, <>, N=3)
    

    View Slide

17. Replicas & Quorums
    0
    2160 hash(<>,<>)
    read(<>, <>, N=3, R=2)
    

    View Slide

18. Replicas & Quorums
    0
    2160 hash(<>,<>)
    read(<>, <>, N=3, R=2)
    

    View Slide

19. Quorum Options
    n r w
    Also: Primary & Durable
    

    View Slide

20. Nodes & Virtual Nodes
    riak1.example.com
    riak2.example.com
    riak3.example.com
    riak4.example.com
    

    View Slide

21. Nodes & Virtual Nodes
    riak1.example.com
    riak2.example.com
    riak3.example.com
    riak4.example.com
    

    View Slide

22. Nodes & Virtual Nodes
    riak1.example.com 11
    riak2.example.com 11
    riak3.example.com 10
    riak4.example.com 0
    

    View Slide

23. Nodes & Virtual Nodes
    riak1.example.com 8
    riak2.example.com 8
    riak3.example.com 8
    riak4.example.com 8
    

    View Slide

24. Nodes & Virtual Nodes
    riak1.example.com
    riak2.example.com
    riak3.example.com
    riak4.example.com
    

    View Slide

25. Extras
    Not only a Key-Value store. We also have:
    • MapReduce (Erlang & JS)
    • Links & Link-walking
    • Secondary Indexes
    • Fulltext Search (& More Soon)
    • CRDTs
    

    View Slide

26. Extras
    Not only a Key-Value store. We also have:
    • MapReduce (Erlang & JS)
    • Links & Link-walking
    • Secondary Indexes
    • Fulltext Search (& More Soon)
    • CRDTs
    

    View Slide

27. MapReduce
    

    View Slide

28. Riak Search
    A hand-rolled Solr-like Search Engine, built into Riak
    

    View Slide

29. Riak Search - Mk II (Coming Soon)
    Yokozuna
    Integrate Solr itself with a Riak Cluster: use Riak for distribution
    and Solr for searching.
    https://github.com/basho/yokozuna
    

    View Slide

30. Concurrent Updates
    1
    

    View Slide

31. Concurrent Updates
    1 2
    

    View Slide

32. Concurrent Updates
    1 2
    3a
    

    View Slide

33. Concurrent Updates
    1 2
    3a
    3b
    

    View Slide

34. Concurrent Updates
    1 2
    3a
    3b
    4
    

    View Slide

35. Concurrent Updates
    1 2
    3a
    3b
    4
    

    View Slide

36. Concurrent Updates
    1 2
    3a
    3b
    4
    

    View Slide

37. An Approach
    One Approach to Eventual Consistency:
    • Use allow_mult=true (not the default)
    • Teach your application how to resolve conflicts
    • Fetch all siblings
    • Replace siblings with merged value
    

    View Slide

38. Another Approach
    • Each “actor” maintains their own per-actor data
    • Upon fetch, merged into a single object
    In General:
    • Store the data by “actor”
    • Teach your application how to resolve that structure into an
    object
    

    View Slide

39. Another Approach
    • Each “actor” maintains their own per-actor data
    • Upon fetch, merged into a single object
    In General:
    • Store the data by “actor”
    • Teach your application how to resolve that structure into an
    object
    • CRDTs: Convergent Replicated Data Types
    

    View Slide

40. CRDTs
    Convergent Replicated Data Types
    Data Structures that Riak knows how to merge, preserving all
    operations.
    • Counters (PN-Counter, ...)
    • Sets (OR-Set, ...)
    • Graphs
    • TreeDoc (Collaborative Doc)
    • Maps (field name → CRDT)
    

    View Slide

41. Riak EE
    Enterprise Edition
    Riak with:
    • Multi Data Centre Replication
    • Better Statistics
    

    View Slide

42. Riak CS
    Riak CS
    An S3-compatible Multi-tenant Storage System
    

    View Slide

43. Conclusions
    • Distributed databases aren’t trivial
    • Riak doesn’t hide anything
    • CRDTs are where it’s at for EC
    

    View Slide

44. Any Questions?
    sam.elliott@basho.com
    

    View Slide