Riak Search: The Next Generation

Presentation on Yokozuna (https://github.com/basho/yokozuna) at the NYC Riak Meetup group

Transcript

1. Riak
   Search
   the next generation
   Tuesday, September 17, 13
   

   View Slide

2. tsantero
   @
   basho.com
   Tuesday, September 17, 13
   

   View Slide

3. Tuesday, September 17, 13
   

   View Slide

4. Tuesday, September 17, 13
   

   View Slide

5. 2.0
   coming soon..
   Tuesday, September 17, 13
   

   View Slide

6. the history of
   Riak Search
   Tuesday, September 17, 13
   

   View Slide

7. home grown
   full-text search
   Tuesday, September 17, 13
   

   View Slide

8. lucene
   Tuesday, September 17, 13
   

   View Slide

9. SCALE
   Tuesday, September 17, 13
   

   View Slide

10. NODE # = HASH(KEY) % NUM_NODES
    NH(Ka) = 0
    NH(Kb) = 1
    NH(Kc) = 2
    NH(Kd) = 0
    ...
    Naive Hashing
    Tuesday, September 17, 13
    

    View Slide

11. NODE 0 NODE 1 NODE 2
    Ka Kb Kc
    Kd Ke Kf
    Kg Kh Ki
    Kj Kk
    Km Kl
    Kp Kn Ko
    Kq Kr
    Naive Hashing
    Tuesday, September 17, 13
    

    View Slide

12. NODE 0 NODE 1 NODE 2
    Ka Kb Kc Kd
    Kg
    Ki
    NODE 3
    Ke Kf Kh
    Kj Kk Kl
    Km Kn Ko Kp
    Kq Kr
    Naive Hashing
    Tuesday, September 17, 13
    

    View Slide

13. K * (NN - 1) / NN => K
    • K = # OF KEYS
    • NN = # OF NODES
    • AS NN GROWS FACTOR ESSENTIALLY BECOMES 1, THUS
    ALL KEYS MOVE
    Naive Hashing
    Tuesday, September 17, 13
    

    View Slide

14. PARTITION # = HASH(KEY) % PARTITIONS
    • # PARTITIONS REMAINS CONSTANT
    • KEY ALWAYS MAPS TO SAME PARTITION
    • NODES OWN PARTITIONS
    • PARTITIONS CONTAIN KEYS
    • EXTRA LEVEL OF INDIRECTION
    Consistent Hashing
    Tuesday, September 17, 13
    

    View Slide

15. P9
    P6
    P3
    P8
    P5
    P2
    P7
    P4
    P1
    NODE 0 NODE 1 NODE 2
    Ka Kb Kc
    Kd Ke Kf
    Kg Kh Ki
    Kj Kk
    Km Kl
    Kp Kn Ko
    Kq Kr
    Consistent Hashing
    Tuesday, September 17, 13
    

    View Slide

16. P9
    P6
    P3
    P8 P5
    P2
    P7
    P4 P1
    NODE 0 NODE 1 NODE 2
    Ka
    Kb Kc
    Kd Ke
    Kf
    Kg Kh Ki
    Kj
    Kk
    Km Kl
    Kp Kn
    Ko
    Kq Kr
    NODE 3
    Consistent Hashing
    Tuesday, September 17, 13
    

    View Slide

17. NN * K/Q => K/Q
    • K = # OF KEYS
    • NN = # OF NODES
    • Q = # OF PARTITIONS
    • AS K GROWS NN BECOMES CONSTANT, THUS K/Q KEYS
    MOVE
    Consistent Hashing
    Tuesday, September 17, 13
    

    View Slide

18. uniform distribution
    Consistent
    Hashing
    {logical vs physical
    partitioning scheme
    even division of keys
    Tuesday, September 17, 13
    

    View Slide

19. the future of
    Riak Search
    Tuesday, September 17, 13
    

    View Slide

20. Tuesday, September 17, 13
    

    View Slide

21. persistence
    distributing Solr
    querying
    indexing
    Tuesday, September 17, 13
    

    View Slide

22. Each Riak node runs an instance of Solr
    Tuesday, September 17, 13
    

    View Slide

23. Solr index = riak bucket
    document = RObj value
    plaintext, JSON, XML
    Tuesday, September 17, 13
    

    View Slide

24. Distributed Searching in Solr
    query
    faceting
    highlighting
    stats
    spell check
    term vectors
    Tuesday, September 17, 13
    

    View Slide

25. SolrCloud
    Tuesday, September 17, 13
    

    View Slide

26. SolrCloud
    Tuesday, September 17, 13
    

    View Slide

27. Harvest vs Yield
    Tuesday, September 17, 13
    

    View Slide

28. A better measure of
    Availability
    Tuesday, September 17, 13
    

    View Slide

29. Queries Issues
    Queries Offered
    Yield =
    Tuesday, September 17, 13
    

    View Slide

30. Harvest = Data Available
    Total Dataset
    Tuesday, September 17, 13
    

    View Slide

31. Harvest
    Yield
    Tuesday, September 17, 13
    

    View Slide

32. Manage Harvest by storing Index Replicas
    Tuesday, September 17, 13
    

    View Slide

33. Term vs Document
    Partitioning Schemes
    Tuesday, September 17, 13
    

    View Slide

34. Node 0
    Node 1
    Node 2
    Term Based Partitioning
    Tuesday, September 17, 13
    

    View Slide

35. Node 0
    Node 1
    Node 2
    Document Based Partitioning
    Tuesday, September 17, 13
    

    View Slide

36. Replicas
    Node 0
    Node 1
    Node 2
    Tuesday, September 17, 13
    

    View Slide

37. Quorums
    Tuesday, September 17, 13
    

    View Slide

38. Concurrency
    =>
    Siblings
    Tuesday, September 17, 13
    

    View Slide

39. Read Repair
    (Anti-Entropy)
    Tuesday, September 17, 13
    

    View Slide

40. replica replica replica
    Tuesday, September 17, 13
    

    View Slide

41. replica replica replica
    X
    Tuesday, September 17, 13
    

    View Slide

42. replica replica replica
    replica replica replica
    Tuesday, September 17, 13
    

    View Slide

43. Active Anti-Entropy
    (self healing clusters)
    Tuesday, September 17, 13
    

    View Slide

44. real-time updates
    persistent
    non-blocking
    disk-based
    Tuesday, September 17, 13
    

    View Slide

45. Tuesday, September 17, 13
    

    View Slide

46. = hashes marked “dirty”
    Tuesday, September 17, 13
    

    View Slide

47. Tuesday, September 17, 13
    

    View Slide

48. Tuesday, September 17, 13
    

    View Slide

49. Tuesday, September 17, 13
    

    View Slide

50. Tuesday, September 17, 13
    

    View Slide

51. = keys to read-repair
    Tuesday, September 17, 13
    

    View Slide

52. Questions?
    make it so!
    Tuesday, September 17, 13
    

    View Slide