Yokozuna, Distributed Search You Don't Think About

Discussion and demos of Yokozuna at RICON|East 2013.

Transcript

1. Yokozuna
   Distributed Search You Don’t Think About
   Ryan Zezeski May 14th 2013
   1
   Tuesday, May 21, 13
   

   View Slide

2. Live Demo
   2
   Tuesday, May 21, 13
   

   View Slide

3. Live Demos
   3
   Tuesday, May 21, 13
   

   View Slide

4. PROBLEM?
   4
   Tuesday, May 21, 13
   

   View Slide

5. PROBLEM?
   SOLUTION!
   5
   Tuesday, May 21, 13
   

   View Slide

6. Solution Pre-made
   6
   Tuesday, May 21, 13
   

   View Slide

7. Piece At A Time
   7
   Tuesday, May 21, 13
   

   View Slide

8. Goals
   • Don’t screw up
   • Show how Yokozuna
   doesn’t make you think
   (too hard)
   • Teach you about Search
   • Neat things you can do
   with Yokozuna
   8
   Tuesday, May 21, 13
   

   View Slide

9. PROBLEM: SEARCH
   FOR COMMITS
   ABOUT SPECIFIC
   FEATURE/BUG
   MAKE IT GOOGLE-LIKE
   9
   Tuesday, May 21, 13
   

   View Slide

10. SOLUTION: INDEX
    COMMITS IN
    YOKOZUNA -
    “COMMIT LOG
    SEARCHER” (CLS)
    10
    Tuesday, May 21, 13
    

    View Slide

11. Anatomy of a Commit
    Msg
    11
    Tuesday, May 21, 13
    

    View Slide

12. Primary Key
    12
    Tuesday, May 21, 13
    

    View Slide

13. Any Node Will Do
    13
    Tuesday, May 21, 13
    

    View Slide

14. Term Query
    14
    Tuesday, May 21, 13
    

    View Slide

15. Query Any Node
    15
    Tuesday, May 21, 13
    

    View Slide

16. Boolean (1)
    repo:riak_kv repo:riak_core
    16
    Tuesday, May 21, 13
    

    View Slide

17. Boolean (2)
    repo:riak_kv AND author:”Ryan Zezeski”
    17
    Tuesday, May 21, 13
    

    View Slide

18. Boolean (3)
    commit_author:"Ryan Zezeski" OR
    commit_author:"Joseph Blomstedt" NOT
    commit_repo:riak_kv
    18
    Tuesday, May 21, 13
    

    View Slide

19. Range (1)
    commit_repo:riak_core
    AND commit_dt:[NOW-1YEAR TO NOW]
    19
    Tuesday, May 21, 13
    

    View Slide

20. Range (2)
    commit_repo:riak_core
    AND commit_dt:[NOW-1YEAR TO NOW]
    I RAN THIS ON
    2013-05-10
    sort=dt asc
    20
    Tuesday, May 21, 13
    

    View Slide

21. Wildcard (1)
    *:*
    GET TOTAL
    COUNT FIRST
    21
    Tuesday, May 21, 13
    

    View Slide

22. Wildcard (2)
    commit_repo:riak_*
    NOTICE COUNT
    IS LOWER
    22
    Tuesday, May 21, 13
    

    View Slide

23. WHAT ABOUT
    SEARCHING
    SUMMARY AND
    BODY?
    23
    Tuesday, May 21, 13
    

    View Slide

24. THE INVERTED
    INDEX
    24
    Tuesday, May 21, 13
    

    View Slide

25. AN INDEX - BUT
    INVERTED
    25
    Tuesday, May 21, 13
    

    View Slide

26. EVERYONE KNOWS
    WHAT IT IS
    26
    Tuesday, May 21, 13
    

    View Slide

27. EVEN NON-TECH
    PEOPLE
    27
    Tuesday, May 21, 13
    

    View Slide

28. YES...EVEN YOUR
    PARENTS
    28
    Tuesday, May 21, 13
    

    View Slide

29. What’s In A Book?
    29
    Tuesday, May 21, 13
    

    View Slide

30. • WORDS
    • PARAGRAPHS
    • SECTIONS
    • CHAPTERS
    • ETC.
    30
    Tuesday, May 21, 13
    

    View Slide

31. AND PAGE
    NUMBERS
    31
    Tuesday, May 21, 13
    

    View Slide

32. 32
    Tuesday, May 21, 13
    

    View Slide

33. PAGE NUMBERS ARE
    AN IMPLICIT INDEX
    33
    Tuesday, May 21, 13
    

    View Slide

34. PAGE NUMBER TO
    WORDS
    WORD TO PAGE
    NUMBERS
    INVERTED
    34
    Tuesday, May 21, 13
    

    View Slide

35. STOLEN FROM BLOG OF RICKY HO: http://
    horicky.blogspot.com/2013/02/text-processing-
    part-2-inverted-index.html
    35
    Tuesday, May 21, 13
    

    View Slide

36. HOW DO YOU GET
    THE WORDS IN THE
    FIRST PLACE?
    36
    Tuesday, May 21, 13
    

    View Slide

37. Analysis - The Iceberg
    That Sunk The Titanic
    37
    Tuesday, May 21, 13
    

    View Slide

38. Phrase (1)
    subject:hinted OR subject:handoff
    OR body:hinted OR body:handoff
    38
    Tuesday, May 21, 13
    

    View Slide

39. Phrase (2)
    subject:”hinted handoff”
    OR body:”hinted handoff”
    39
    Tuesday, May 21, 13
    

    View Slide

40. Phrase (3)
    subject:”partition vnode”
    OR body:”partition vnode”
    40
    Tuesday, May 21, 13
    

    View Slide

41. Phrase (4)
    subject:”partition vnode”~4
    OR body:”partition vnode”~4
    41
    Tuesday, May 21, 13
    

    View Slide

42. Exact Term
    subject:behavior OR body:behavior
    42
    Tuesday, May 21, 13
    

    View Slide

43. Fuzzy Term
    subject:behavior~1 OR body:behavior~1
    43
    Tuesday, May 21, 13
    

    View Slide

44. Ranking
    ADD SCORE TO
    FL
    SCORE ADDED
    TO EVERY
    RESULT
    44
    Tuesday, May 21, 13
    

    View Slide

45. RECALL, PRECISION,
    AND RELEVANCY, OH
    MY!
    45
    Tuesday, May 21, 13
    

    View Slide

46. RELEVANCY - FOR A
    GIVEN QUERY &
    DOC SET THERE IS
    AN IDEAL ANSWER
    OF ONLY RELEVANT
    DOCS
    46
    Tuesday, May 21, 13
    

    View Slide

47. RECALL = WHAT %
    OF IDEAL ANSWER
    SET WAS RETRIEVED
    47
    Tuesday, May 21, 13
    

    View Slide

48. PRECISION = WHAT
    % OF ANSWER IS
    RELEVANT
    48
    Tuesday, May 21, 13
    

    View Slide

49. RECALLvs.PRECISION
    AS YOU INCREASE
    RECALL YOU
    DEGRADE PRECISION
    49
    Tuesday, May 21, 13
    

    View Slide

50. SOLR DETERMINES
    RELEVANCY VIA THE
    NOTION OF
    SIMILARITY
    50
    Tuesday, May 21, 13
    

    View Slide

51. SOLR USES TF-IDF:
    TERM FREQUENCY,
    INVERSE DOCUMENT
    FREQUENCY
    51
    Tuesday, May 21, 13
    

    View Slide

52. Dismax + Facets +
    Highlighting
    FACETS
    HIGHLIGHTING
    DISMAX
    52
    Tuesday, May 21, 13
    

    View Slide

53. FACET - A
    TAXONOMY OF
    YOUR QUERY BASED
    ON FIELD’S VALUES
    53
    Tuesday, May 21, 13
    

    View Slide

54. FACETS ALLOW
    “DRILL DOWN” -
    THEY GUIDE THE
    USER
    54
    Tuesday, May 21, 13
    

    View Slide

55. HIGHLIGHTING
    GIVES YOUR RESULTS
    CONTEXT - ALLOWS
    QUICKER
    DETERMINATION OF
    RELEVANCY
    55
    Tuesday, May 21, 13
    

    View Slide

56. DISMAX - DISjunction
    MAX - A QUERY
    HANDLER MEANT
    FOR DIRECT USER
    INPUT
    56
    Tuesday, May 21, 13
    

    View Slide

57. All Nodes Up
    57
    Tuesday, May 21, 13
    

    View Slide

58. All Nodes Up - Query
    58
    Tuesday, May 21, 13
    

    View Slide

59. Node 4 Down
    59
    Tuesday, May 21, 13
    

    View Slide

60. Node 4 Down - Query
    60
    Tuesday, May 21, 13
    

    View Slide

61. Node 3 & 4 Down
    61
    Tuesday, May 21, 13
    

    View Slide

62. Node 3 & 4 Down -
    Query
    62
    Tuesday, May 21, 13
    

    View Slide

63. REPLICATION
    PROVIDES HIGH
    AVAILABILITY
    2 3 4
    1
    START WITH 4 NODES
    63
    Tuesday, May 21, 13
    

    View Slide

64. Write 3 Replicas
    2 3 4
    1
    64
    Tuesday, May 21, 13
    

    View Slide

65. Take 2 Nodes Down
    2 3 4
    1
    1 REPLICA STILL
    AVAILABLE
    65
    Tuesday, May 21, 13
    

    View Slide

66. WHAT IF DATA IS
    WRITTEN WHILE
    NODES ARE DOWN?
    66
    Tuesday, May 21, 13
    

    View Slide

67. YZ Not Stored Yet
    67
    Tuesday, May 21, 13
    

    View Slide

68. Store YZ Log
    68
    Tuesday, May 21, 13
    

    View Slide

69. Query YZ - Node 1& 2
    69
    Tuesday, May 21, 13
    

    View Slide

70. Set XFer Limit To 0
    70
    Tuesday, May 21, 13
    

    View Slide

71. Start Nodes 3 & 4
    71
    Tuesday, May 21, 13
    

    View Slide

72. Query Solr Direct
    WHEN MAKING THIS
    DEMO I WAS
    EXPECTING THIS TO BE
    0 BUT I FORGOT ABOUT
    AAE WHICH STARTED
    KICKING IN BEFORE
    HANDOFF - SELF
    HEALING FTW!
    72
    Tuesday, May 21, 13
    

    View Slide

73. Set Xfer Limit To 64
    73
    Tuesday, May 21, 13
    

    View Slide

74. Handoff Occurs
    74
    Tuesday, May 21, 13
    

    View Slide

75. 0 Pending Xfers
    75
    Tuesday, May 21, 13
    

    View Slide

76. Solr Direct (Again)
    NOTICE IT’S NOW 301,
    UP FROM 54, MORE
    PROOF THAT HANDOFF
    OCCURRED - NOTE
    THIS QUERY IS GOING
    DIRECT TO ONLY 1
    SHARD
    76
    Tuesday, May 21, 13
    

    View Slide

77. Query Node 4 YZ
    NOW HIT YOKOZUNA
    ON NODE4 (NOTICE
    CHANGE IN PORT #) -
    THIS WILL RUN A DIST
    SEARCH AND THUS
    RETURN CORRECT
    COUNT
    77
    Tuesday, May 21, 13
    

    View Slide

78. Data Ownership
    A VNODE
    THE RING
    78
    Tuesday, May 21, 13
    

    View Slide

79. Node Down
    X
    X
    X
    X
    X
    X
    X
    X
    X
    X
    79
    Tuesday, May 21, 13
    

    View Slide

80. Write Fallback
    X
    X
    X
    X
    X
    X
    X
    X
    X
    X
    80
    Tuesday, May 21, 13
    

    View Slide

81. Node Up
    HINTED
    HANDOFF WILL
    MOVE REPLICA
    TO PRIMARY
    81
    Tuesday, May 21, 13
    

    View Slide

82. WHAT IF YOU RM -RF
    THE INDEX DIR?
    82
    Tuesday, May 21, 13
    

    View Slide

83. Kill The Data
    RM -RF THE INDEX DIRECTORY
    KILL THE SOLR PROC
    83
    Tuesday, May 21, 13
    

    View Slide

84. Auto Restart
    YOKOZUNA NOTICES SOLR DIED
    AND AUTOMATICALLY RESTARTS
    IT
    84
    Tuesday, May 21, 13
    

    View Slide

85. Node 4 - 0 Results
    85
    Tuesday, May 21, 13
    

    View Slide

86. AAE Notices Missing
    Data
    86
    Tuesday, May 21, 13
    

    View Slide

87. Node 4 - 13 Results
    DATA IS RE-INDEXED OVER TIME
    87
    Tuesday, May 21, 13
    

    View Slide

88. More AAE Repair
    88
    Tuesday, May 21, 13
    

    View Slide

89. Node 4 - 128 Results
    MORE INDEXES ARE REPAIRED,
    THIS CONTINUES UNTIL AAE
    REPAIRS ALL INDEXES
    89
    Tuesday, May 21, 13
    

    View Slide

90. WHAT EVEN IS
    ACTIVE ANTI-
    ENTROPY?
    90
    Tuesday, May 21, 13
    

    View Slide

91. Mo Systems Mo Failure
    • index update could get
    lost
    • files can become
    truncated/corrupted
    • accidental `rm -rf`
    • segfault at right time
    • etc...
    91
    Tuesday, May 21, 13
    

    View Slide

92. MYRAID OF FAILURE
    SCENARIOS - FROM
    OBVIOUS TO NEARLY
    INVISIBLE
    92
    Tuesday, May 21, 13
    

    View Slide

93. ENTROPY IS DAMAGE
    AAE IS SELF HEALING
    STRIKER!!!! EHEM, I
    MEAN, ENTROPY!!!!
    93
    Tuesday, May 21, 13
    

    View Slide

94. REPAIR EFFICIENTLY -
    NOT STUPIDLY
    94
    Tuesday, May 21, 13
    

    View Slide

95. Learn You Some Merkle
    For A Great Good
    BIG UPS TO @jtuple FOR THE AAE DIAGRAMS
    95
    Tuesday, May 21, 13
    

    View Slide

96. Segments
    EACH SEGMENT IS LIST OF
    KEY-HASH PAIRS
    96
    Tuesday, May 21, 13
    

    View Slide

97. Segment Hashes
    HASH OF HASHES IN
    SEGMENT
    97
    Tuesday, May 21, 13
    

    View Slide

98. Hash O’ Hashes
    98
    Tuesday, May 21, 13
    

    View Slide

99. WHAT HAPPENS
    DURING
    EXCHANGE?
    99
    Tuesday, May 21, 13
    

    View Slide

100. Start With 2 Trees
     100
     Tuesday, May 21, 13
     

     View Slide

101. Compare Top Hashes
     TOP HASHES DON’T
     MATCH -
     SOMETHING IS
     DIFFERENT
     101
     Tuesday, May 21, 13
     

     View Slide

102. Compare Child Hashes
     NARROW DOWN
     THE DIVERGENT
     SEGMENT
     102
     Tuesday, May 21, 13
     

     View Slide

103. Recur
     NARROW DOWN
     THE DIVERGENT
     SEGMENT CONT...
     103
     Tuesday, May 21, 13
     

     View Slide

104. Iter Key-Hash Pairs
     ITER FINAL LIST OF
     HASHES TO FIND
     DIVERGENT KEYS
     104
     Tuesday, May 21, 13
     

     View Slide

105. Repair Divergent Keys
     REPAIR (RE-INDEX)
     KEYS THAT ARE
     DIVERGENT (RED)
     105
     Tuesday, May 21, 13
     

     View Slide

106. CODE FOR
     DETECTION AND
     REPAIR - NOT
     PREVENTION
     106
     Tuesday, May 21, 13
     

     View Slide

107. WHAT HAPPENS IF 3
     NODES GO DOWN?
     107
     Tuesday, May 21, 13
     

     View Slide

108. Stop 3 Nodes
     108
     Tuesday, May 21, 13
     

     View Slide

109. Query
     109
     Tuesday, May 21, 13
     

     View Slide

110. CONSISTENCY
     vs.
     AVAILABILITY
     110
     Tuesday, May 21, 13
     

     View Slide

111. Uptime - Story of 9s
     UPTIME = (MTBF - MTTR) / MTBF
     111
     Tuesday, May 21, 13
     

     View Slide

112. Uptime is Flawed
     IF THE SYSTEM IS
     DOWN, BUT NO
     ONE MAKES A
     REQUEST, IS IT
     REALLY DOWN?
     112
     Tuesday, May 21, 13
     

     View Slide

113. Yield - Uptime of the
     People
     YIELD = QUERIES COMPLETED / QUERIES
     OFFERED
     113
     Tuesday, May 21, 13
     

     View Slide

114. Harvest vs. Yield
     HARVEST = DATA AVAIL / COMPLETE DATA
     IF FACE OF FAILURE YOU CAN’T HAVE BOTH
     FOR A SINGLE REQUEST
     114
     Tuesday, May 21, 13
     

     View Slide

115. IN TIMES OF
     TROUBLE -
     YOKOZUNA
     CHOOSES HARVEST
     FOR QUERIES
     115
     Tuesday, May 21, 13
     

     View Slide

116. TECHNICALLY -
     YOKOZUNA IS
     ALWAYS < 100%
     HARVEST IN A NON-
     QUIESCENT CLUSTER
     116
     Tuesday, May 21, 13
     

     View Slide

117. YOKOZUNA FAVORS
     YIELD FOR WRITES
     117
     Tuesday, May 21, 13
     

     View Slide

118. ONCE RIAK 1.4 SHIPS
     - YOKOZUNA LANDS
     IN MASTER
     118
     Tuesday, May 21, 13
     

     View Slide

119. THANK YOU
     HTTP://GITHUB.COM/BASHO/YOKOZUNA
     119
     Tuesday, May 21, 13
     

     View Slide