Yokozuna, Distributed Search You Don't Think About

Slide 1

Slide 1 text

Yokozuna Distributed Search You Don’t Think About Ryan Zezeski May 14th 2013 1 Tuesday, May 21, 13

Slide 2

Slide 2 text

Live Demo 2 Tuesday, May 21, 13

Slide 3

Slide 3 text

Live Demos 3 Tuesday, May 21, 13

Slide 4

Slide 4 text

PROBLEM? 4 Tuesday, May 21, 13

Slide 5

Slide 5 text

PROBLEM? SOLUTION! 5 Tuesday, May 21, 13

Slide 6

Slide 6 text

Solution Pre-made 6 Tuesday, May 21, 13

Slide 7

Slide 7 text

Piece At A Time 7 Tuesday, May 21, 13

Slide 8

Slide 8 text

Goals • Don’t screw up • Show how Yokozuna doesn’t make you think (too hard) • Teach you about Search • Neat things you can do with Yokozuna 8 Tuesday, May 21, 13

Slide 9

Slide 9 text

PROBLEM: SEARCH FOR COMMITS ABOUT SPECIFIC FEATURE/BUG MAKE IT GOOGLE-LIKE 9 Tuesday, May 21, 13

Slide 10

Slide 10 text

SOLUTION: INDEX COMMITS IN YOKOZUNA - “COMMIT LOG SEARCHER” (CLS) 10 Tuesday, May 21, 13

Slide 11

Slide 11 text

Anatomy of a Commit Msg 11 Tuesday, May 21, 13

Slide 12

Slide 12 text

Primary Key 12 Tuesday, May 21, 13

Slide 13

Slide 13 text

Any Node Will Do 13 Tuesday, May 21, 13

Slide 14

Slide 14 text

Term Query 14 Tuesday, May 21, 13

Slide 15

Slide 15 text

Query Any Node 15 Tuesday, May 21, 13

Slide 16

Slide 16 text

Boolean (1) repo:riak_kv repo:riak_core 16 Tuesday, May 21, 13

Slide 17

Slide 17 text

Boolean (2) repo:riak_kv AND author:”Ryan Zezeski” 17 Tuesday, May 21, 13

Slide 18

Slide 18 text

Boolean (3) commit_author:"Ryan Zezeski" OR commit_author:"Joseph Blomstedt" NOT commit_repo:riak_kv 18 Tuesday, May 21, 13

Slide 19

Slide 19 text

Range (1) commit_repo:riak_core AND commit_dt:[NOW-1YEAR TO NOW] 19 Tuesday, May 21, 13

Slide 20

Slide 20 text

Range (2) commit_repo:riak_core AND commit_dt:[NOW-1YEAR TO NOW] I RAN THIS ON 2013-05-10 sort=dt asc 20 Tuesday, May 21, 13

Slide 21

Slide 21 text

Wildcard (1) *:* GET TOTAL COUNT FIRST 21 Tuesday, May 21, 13

Slide 22

Slide 22 text

Wildcard (2) commit_repo:riak_* NOTICE COUNT IS LOWER 22 Tuesday, May 21, 13

Slide 23

Slide 23 text

WHAT ABOUT SEARCHING SUMMARY AND BODY? 23 Tuesday, May 21, 13

Slide 24

Slide 24 text

THE INVERTED INDEX 24 Tuesday, May 21, 13

Slide 25

Slide 25 text

AN INDEX - BUT INVERTED 25 Tuesday, May 21, 13

Slide 26

Slide 26 text

EVERYONE KNOWS WHAT IT IS 26 Tuesday, May 21, 13

Slide 27

Slide 27 text

EVEN NON-TECH PEOPLE 27 Tuesday, May 21, 13

Slide 28

Slide 28 text

YES...EVEN YOUR PARENTS 28 Tuesday, May 21, 13

Slide 29

Slide 29 text

What’s In A Book? 29 Tuesday, May 21, 13

Slide 30

Slide 30 text

• WORDS • PARAGRAPHS • SECTIONS • CHAPTERS • ETC. 30 Tuesday, May 21, 13

Slide 31

Slide 31 text

AND PAGE NUMBERS 31 Tuesday, May 21, 13

Slide 32

Slide 32 text

32 Tuesday, May 21, 13

Slide 33

Slide 33 text

PAGE NUMBERS ARE AN IMPLICIT INDEX 33 Tuesday, May 21, 13

Slide 34

Slide 34 text

PAGE NUMBER TO WORDS WORD TO PAGE NUMBERS INVERTED 34 Tuesday, May 21, 13

Slide 35

Slide 35 text

STOLEN FROM BLOG OF RICKY HO: http:// horicky.blogspot.com/2013/02/text-processing- part-2-inverted-index.html 35 Tuesday, May 21, 13

Slide 36

Slide 36 text

HOW DO YOU GET THE WORDS IN THE FIRST PLACE? 36 Tuesday, May 21, 13

Slide 37

Slide 37 text

Analysis - The Iceberg That Sunk The Titanic 37 Tuesday, May 21, 13

Slide 38

Slide 38 text

Phrase (1) subject:hinted OR subject:handoff OR body:hinted OR body:handoff 38 Tuesday, May 21, 13

Slide 39

Slide 39 text

Phrase (2) subject:”hinted handoff” OR body:”hinted handoff” 39 Tuesday, May 21, 13

Slide 40

Slide 40 text

Phrase (3) subject:”partition vnode” OR body:”partition vnode” 40 Tuesday, May 21, 13

Slide 41

Slide 41 text

Phrase (4) subject:”partition vnode”~4 OR body:”partition vnode”~4 41 Tuesday, May 21, 13

Slide 42

Slide 42 text

Exact Term subject:behavior OR body:behavior 42 Tuesday, May 21, 13

Slide 43

Slide 43 text

Fuzzy Term subject:behavior~1 OR body:behavior~1 43 Tuesday, May 21, 13

Slide 44

Slide 44 text

Ranking ADD SCORE TO FL SCORE ADDED TO EVERY RESULT 44 Tuesday, May 21, 13

Slide 45

Slide 45 text

RECALL, PRECISION, AND RELEVANCY, OH MY! 45 Tuesday, May 21, 13

Slide 46

Slide 46 text

RELEVANCY - FOR A GIVEN QUERY & DOC SET THERE IS AN IDEAL ANSWER OF ONLY RELEVANT DOCS 46 Tuesday, May 21, 13

Slide 47

Slide 47 text

RECALL = WHAT % OF IDEAL ANSWER SET WAS RETRIEVED 47 Tuesday, May 21, 13

Slide 48

Slide 48 text

PRECISION = WHAT % OF ANSWER IS RELEVANT 48 Tuesday, May 21, 13

Slide 49

Slide 49 text

RECALLvs.PRECISION AS YOU INCREASE RECALL YOU DEGRADE PRECISION 49 Tuesday, May 21, 13

Slide 50

Slide 50 text

SOLR DETERMINES RELEVANCY VIA THE NOTION OF SIMILARITY 50 Tuesday, May 21, 13

Slide 51

Slide 51 text

SOLR USES TF-IDF: TERM FREQUENCY, INVERSE DOCUMENT FREQUENCY 51 Tuesday, May 21, 13

Slide 52

Slide 52 text

Dismax + Facets + Highlighting FACETS HIGHLIGHTING DISMAX 52 Tuesday, May 21, 13

Slide 53

Slide 53 text

FACET - A TAXONOMY OF YOUR QUERY BASED ON FIELD’S VALUES 53 Tuesday, May 21, 13

Slide 54

Slide 54 text

FACETS ALLOW “DRILL DOWN” - THEY GUIDE THE USER 54 Tuesday, May 21, 13

Slide 55

Slide 55 text

HIGHLIGHTING GIVES YOUR RESULTS CONTEXT - ALLOWS QUICKER DETERMINATION OF RELEVANCY 55 Tuesday, May 21, 13

Slide 56

Slide 56 text

DISMAX - DISjunction MAX - A QUERY HANDLER MEANT FOR DIRECT USER INPUT 56 Tuesday, May 21, 13

Slide 57

Slide 57 text

All Nodes Up 57 Tuesday, May 21, 13

Slide 58

Slide 58 text

All Nodes Up - Query 58 Tuesday, May 21, 13

Slide 59

Slide 59 text

Node 4 Down 59 Tuesday, May 21, 13

Slide 60

Slide 60 text

Node 4 Down - Query 60 Tuesday, May 21, 13

Slide 61

Slide 61 text

Node 3 & 4 Down 61 Tuesday, May 21, 13

Slide 62

Slide 62 text

Node 3 & 4 Down - Query 62 Tuesday, May 21, 13

Slide 63

Slide 63 text

REPLICATION PROVIDES HIGH AVAILABILITY 2 3 4 1 START WITH 4 NODES 63 Tuesday, May 21, 13

Slide 64

Slide 64 text

Write 3 Replicas 2 3 4 1 64 Tuesday, May 21, 13

Slide 65

Slide 65 text

Take 2 Nodes Down 2 3 4 1 1 REPLICA STILL AVAILABLE 65 Tuesday, May 21, 13

Slide 66

Slide 66 text

WHAT IF DATA IS WRITTEN WHILE NODES ARE DOWN? 66 Tuesday, May 21, 13

Slide 67

Slide 67 text

YZ Not Stored Yet 67 Tuesday, May 21, 13

Slide 68

Slide 68 text

Store YZ Log 68 Tuesday, May 21, 13

Slide 69

Slide 69 text

Query YZ - Node 1& 2 69 Tuesday, May 21, 13

Slide 70

Slide 70 text

Set XFer Limit To 0 70 Tuesday, May 21, 13

Slide 71

Slide 71 text

Start Nodes 3 & 4 71 Tuesday, May 21, 13

Slide 72

Slide 72 text

Query Solr Direct WHEN MAKING THIS DEMO I WAS EXPECTING THIS TO BE 0 BUT I FORGOT ABOUT AAE WHICH STARTED KICKING IN BEFORE HANDOFF - SELF HEALING FTW! 72 Tuesday, May 21, 13

Slide 73

Slide 73 text

Set Xfer Limit To 64 73 Tuesday, May 21, 13

Slide 74

Slide 74 text

Handoff Occurs 74 Tuesday, May 21, 13

Slide 75

Slide 75 text

0 Pending Xfers 75 Tuesday, May 21, 13

Slide 76

Slide 76 text

Solr Direct (Again) NOTICE IT’S NOW 301, UP FROM 54, MORE PROOF THAT HANDOFF OCCURRED - NOTE THIS QUERY IS GOING DIRECT TO ONLY 1 SHARD 76 Tuesday, May 21, 13

Slide 77

Slide 77 text

Query Node 4 YZ NOW HIT YOKOZUNA ON NODE4 (NOTICE CHANGE IN PORT #) - THIS WILL RUN A DIST SEARCH AND THUS RETURN CORRECT COUNT 77 Tuesday, May 21, 13

Slide 78

Slide 78 text

Data Ownership A VNODE THE RING 78 Tuesday, May 21, 13

Slide 79

Slide 79 text

Node Down X X X X X X X X X X 79 Tuesday, May 21, 13

Slide 80

Slide 80 text

Write Fallback X X X X X X X X X X 80 Tuesday, May 21, 13

Slide 81

Slide 81 text

Node Up HINTED HANDOFF WILL MOVE REPLICA TO PRIMARY 81 Tuesday, May 21, 13

Slide 82

Slide 82 text

WHAT IF YOU RM -RF THE INDEX DIR? 82 Tuesday, May 21, 13

Slide 83

Slide 83 text

Kill The Data RM -RF THE INDEX DIRECTORY KILL THE SOLR PROC 83 Tuesday, May 21, 13

Slide 84

Slide 84 text

Auto Restart YOKOZUNA NOTICES SOLR DIED AND AUTOMATICALLY RESTARTS IT 84 Tuesday, May 21, 13

Slide 85

Slide 85 text

Node 4 - 0 Results 85 Tuesday, May 21, 13

Slide 86

Slide 86 text

AAE Notices Missing Data 86 Tuesday, May 21, 13

Slide 87

Slide 87 text

Node 4 - 13 Results DATA IS RE-INDEXED OVER TIME 87 Tuesday, May 21, 13

Slide 88

Slide 88 text

More AAE Repair 88 Tuesday, May 21, 13

Slide 89

Slide 89 text

Node 4 - 128 Results MORE INDEXES ARE REPAIRED, THIS CONTINUES UNTIL AAE REPAIRS ALL INDEXES 89 Tuesday, May 21, 13

Slide 90

Slide 90 text

WHAT EVEN IS ACTIVE ANTI- ENTROPY? 90 Tuesday, May 21, 13

Slide 91

Slide 91 text

Mo Systems Mo Failure • index update could get lost • ﬁles can become truncated/corrupted • accidental `rm -rf` • segfault at right time • etc... 91 Tuesday, May 21, 13

Slide 92

Slide 92 text

MYRAID OF FAILURE SCENARIOS - FROM OBVIOUS TO NEARLY INVISIBLE 92 Tuesday, May 21, 13

Slide 93

Slide 93 text

ENTROPY IS DAMAGE AAE IS SELF HEALING STRIKER!!!! EHEM, I MEAN, ENTROPY!!!! 93 Tuesday, May 21, 13

Slide 94

Slide 94 text

REPAIR EFFICIENTLY - NOT STUPIDLY 94 Tuesday, May 21, 13

Slide 95

Slide 95 text

Learn You Some Merkle For A Great Good BIG UPS TO @jtuple FOR THE AAE DIAGRAMS 95 Tuesday, May 21, 13

Slide 96

Slide 96 text

Segments EACH SEGMENT IS LIST OF KEY-HASH PAIRS 96 Tuesday, May 21, 13

Slide 97

Slide 97 text

Segment Hashes HASH OF HASHES IN SEGMENT 97 Tuesday, May 21, 13

Slide 98

Slide 98 text

Hash O’ Hashes 98 Tuesday, May 21, 13

Slide 99

Slide 99 text

WHAT HAPPENS DURING EXCHANGE? 99 Tuesday, May 21, 13

Slide 100

Slide 100 text

Start With 2 Trees 100 Tuesday, May 21, 13

Slide 101

Slide 101 text

Compare Top Hashes TOP HASHES DON’T MATCH - SOMETHING IS DIFFERENT 101 Tuesday, May 21, 13

Slide 102

Slide 102 text

Compare Child Hashes NARROW DOWN THE DIVERGENT SEGMENT 102 Tuesday, May 21, 13

Slide 103

Slide 103 text

Recur NARROW DOWN THE DIVERGENT SEGMENT CONT... 103 Tuesday, May 21, 13

Slide 104

Slide 104 text

Iter Key-Hash Pairs ITER FINAL LIST OF HASHES TO FIND DIVERGENT KEYS 104 Tuesday, May 21, 13

Slide 105

Slide 105 text

Repair Divergent Keys REPAIR (RE-INDEX) KEYS THAT ARE DIVERGENT (RED) 105 Tuesday, May 21, 13

Slide 106

Slide 106 text

CODE FOR DETECTION AND REPAIR - NOT PREVENTION 106 Tuesday, May 21, 13

Slide 107

Slide 107 text

WHAT HAPPENS IF 3 NODES GO DOWN? 107 Tuesday, May 21, 13

Slide 108

Slide 108 text

Stop 3 Nodes 108 Tuesday, May 21, 13

Slide 109

Slide 109 text

Query 109 Tuesday, May 21, 13

Slide 110

Slide 110 text

CONSISTENCY vs. AVAILABILITY 110 Tuesday, May 21, 13

Slide 111

Slide 111 text

Uptime - Story of 9s UPTIME = (MTBF - MTTR) / MTBF 111 Tuesday, May 21, 13

Slide 112

Slide 112 text

Uptime is Flawed IF THE SYSTEM IS DOWN, BUT NO ONE MAKES A REQUEST, IS IT REALLY DOWN? 112 Tuesday, May 21, 13

Slide 113

Slide 113 text

Yield - Uptime of the People YIELD = QUERIES COMPLETED / QUERIES OFFERED 113 Tuesday, May 21, 13

Slide 114

Slide 114 text

Harvest vs. Yield HARVEST = DATA AVAIL / COMPLETE DATA IF FACE OF FAILURE YOU CAN’T HAVE BOTH FOR A SINGLE REQUEST 114 Tuesday, May 21, 13

Slide 115

Slide 115 text

IN TIMES OF TROUBLE - YOKOZUNA CHOOSES HARVEST FOR QUERIES 115 Tuesday, May 21, 13

Slide 116

Slide 116 text

TECHNICALLY - YOKOZUNA IS ALWAYS < 100% HARVEST IN A NON- QUIESCENT CLUSTER 116 Tuesday, May 21, 13