検索エンジン自作入門 Go Conference 2021 Spring

by kotaroooo0

Slide 1

Slide 1 text

Go Conference 2021 Spring 2021/04/24 @kotaroooo0 ݕࡧΤϯδϯ ࣗ࡞ೖ໳

Slide 2

Slide 2 text

҆ୡޫଠ࿠ @kotaroooo0 αʔόʔαΠυΤϯδχΞ CLIπʔϧ, Bot, ݕࡧΤϯδϯ ࣗݾ঺հ 2

Slide 3

Slide 3 text

͜ͷτʔΫ: ࣗ࡞ݕࡧΤϯδϯͷ࿩ https://kotaroooo0-dev.hatenablog.com/ 3

Slide 4

Slide 4 text

λʔήοτ ݕࡧΤϯδϯʹڵຯ͋Δ ݕࡧΤϯδϯΛࣗ࡞͍ͨ͠ ΰʔϧ ݕࡧΤϯδϯͷ࢓૊Έ͕෼͔Δ GoͰݕࡧΤϯδϯΛࣗ࡞࢝͠ΊΒΕΔ 4

Slide 5

Slide 5 text

Ξ΢τϥΠϯ 1. ݕࡧΤϯδϯͱ͸ ҰൠతͳݕࡧΤϯδϯͷ࢓૊Έͱߏ੒ཁૉ 2. ࣗ࡞ͨ͠ݕࡧΤϯδϯͷ঺հ ۩ମతʹࣗ࡞ͨ͠ݕࡧΤϯδϯͷߏ੒ཁૉͱಈ࡞ྫ 3. ࣗ࡞ͨ͠ݕࡧΤϯδϯͷ࣮૷ ΞϧΰϦζϜͱσʔλߏ଄ɺϥΠϒϥϦ 4. ͓ΘΓʹ ݕࡧΤϯδϯΛࣗ࡞ͨ͠ײ૝ 5

Slide 6

Slide 6 text

Ξ΢τϥΠϯ 1. ݕࡧΤϯδϯͱ͸ ҰൠతͳݕࡧΤϯδϯͷ࢓૊Έͱߏ੒ཁૉ 2. ࣗ࡞ͨ͠ݕࡧΤϯδϯͷ঺հ 3. ࣗ࡞ͨ͠ݕࡧΤϯδϯͷ࣮૷ 4. ͓ΘΓʹ 6

Slide 7

Slide 7 text

ݕࡧΤϯδϯͱ͸ υΩϡϝϯτͷू߹͔ΒΫΤϦʹద͢ΔυΩϡϝϯτΛฦ͢ ݕࡧΤϯδϯ ΫΤϦೖྗ υΩϡϝϯτฦ٫ υΩϡϝϯτొ࿥ 🤖 👦 Ϣʔβʔଆ ఏڙଆ ಛʹෳ਺υΩϡϝϯτ͔ΒಛఆͷจࣈྻΛݕࡧ͢Δ͜ͱΛશจݕࡧ 7 (GoogleݕࡧͰ͸Webϖʔδ) (GoogleݕࡧͰ͸ΫϩʔϦϯά)

Slide 8

Slide 8 text

શจݕࡧͷछྨ G O R U B Y J S J S J S J S J S J S G O R U B Y G O R U B Y J S Grepܕ ΠϯσοΫεܕ ྫ: υΩϡϝϯτʮGO RUBY JSʯʹจࣈྻʮJSʯؚ͕·Ε͍ͯΔ͔ݕࡧ 8 ࣄલʹݕࡧର৅ͷυΩϡϝϯτͷࡧҾσʔλΛ࡞੒

Slide 9

Slide 9 text

ΠϯσοΫεܕݕࡧΤϯδϯͷߏ੒ ΠϯσοΫε ΫΤϦೖྗ υΩϡϝϯτొ࿥ Analyzer Analyzer Indexer Searcher υΩϡϝϯτฦ٫ 9 Sorter ҎԼͷཁૉͰߏ੒͞Ε͍ͯ·͢(ৄࡉ͸͜ͷޙઆ໌͠·͢) ※Sorter͸࣮૷͍ͯ͠ͳ͍ͨΊɺҎޙͷઆ໌Ͱ͸লུ

Slide 10

Slide 10 text

ΠϯσοΫε సஔΠϯσοΫε(ࣙॻ+ϙεςΟϯάϦετ)͕Ұൠత go js ruby 1 1 2 3 4 5 5 ࣙॻ ϙεςΟϯάϦετ 10

Slide 11

Slide 11 text

Indexing “I have pens.” “We have a desk.” Indexer Analyzer จࣈྻΛ୯ޠ΁෼ղ సஔΠϯσοΫε࡞੒ 11 1 2 have pen we desk 1 1 2 2 2 సஔΠϯσοΫε υΩϡϝϯτΛసஔΠϯσοΫεʹొ࿥͢Δ

Slide 12

Slide 12 text

Search ΫΤϦ “PENS” υΩϡϝϯτ1͕ώοτ Searcher Analzyer จࣈྻΛ୯ޠ΁෼ղ “PENS” → “pen” సஔΠϯσοΫεΛ୳ࡧ 12 have pen we desk 1 1 2 2 2 సஔΠϯσοΫεΛ୳ࡧͯ͠ద߹͢ΔυΩϡϝϯτΛ୳͢

Slide 13

Slide 13 text

Analyzer Char Filter จࣈྻΛ୯ޠ΁෼ׂ͢Δલͷจࣈͷௐ੔ ྫ: & → & 13 Token Filter ෼ׂͨ͠୯ޠΛௐ੔ ྫ: খจࣈ΁౷Ұɺετοϓϫʔυআڈ Tokenizer จࣈྻΛ୯ޠʹ෼ׂɺදه༳Εٵऩ ྫ: I have pens → I, have, pens Char Filter Tokenizer Token Filter 3ͭ߹ΘͤͯAnalyzer จࣈྻ ෳ਺୯ޠ දه༳ΕΛٵऩ &จࣈྻΛ୯ޠʹ෼ׂ Indexing, Search྆ํͰར༻ ※྆ํͰಉ͡AnalyzerΛ࢖Θͳ͍ͱޮՌ͸ബ͍

Slide 14

Slide 14 text

Ξ΢τϥΠϯ 1. ݕࡧΤϯδϯͱ͸ 2. ࣗ࡞ͨ͠ݕࡧΤϯδϯͷ঺հ ۩ମతʹࣗ࡞ͨ͠ݕࡧΤϯδϯͷߏ੒ཁૉͱಈ࡞ྫ 3. ࣗ࡞ͨ͠ݕࡧΤϯδϯͷ࣮૷ 4. ͓ΘΓʹ 14

Slide 15

Slide 15 text

ͳͥݕࡧΤϯδϯΛࣗ࡞͍ͯ͠Δͷ͔ Elasticsearch͕ٕज़తɺϓϩμΫτతʹ໘ന͍ → શจݕࡧΤϯδϯͷ࢓૊ΈΛ஌Γ͍ͨ GoݚमΛडߨ → GoͰԿ͔࡞Γ͍ͨ GoͰݕࡧΤϯδϯΛ࣮૷͠Α͏💪 15

Slide 16

Slide 16 text

ࣗ࡞ͨ͠ݕࡧΤϯδϯͷ঺հ 16 ※Sorter͸ະ࣮૷

Slide 17

Slide 17 text

ಈ࡞ྫ Analyzer Analyzeલ MappingCharFilter StandardTokenizer LowercaseFilter StemmerFilter StopWordFilter I feel TIRED :( I feel TIRED sad I, feel, TIRED, sad i, feel, tired, sad i, feel, tire, sad feel, tire, sad ॲཧͷྲྀΕ 17

Slide 18

Slide 18 text

Analyzeલ MorphologicalTokenizer ReadingformFilter ౦ژͱژ౎ ౦ژ, ͱ, ژ౎ tokyo, to, kyoto ॲཧͷྲྀΕ ಈ࡞ྫ Analyzer 18

Slide 19

Slide 19 text

ಈ࡞ྫ Indexing & Search 19 ΠϯσοΫε΁௥Ճ 1: “Go Ruby PHP” 2: “Go PHP Python” ϑϨʔζݕࡧ ”go php” ANDݕࡧ ”go php” 2ͷΈώοτ 1,2͕ώοτ 1⃣ 2⃣ 3⃣ 1⃣ 2⃣ 3⃣ 3: “Go Python Ruby”

Slide 20

Slide 20 text

ಈ࡞ྫ ઃఆ΍ॳظԽ • సஔΠϯσοΫεΛӬଓԽ͢ΔMySQLઃఆ • MySQL΁ΞΫηε͢ΔӬଓԽ૚ͷॳظԽ • ΞφϥΠβͷॳظԽ • CharFilter: ͳ͠ • Tokenizer: εϖʔεͰ۠੾Δ(Standard Tokenizer) • Token Filter: খจࣈԽ 20

Slide 21

Slide 21 text

ಈ࡞ྫ Indexing • సஔΠϯσοΫεΛ࡞੒͢ΔIndexerͷॳظԽ • ҎԼͷυΩϡϝϯτΛIndexing(సஔΠϯσοΫεΛੜ੒͠MySQL΁อଘ) 1: “Go Ruby PHP” • Analyzeޙ: go, ruby, php 2: “Go PHP Python” • Analyzeޙ: go, php, python 3: “Go Python Ruby” • Analyzeޙ: go, python, ruby 21 go 1 1:1 ruby php 2:1 3:1 1:2 1:3 3:3 2:2 ɾ ɾ ɾ υΩϡϝϯτID:ग़ݱҐஔ

Slide 22

Slide 22 text

ಈ࡞ྫ Search ANDݕࡧ(“GO PHP”) • ΞφϥΠβͰେจࣈখจࣈͷදه༳ΕΛٵऩ • “Go”ͱ”PHP”͕྆ํؚ·ΕΔυΩϡϝϯτ1,2͕ώοτ 22 1: “Go Ruby PHP” 2: “Go PHP Python”

Slide 23

Slide 23 text

ಈ࡞ྫ Search ϑϨʔζݕࡧ(“GO PHP”) • ΞφϥΠβͰେจࣈখจࣈͷදه༳ΕΛٵऩ • ϑϨʔζ”Go PHP”ΛؚΉυΩϡϝϯτ2ͷΈώοτ 23 2: “Go PHP Python”

Slide 24

Slide 24 text

Ξ΢τϥΠϯ 1. ݕࡧΤϯδϯͱ͸ 2. ࣗ࡞ͨ͠ݕࡧΤϯδϯͷ঺հ 3. ࣗ࡞ͨ͠ݕࡧΤϯδϯͷ࣮૷ ΞϧΰϦζϜͱσʔλߏ଄ɺϥΠϒϥϦ 4. ͓ΘΓʹ 24

Slide 25

Slide 25 text

Analyzerͷ࣮૷ 25 Analyzerߏ଄ମ • CharFilter InterfaceͷSlice • Tokenizer Interface(1ͭͷΈ) • ToknenFilter InterfaceͷSlice Analyzeϝιου • string → TokenStream(TokenͷSlice) • CharFilter, Tokenizer, TokenFilterͷॱʹ దԠ

Slide 26

Slide 26 text

Char Filterͷ࣮૷ 26 CharFilter Interface • Filterϝιου ࣮૷ྫ: MappingCharFilter • Key → Value΁Ϛοϐϯά͢Δ

Slide 27

Slide 27 text

Tokenizerͷ࣮૷ 27 Tokenizer Interface • Tokenizeϝιου ࣮૷ྫ: StandardTokenizer • String → TokenStream • จࣈͰ΋਺ࣈͰ΋ͳ͚Ε͹෼ׂ

Slide 28

Slide 28 text

Token Filterͷ࣮૷ 28 TokenFilter Interface • Filterϝιου ࣮૷ྫ: LowercaseFilter • strings.ToLowerͰখจࣈԽ ࣮૷ྫ: StopWordFilter • ର৅ͷจࣈΛTokenStream͔Βল͘

Slide 29

Slide 29 text

Token Filterͷ࣮૷ ReadingformFilter • ׽ࣈ → ϩʔϚࣈ΁ม׵͢ΔFilter • ׽ࣈ → ฏԾ໊ → ϩʔϚࣈ • ׽ࣈ → ฏԾ໊(ܗଶૉղੳث͕ߦ͏) • ฏԾ໊ → ϩʔϚࣈ(ࣗ࡞) ࣮૷ • ۪௚ͳϚοϐϯά+ ϔϘϯࣜͷϧʔϧ 29

Slide 30

Slide 30 text

సஔΠϯσοΫεͷ࣮૷ సஔΠϯσοΫε(ࣙॻ+ϙεςΟϯάϦετ) • Ϛοϓ: τʔΫϯID → ϙεςΟϯάϦετ ϙεςΟϯάϦετ • ϙεςΟϯάͷϦϯΫτϦετ ϙεςΟϯά • υΩϡϝϯτID • υΩϡϝϯτதͷҐஔ৘ใ(ϑϨʔζݕࡧ༻) • ࣍ͷϙεςΟϯά΁ͷϙΠϯλ 30

Slide 31

Slide 31 text

సஔΠϯσοΫεͷ࣮૷ 31 0 TokenID(mapͷkey) 3 PostingListͷPostings Postings 1 6 17 90 198 5 8 14 Postings Postings nil Next Next Next Next ※ऴ୺͸nilΛࢦ͢ TokenID 0ͷτʔΫϯ͕ υΩϡϝϯτID3,5,8,14ʹؚ·Ε͍ͯΔ͜ͱΛࣔ͢

Slide 32

Slide 32 text

సஔΠϯσοΫεͷ࣮૷ 32 0 TokenID 3 PostingListͷPostings Postings 1 3 7 90 198 5 8 14 Postings Postings nil Next Next Next Next ※ऴ୺͸nilΛࢦ͢ 7 Postings Next ϦϯΫτϦετ: ͋Δϊʔυ௚ޙͷσʔλͷ௥Ճ࡟আO(1) ϙεςΟϯάϦετͰ͸υΩϡϝϯτIDͷঢॱΛΩʔϓ͢ΔͨΊ ͜ͷૢ࡞͕ଟ͘ϦϯΫτϦετ͕޲͍͍ͯΔ ϙΠϯλͷ෇͚ସ͑ͷΈ

Slide 33

Slide 33 text

Indexerͷ࣮૷ ( ͷ۩ମྫ) ϝϞϦ্ͷసஔΠϯσοΫε ௥Ճ͍ͨ͠υΩϡϝϯτ go php go php MySQL্ͷసஔΠϯσοΫε సஔΠϯσοΫε΁௥Ճ ϝϞϦͱMySQLͷసஔΠϯσοΫε Ϛʔδ ӬଓԽ ”Go PHP” go, php Analyze 33 Analyzer Indexer MySQL͔ΒRead 1⃣ 2⃣ 3⃣ ϝϞϦ্ͷసஔΠϯσοΫεͷ αΠζ͕ᮢ஋Λ௒͑Δͱ… 4⃣ 3⃣ MySQL΁Write (ᮢ஋͸ετϨʔδ΁ͷΞΫηεճ਺ͱϝϞϦ࢖༻ྔͷτϨʔυΦϑ)

Slide 34

Slide 34 text

Indexerͷ࣮૷ Indexerͷιʔείʔυ͸ ϙεςΟϯάϦετͷ ϦϯΫτϦετΛ૸͕ࠪओ 34

Slide 35

Slide 35 text

Searcherͷ࣮૷ 35 go php go php MySQL্ͷసஔΠϯσοΫε ݕࡧΫΤϦ ”Go PHP” go, php Analyzer Analyze Searcher ద߹ͨ͠υΩϡϝϯτ τʔΫϯʹରԠ͢ΔϙεςΟϯάϦετΛ MySQL͔ΒϝϞϦ΁Read ϙεςΟϯάϦετ ૸ࠪ

Slide 36

Slide 36 text

ద߹͢ΔυΩϡϝϯτΛ୳ͨ͢ΊʹͲ͏ϙεςΟϯάϦετΛͲ͏૸ࠪ͢Δ͔? DAAT(Document At A Time) • ΫΤϦʹؚ·ΕΔτʔΫϯͷϙεςΟϯάϦετͷΧʔιϧΛ͢΂ͯ։͖ಉ ࣌ʹॲཧ • υΩϡϝϯτ͝ͱʹ૸ࠪ͢Δ • Elasticsearch(Lucene)Ͱ΋࠾༻ ※DAATʹରͯ͠TAAT(Team At A Time)͕͋Δ͕ࠓճ͸ϊʔλον ࢀߟࢿྉΛఴ෇ https://www.slideshare.net/tsubosaka/top-kquery Searcherͷ࣮૷ 36

Slide 37

Slide 37 text

DAATͰͷANDݕࡧ Χʔιϧ͕ࢦ͢υΩϡϝϯτ͕ҟͳΔͷͰΧʔιϧΛਐΊΔ Result = [] 37 ྫ “PHP” AND “Ruby” ݕࡧ ruby php 1 2 5 5 2 4

Slide 38

Slide 38 text

DAATͰͷANDݕࡧ Χʔιϧ͕ࢦ͢υΩϡϝϯτ౳͍͠ͷͰ݁ՌʹυΩϡϝϯτΛՃ͑Δ Result = [2] 38 ྫ “PHP” AND “Ruby” ݕࡧ ruby php 1 5 5 2 4 2

Slide 39

Slide 39 text

DAATͰͷANDݕࡧ Χʔιϧ͕ࢦ͢υΩϡϝϯτ͕ҟͳΔͷͰΧʔιϧΛਐΊΔ Result = [2] 39 ྫ “PHP” AND “Ruby” ݕࡧ ruby php 1 5 2 4 2 5

Slide 40

Slide 40 text

DAATͰͷANDݕࡧ Χʔιϧ͕ࢦ͢υΩϡϝϯτ౳͍͠ͷͰ݁ՌʹυΩϡϝϯτΛՃ͑Δ Result = [2, 5] 40 ྫ “PHP” AND “Ruby” ݕࡧ ruby php 1 5 2 4 2 5

Slide 41

Slide 41 text

DAATͰͷORݕࡧ ΧʔιϧΛಈ͔ͯ͠ɺશͯͷIDΛॏෳͳ͘௥Ճɹ ✌︎ (‘ω' ✌︎ ) ݪ࢝త ( ✌︎ ’ω') ✌︎ ※࣮ࡍͷݕࡧΤϯδϯͰ͸ORݕࡧ͸࠷దԽ͕ߦΘΕ͍ͯΔ(͜ͷτʔΫͰ͸ϊʔλον) 41 ྫ “PHP” OR “Ruby” ݕࡧ ruby php 1 2 5 5 2 4

Slide 42

Slide 42 text

ҎԼೋͭͷυΩϡϝϯτ΁“Amazon Prime”ͱݕࡧ • D1: “Amazon Prime movies” • D2: “a prime concern of Amazon” ϙεςΟϯάʹυΩϡϝϯτதͷҐஔ৘ใ ୯ޠͷॱংΛߟྀͰ͖Δ D1ͷΈΛώοτͤ͞ΒΕΔ 42 ϑϨʔζݕࡧ

Slide 43

Slide 43 text

ΫΤϦ“Go PHP”Ͱݕࡧ͢Δ৔߹ 43 ϑϨʔζݕࡧ go 1:1 php 2:1 1:3 2:2 υΩϡϝϯτID:ग़ݱҐஔ go 1:1-0 php 2:1-0 1:3-1 2:2-1 υΩϡϝϯτID:૬ରग़ݱҐஔ go 1:1 php 2:1 1:2 2:1 υΩϡϝϯτID:૬ରग़ݱҐஔ ૬ରग़ݱҐஔ͕౳͍͠ → ϑϨʔζΛؚΜͰ͍Δ 1: “Go Ruby PHP” 2: “Go PHP Python” ΫΤϦ্ͰGo͸0൪໨ɺPHP͸1൪໨ͳͷͰ૬ରग़ݱҐஔΛܭࢉ͢Δ సஔΠϯσοΫεʹొ࿥ͨ͠υΩϡϝϯτ

Slide 44

Slide 44 text

Searcherͷιʔείʔυ΋ ϙεςΟϯάϦετͷ ϦϯΫτϦετͷૢ࡞͕ओ 44 Searcherͷ࣮૷

Slide 45

Slide 45 text

Storageͷ࣮૷ ӬଓԽʹMySQLΛ࠾༻ • τʔΫϯ΍υΩϡϝϯτͷ࠾൪Λ؆୯ʹߦ͍͔ͨͬͨ • AUTO_INCREMENT • ӬଓԽʹ࿑ྗΛׂ͖ͨ͘ͳ͍ • ࢖͍׳Ε͍ͯΔMySQL • औΓସ͑΍͍͢Α͏ʹӬଓԽ૚Λந৅Խ 45

Slide 46

Slide 46 text

సஔΠϯσοΫεͷѹॖ సஔΠϯσοΫεΛετϨʔδ͔ΒಡΈग़͕࣌ؒ͢ϘτϧωοΫʹͳΔ͜ͱ͕͋Δ → సஔΠϯσοΫεΛѹॖͯ͠อଘ͢Δ 46 ѹॖࡁͷసஔΠϯσοΫεΛ ετϨʔδ͔ΒಡΈग़࣌ؒ͢ ະѹॖͷసஔΠϯσοΫεΛ ετϨʔδ͔ΒಡΈग़࣌ؒ͢ < ্ͷෆ౳͕ࣜ੒ΓཱͭΑ͏ʹѹॖ ※ࠓճ͸ܭଌ·Ͱ͸͍ͯ͠ͳ͍ 600MB 1GB ׬ྃ ಡΈग़͠ ෮ݩ ಡΈग़͠ ׬ྃ సஔΠϯσοΫεͷ෮ݩ࣌ؒ ʴ

Slide 47

Slide 47 text

సஔΠϯσοΫεͷѹॖ go υΩϡϝϯτID ̍ 4 21 241 412 500 560 600 888 1324 2000 2321 10940 23131 29898 3001 8090 9012 3001 8090 9012 901 1201 ϙεςΟϯάϦετʹυΩϡϝϯτID͕Կेສͱ͋Δ৔߹΋͋Γ༰ྔ͕େ͖͘ͳΔ → Մม௕ූ߸Խͱࠩ෼ྻΛ૊Έ߹Θͤͯѹॖ 47

Slide 48

Slide 48 text

సஔΠϯσοΫεͷѹॖ Մม௕ූ߸ ੔਺஋ʹରͯ͠4όΠτ΍8όΠτͷݻఆ௕ූ߸ΛׂΓ౰ͯΔͷ͕ී௨ ྫ͑͹64ϏοτͷintͰ͸1΋9223372036854775807΋ಉ͡όΠτ਺ ͔͠͠ ϙεςΟϯάϦετ͸খ͍͞੔਺஋Λଟ͘ѻ͏ 48 খ͍͞੔਺஋ΛΑΓখ͍͞όΠτ਺Ͱදݱ͢ΔՄม௕ූ߸ԽΛߦ͏

Slide 49

Slide 49 text

సஔΠϯσοΫεͷѹॖ ࠩ෼ྻ go ̍ 5 50 250 500 800 1000 2500 2800 υΩϡϝϯτID͸ঢॱʹιʔτ͞Ε͍ͯΔ ࠩ෼ΛͱΔ → Մม௕ූ߸Խʹ͓͍ͯ৘ใྔ͕গͳ͘ͳΔ go ̍ 4 45 200 250 300 200 1500 300 υΩϡϝϯτIDࠩ෼ 49

Slide 50

Slide 50 text

సஔΠϯσοΫεͷѹॖ Մม௕ූ߸Խ • Մม௕ූ߸Խ͸ࣗ࡞ͤͣʹඪ४ύοέʔδ”encoding/gob"Λར༻ • JSON΍XML, Protocol buffersͳͲ͋Δ͕Goʹดͨ͡؀ڥͰ͸gobͷํ͕ศར • json:”hoge"ͷΑ͏ͳλά͕ෆཁ • ςΩετϕʔεͰͳ͘όΠφϦʔͰແବ͕ͳ͍ • ࣗݾهड़త • Goͷม਺ͳΒstructͰ΋nilͰ΋ΤϯίʔυͰ͖Δ(ϦϯΫτϦετ΋Ͱ͖ͨ) • gobͷunsigned intͷΤϯίʔυ • 0 → 00000000(1byte) • 7 → 00000111(1byte) • 256 → 11111110 000000001 00000000(3byte) https://blog.golang.org/gob https://golang.org/pkg/encoding/gob/ 50

Slide 51

Slide 51 text

సஔΠϯσοΫεͷѹॖ 51

Slide 52

Slide 52 text

Ξ΢τϥΠϯ 1. ݕࡧΤϯδϯͱ͸ 2. ࣗ࡞ͨ͠ݕࡧΤϯδϯͷ঺հ 3. ࣗ࡞ͨ͠ݕࡧΤϯδϯͷ࣮૷ 4. ͓ΘΓʹ ݕࡧΤϯδϯΛࣗ࡞ͨ͠ײ૝ 52

Slide 53

Slide 53 text

͓ΘΓʹ 53 • ंྠͷ࠶ൃ໌ͰΠϯϓοτͱΞ΢τϓοτͷαΠΫϧΛճͤͨ • ࣗ࡞ָ͕͍͠(Ϟνϕʔγϣϯ) • ৄ͘͠ͳ͍ٕज़ʹ৮ΕΔඞཁ(Πϯϓοτ) • (ྫ)ΤϯίʔσΟϯά • encoding/gob • encoding/binary • protobuf • धཁ͕͋Δ(Ξ΢τϓοτ) • ϒϩά౤ߘ→ొஃ • Ξ΢τϓοτۦಈ։ൃ INPUT OUTPUT

Slide 54

Slide 54 text

ࢀߟ 54 ݕࡧΤϯδϯࣗ࡞ೖ໳ɹʙखΛಈ͔͠ͳ͕Βݟ౉͢ݕࡧͷ෣୆ཪ ৘ใݕࡧ ݕࡧΤϯδϯͷ࣮૷ͱධՁ https://logmi.jp/tech/articles/322211 https://www.slideshare.net/tsubosaka/top-kquery https://artem.krylysov.com/blog/2020/07/28/lets-build-a-full-text-search-engine/ https://github.com/blevesearch/bleve ϦϙδτϦ: https://github.com/kotaroooo0/stalefish ϒϩάهࣄ: https://kotaroooo0-dev.hatenablog.com/entry/toy-search-engine

Slide 55

Slide 55 text

Appendix 55

Slide 56

Slide 56 text

• DAATͱରʹͳΔϙεςΟϯάϦετͷ૸ࠪํ๏ • ϙεςΟϯάϦετΛ̍ͭͣͭॲཧ • ಉ࣌ʹ։͘ϙεςΟϯάϦετͷΧʔιϧ͸̍ͭͷΈ • ୯ޠ͝ͱʹ૸ࠪ 56 TAAT(Term At A Time)

Slide 57

Slide 57 text

TAATͰͷANDݕࡧ 1. ϙεςΟϯάϦετͷαΠζ͕࠷খͷ΋ͷ”ruby”Λબ୒͠ɺAccumulator࡞੒ [2, 5] 2. “php”ͷϦετΛ૸ࠪ ɾID2͸ؚ·Ε͍ͯΔ͔?ID5͸ؚ·Ε͍ͯΔ͔?ͷΈͷνΣοΫͰOK go ruby php 1 1 2 4 5 5 2 57 “PHP” AND “Ruby” ݕࡧ

Slide 58

Slide 58 text

TAATͰͷORݕࡧ 1. ͲͷΩʔͰ΋ྑ͍ͷͰAccumlatorΛ࡞੒ [1,2,5] (php) 2. ॏෳ͠ͳ͍શͯͷIDΛAccumlatorʹ௥Ճ ✌︎ (‘ω' ✌︎ ) ݪ࢝త ( ✌︎ ’ω') ✌︎ ※࣮ࡍͷݕࡧΤϯδϯͰ͸ORݕࡧ͸࠷దԽ͕ߦΘΕ͍ͯΔ(͜ͷτʔΫͰ͸৮Ε·ͤΜ) 58 “PHP” OR “Ruby” ݕࡧ ruby php 1 2 5 5 2