Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Riak 2.0における全文検索 と整合性の問題

E1923013dacab39eb231a2fffbf7b33c?s=47 UENISHI Kota
November 19, 2014

Riak 2.0における全文検索 と整合性の問題

WebDB Forum 2014 のBashoジャパンの発表スライドです

E1923013dacab39eb231a2fffbf7b33c?s=128

UENISHI Kota

November 19, 2014
Tweet

Transcript

  1. Riak 2.0ʹ͓͚Δશจݕࡧ ͱ੔߹ੑͷ໰୊ WebDB Forum 2014 Basho Japan ্੢ 2014/11/19

  2. ࣗݾ঺հ • ෼ࢄγεςϜྺ6೥ • Bashoδϟύϯͷํ͔Βདྷ·ͨ͠ • Riak CSͷ։ൃ • ͦͷଞ೔ຊͷ͜ͱ

  3. BashoͱRiak •෼ࢄσʔλϕʔεʁ •RiakΛ஌͍ͬͯΔʁ •BashoΛ஌͍ͬͯΔʁ

  4. We are hiring •࣮ੈքͷ෼ࢄγεςϜͷ ໰୊ʹڵຯ͋Δਓʂ •@BashoJapan •kota@basho.com

  5. •APࢦ޲ͷσʔλϕʔεʢ݁Ռ੔߹ੑʣ •҆ఆੑɺ༧ଌՄೳੑ •ʮσʔλΛઈରʹͳ͘͞ͳ͍ʯ •৽͍͠ݕࡧΤϯδϯͱͯ͠SolrΛόϯυϧ

  6. ͜Μͳͱ͜ΖͰ ಈ͍͍ͯ·͢Riak •Rovio (Angry Birds) •Yahoo!JAPAN ͷΫϥ΢υετϨʔδ •NHS (ΠΪϦε ࠃຽอݥαʔϏε)

    •League of Legends (MMORPG) •ۜߦɺήʔϜɺখചɺηϯαʔɺetc…
  7. CAPఆཧͱཧ૝ͷDB •ͲΜͳނোʹରͯ͠΋ (partition tolerance) •σʔλ͸ৗʹ੔߹͓ͯ͠Γ (consistency) •γεςϜ͕ࢭ·Δ͜ͱ͸ͳ͍ (availability) ͜ͷ3ͭΛಉ࣌ʹຬͨ͢γεςϜ͸ଘࡏ͠ͳ͍

  8. CAP Theorem • C: ෳ਺ͷAtomic Objectʹର͢Δ࿈ ଓͨ͠ૢ࡞ (w1, w3, w4,

    ….) ͕શ ͯಉҰͰ͋Δ͜ͱ (linearizable) • A: Atomic Objectʹૢ࡞ w1, w2, …Λ࣮ߦͰ͖Δ͜ͱ • P: ωοτϫʔΫ͕੾ΕΔͱAtomic Object΁ͷϝοηʔδ͕౸ୡ͠ͳ͍
  9. Is the network reliable?

  10. None
  11. None
  12. How Riak Works (mostly about search)

  13. run Riak with Search $ which java $ sudo apt-get

    install riak $ echo “search = on” >> /etc/riak/riak.conf $ riak start $ riak-admin bucket-type create tt $ riak-admin bucket-type activate tt $ curl -XPUT http://localhost:8098/search/index/t $ curl -XPUT -H ‘content-type: application/json’ \ http://localhost:8098/types/tt/props \ -d ‘{“props”:{“search_index”:”t”}}’
  14. Consistent Hashing • 160-bit Ωʔۭؒ • ۭؒΛ౳෼͢Δ • ύʔςΟγϣϯ͸ϊʔ υ͕ݸผ؅ཧ

    • ϨϓϦΧ͸Nݸͷύʔ ςΟγϣϯʹίϐʔ͞ ΕΔ OPEF OPEF OPEF OPEF hash(“meetups/spamham”) N=3
  15. σʔλϕʔεͷݕࡧ • DB૊ࠐܕ • groongaͳͲ • “LIKE” • Pros •

    ੔߹ੑΛҡ࣋͠΍͍͢ • Cons • ີ݁߹ʹͳΔ • ֎෦࿈ܞܕ • Solr • Elasticsearch • Pros • ૄ݁߹ʹͰ͖Δ • Cons • ੔߹ੑΛҡ࣋͠ʹ͍͘
  16. DBͱΠϯσοΫεͷ੔߹ੑ • DBͷσʔλຊମͱɺΠϯσοΫεΛ౰࣌ʹ҆શʹߋ ৽͢ΔͨΊʹ͸τϥϯβΫγϣϯ͕ඞཁ • ີ݁߹ͷγεςϜͰ͋Ε͹ϩοΫͳͲΛ࢖ͬͯ੔߹ੑ Λอো͠΍͍͢

  17. ૄ݁߹ʹ͢ΔϝϦοτ • ಉ͡ݕࡧΤϯδϯͰ͍ΖΜͳDBʹରԠͰ͖Δ • ӡ༻ϊ΢ϋ΢΍πʔϧɺ։ൃΛूதͰ͖Δ • όʔδϣϯ੔߹ͷ੍໿͕؇࿨͞ΕΔ • ยํ͕ނো͍ͯͯ͠΋΋͏ยํ͸ಈ࡞Ͱ͖Δ •

    ֦ு͠΍͘͢ͳΔ
  18. ྆ํͱ΋ ͍͍ͱ͜ͲΓ͍ͨ͠

  19. Yokozuna

  20. as Riak Search 2.0 = + σʔλͷӬଓԽ Ϋϥελͷ؅ཧ γϯϓϧͳૢ࡞ RVFSZ

    JOEFYJOH
  21. Riak Search 2.0 • RiakʹSolrΛόϯυϧ • RiakͷKVΛߋ৽͢Δͱσʔλ͕ࣗಈతʹΠϯσο ΫεԽ͞ΕΔ • SolrͷϓϥάΠϯ΍εΩʔϚɺΫΤϦΛͦͷ··࢖

    ͏͜ͱ͕Ͱ͖Δ • ৑௕Խɺϊʔυ௥Ճ࡟আ͸Riakͷ୲౰
  22. ૄ݁߹ʁີ݁߹ʁ • ύοέʔδ͸ಉࠝ • ϓϩηεͱͯ͠͸ૄ݁߹ • ΠϯσοΫεͱσʔλؒ͸݁Ռ੔߹ੑΛอো • ΠϯσοΫεͷ৑௕Խ͸݁Ռ੔߹ੑΛอো

  23. Solrͱ͸ૄ݁߹ • Riak ϓϩηε͕ Solr ͷϓϩηεΛ fork ͢Δ • ؅ཧɺ؂ࢹ͸શͯRiak͕ߦ͏

    • ผϓϩηεɺϝϞϦۭؒ΋ผ
  24. ΠϯσοΫεԽͷྲྀΕ • Riak΁ͷߋ৽ͱҰ৺ಉମʢdocument-based indexingʣ • ΠϯσοΫεԽʹࣦഊ͢ΔͱPUT͕ࣦഊ

  25. ݕࡧΫΤϦͷॲཧ • Riakͷ2i΍MapReduceͱಉछͷΫΤϦ͕૸Δ • 3͓͖ͭͷ vnode ʹΞΫηε͢ΔͷͰɺॏෳ͸ͳ͍ ू໿

  26. ੔߹ੑͷ໰୊ ͱҟৗܥ ੔߹ੑҡ࣋ͷͨΊʹߋ৽ϩάΛอ࣋ͯ͠ ϦΧόϦ͢Δͱ͍͏ํ๏Ͱ͸ɺՄ༻ੑͱ ෼அ଱ੑΛ୲อͰ͖ͳ͍ɻRiakͰ͸ଞͷ ํ๏Ͱσʔλͷ੔߹ੑΛҡ࣋͢Δɻ

  27. Α͋͘Δҟৗܥ #1 •ϊʔυނোͰෳ੡Λ3ͭ࡞Εͳ͍ •෮چͯ͠΋ɺ2ͭͷ··ʹͳͬͯ͠·͏

  28. Α͋͘Δҟৗܥ #1 • ෮چͨ͋͠ͱʹRead Repair͢Δ • GETͨ͠ͱ͖ʹɺσʔλʹ͕ܽؕ͋ͬͨΒ RiakଆͰPUT͠௚͢ v2 v2

    not found v2
  29. Read Repair͕ ݺ͹Εͳ͍ͱ σʔλ͕௚Βͳ͍ʁ YES

  30. ΫϥΠΞϯτ͔Β GET͕ݺ͹Εͳ͍ͱ σʔλ͕௚Βͳ͍ʁ NO

  31. AAE (Active Anti Entropy) https://www.flickr.com/photos/51pct/7507525118/

  32. Active Anti Entropy • APࢦ޲ͷDBͷσʔλྼԽΛ๷͙ ͨΊͷόοΫάϥ΢ϯυॲཧ • Merkle-TreeΛ࢖ͬͯύʔςΟγϣ ϯຖͷʮνΣοΫαϜʯΛܭࢉ •

    ࠩ෼Λݟ͚ͭͨΒͦ͜ΛRead Repair͢Δ hash(vnode=0, pid=0) hash(vnode=1, pid=0) hash(vnode=2, pid=0)
  33. Α͋͘Δҟৗܥ #2 • Solr͕ΠϯσοΫεԽ͢Δલʹམͪͨ • Key, Value͸อଘ͞Ε͕ͨΠϯσοΫε͸ͳ͍

  34. Read Repair͕ ݺ͹ΕΕ͹Πϯσο Ϋε͕म෮͞ΕΔʁ NO

  35. AAE (Active Anti Entropy) https://www.flickr.com/photos/51pct/7507525118/

  36. YZ Active Anti Entropy • ݕࡧΠϯσοΫεʹ͍ͭͯ΋ Merkle TreeΛ࡞੒ • ύʔςΟγϣϯຖʹKey-Valueͷ

    Treeͱൺֱ͠ɺ͕ࠩ͋Ε͹Πϯ σοΫεΛमਖ਼ hash(vnode=0, pid=0, kv) hash(vnode=0, pid=0, yz)
  37. Α͋͘Δҟৗܥ #2 •Searchઐ༻AAE͕ಈ࡞͍ͯ͠Δ •ΠϯσοΫεͱσʔλͷෆ੔߹Λݟ͚ͭͯ म෮͢Δ AAE

  38. Α͋͘Δҟৗܥ #3 • ωοτϫʔΫ͕੾Ε͍ͯͨͱ͖ʹ྆ଆͰߋ৽͕ى ͖ͨʢSplit Brainʣ • Hinted HandoffʹΑΓͲͪΒ΋ॻ͖ࠐΈ੒ޭ

  39. Α͋͘Δҟৗܥ #3 • ෮چ࣌ʹHandoffͰฦ͞Εɺ྆ํͷ஋Λอ࣋ • GET·ͨ͸AAEʹΑͬͯRead Repair͞ΕΔ

  40. ҟৗܥ #1, #2, #3 •Ͳͷҟৗ΋ɺಛʹۓٸͷରॲ͸ඞཁͳ͍ •੔߹ੑݕࠪͱम෮͕ࣗಈͰߦΘΕΔ •ϧʔνϯϫʔΫͷӡ༻࡞ۀ͕΄ͱΜͲͳ͍

  41. ·ͱΊ • σʔλϕʔεͱશจݕࡧͷ໰୊ • ੔߹ੑΛͱΔ͔ɺૄ݁߹ΛͱΔ͔ͷ໰୊ • ৽͍͠Riak Search 2.0 •

    ਫฏ෼ࢄͯ͠εέʔϧΞ΢τͰ͖Δݕࡧ • ݁Ռ੔߹ੑΛબͿ͜ͱʹΑΓૄ݁߹Խͱ෼ࢄԽ ʹ੒ޭʢ͔͠΋APܕʹʂʣ • ݁Ռ੔߹ੑΛબ୒ͨ͠ઃܭʹΑΓӡ༻͕؆୯ʹ
  42. Questions?

  43. FAQ •ΦϯϥΠϯͷεΩʔϚมߋʁ •μΠφϛοΫεΩʔϚʁ

  44. outline • σʔλϕʔεʹೖ͍ͬͯΔσʔλΛશจݕࡧ͍ͨ͠→୔ࢁͷΠϯσοΫε͕Ͱ͖ΔɺFKeyͰ ΠϯσοΫε…ͱ͍͏͚ͩͰ͸ • શจݕࡧΤϯδϯͷछྨ • ֎෦ܕʢ੔߹ੑΛҡ࣋͠ʹ͍͕͘Ԡ༻͕ޮ͘ɺރΕͨιϑτ΢ΣΞ΍ϓϥάΠϯ͕ଟ͍ʣ • ૊ΈࠐΈܕʢ੔߹ੑΛҡ࣋͠΍͍͕͢ɺϝϯςφϯε͕೉͍͠ʣ

    • Riak͸ͦͷதؒɹόϯυϧͯ͠ΔͷͰָɺ͚ͩͲ࣮ମ͸Solrɺ੔߹ੑΛࣗಈతʹҡ࣋͢Δ • όϯυϧͷํ๏ɺ੔߹ੑɺ෼ࢄͷํ๏ • ݕࡧΠϯσοΫε͚ͩΛ෼ࢄͤ͞Δͷ͸೉͍͠ • DBͱΠϯσοΫεͷ੔߹ɺϨϓϦΧͱϨϓϦΧͷ੔߹ɺJepsenͷ͸ͳ͠