Aerospikeで大規模データを扱う

by miuranobuaki

Slide 1

Slide 1 text

Aerospike Ͱ େن໛σʔλΛѻ͏ ࡾӜ৳໌ גࣜձࣾαΠόʔΤʔδΣϯτ / גࣜձࣾRightSegment 2016/08/02 - Aerospike meetup

Slide 2

Slide 2 text

ࣗݾ঺հ ࡾӜ৳໌ (@miuranobuak) גࣜձࣾαΠόʔΤʔδΣϯτ ΞυςΫελδΦ גࣜձࣾRightSegment ʹͯ PrivateDMP Λ։ൃ ΤϯδχΞ Podcast (http:/ /lean-agile.fm/)

Slide 3

Slide 3 text

ࠓ೔͓࿩͢Δ͜ͱ ɾAerospike ΛͲ͏࢖͍ͬͯΔ͔ ɾղܾ͖ͯͨ͠໰୊ ɾ࠷ۙࢼͯ͠Έͨ͜ͱ ɾࠓޙ΍ͬͯΈΑ͏ͱ͍ͯ͠Δ͜ͱ

Slide 4

Slide 4 text

AerospikeΛͲ͏࢖͍ͬͯΔ͔

Slide 5

Slide 5 text

RightSegmentͰͷ࢖͍ํ ϦΞϧλΠϜϢʔβ൑ఆ ৚͔݅ΒϢʔβΛநग़

Slide 6

Slide 6 text

RightSegment ߦಈ৘ใ΍ձһ৘ใɾ޿ࠂޮՌσʔλͳͲΛ஝ੵ ͠ɺDSP΍ΞυωοτϫʔΫͱ࿈ಈͯ͠Ϣʔβͷଐ ੑ΍޿ࠂ઀৮Λϕʔεʹߴ౓ͳ෼ੳʗϚʔέςΟϯ άΛߦ͏͜ͱ͕ՄೳͳϓϥΠϕʔτDMP ແཧ໼ཧγϯϓϧʹݴ͏ͱ • Ϣʔβʹద੾ͳΞϓϩʔνऔΕΔΑ͏ʹ൑ఆ • ద੾ͳ੾ΓޱͰ޿ࠂ഑৴

Slide 7

Slide 7 text

ϦΞϧλΠϜϢʔβ൑ఆ

Slide 8

Slide 8 text

ϦΞϧλΠϜϢʔβ൑ఆ • ࣄલʹઃఆͨ͠৚݅ʹ͋ͯ͸·Δ͔൑ఆ • ໿ 35,000 ʙ 40,000 TPS

Slide 9

Slide 9 text

શମͷΠϝʔδ

Slide 10

Slide 10 text

৚͔݅ΒϢʔβΛநग़

Slide 11

Slide 11 text

৚͔݅ΒϢʔβΛநग़ • ৚݅ʹ߹க͢ΔશϢʔβͷϦετΛ࡞੒ • 1ϢʔβϦετ͸ ਺ඦ ʙ ਺ઍສ݅ • நग़ͨ͠ϢʔβϦετΛ࿈ܞઌʹૹ৴ • Secondary IndexΛ࢖༻

Slide 12

Slide 12 text

શମΠϝʔδ

Slide 13

Slide 13 text

ղܾ͖ͯͨ͠໰୊

Slide 14

Slide 14 text

ղܾ͖ͯͨ͠໰୊ • ਪ঑SSDͰͳ͍ͱύϑΥʔϚϯε͕ग़ͳ͍(স) • ίϨΫγϣϯͷIndex͕࢖͑ͳ͍(౰࣌) • ΫϥΠΞϯτͷઃఆ΍εϖοΫ • αʔόͷઃఆ • etc... AerospikeͷαϙʔτͱҰॹʹղܾ શྗͰαϙʔτͯ͘͠ΕΔ

Slide 15

Slide 15 text

ղܾͨ͠໰୊ΛϐοΫΞοϓ

Slide 16

Slide 16 text

ղܾͨ͠໰୊ ͦͷ1 λΠϜΞ΢τසൃ • 1ϢʔβͰ਺ઍͷ৚݅ʹ౰ͯ͸·Δ৔߹΋͋Δ • Secondary Index ͕ ݪҼ • େྔRead/Write ͩͱ଱͑ΒΕͳ͍Β͍͠(౰࣌) ղܾํ๏ • Secondary Index༗ແͷ ΫϥελΛ࡞Δ • ͖ͬͱࠓͳΒେৎ෉ʁ

Slide 17

Slide 17 text

ղܾͨ͠໰୊ ͦͷ2 Ϋϥελʹউखʹ૊Έࠐ·ΕΔ • AerospikeͷΫϥελ͍͢͝ศར • ࡞ۀϛε΋͋Γɺผͷ2ͭͷΫϥελ͕ͬͭ͘͘ • ϊʔυΛΫϥελؒͰҠಈͤ͞Δͱ͖ʹൃੜ ղܾํ๏ • શ؀ڥͰNameSpaceΛΘ͚Δ

Slide 18

Slide 18 text

ղܾͨ͠໰୊ ͦͷ3 σʔλ࡟আ͕௥͍͔ͭͳ͍ • ࡟আ͕ଟ͍ͱ࣮ࡍʹফ͑Δ·Ͱ͕͔͔࣌ؒΔ • ૣ͘͢Δ͜ͱ΋Ͱ͖Δ͕ύϑΥʔϚϯε͕མͪΔ • େྔʹ࡟আ͢ΔͳΒؾΛ͚ͭͨํ͕ྑͦ͞͏ ղܾํ๏ • σʔλͷੜଘظؒΛͪΌΜͱઃఆ͢Δ • ύʔςΟγϣϯαΠζม͑ͨΒྑ͘ͳͬͨΓ͠·

Slide 19

Slide 19 text

࠷ۙ΍ͬͯΈͨ͜ͱ

Slide 20

Slide 20 text

࠷৽όʔδϣϯΛࢼͯ͠Έͨ ɾ3.5.15 → 3.9.0.1 ΁όʔδϣϯΞοϓ ɾόʔδϣϯΞοϓҎ֎Կ΋͍ͯ͠ͳ͍ ͜ΕͰͲΕ͚ͩมΘΔͷ͔

Slide 21

Slide 21 text

Migration Node௥Ճ΍੾Γ཭͠Λͯ͠IN/OUT͕0ʹͳΔ·Ͱ ɾMigrationThread=1 20h -> 5h ɾMigrationThread=50 4h -> 50min ໿5ഒͷεϐʔυ!!

Slide 22

Slide 22 text

ॻ͖ࠐΈੑೳ • Secondary Index Λ࢖༻ • UDFΛར༻ͨ͠ॻ͖ࠐΈ 3.5.15 -> 22,000TPS 3.9.0.1 -> 45,000TPSʂʂ ݶք·Ͱ͸ݕূͰ͖ͳ͔ͬͨ ͚Ͳɺ·ͩ·͍͚ͩͦ͏

Slide 23

Slide 23 text

ࠓޙ΍ͬͯΈΑ͏ͱ͍ͯ͠Δ͜ͱ

Slide 24

Slide 24 text

Migration໰୊ • σʔλྔ͕ଟ͍ͱ Migration ͷෛՙ͕ߴ͍ • σʔλྔ͕ଟ͍ͱόΧʹͳΒͳ͍ • ϦΞϧλΠϜॲཧʹ͸க໋తʹ • Nodeͷ௥Ճ΋ා͍ • NodeΛൈ͘ͷ͸΋ͬͱා͍

Slide 25

Slide 25 text

ϋʔυো֐ͷରԠ • ͱ͸ݴͬͯ΋ϋʔυ͸ઈରʹյΕΔ ͯ͞ɺͲ͏ͨ͠΋ͷ͔ • λΠϜΞ΢τ͕େྔൃੜ͢Δঢ়گ͸ආ͚͍ͨ • ΋͏ҰݸΫϥελͭ͘Ε͹ྑ͍Μ͡Ό? • XDRͰ͖Δ͡ΌΜʂ

Slide 26

Slide 26 text

ͱ͍͏Θ͚Ͱ • ݱࡏXDR؀ڥΛߏஙɺݕূத • ݩͷΫϥελΛ2ͭʹ෼͚ͯ࡞੒ • ग़དྷΔݶΓmigrationΛආ͚Δ • όʔδϣϯΞοϓʹ΋ॊೈʹରԠͰ͖ΔͷͰ͸ʁ

Slide 27

Slide 27 text

৭ʑͱ໰୊Λ͖͋͛ͯ·͕ͨ͠

Slide 28

Slide 28 text

ݱঢ়ͷ࢓૊Έ͸ Aerospike Ͱͳ͍ͱ࣮ݱͰ͖͍ͯͳ͍

Slide 29

Slide 29 text

͔ͳΓཁٻ͕ߴ͘ͳ͍ͬͯ·͢ ࣮ࡍɺ௒଎͍Ͱ͢ɻ αϙʔτ΋ੌ͍ʂ

Slide 30

Slide 30 text

·ͱΊ ɾ௒଎͍͚Ͳɺۜͷ஄ؙ͡Όͳ͍ ɾσʔλಛੑ Λཧղͯ͠ઃܭ͢Δ ɾmigrationൃੜ࣌ͷෛՙݕূ Λ ɾසൟʹ࡟আ͞ΕΔ ৔߹΋ݕূ͢΂͖

Slide 31

Slide 31 text

࠷ޙʹ ҰॹʹؤுΔ ஥ؒ Λืू͍ͯ͠·͢ʂ େن໛σʔλ Λ ࠷৽ٕज़ Ͱ ࠷ߴͷՁ஋ Λಧ͚Δ͜ͱʹڵຯ͕͋Δํ ੋඇɺ͝࿈བྷ͍ͩ͘͞ʂ

Slide 32

Slide 32 text

͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠