Scaling Memcache at Facebook

id:y_uuki 論文輪読会#8

Transcript

1. Scaling Memcache at Facebook Rajesh Nishtala, Hans Fugal, Steven Grimm, Marc Kwiatkowski, Herman Lee, Harry C. Li, Ryan McElroy, Mike Paleczny, Daniel Peek, Paul Saab, David Stafford, Tony Tung, Venkateshwaran Venkataramani JEZ@VVLJ ࿦จྠಡձ NSDI 13 In Proceedings of the 10th USENIX conference on Networked Systems Design and Implementation
2. Infrastructure Requirements • ϦΞϧλΠϜʹ͍ۙίϛϡχέʔγϣϯ • ෳ਺ͷσʔλιʔε͔ΒίϯςϯπΛଋͶΔ • ਓؾͷ͋ΔγΣΞ͞ΕͨίϯςϯπΛࢀর/ߋ৽ Ͱ͖Δ • ຖඵ਺ेԯϦΫΤετ·Ͱεέʔϧ͢Δ
3. Design Requirements • ඇৗʹॏ͍ read ෛՙͷαϙʔτ • over 1 G read per second • όοΫΤϯυαʔϏεΛॏ͍ read ͔Βִ཭ • ஍ཧతͳ෼ࢄ • ఆظతͳϓϩμΫτͷ૑ग़ • ༷ʑͳϢʔεέʔεʹରͯ͠ϑϨΩγϒϧ • ৽ػೳͷߴ଎։ൃαϙʔτ • Ӭଓ૚ͱΩϟογϡ૚ͷ෼཭
4. memcached • Facebook ͷ෼ࢄKVSͷجຊ୯Ґ • Trillions of items • Billions of requests / second • ωοτϫʔΫΞΫηεͰ͖ΔΠϯϝϞϦϋογϡ ςʔϒϧ • LRUͰϝϞϦ͔Β௥͍ग़͠
5. Roadmap 1. ୯ҰϑϩϯτΤϯυΫϥελ - ॏ͍ read ϫʔΫϩʔυ - Wide fanout - ϦΫΤετࣦഊͷϋϯυϦϯά 2. ෳ਺ϑϩϯτΤϯυΫϥελ - σʔλϨϓϦέʔγϣϯ - σʔλͷҰ؏ੑ 3. ෳ਺Ϧʔδϣϯ - σʔλͷҰ؏ੑ ࠷ऴܗ
6. memcache scale steps  NFNDBDIFEαʔόͳ͠  ਺୆ͷ
   NFNDBDIFE
   αʔό  γϯάϧΫϥελ  ϚϧνΫϥελ  ஍ཧత෼ࢄΫϥελ
7. Query Cache (1) • 1. memcache ʹ get ϦΫΤετ • 2. Ωϟογϡϛεͨ͠ΒɺDB ʹ ΫΤϦΛ౤͛Δ • 3. memcache ʹΫΤϦ݁ՌΛ Ωϟογϡ • ී௨ͷϑϩʔ
8. Query Cache (2) • DBʹߋ৽͕͋ͬͨͱ͖Ωϟο γϡΛ”ແޮԽ”(invalidation)͢Δ ඞཁ͕͋Δ • ΩϟογϡΛߋ৽͢ΔΑΓ࡟আ • ࡟আ͸ႈ౳ੑ͕͋Δ • “demand-filled look-aside cache”
9. The problem of look-aside cache • ෳ਺ͷWebαʔό͕ set͢Δͱ͖ෆ੔߹ ͕ى͖Δ • ҟͳΔ஋Λฒྻʹset͠Α͏ͱ͢Δ • memcache ϓϩτίϧ֦ு “leases” • Ωϟογϡϛε࣌ʹτʔΫϯ(lease)Λ ൃߦ • Ωϟογϡϛεͨ͠ਓ͕set͢Δ • delete ࣌ʹτʔΫϯΛແޮԽ • τʔΫϯͷೝূʹࣦഊ͢ΔͱɺsetෆՄ TUBMF
   TFUT SFG
   IUUQTXXXVTFOJYPSHTJUFTEFGBVMUpMFTDPOGFSFODFQSPUFDUFEpMFTOJTIUBMB@OTEJ@TMJEFTQEG
10. The problem of look-aside cache • ಛఆͷΩʔ͕ແޮԽ → DBʹ Ұ੪ϑΥʔϧόοΫ • “leases”Λগ͠मਖ਼ • Ωϟογϡϛε࣌ʹsetΛ଴ͭ or ݹ͍ΩϟογϡΛࢀর͢Δ • ݹ͍ΩϟογϡΛઐ༻ͷσʔ λߏ଄ʹूΊ͓ͯ͘ 5IVOEFSJOH
    )FSET SFG
    IUUQTXXXVTFOJYPSHTJUFTEFGBVMUpMFTDPOGFSFODFQSPUFDUFEpMFTOJTIUBMB@OTEJ@TMJEFTQEG
11. memcache scaling steps  NFNDBDIFEαʔόͳ͠  ਺୆ͷ
    NFNDBDIFE
    αʔό  γϯάϧΫϥελ  ϚϧνΫϥελ  ஍ཧత෼ࢄΫϥελ
12. Many memcached servers • Ωʔʹର͢Δ consistent-hashing Ͱ෼ࢄ • Ұ෦ͷΩʔ͚ͩΞΫηεස౓͕ߴ͍ • ϨϓϦέʔγϣϯͩͱϝϞϦޮ཰ѱ͍ • શWebαʔόͱmemcachedαʔό͕ଟରଟͰ௨৴ • hundreads of memcache gets per user request (eg. avg 521 fetches)
13. The problem of Consistent-Hashing 1BDLFU
    %SPQ • શ Web αʔό͔Βશ memcached ʹόʔετΞΫηε • ωοτϫʔΫεΠονͷόοϑΝ͕ᷓΕͯύέοτϩε • ᫔᫓੍ޚʹΑΓ TCP ίωΫγϣϯͷεϧʔϓοτ௿Լ • εϥΠσΟϯά΢Οϯυ΢Ͱಉ࣌ϦΫΤετ਺Λ੍ޚ • Ϩεϙϯε͕ฦ͖ͬͯͨΒ࣍ͷϦΫΤετΛ౤͛Δ • ϦΫΤετ੒ޭ਺ʹԠͯ͡΢Οϯυ΢αΠζΛখ͘͢͞Δ 5$1
    *ODBTU
    $POHFTUJPO
14. memcache scaling steps  NFNDBDIFEαʔόͳ͠  ਺୆ͷ
    NFNDBDIFE
    αʔό  γϯάϧΫϥελ  ϚϧνΫϥελ  ஍ཧత෼ࢄΫϥελ
15. Multiple Clusters • ୆਺૿΍͚ͩ͢ͷεέʔϦϯ άͩͱ͖ͼ͍͠ • ෳ਺ͷϑϩϯτΤϯυΫϥε λ + ετϨʔδΫϥελ • ඞཁͳ͜ͱ • ֤ΫϥελؒͰҰ؏ੑͷҡ ࣋ • σʔλϨϓϦέʔγϣϯ
16. DB invalidate caches • DBߋ৽ޙɺ֤ϑϩϯτΤϯυΫϥελͷΩϟο γϡΛແޮԽ͠ͳ͚Ε͹ͳΒͳ͍ • MySQL ίϛοτϩάΛ tail ͯ͠ શ memcached ͷ ΩϟογϡΛແޮԽ͢ΔσʔϞϯʢMcSqueal) • ͋Β͔͡ΊɺແޮԽ͢΂͖ΩʔΛSQLʹຒΊࠐΉ
17. invalidate pipeline • memcached ͱ McSquealͷଟର ଟ௨৴ͰύέοτϨʔτ͕໰୊ʹ • ઐ༻ͷϧʔλʢmcrouter)ΛڬΉ • deleteཁٻͷѹॖ΋ • ύέοτ͋ͨΓͷdelete਺͕18ഒ 5PP
    NBOZ
    QBDLFUT
18. memcache scaling steps  NFNDBDIFEαʔόͳ͠  ਺୆ͷ
    NFNDBDIFE
    αʔό  γϯάϧΫϥελ  ϚϧνΫϥελ  ஍ཧత෼ࢄΫϥελ
19. Geographically distributed clusters • ϨϓϦΧ = ෳ਺ϑϩϯτΤϯυΫϥελ + ετ ϨʔδΫϥελ • ϨϓϦΧΛ֤Ϧʔδϣϯʹ഑ஔ • ϚελϨϓϦΧ • ϚελετϨʔδΫϥελΛ΋ͭϨϓϦΧ • ॻ͖ࠐΈ͸ϚελϨϓϦΧʹ
20. Writes in non-master replica • εϨʔϒϨϓϦΧ͔ΒϚελDB΁ͷॻ͖ࠐΈ • ϨϓϦ஗Ԇ͍ͯ͠ΔؒʹɺผίϯςΩετͰ Ωϟογϡϛε → Ωϟογϡ஋ set • ϚελDBʹ௥͍͍ͭͯͳ͍ͳ͍εϨʔϒDBͷ ஋͕Ωϟογϡ͞Εͯ͠·͏
21. Remote Markers • εϨʔϒϨϓϦ͔ΒϚελ΁ͷॻ͖ࠐΈ࣌ʹ͸ɺ ϚʔΧʔΛ͚ͭΔ • ϚʔΧʔ͕͍͍ͭͯͨΒɺϚελDB͔Β read • ϚʔΧʔ͕͍ͭͯͳ͔ͬͨΒɺεϨʔϒDB͔ Β read
22. Conclusion • Facbeookͷ੒௕ʹ͋ΘͤͯͲ͏΍ͬͯMemcache Λεέʔϧͤ͞Δ͔ • ΤϯδχΞϦϯάϦιʔεͷόϥϯεΛͱΕΔΑ͏ʹɺݱ࣮ʹଇͨ͠ ΍Γํ 1. ΩϟογϡͱӬଓετϨʔδΛ෼཭ͯ͠ɺಠཱͯ͠εέʔϧͤ͞Δ 2. ϞχλϦϯάɺσόοΪϯάɺΦϖϨʔγϣϯޮ཰Λվળ͢Δػೳ͸ ύϑΥʔϚϯεͱಉ͘͡Β͍ॏཁ 3. ϩδοΫ͸ stateless ͳΫϥΠΞϯτʹஔ͘΄͏͕ࠞཚ͠ͳ͍ 4. γεςϜ͸৽ػೳͷஈ֊తͳϩʔϧΞ΢τͱϩʔϧόοΫΛαϙʔτ ͠ͳ͚Ε͹ͳΒͳ͍ 5. Simplicity is vital.
23. ٙ໰ • ϨϓϦέʔγϣϯͷ࣮૷͸ʁ • memcached ͸ϨϓϦέʔγϣϯͷػೳͳ͍ • ΫϥΠΞϯτϥΠϒϥϦͷ࣮૷͸ʁ • Ͳ͜·ͰΞϓϦͷϩδοΫͱ෼཭͍ͯ͠Δͷ͔