Scaling Memcache at Facebook Rajesh Nishtala, Hans Fugal, Steven Grimm, Marc Kwiatkowski, Herman Lee, Harry C. Li, Ryan McElroy, Mike Paleczny, Daniel Peek, Paul Saab, David Stafford, Tony Tung, Venkateshwaran Venkataramani JEZ@VVLJ ࿦จྠಡձ NSDI 13 In Proceedings of the 10th USENIX conference on Networked Systems Design and Implementation

Infrastructure Requirements • ϦΞϧλΠϜʹ͍ۙίϛϡχέʔγϣϯ • ෳ਺ͷσʔλιʔε͔ΒίϯςϯπΛଋͶΔ • ਓؾͷ͋ΔγΣΞ͞ΕͨίϯςϯπΛࢀর/ߋ৽ Ͱ͖Δ • ຖඵ਺ेԯϦΫΤετ·Ͱεέʔϧ͢Δ

Design Requirements • ඇৗʹॏ͍ read ෛՙͷαϙʔτ • over 1 G read per second • όοΫΤϯυαʔϏεΛॏ͍ read ͔Βִ཭ • ஍ཧతͳ෼ࢄ • ఆظతͳϓϩμΫτͷ૑ग़ • ༷ʑͳϢʔεέʔεʹରͯ͠ϑϨΩγϒϧ • ৽ػೳͷߴ଎։ൃαϙʔτ • Ӭଓ૚ͱΩϟογϡ૚ͷ෼཭

memcached • Facebook ͷ෼ࢄKVSͷجຊ୯Ґ • Trillions of items • Billions of requests / second • ωοτϫʔΫΞΫηεͰ͖ΔΠϯϝϞϦϋογϡ ςʔϒϧ • LRUͰϝϞϦ͔Β௥͍ग़͠

Roadmap 1. ୯ҰϑϩϯτΤϯυΫϥελ - ॏ͍ read ϫʔΫϩʔυ - Wide fanout - ϦΫΤετࣦഊͷϋϯυϦϯά 2. ෳ਺ϑϩϯτΤϯυΫϥελ - σʔλϨϓϦέʔγϣϯ - σʔλͷҰ؏ੑ 3. ෳ਺Ϧʔδϣϯ - σʔλͷҰ؏ੑ ࠷ऴܗ

memcache scale steps NFNDBDIFEαʔόͳ͠ ਺୆ͷNFNDBDIFEαʔό γϯάϧΫϥελ ϚϧνΫϥελ ஍ཧత෼ࢄΫϥελ

Query Cache (1) • 1. memcache ʹ get ϦΫΤετ • 2. Ωϟογϡϛεͨ͠ΒɺDB ʹ ΫΤϦΛ౤͛Δ • 3. memcache ʹΫΤϦ݁ՌΛ Ωϟογϡ • ී௨ͷϑϩʔ

Query Cache (2) • DBʹߋ৽͕͋ͬͨͱ͖Ωϟο γϡΛ”ແޮԽ”(invalidation)͢Δ ඞཁ͕͋Δ • ΩϟογϡΛߋ৽͢ΔΑΓ࡟আ • ࡟আ͸ႈ౳ੑ͕͋Δ • “demand-filled look-aside cache”

The problem of look-aside cache • ෳ਺ͷWebαʔό͕ set͢Δͱ͖ෆ੔߹ ͕ى͖Δ • ҟͳΔ஋Λฒྻʹset͠Α͏ͱ͢Δ • memcache ϓϩτίϧ֦ு “leases” • Ωϟογϡϛε࣌ʹτʔΫϯ(lease)Λ ൃߦ • Ωϟογϡϛεͨ͠ਓ͕set͢Δ • delete ࣌ʹτʔΫϯΛແޮԽ • τʔΫϯͷೝূʹࣦഊ͢ΔͱɺsetෆՄ TUBMFTFUT SFGIUUQTXXXVTFOJYPSHTJUFTEFGBVMUpMFTDPOGFSFODFQSPUFDUFEpMFTOJTIUBMB@OTEJ@TMJEFTQEG

The problem of look-aside cache • ಛఆͷΩʔ͕ແޮԽ → DBʹ Ұ੪ϑΥʔϧόοΫ • “leases”Λগ͠मਖ਼ • Ωϟογϡϛε࣌ʹsetΛ଴ͭ or ݹ͍ΩϟογϡΛࢀর͢Δ • ݹ͍ΩϟογϡΛઐ༻ͷσʔ λߏ଄ʹूΊ͓ͯ͘ 5IVOEFSJOH)FSET SFGIUUQTXXXVTFOJYPSHTJUFTEFGBVMUpMFTDPOGFSFODFQSPUFDUFEpMFTOJTIUBMB@OTEJ@TMJEFTQEG

memcache scaling steps NFNDBDIFEαʔόͳ͠ ਺୆ͷNFNDBDIFEαʔό γϯάϧΫϥελ ϚϧνΫϥελ ஍ཧత෼ࢄΫϥελ

Many memcached servers • Ωʔʹର͢Δ consistent-hashing Ͱ෼ࢄ • Ұ෦ͷΩʔ͚ͩΞΫηεස౓͕ߴ͍ • ϨϓϦέʔγϣϯͩͱϝϞϦޮ཰ѱ͍ • શWebαʔόͱmemcachedαʔό͕ଟରଟͰ௨৴ • hundreads of memcache gets per user request (eg. avg 521 fetches)

The problem of Consistent-Hashing 1BDLFU%SPQ • શ Web αʔό͔Βશ memcached ʹόʔετΞΫηε • ωοτϫʔΫεΠονͷόοϑΝ͕ᷓΕͯύέοτϩε • ᫔᫓੍ޚʹΑΓ TCP ίωΫγϣϯͷεϧʔϓοτ௿Լ • εϥΠσΟϯά΢Οϯυ΢Ͱಉ࣌ϦΫΤετ਺Λ੍ޚ • Ϩεϙϯε͕ฦ͖ͬͯͨΒ࣍ͷϦΫΤετΛ౤͛Δ • ϦΫΤετ੒ޭ਺ʹԠͯ͡΢Οϯυ΢αΠζΛখ͘͢͞Δ 5$1*ODBTU$POHFTUJPO

memcache scaling steps NFNDBDIFEαʔόͳ͠ ਺୆ͷNFNDBDIFEαʔό γϯάϧΫϥελ ϚϧνΫϥελ ஍ཧత෼ࢄΫϥελ

Multiple Clusters • ୆਺૿΍͚ͩ͢ͷεέʔϦϯ άͩͱ͖ͼ͍͠ • ෳ਺ͷϑϩϯτΤϯυΫϥε λ + ετϨʔδΫϥελ • ඞཁͳ͜ͱ • ֤ΫϥελؒͰҰ؏ੑͷҡ ࣋ • σʔλϨϓϦέʔγϣϯ

DB invalidate caches • DBߋ৽ޙɺ֤ϑϩϯτΤϯυΫϥελͷΩϟο γϡΛແޮԽ͠ͳ͚Ε͹ͳΒͳ͍ • MySQL ίϛοτϩάΛ tail ͯ͠ શ memcached ͷ ΩϟογϡΛແޮԽ͢ΔσʔϞϯʢMcSqueal) • ͋Β͔͡ΊɺແޮԽ͢΂͖ΩʔΛSQLʹຒΊࠐΉ

invalidate pipeline • memcached ͱ McSquealͷଟର ଟ௨৴ͰύέοτϨʔτ͕໰୊ʹ • ઐ༻ͷϧʔλʢmcrouter)ΛڬΉ • deleteཁٻͷѹॖ΋ • ύέοτ͋ͨΓͷdelete਺͕18ഒ 5PPNBOZQBDLFUT

memcache scaling steps NFNDBDIFEαʔόͳ͠ ਺୆ͷNFNDBDIFEαʔό γϯάϧΫϥελ ϚϧνΫϥελ ஍ཧత෼ࢄΫϥελ

Geographically distributed clusters • ϨϓϦΧ = ෳ਺ϑϩϯτΤϯυΫϥελ + ετ ϨʔδΫϥελ • ϨϓϦΧΛ֤Ϧʔδϣϯʹ഑ஔ • ϚελϨϓϦΧ • ϚελετϨʔδΫϥελΛ΋ͭϨϓϦΧ • ॻ͖ࠐΈ͸ϚελϨϓϦΧʹ

Writes in non-master replica • εϨʔϒϨϓϦΧ͔ΒϚελDB΁ͷॻ͖ࠐΈ • ϨϓϦ஗Ԇ͍ͯ͠ΔؒʹɺผίϯςΩετͰ Ωϟογϡϛε → Ωϟογϡ஋ set • ϚελDBʹ௥͍͍ͭͯͳ͍ͳ͍εϨʔϒDBͷ ஋͕Ωϟογϡ͞Εͯ͠·͏

Remote Markers • εϨʔϒϨϓϦ͔ΒϚελ΁ͷॻ͖ࠐΈ࣌ʹ͸ɺ ϚʔΧʔΛ͚ͭΔ • ϚʔΧʔ͕͍͍ͭͯͨΒɺϚελDB͔Β read • ϚʔΧʔ͕͍ͭͯͳ͔ͬͨΒɺεϨʔϒDB͔ Β read

Conclusion • Facbeookͷ੒௕ʹ͋ΘͤͯͲ͏΍ͬͯMemcache Λεέʔϧͤ͞Δ͔ • ΤϯδχΞϦϯάϦιʔεͷόϥϯεΛͱΕΔΑ͏ʹɺݱ࣮ʹଇͨ͠ ΍Γํ 1. ΩϟογϡͱӬଓετϨʔδΛ෼཭ͯ͠ɺಠཱͯ͠εέʔϧͤ͞Δ 2. ϞχλϦϯάɺσόοΪϯάɺΦϖϨʔγϣϯޮ཰Λվળ͢Δػೳ͸ ύϑΥʔϚϯεͱಉ͘͡Β͍ॏཁ 3. ϩδοΫ͸ stateless ͳΫϥΠΞϯτʹஔ͘΄͏͕ࠞཚ͠ͳ͍ 4. γεςϜ͸৽ػೳͷஈ֊తͳϩʔϧΞ΢τͱϩʔϧόοΫΛαϙʔτ ͠ͳ͚Ε͹ͳΒͳ͍ 5. Simplicity is vital.

ٙ໰ • ϨϓϦέʔγϣϯͷ࣮૷͸ʁ • memcached ͸ϨϓϦέʔγϣϯͷػೳͳ͍ • ΫϥΠΞϯτϥΠϒϥϦͷ࣮૷͸ʁ • Ͳ͜·ͰΞϓϦͷϩδοΫͱ෼཭͍ͯ͠Δͷ͔