Share
Embed
コンテナの研究開発から学ぶLinuxの要素技術
コンテナの研究開発から学ぶLinuxの要素技術
IEEE Computer Society Flagship Conference 採録を通じて
3-shake SRE Tech Talk #3 スリーシェイク技術顧問 松本亮介 / まつもとりー 2022/03/18
MATSUMOTO Ryosuke PRO
March 18, 2022
Transcript
56 slides
-
3-shake SRE Tech Talk #3 εϦʔγΣΠΫٕज़ސ দຊ྄հ / ·ͭͱΓʔ 2022/03/18
ίϯςφͷݚڀ։ൃ͔ΒֶͿLinuxͷཁૉٕज़ IEEE Computer Society Flagship Conference ࠾Λ௨ͯ͡ -
2 ɾגࣜձࣾεϦʔγΣΠΫ ٕज़ސ ɾ͘͞ΒΠϯλʔωοτݚڀॴ ্ڃݚڀһɺͦͷଞෳࣾͷٕज़ސ ɾใॲཧֶձ IOTݚڀձ OSݚڀձ ҕһɾװࣄ ɾΠϯλʔωοτٕज़ୈ163ҕһձ
ӡӦҕһ ɾIEEE / ACM / USENIX ֤छձһ ɾژେֶത࢜ʢใֶʣ ɾhttps://research.matsumoto-r.jp/ দຊ྄հ / ·ͭͱΓʔ / @matsumotory -
• COMPSAC: IEEE Computer Society Flagship International Conference • COMPSAC
2020 Message from the 2020 Program Chairs-in-Chief ※1 • over 450 submissions this year, to both our conference tracks and associated workshops • accepted 69 regular papers and 69 short papers • 76 papers that were not accepted for the main conference were referred to COMPSAC workshops • An additional 146 papers were submitted directly to our associated workshops • ͜ΕΒΛಡΉݶΓRegular Paperͷ࠾ 69 / (450 - 146) ͷ23%ҎԼ 4 COMPSAC 2020 Regular Paper ˞.FTTBHFGSPNUIF1SPHSBN$IBJSTJO$IJFG IUUQTJFFFDPNQTBDDPNQVUFSPSH -
• ݸਓ͕ͨΓલʹଟछଟ༷ͳWebαΠτΛ࣋ͭ࣌ • ΫϥυɾVPSͷΑ͏ͳࣗ༝ͱִڥʢΠϯελϯεʣʹର͢Δཁٻ • SNSΛհͯ͠ݸਓͷίϯςϯπΛ֦ࢄ͍࣌͢͠ • ݸਓͷWebαΠτͷΞΫηεूத͢Δػձͷ૿େ • ݸਓαΠτͰΞΫηεूதোͱ͍ͬͨมԽʹڧ͍ج൫͕ඞཁ
• ༧ଌͰ͖ͳ͍ΞΫηεͱϦιʔεׂΛϦΞΫςΟϒʹߦ͍͍ͨ • ଟछଟ༷ͳWebΞϓϦʹͰ͖Δ͚ͩରԠͯ͠બࢶΛఏڙ͍ͨ͠ 7 ݸਓͷWebαΠτͷΞΫηεूத৴པੑ -
1. Πϯελϯε্ͰWordPressͷΑ͏ͳҰൠత͔ͭଟ༷ͳWebΞϓϦ͕ಈ࡞Մೳ • ઐతͳ͕ࣝͳͯ͘ར༻Ͱ͖Δ҆ՁͳαʔϏεΛ࣮ݱ͍ͨ͠ 2. Πϯελϯεͷঢ়ଶมߋॲཧ͕ߴ • Πϯελϯε(ίϯςφ)ͷঢ়ଶͷఀࢭɾىಈɾεέʔϦϯάΛߴʹ॥ • ϦΫΤετ୯ҐͰϦΞΫςΟϒʹঢ়ଶΛܾఆ
→ มԽʹڧ͍ج൫ 3. ϋʔυΣΞϦιʔεͷར༻ޮΛ্ • ϦΫΤετ͕ແ͍ΠϯελϯεҰఆظؒىಈޙʹఀࢭ ΠϯελϯεΛߴूੵʹऩ༰ՄೳͰมԽʹڧ͍ԾԽج൫FastContainer 8 ߃ৗੑͷ͋ΔมԽʹڧ͍ج൫ͷؔ࿈ݚڀ※1 ˞3ZPTVLF.BUTVNPUP 6DIJP,POEP ,FOUBSP,VSJCBZBTIJ 'BTU$POUBJOFS")PNFPTUBUJD4ZTUFN"SDIJUFDUVSF)JHITQFFE"EBQUJOH&YFDVUJPO &OWJSPONFOU$IBOHFT 5IFSE"OOVBM*&&&*OUFSOBUJPOBM$PNQVUFST 4PGUXBSF BOE"QQMJDBUJPOT$POGFSFODF $0.14"$ +VMZ -
• Մ༻ੑ୲อͷͨΊʹෳΠϯελϯεՔಇͤ͞Δ͜ͱʹΑΔίετͷ૿େ • ෳͷऩ༰αʔόʹΠϯελϯεΛͦΕͧΕՔಇͤ͞Δ͜ͱͰՄ༻ੑΛ୲อ • ߴूੵʹΠϯελϯεऩ༰͢Δ؍͔ΒͰ͖Δ͚ͩϦιʔεΛઅ͍ͨ͠ • ར༻ऀઢͰՄ༻ੑΛ୲อͭͭ͠අ༻Λݮ͍ͨ͠ • ґવͱͯ͠ɺHTTPͷϨεϙϯελΠϜͷ؍͔ΒɺFastContainerͷHTTPϦ
ΫΤετܖػͰঢ়ଶΛมߋ͢Δ࣌ؒͷॖ՝ • ίϯςφࣗମ͘ىಈͯ͠தͰಈ͘αʔόϓϩηεͷىಈ͕͍ 9 FastContainer͓Αͼؔ࿈ݚڀͷՄ༻ੑͷ՝ -
1. Πϯελϯε্ͰWordPressͷΑ͏ͳҰൠతͳWebΞϓϦ͕ಈ࡞Մೳ 2. ୯ҰΠϯελϯεͰऩ༰αʔόো࣌ʹผαʔόࣗಈతʹ࠶ஔՄೳ 3. Πϯελϯεͷ࠶ஔͷ࣮ߦ࣌Ͱ͋ͬͯඵͷԆͰHTTPλΠϜΞτ͢ Δ͜ͱͳ͘ΦϯϥΠϯͰϨεϙϯεΛૹ৴Մೳ • ྫ͑εϚʔτϑΥϯͰ4GճઢΛܦ༝͢Δͱਓ͕ؒؾ͔ͮͳ͍Α͏ͳԆ ΠϯελϯεΛߴʹ࠶ஔՄೳʹ͢Δ͜ͱͰूੵΛ্ͤ͞
ίετͳج൫Λ࣮ݱ͢ΔεέδϡʔϦϯάख๏ͷఏҊ 10 ݚڀͷత: ϦΫΤετ୯Ґ ͰͷΠϯελϯε࠶ஔ -
• FastContainer ※1HTTPϦΫΤετʹԠͯ͡Ԡత͔ͭߴʹΠϯελϯε ͷঢ়ଶʢىಈɺఀࢭɺҠಈɺෳɺϦιʔε૿ݮʣΛܾఆ • αʔϏεར༻ऀWordpressͱ͔WebΞϓϦΛී௨ʹ͏Α͏ͳ͍ํ • ΞΫηεʹԠͨ͡ϦΞΫςΟϒͳεέʔϦϯάॲཧ͕Մೳ • ΫϥυαʔϏεج൫༧ΊΠϯελϯεΛىಈͤͯ͞ϦΫΤετΛॲཧ
• ΞΫηεूத࣌༧ଌత͔ͭϓϩΞΫςΟϒͳεέʔϦϯάॲཧ͕ඞཁ 12 FastContainerͱΫϥυαʔϏεج൫ͷಛ ˞3ZPTVLF.BUTVNPUP 6DIJP,POEP ,FOUBSP,VSJCBZBTIJ 'BTU$POUBJOFS")PNFPTUBUJD4ZTUFN"SDIJUFDUVSF)JHITQFFE"EBQUJOH&YFDVUJPO &OWJSPONFOU$IBOHFT 5IFSE"OOVBM*&&&*OUFSOBUJPOBM$PNQVUFST 4PGUXBSF BOE"QQMJDBUJPOT$POGFSFODF $0.14"$ +VMZ -
• ༷ʑͳWebΞϓϦέʔγϣϯ͕ར༻Մೳ • IaaSPaaSΛར༻ͨ͠Πϯελϯε୯ҐͰͷߏங͓ΑͼՄ༻ੑ୲อ͕Ұൠత • ຊݚڀͷՄ༻ੑ͕લఏͷ߹ɺෳऩ༰αʔόʹͦΕͧΕΠϯελϯεΛϗο τελϯόΠํࣜʢىಈࡁΈͷػΠϯελϯεʣͰػ͢Δ͜ͱͰ࣮ݱ • ऩ༰αʔόఀࢭ࣌ʹଈ࣌αʔϏεΛܧଓ͢ΔͨΊ •
ෳΠϯελϯεͷىಈ͕લఏͱͳΓϦιʔε༗ͷίετ͕૿େ • ϦΞΫςΟϒʹՄ༻ੑΛ୲อ͢ΔʹϨεϙϯελΠϜͷӨڹ͕େ͖͍ 14 FastContainerΫϥυج൫ͷՄ༻ੑ -
4UPSBHF $MJFOU 4FSWFS JOTUBODF" JOTUBODF# JOTUBODF$ 4FSWFS JOTUBODF" 'BTU$POUBJOFSΫϥυج൫ͷՄ༻ੑ )551ϦΫΤετ
4UPSBHF $MJFOU 4FSWFS JOTUBODF" JOTUBODF# JOTUBODF$ 4FSWFS JOTUBODF" )551ϦΫΤετ ✗ 15 αʔόো -
18 4UPSBHF $MJFOU 4FSWFS JOTUBODF" JOTUBODF# JOTUBODF$ 4FSWFS ఏҊख๏ʴ'BTU$POUBJOFSͷՄ༻ੑ )551ϦΫΤετ
4UPSBHF $MJFOU 4FSWFS JOTUBODF" JOTUBODF# JOTUBODF$ 4FSWFS )551ϦΫΤετ JOTUBODF" ✗Ԡతʹ ࠶ஔ αʔόো -
HTTP FastContainerͷجຊϑϩʔ 20 8FC1SPYZ ʢOHY@NSVCZ $.%# ʴ "1* 8FC%JTQBUDIFS OHY@NSVCZ
$MJFOU ίϯςφ ίϯςφ ίϯςφ w )551ϦΫΤετͷ)PTUOBNF ΛΩʔʹɺ$.%# ߏཧ%# ͔ΒίϯςφͷใΛऔಘ )551 4 ϦΫΤετ w ίϯςφͷ*1ͱϙʔτʹج͍ ͯίϯςφʹϓϩΩγ w ίϯςφ͕-JTUFO͍ͯ͠ͳ͍ ߹$.%#͔Βίϯςφ ใΛಘͯىಈ $POUBJOFS&OHJOF IBDPOJXB ऩ༰ϗετ" -
blocking each request with mruby 21 SFRVFTU NSVCZ NSVCZ SFTQPOTF
SFRVFTU SFRVFTU SFTQPOTF SFTQPOTF NSVCZ TFOESFTQPOTF SFDWSFRVFTU BUUIFTBNFUJNF Other responses are delayed in proportion to the time of processing of mruby blocking OPOCMPDLJOHNJEEMFXBSFMJLFOHJOYJOTJOHMFQSPDFTT -
non-blocking each request with mruby 23 SFRVFTU SFTQPOTF SFRVFTU SFRVFTU
SFTQPOTF SFTQPOTF TFOESFTQPOTF SFDWSFRVFTU BUUIFTBNFUJNF CMPDLJOH PQFSBJUPO NSVCZ CMPDLJOH PQFSBJUPO NSVCZ NSVCZ CMPDLJOH PQFSBJUPO OPOCMPDLJOHNJEEMFXBSFMJLFOHJOYJOTJOHMFQSPDFTT -
)PTU04 8FC1SPYZ $.%#"1* $POUBJOFS %JTQBUDIFS $POUBJOFS )PTU04 $POUBJOFS %JTQBUDIFS $POUBJOFS
$MJFOU )551 *$.1 )551 )551 )551 *$.1PS5$1 -
)PTU04 8FC1SPYZ $.%#"1* $POUBJOFS %JTQBUDIFS $POUBJOFS )PTU04 $POUBJOFS %JTQBUDIFS $POUBJOFS
$MJFOU )551 *$.1PS5$1 *$.1PS5$1 )551 )551 )551 ✗ ࠷ॳͷ࠶ஔ࣌ίϯςφͷىಈ͕ඞཁͰ͋Δ͕ɺ ىಈޙҰఆظؒىಈ͠ଓ͚Δɻ -
• ICMP/TCPͰᮢνΣοΫ͕Ұ࣌తʹޡݕͯ͠Өڹ͕গͳ͍ • TCPͷ߹ࣗ࡞TCPελοΫͰԟ෮3ύέοτͰνΣοΫ[3][4] • FastContainerͳͷͰޡݕͷ࠶ஔ͕ੜͯ͡αʔϏε͕ܧଓ͞ΕΔ • ޡݕͰଞαʔόʹىಈͯ͠͠·ͬͯҰఆ࣌ؒىಈͨ͠Βఀࢭ͢Δ • ݩαʔόʹ࠶ஔ͞ΕͯCMDB্ݩαʔόͷΈʹϦΫΤετ͕ྲྀΕΔ
• Ԡ࣌ؒͷᮢλΠϜΞτΛΪϦΪϦ·ͰνϡʔχϯάՄೳ 31 ఏҊख๏ͷϙΠϯτʢICMP/TCPࢹʣ <>NBUTVNPUPSZ NSVCZGBTUSFNPUFDIFDL IUUQTHJUIVCDPNNBUTVNPUPSZNSVCZGBTUSFNPUFDIFDL <>-JOVYΧʔωϧͷ5$1ελοΫͱγεςϜίʔϧͷΈ߹ΘͤʹΑΔख๏ΑΓߴʹϙʔτͷ-JTUFOνΣοΫΛ ߦ͏ IUUQTICNBUTVNPUPSKQFOUSZ -
• ίϯςφͷWebαʔόϓϩηεΛىಈྃલͰΠϝʔδԽ(Checkpoint) • ఆظతʹඇಉظͰϓϩηεͷىಈྃલΠϝʔδΛ࡞͓ͯ͘͠ • ϦΫΤετड৴࣌ʹΠϝʔδΛϓϩηεʹ෮ݩ(Restore) • αʔόϓϩηεͷॳظԽॲཧΛεΩοϓ • ىಈʹ࣌ؒͷ͔͔ΔΞϓϦέʔγϣϯαʔόʹ༗ར
• Ruby on RailsɼDjangoͳͲ 34 ఏҊख๏ͷϙΠϯτʢCheckpoint/Restoreʣ -
• αʔόϓϩηεͷىಈޙΛίϯςφϥϯλΠϜͰϑοΫͯ͠Checkpoint • ࢀߟ: seccompͰγεςϜίʔϧΛࢹ͠ptraceͰҰ࣌ఀࢭ͔ͯ͠ΒCRIUͰ CheckpointʹΑΔΠϝʔδԽͱ͍͏ํ๏͋Δ • CRIUͷதͰseccompΛ͓ͬͯΓύον͕ඞཁͰ൚༻ੑʹ͔͚Δ • seccompͷϓϩηεఀࢭʹCRIUͷػೳΛ͍ͬͯΔͳͲ
• seccomp࣮ߦޙʹݖݶΛམͱ͍ͯ͠ΔͨΊseccomp͕͑ͳ͍ͳͲ 37 CRIU+seccompʹΑΔFastContainerͷىಈ 04ϨΠϠͰ8FCαʔό͕ىಈ࣌ʹ࣮ߦ͢ΔγεςϜίʔϧΛࢹ͠ىಈྃલͷϓϩηεΛΠϝʔδԽ͢Δ IUUQTICNBUTVNPUPSKQFOUSZ -
• WebαʔόιϑτΣΞͷىಈॲཧྃͰɺ͔ͭωοτϫʔΫ͕Listen͍ͯ͠ͳ͍ ঢ়ଶͷϓϩηεΛΠϝʔδԽ͢Δ͜ͱΛࢦ͢ • seccompͰࢹ͢ΔγεςϜίʔϧlisten()Λઃఆ͠ɺΠϝʔδԽ͍ͨ͠αʔόϓϩ ηεΛfork()͔ͯ͠Βexecv() • ϓϩηε͔ΒରͷαʔόϓϩηεͷseccompΠϕϯτΛptrace()Ͱࢹ͠ɺ Listen()࣮ߦલʹΠϕϯτ͕ൃੜ •
Πϕϯτൃੜ࣌ʹϓϩηεΛCRIUͰΠϝʔδԽͯ͠อଘ 38 γεςϜίʔϧΛࢹͯ͠લͰΠϝʔδԽ 04ϨΠϠͰ8FCαʔό͕ىಈ࣌ʹ࣮ߦ͢ΔγεςϜίʔϧΛࢹ͠ىಈྃલͷϓϩηεΛΠϝʔδԽ͢Δ IUUQTICNBUTVNPUPSKQFOUSZ -
• ༧උ࣮ݧ: CRIUͱCheckpoint/Restore͢ΔϓϩηεͷϝϞϦαΠζͱͷؔ • දతͳΞϓϦέʔγϣϯΛͬͨίϯςφ࠶ஔ࣌ͷϨεϙϯελΠϜ • Apache 2.4.18ɼPHP 7.3.0ɼWordpress 5.0.3ʢσϑΥϧτϖʔδʣ
• Python 3.7.1ɼDjango 2.1.4ɼgunicorn 19.9.0※1 • Ruby 2.5.1ɼRails 5.2.1ɼPuma 3.12.0※2 43 ࣮ݧ༰ ˞IUUQTNDMPMJQPQ[FOEFTLDPNIDKBBSUJDMFT ˞IUUQTHJUIVCDPNFWFSZMFBGFMUSBJOJO ݱ࣮తͳنʢݸਓάϧʔϓͰͷར༻ͷΞϓϦέʔγϣϯఔʣͰ%#Λར༻ͨ͠ͷΛ࠾ -
• ComputeͰWebαʔόͷCheckpoint/RestoreͷΛܭଌ • mruby-simplehttpserver※1ͰWebαʔόΛىಈͤ͞setsockopt()Λࢹ • setsockopt()࣮ߦલʹCheckpoint • setsockopt()࣮ߦલʹϝϞϦΛ֬อͯ͠ɺϝϞϦαΠζʹԠͯ͡ Checkpoint/RestoreͷͷมԽΛܭଌ 44
༧උ࣮ݧɿϓϩηεͷΠϝʔδԽͷ࣮ݧ ˞NBUTVNPUPSZNSVCZTJNQMFIUUQTFSWFS IUUQTHJUIVCDPNNBUTVNPUPSZNSVCZTJNQMFIUUQTFSWFS -
αʔόϓϩηεͷΠϝʔδԽ(Checkpoint/Restore) 45 $IFDLQPJOU3FTUPSF1SPDFTTJOH5JNFEVFUP.FNPSZ6TBHF 1SPDFTTJOHUJNF<TFD>
.FNPSZVTBHFQFSQSPDFTT<.#> $IFDLQPJOU 3FTUPSF ୯ҰͷαʔόϓϩηεͷϝϞϦ༻ྔʹԠͨ͡$IFDLQPJOU3FTUPSFʹඞཁ -
• ComputeͰApache httpdͷϓϩηεΛมԽ • HTTPϦΫΤετΛܖػʹApache httpdΛىಈͤͯ͞ϨεϙϯεΛฦ͢ • ेόΠτͷ੩తͳindex.htmlʹର͢ΔϦΫΤετ • CRIUΛ͏߹
• CRIUΛΘͳ͍߹ 46 ༧උ࣮ݧ2ɿϓϩηεͱCRIUͷؔ -
• Apache 2.4.18ɼPHP 7.3.0ɼWordpress 5.0.3 • ϓϩηε3ɼ୯ҰͷϓϩηεͷϝϞϦαΠζ(RSS)35MBytes • Python 3.7.1ɼDjango
2.1.4ɼgunicorn 19.9.0 ※1 • ϓϩηε2ɼεϨου2ɼ୯ҰͷϓϩηεͷRSS33MBytes • Ruby 2.5.1ɼRails 5.2.1ɼPuma 3.12.0 ※2 • ϓϩηε2ɼεϨου14ɼ୯ҰͷϓϩηεͷRSS89MBytes • gemΛࣄલίϯύΠϧ͓ͯ͘͠bootsnapͱൺֱ 48 ίϯςφ࠶ஔ࣌ͷϨεϙϯελΠϜ -
• ୯ҰΠϯελϯεͰՄ༻ੑΛ୲อ͢ΔߴͳεέδϡʔϦϯάख๏ΛఏҊ • ෳΠϯελϯεΛඞཁͱ͠ͳ͍ͨΊϦιʔείετ͕͍ • ࣮ݧ͔Βݱ࣌Ͱ࣮༻ՄೳͳϨϕϧͷ࠶ஔͷੑೳ͕ಘΒΕͨ • ϓϩμΫγϣϯڥͰԠ༻ • ϗετো࣌Ͱ͋ͬͯϢʔβ͕ؾ͔ͮͳ͍ϨϕϧͰͷՄ༻ੑ
• ΦʔτεέʔϦϯά࣌ʹγʔϜϨεʹίϯςφΛ૿ͯ͠ෛՙରࡦՄೳʹ • ΞΫηεͱϦιʔεׂΓ͕ͯਖ਼֬ʹਵՄೳʹ • εέʔϦϯάϋʔυΣΞϓʔϧͷϦιʔεׂΓͯ࠷దԽ 53 ·ͱΊ -
• ݚڀ։ൃཁૉٕज़ͷධՁ͢Βඞཁͳ໘͕ଟ͍ • ཁૉٕज़ͷ࣮͚ͩͰͳ༷͘ʑͳڥͰͷධՁΛߦ͏ • ࣗવͱৄ͘͠ͳ͍ͬͯ͘ • ࠓճͷΑ͏ʹҰͭͷݚڀͰࢁͷཁૉٕज़ΛΓɺָ͘͠ͳΔ • ΈࠐΈεΫϦϓτݴޠ෦ɺϛυϧΣΞ෦ɺϊϯϒϩοΫI/Oɺύ
έοτɺCIRUɺseccompɺptrace • ઐՈʹͳͬͨؾʂʂʂʂ 54 ·ͱΊ -
• ࠷ॳͷWWW2020ʹఏग़ͯ͠Reject͞Εͨཧ༝ • ݚڀͷཱͪҐஔ͕ෆ໌֬ɺ৽نੑ͕͍·͍ͪΑ͘Θ͔Βͳ͍ • ຊݚڀͷཱͪҐஔલఏͷ໌֬Խ • ൺֱ͖ؔ͢࿈ݚڀ͕ෆ໌ྎ • ຊݚڀͱൺֱ͖ؔ͢࿈ݚڀΛॆ࣮ͤͯࠩ͞Λ໌֬Խ
ڭ܇: ΠϯλʔωοτɾWebٕज़ͰൃදจOSSͷٕज़ͷมԽ ͕ඇৗʹ͍ͨΊɺݚڀͷείʔϓͱ࠷৽ͷ՝Λ໌֬ʹ্ͨ͠ͰࠩΛ͔ͬ͠ Γͱࣔ͠ɺͦͷ՝͕ݱ࣮తʹͲΕ΄Ͳҙ͕ٛ͋Δ͜ͱͳͷ͔Λࣔ͢͜ͱ͕େࣄ 56 Accept·Ͱͷաఔ