Rapid Container Scheduling for Reactive Relocation of Individual HTTP Requests

Rapid Container Scheduling for Reactive Relocation of Individual HTTP Requests

Rapid Container Scheduling for Reactive Relocation of Individual HTTP Requests
FIT2020 トップカンファレンスセッション

2020/09/03
さくらインターネット株式会社
さくらインターネット研究所
上級研究員
松本亮介 / @matsumotory / Ryosuke Matsumoto

2b692bd83f4418103142a053ecf5ff59?s=128

MATSUMOTO Ryosuke

September 03, 2020
Tweet

Transcript

  1. ͘͞ΒΠϯλʔωοτגࣜձࣾ (C) Copyright 1996-2020 SAKURA Internet Inc ͘͞ΒΠϯλʔωοτݚڀॴ Rapid Container

    Scheduling for Reactive Relocation of Individual HTTP Requests 2020/09/03 ্ڃݚڀһ দຊ ྄հ FIT2020 τοϓΧϯϑΝϨϯεηογϣϯ
  2. • COMPSAC: IEEE Computer Society Flagship International Conference • COMPSAC

    2020 Message from the 2020 Program Chairs-in-Chief ※1 • over 450 submissions this year, to both our conference tracks and associated workshops • accepted 69 regular papers and 69 short papers • 76 papers that were not accepted for the main conference were referred to COMPSAC workshops • An additional 146 papers were submitted directly to our associated workshops • ͜ΕΒΛಡΉݶΓRegular Paperͷ࠾୒཰͸ 69 / (450 - 146) ͷ23%ҎԼ 2 COMPSAC 2020 Regular Paper ˞.FTTBHFGSPNUIF1SPHSBN$IBJSTJO$IJFG IUUQTJFFFDPNQTBDDPNQVUFSPSH
  3. 1. എܠͱ໨త 2. ؔ࿈ݚڀͷ՝୊ 3. ఏҊख๏ 4. ࣮ݧͱߟ࡯ 5. ·ͱΊͱaccept·Ͱͷաఔ

    3 ໨࣍
  4. 1. എܠͱ໨త

  5. • ݸਓ͕౰ͨΓલʹଟछଟ༷ͳWebαΠτΛ࣋ͭ࣌୅ • Ϋϥ΢υɾVPSͷΑ͏ͳࣗ༝౓ͱִ཭؀ڥʢΠϯελϯεʣʹର͢Δཁٻ • SNSΛհͯ͠ݸਓͷίϯςϯπΛ֦ࢄ͠΍͍࣌͢୅ • ݸਓͷWebαΠτ΁ͷΞΫηεूத͢Δػձͷ૿େ • ݸਓαΠτͰ΋ΞΫηεूத΍ো֐ͱ͍ͬͨมԽʹڧ͍ج൫͕ඞཁ

    • ༧ଌͰ͖ͳ͍ΞΫηε܏޲ͱϦιʔεׂ౰ΛϦΞΫςΟϒʹߦ͍͍ͨ • ଟछଟ༷ͳWebΞϓϦʹͰ͖Δ͚ͩରԠͯ͠બ୒ࢶΛఏڙ͍ͨ͠ 5 ݸਓͷWebαΠτ΁ͷΞΫηεूத΍৴པੑ
  6. 1. Πϯελϯε্ͰWordPressͷΑ͏ͳҰൠత͔ͭଟ༷ͳWebΞϓϦ͕ಈ࡞Մೳ • ઐ໳తͳ஌͕ࣝͳͯ͘΋ར༻Ͱ͖Δ҆ՁͳαʔϏεΛ࣮ݱ͍ͨ͠ 2. Πϯελϯεͷঢ়ଶมߋॲཧ͕ߴ଎ • Πϯελϯε(ίϯςφ)ͷঢ়ଶͷఀࢭɾىಈɾεέʔϦϯάΛߴ଎ʹ॥؀ • ϦΫΤετ୯ҐͰϦΞΫςΟϒʹঢ়ଶΛܾఆ

    → มԽʹڧ͍ج൫΁ 3. ϋʔυ΢ΣΞϦιʔεͷར༻ޮ཰Λ޲্ • ϦΫΤετ͕ແ͍Πϯελϯε͸Ұఆظؒىಈޙʹఀࢭ ΠϯελϯεΛߴूੵʹऩ༰ՄೳͰมԽʹڧ͍Ծ૝Խج൫FastContainer 6 ߃ৗੑͷ͋ΔมԽʹڧ͍ج൫ͷؔ࿈ݚڀ※1 ˞3ZPTVLF.BUTVNPUP 6DIJP,POEP ,FOUBSP,VSJCBZBTIJ 'BTU$POUBJOFS")PNFPTUBUJD4ZTUFN"SDIJUFDUVSF)JHITQFFE"EBQUJOH&YFDVUJPO &OWJSPONFOU$IBOHFT 5IFSE"OOVBM*&&&*OUFSOBUJPOBM$PNQVUFST 4PGUXBSF BOE"QQMJDBUJPOT$POGFSFODF $0.14"$ +VMZ
  7. • Մ༻ੑ୲อͷͨΊʹෳ਺ΠϯελϯεՔಇͤ͞Δ͜ͱʹΑΔίετͷ૿େ • ෳ਺ͷऩ༰αʔόʹΠϯελϯεΛͦΕͧΕՔಇͤ͞Δ͜ͱͰՄ༻ੑΛ୲อ • ߴूੵʹΠϯελϯεऩ༰͢Δ؍఺͔ΒͰ͖Δ͚ͩϦιʔεΛઅ໿͍ͨ͠ • ར༻ऀ໨ઢͰ΋Մ༻ੑΛ୲อͭͭ͠අ༻Λ࡟ݮ͍ͨ͠ • ґવͱͯ͠ɺHTTPͷϨεϙϯελΠϜͷ؍఺͔ΒɺFastContainerͷHTTPϦ

    ΫΤετܖػͰঢ়ଶΛมߋ͢Δ࣌ؒͷ୹ॖ΋՝୊ • ίϯςφࣗମ͸଎͘ىಈͯ͠΋தͰಈ͘αʔόϓϩηεͷىಈ͕஗͍ 7 FastContainer͓Αͼؔ࿈ݚڀͷՄ༻ੑͷ՝୊
  8. 1. Πϯελϯε্ͰWordPressͷΑ͏ͳҰൠతͳWebΞϓϦ͕ಈ࡞Մೳ 2. ୯ҰΠϯελϯεͰ΋ऩ༰αʔόো֐࣌ʹ͸ผαʔό΁ࣗಈతʹ࠶഑ஔՄೳ 3. Πϯελϯεͷ࠶഑ஔͷ࣮ߦ࣌Ͱ͋ͬͯ΋਺ඵͷ஗ԆͰHTTPλΠϜΞ΢τ͢ Δ͜ͱͳ͘ΦϯϥΠϯͰϨεϙϯεΛૹ৴Մೳ • iPhoneͰ4GճઢΛܦ༝͢Δͱਓ͕ؒؾ͔ͮͳ͍Α͏ͳ஗Ԇ ΠϯελϯεΛߴ଎ʹ࠶഑ஔՄೳʹ͢Δ͜ͱͰूੵ཰Λ޲্ͤ͞

    ௿ίετͳج൫Λ࣮ݱ͢ΔεέδϡʔϦϯάख๏ͷఏҊ 8 ݚڀͷ໨త: ϦΫΤετ୯Ґ ͰͷΠϯελϯε࠶഑ஔ
  9. 2. ؔ࿈ݚڀͷ՝୊

  10. • FastContainer ※1͸HTTPϦΫΤετʹԠͯ͡൓Ԡత͔ͭߴ଎ʹΠϯελϯε ͷঢ়ଶʢىಈɺఀࢭɺҠಈɺෳ੡ɺϦιʔε૿ݮ౳ʣΛܾఆ • αʔϏεར༻ऀ͸Wordpressͱ͔WebΞϓϦΛී௨ʹ࢖͏Α͏ͳ࢖͍ํ • ΞΫηε਺ʹԠͨ͡ϦΞΫςΟϒͳεέʔϦϯάॲཧ͕Մೳ • Ϋϥ΢υαʔϏεج൫͸༧ΊΠϯελϯεΛىಈͤͯ͞ϦΫΤετΛॲཧ

    • ΞΫηεूத࣌͸༧ଌత͔ͭϓϩΞΫςΟϒͳεέʔϦϯάॲཧ͕ඞཁ 10 FastContainerͱΫϥ΢υαʔϏεج൫ͷಛ௃ ˞3ZPTVLF.BUTVNPUP 6DIJP,POEP ,FOUBSP,VSJCBZBTIJ 'BTU$POUBJOFS")PNFPTUBUJD4ZTUFN"SDIJUFDUVSF)JHITQFFE"EBQUJOH&YFDVUJPO &OWJSPONFOU$IBOHFT 5IFSE"OOVBM*&&&*OUFSOBUJPOBM$PNQVUFST 4PGUXBSF BOE"QQMJDBUJPOT$POGFSFODF $0.14"$ +VMZ
  11. 1. WordPressͷΑ͏ͳҰൠతͳWebΞϓϦέʔγϣϯΛར༻Մೳ 2. Πϯελϯε΍ίϯςφͷऩ༰αʔόͷো֐࣌ʹHTTPλΠϜΞ΢τ͕ੜ͡ͳ ͍ϨϕϧͰαʔϏεΛܧଓͰ͖Δఔ౓ͷՄ༻ੑΛ૝ఆ 11 ຊݚڀʹ͓͚ΔԾ૝Խج൫ͱՄ༻ੑͷલఏ

  12. • ༷ʑͳWebΞϓϦέʔγϣϯ͕ར༻Մೳ • IaaS΍PaaSΛར༻ͨ͠Πϯελϯε୯ҐͰͷߏங͓ΑͼՄ༻ੑ୲อ͕Ұൠత • ຊݚڀͷՄ༻ੑ͕લఏͷ৔߹ɺෳ਺ऩ༰αʔόʹͦΕͧΕΠϯελϯεΛϗο τελϯόΠํࣜʢىಈࡁΈͷ଴ػΠϯελϯεʣͰ଴ػ͢Δ͜ͱͰ࣮ݱ • ऩ༰αʔόఀࢭ࣌ʹ΋ଈ࣌αʔϏεΛܧଓ͢ΔͨΊ •

    ෳ਺Πϯελϯεͷىಈ͕લఏͱͳΓϦιʔε઎༗ͷίετ͕૿େ • ϦΞΫςΟϒʹՄ༻ੑΛ୲อ͢Δʹ͸ϨεϙϯελΠϜ΁ͷӨڹ͕େ͖͍ 12 FastContainer΍Ϋϥ΢υج൫ͷՄ༻ੑ
  13. 4UPSBHF $MJFOU 4FSWFS JOTUBODF" JOTUBODF# JOTUBODF$ 4FSWFS JOTUBODF" 'BTU$POUBJOFS΍Ϋϥ΢υج൫ͷՄ༻ੑ )551ϦΫΤετ

    4UPSBHF $MJFOU 4FSWFS JOTUBODF" JOTUBODF# JOTUBODF$ 4FSWFS JOTUBODF" )551ϦΫΤετ ✗ 13 αʔόো֐
  14. 3. ఏҊख๏

  15. 1. Πϯελϯε্ͰWordPressͷΑ͏ͳҰൠతͳWebΞϓϦ͕ಈ࡞Մೳ 2. ୯ҰΠϯελϯεͰ΋ऩ༰αʔόো֐࣌ʹ͸ผαʔό΁ࣗಈతʹ࠶഑ஔՄೳ 3. Πϯελϯεͷ࠶഑ஔͷ࣮ߦ࣌Ͱ͋ͬͯ΋਺ඵͷ஗ԆͰHTTPλΠϜΞ΢τ͢ Δ͜ͱͳ͘ΦϯϥΠϯͰϨεϙϯεΛૹ৴Մೳ 15 ఏҊख๏: ϦΫΤετ୯Ґ

    ͰͷΠϯελϯε࠶഑ஔ
  16. 16 4UPSBHF $MJFOU 4FSWFS JOTUBODF" JOTUBODF# JOTUBODF$ 4FSWFS ఏҊख๏ʴ'BTU$POUBJOFSͷՄ༻ੑ )551ϦΫΤετ

    4UPSBHF $MJFOU 4FSWFS JOTUBODF" JOTUBODF# JOTUBODF$ 4FSWFS )551ϦΫΤετ JOTUBODF" ✗൓Ԡతʹ ࠶഑ஔ αʔόো֐
  17. HTTP FastContainerͷجຊϑϩʔ 17 8FC1SPYZ ʢOHY@NSVCZ $.%# ʴ "1* 8FC%JTQBUDIFS OHY@NSVCZ

    $MJFOU ίϯςφ ίϯςφ ίϯςφ w )551ϦΫΤετͷ)PTUOBNF ΛΩʔʹɺ$.%# ߏ੒؅ཧ%# ͔Βίϯςφͷ৘ใΛऔಘ )551 4  ϦΫΤετ w ίϯςφͷ*1ͱϙʔτʹج͍ ͯίϯςφʹϓϩΩγ w ίϯςφ͕-JTUFO͍ͯ͠ͳ͍ ৔߹͸$.%#͔Βίϯςφ ৘ใΛಘͯىಈ $POUBJOFS&OHJOF IBDPOJXB ऩ༰ϗετ"
  18. 1. Proxy͔Βऩ༰αʔόʹICMP/TCPͰ࠷௿ݶͷύέοτͰԠ౴଎౓Λଌఆ 2. Ԡ౴଎౓͕ᮢ஋Λ௒͍͑ͯͨΒผͷऩ༰αʔό΁࠶഑ஔ 3. ίϯςφىಈ࣌͸ɺىಈ׬ྃ௚લͷঢ়ଶͷϓϩηεΠϝʔδ͔Β෮ݩ → ඇৗʹγϯϓϧͳํࣜͰߴ଎ͳ࠶഑ஔΛ࣮ݱՄೳ → ୯ҰͷίϯςφͰ࣮༻తͳՄ༻ੑΛ୲อՄೳʢϦιʔείετͷ࡟ݮʣ

    18 ߴ଎ͳऩ༰ϗετͷࢮ׆؂ࢹͱ࠶഑ஔ
  19. )PTU04 8FC1SPYZ $.%#"1* $POUBJOFS %JTQBUDIFS $POUBJOFS )PTU04 $POUBJOFS %JTQBUDIFS $POUBJOFS

    $MJFOU )551 *$.1 )551 )551 )551 *$.1PS5$1
  20. )PTU04 8FC1SPYZ $.%#"1* $POUBJOFS %JTQBUDIFS $POUBJOFS )PTU04 $POUBJOFS %JTQBUDIFS $POUBJOFS

    $MJFOU )551 *$.1PS5$1 *$.1PS5$1 )551 )551 )551 ✗ ࠷ॳͷ࠶഑ஔ࣌͸ίϯςφͷىಈ͕ඞཁͰ͋Δ͕ɺ ىಈޙ͸Ұఆظؒىಈ͠ଓ͚Δɻ
  21. • ICMP/TCPͰᮢ஋νΣοΫ͕Ұ࣌తʹޡݕ஌ͯ͠΋Өڹ͕গͳ͍ • TCPͷ৔߹͸ࣗ࡞TCPελοΫͰԟ෮3ύέοτͰνΣοΫ[3][4] • FastContainerͳͷͰޡݕ஌ͷ࠶഑ஔ͕ੜͯ͡΋αʔϏε͕ܧଓ͞ΕΔ • ޡݕ஌Ͱଞαʔόʹىಈͯ͠͠·ͬͯ΋Ұఆ࣌ؒىಈͨ͠Βఀࢭ͢Δ • ݩαʔόʹ࠶഑ஔ͞ΕͯCMDB্͸ݩαʔόͷΈʹϦΫΤετ͕ྲྀΕΔ

    • Ԡ౴࣌ؒͷᮢ஋΍λΠϜΞ΢τΛΪϦΪϦ·ͰνϡʔχϯάՄೳ 21 ఏҊख๏ͷϙΠϯτʢICMP/TCP؂ࢹʣ <>NBUTVNPUPSZ NSVCZGBTUSFNPUFDIFDL IUUQTHJUIVCDPNNBUTVNPUPSZNSVCZGBTUSFNPUFDIFDL <>-JOVYΧʔωϧͷ5$1ελοΫͱγεςϜίʔϧͷ૊Έ߹ΘͤʹΑΔख๏ΑΓ΋ߴ଎ʹϙʔτͷ-JTUFOνΣοΫΛ ߦ͏ IUUQTICNBUTVNPUPSKQFOUSZ
  22. • ίϯςφ಺ͷWebαʔόϓϩηεΛىಈ׬ྃ௚લͰΠϝʔδԽ(Checkpoint) • ఆظతʹඇಉظͰϓϩηεͷىಈ׬ྃ௚લΠϝʔδΛ࡞੒͓ͯ͘͠ • ϦΫΤετड৴࣌ʹΠϝʔδΛϓϩηεʹ෮ݩ(Restore) • αʔόϓϩηεͷॳظԽॲཧΛεΩοϓ • ىಈʹ࣌ؒͷ͔͔ΔΞϓϦέʔγϣϯαʔό౳ʹ༗ར

    • Ruby on RailsɼDjangoͳͲ 22 ఏҊख๏ͷϙΠϯτʢCheckpoint/Restoreʣ
  23. • αʔόϓϩηεͷىಈ௚ޙΛίϯςφϥϯλΠϜͰϑοΫͯ͠Checkpoint • ࢀߟ: seccompͰγεςϜίʔϧΛ؂ࢹ͠ptraceͰҰ࣌ఀࢭ͔ͯ͠ΒCRIUͰ CheckpointʹΑΔΠϝʔδԽͱ͍͏ํ๏΋͋Δ • CRIUͷதͰseccompΛ࢖͓ͬͯΓύον͕ඞཁͰ൚༻ੑʹ͔͚Δ • ϦΫΤετΛड͚ͯίϯςφΛىಈ͢Δ࣌͸Πϝʔδ͔ΒRestoreͯ͠ىಈ

    23 Checkpoint/RestoreʹΑΔFastContainerͷىಈ
  24. αʔόϓϩηεͷCheckpoint/Restoreͷ࣮૷ 24 SFGl04ϨΠϠͰ8FCαʔό͕ىಈ࣌ʹ࣮ߦ͢ΔγεςϜίʔϧΛ؂ࢹ͠ىಈ׬ྃ௚લͷϓϩηεΛΠϝʔδԽ͢ΔlIUUQ ICNBUTVNPUPSKQFOUSZ

  25. 4. ࣮ݧͱߟ࡯

  26. FastContainerͷγεςϜߏ੒ 26

  27. 27

  28. • ༧උ࣮ݧ: CRIUͱCheckpoint/Restore͢ΔϓϩηεͷϝϞϦαΠζͱͷؔ܎ • ୅දతͳΞϓϦέʔγϣϯΛ࢖ͬͨίϯςφ࠶഑ஔ࣌ͷϨεϙϯελΠϜ • Apache 2.4.18ɼPHP 7.3.0ɼWordpress 5.0.3ʢσϑΥϧτϖʔδʣ

    • Python 3.7.1ɼDjango 2.1.4ɼgunicorn 19.9.0※1 • Ruby 2.5.1ɼRails 5.2.1ɼPuma 3.12.0※2 28 ࣮ݧ಺༰ ˞IUUQTNDMPMJQPQ[FOEFTLDPNIDKBBSUJDMFT ˞IUUQTHJUIVCDPNFWFSZMFBGFMUSBJOJO ݱ࣮తͳن໛ʢݸਓάϧʔϓ಺Ͱͷར༻ͷΞϓϦέʔγϣϯఔ౓ʣͰ%#Λར༻ͨ͠΋ͷΛ࠾
  29. • ComputeͰWebαʔόͷCheckpoint/Restoreͷ଎౓Λܭଌ • mruby-simplehttpserver※1ͰWebαʔόΛىಈͤ͞setsockopt()Λ؂ࢹ • setsockopt()࣮ߦલʹCheckpoint • setsockopt()࣮ߦલʹϝϞϦΛ֬อͯ͠ɺϝϞϦαΠζʹԠͯ͡ Checkpoint/Restoreͷ଎౓ͷมԽΛܭଌ 29

    ༧උ࣮ݧɿϓϩηεͷΠϝʔδԽͷ࣮ݧ ˞NBUTVNPUPSZNSVCZTJNQMFIUUQTFSWFS IUUQTHJUIVCDPNNBUTVNPUPSZNSVCZTJNQMFIUUQTFSWFS
  30. αʔόϓϩηεͷΠϝʔδԽ(Checkpoint/Restore) 30 $IFDLQPJOU3FTUPSF1SPDFTTJOH5JNFEVFUP.FNPSZ6TBHF 1SPDFTTJOHUJNF<TFD>      

         .FNPSZVTBHFQFSQSPDFTT<.#>          $IFDLQPJOU 3FTUPSF ୯ҰͷαʔόϓϩηεͷϝϞϦ࢖༻ྔʹԠͨ͡$IFDLQPJOU3FTUPSFʹඞཁ
  31. • ComputeͰApache httpdͷϓϩηε਺ΛมԽ • HTTPϦΫΤετΛܖػʹApache httpdΛىಈͤͯ͞ϨεϙϯεΛฦ͢ • ਺ेόΠτͷ੩తͳindex.htmlʹର͢ΔϦΫΤετ • CRIUΛ࢖͏৔߹

    • CRIUΛ࢖Θͳ͍৔߹ 31 ༧උ࣮ݧ2ɿϓϩηε਺ͱCRIUͷؔ܎
  32. Apacheͷworker਺ͱCRIUͷؔ܎ 32 )PUTUBSU͸શͯͷ8PSLFSϓϩηε ͷىಈ׬ྃΛ଴ͨͣʹɼͭͰ΋ XPSLFSϓϩηε͕ىಈ͢Ε͹Ϩεϙ ϯεΛฦ͢͜ͱ͕Ͱ͖ΔͨΊҰఆɽ $3*6͔Βͷىಈ͸8PSLFSϓϩηε ΛશͯΠϝʔδ͔͢ΔͨΊɼશ͕ͯ 3FTUPSF͞Ε͔ͯΒϨεϙϯεΛฦ͢ ͨΊ୯ௐ૿Ճɽ

  33. • Apache 2.4.18ɼPHP 7.3.0ɼWordpress 5.0.3 • ϓϩηε਺͸3ɼ୯ҰͷϓϩηεͷϝϞϦαΠζ(RSS)͸35MBytes • Python 3.7.1ɼDjango

    2.1.4ɼgunicorn 19.9.0 ※1 • ϓϩηε਺2ɼεϨου਺2ɼ୯ҰͷϓϩηεͷRSS͸33MBytes • Ruby 2.5.1ɼRails 5.2.1ɼPuma 3.12.0 ※2 • ϓϩηε਺2ɼεϨου਺14ɼ୯ҰͷϓϩηεͷRSS͸89MBytes • gemΛࣄલίϯύΠϧ͓ͯ͘͠bootsnapͱ΋ൺֱ 33 ίϯςφ࠶഑ஔ࣌ͷϨεϙϯελΠϜ
  34. WordPress on Apache httpd 34 BCίϚϯυͰಉ࣌઀ଓ਺ͷϕϯνϚʔΫΛ͔͚ͳ͕Β ऩ༰ϗετΛJQBUBCMFTͰԾ૝తʹμ΢ϯͤͯ͞ɼผͷऩ༰αʔόʹ࠶഑ஔΛڧ੍తʹൃੜͤͨ͞

  35. Django 35 IUUQTNDMPMJQPQ[FOEFTLDPNIDKBBSUJDMFT ݱ࣮తͳن໛ʢݸਓάϧʔϓ಺Ͱͷར༻ͷΞϓϦέʔγϣϯఔ౓ʣͰ%#Λར༻ͨ͠΋ͷΛ࠾༻

  36. Ruby on Rails 36 IUUQTHJUIVCDPNFWFSZMFBGFMUSBJOJO ݱ࣮తͳن໛ʢݸਓάϧʔϓ಺Ͱͷར༻ͷΞϓϦέʔγϣϯఔ౓ʣͰ%#Λར༻ͨ͠΋ͷΛ࠾༻

  37. 5. ·ͱΊ

  38. • ୯ҰΠϯελϯεͰՄ༻ੑΛ୲อ͢Δߴ଎ͳεέδϡʔϦϯάख๏ΛఏҊ • ෳ਺ΠϯελϯεΛඞཁͱ͠ͳ͍ͨΊϦιʔείετ͕௿͍ • ࣮ݧ͔Βݱ࣌఺Ͱ΋࣮༻ՄೳͳϨϕϧͷ࠶഑ஔͷੑೳ͕ಘΒΕͨ • ϓϩμΫγϣϯ؀ڥͰԠ༻ • ϗετো֐࣌Ͱ͋ͬͯ΋Ϣʔβ͕ؾ͔ͮͳ͍ϨϕϧͰͷՄ༻ੑ

    • ΦʔτεέʔϦϯά࣌ʹ΋γʔϜϨεʹίϯςφΛ૿΍ͯ͠ෛՙରࡦՄೳʹ • ΞΫηε܏޲ͱϦιʔεׂΓ౰͕ͯਖ਼֬ʹ௥ਵՄೳʹ • εέʔϦϯά΍ϋʔυ΢ΣΞϓʔϧͷϦιʔεׂΓ౰ͯ΋࠷దԽ 38 ·ͱΊ
  39. • ࠷ॳͷWWW2020ʹఏग़ͯ͠Reject͞Εͨཧ༝ • ݚڀͷཱͪҐஔ͕ෆ໌֬ɺ৽نੑ͕͍·͍ͪΑ͘Θ͔Βͳ͍ • ຊݚڀͷཱͪҐஔ΍લఏͷ໌֬Խ • ൺֱ͢΂͖ؔ࿈ݚڀ͕ෆ໌ྎ • ຊݚڀͱൺֱ͢΂͖ؔ࿈ݚڀΛॆ࣮ͤͯࠩ͞෼Λ໌֬Խ

    ڭ܇: ΠϯλʔωοτɾWebٕज़෼໺Ͱ͸ൃද࿦จ΍OSSͷ਺΍ٕज़ͷมԽ଎౓ ͕ඇৗʹ଎͍ͨΊɺݚڀͷείʔϓͱ࠷৽ͷ՝୊Λ໌֬ʹ্ͨ͠Ͱࠩ෼Λ͔ͬ͠ Γͱࣔ͠ɺͦͷ՝୊͕ݱ࣮తʹͲΕ΄Ͳҙ͕ٛ͋Δ͜ͱͳͷ͔Λࣔ͢͜ͱ͕େࣄ 39 Accept·Ͱͷաఔ