Upgrade to Pro — share decks privately, control downloads, hide ads and more …

コンテナたちを計測すること ~ ロリポップ!マネージドクラウドの裏側 ~ /wip-how-to-get-container-metrics

KONDO Uchio
September 29, 2017

コンテナたちを計測すること ~ ロリポップ!マネージドクラウドの裏側 ~ /wip-how-to-get-container-metrics

Cookpad Tech Kitchen #12

KONDO Uchio

September 29, 2017
Tweet

More Decks by KONDO Uchio

Other Decks in Technology

Transcript

  1. dϩϦϙοϓʂϚωʔδυΫϥ΢υͷཪଆd
    ۙ౻Ӊஐ࿕(.01FQBCP *OD
    $PPLQBE5FDI,JUDIFO
    ίϯςφͨͪΛܭଌ͢Δ͜ͱ

    View Slide

  2. ΤϯδχΞ
    ۙ౻Ӊஐ࿕!VE[VSB
    ٕज़෦ٕज़ج൫νʔϜ
    ޷͖ͳϥʔϝϯ͸ΒʔΊΜ޻๪ཾ
    (.0ϖύϘॴଐ
    ΍ͬͯΔ෼໺3VCZɺNSVCZɺίϯςφ
    ΍Γ͍ͨ෼໺γεςϜϓϩάϥϛϯάɺ-JOVY֦ு

    View Slide

  3. ؂ࢹɾܭଌʹ͍ͭͯ͸ɺࠓ΋
    ࢼߦࡨޡ͕ଓ͍͍ͯ·͕͢
    ࠓ೔͸ݕূͷաఔΛڞ༗͠·͢

    View Slide

  4. ϩϦϙοϓʂϚωʔδυΫϥ΢υ
    IUUQTNDMPMJQPQKQ
    ΞϧϑΝςετத

    View Slide

  5. IUUQTNDMPMJQPQKQ
    ΞϧϑΝςετத
    ˞͝ڵຯͷ͋Δํ͸ձ৔Ͱ੠Λֻ͓͚Լ͍͞

    View Slide

  6. ؂ࢹͱ͸

    View Slide

  7. *5Πϯϑϥͷ؂ࢹͱ͸ɺຊॻͰ͸࣍ͷ௨Γఆ
    ٛ͠·͢ɻ
    ো֐ൃੜͷ༗ແΛҰఆظؒͰ଺Γͳ֬͘ೝ͢
    Δ͜ͱɻ
    ʰ*5Πϯϑϥ؂ࢹ<࣮ફ>ೖ໳ʱΑΓ

    View Slide

  8. ؂ࢹɾϝτϦΫεͷ໨త
    wӡ༻ͱͯ͠
    wαʔϏεμ΢ϯʹ͙͢ʹؾ෇͖͍ͨɻ·ͨɺ෮چΛૉૣ͍ͨ͘͠
    wαʔϏεͷػೳͱͯ͠
    wίϯςφͷෛՙঢ়گ΍ɺো֐ঢ়ଶʹͳ͍ͬͯΔ͔൱͔ͳͲɻ·ͨͦͷ
    ཤྺΛͪΌΜͱՄࢹԽͯ͠ఏڙͰ͖Δͱخ͍͠
    wˠͦͷͨΊʹɺϝτϦΫεΛܭଌɺ؂ࢹɺෆਖ਼ͳঢ়ଶͷ࣌͸Ξϥʔ
    τͳͲΛ͋͛ΒΕΔΑ͏ʹ͠ͳ͚Ε͹ͳΒͳ͍ɻ

    View Slide

  9. ؂ࢹͷͨΊʹ
    ·ͣ͸ܭଌ͠ͳ͍ͱ͍͚ͳ͍

    View Slide

  10. ࠓɺࠔ͍ͬͯΔ͜ͱ
    wίϯςφ͸ී௨ͷ7.ͷ؂ࢹͷΑ͏ʹ͍͔ͳ͍ͱ͜Ζ΋͋Δ
    wˠϓϩηεʢ܈ʣͳͷͰ-"ͷΑ͏ͳఆ൪ͷ஋͕औΕͳ͔ͬͨΓͳͲ
    wίϯςφϗεςΟϯάͰ͋Δ͜ͱɺ·ͨϚΠΫϩαʔϏεతͰ͋Δ͜
    ͱʹΑΔɺͭͷʮ਺ʯͷ໰୊͕͋Δ

    View Slide

  11. ʮίϯςφ਺ʯͷ໰୊
    wϚωʔδυΫϥ΢υ͸ɺίϯςφΛར༻ͨ͠ߴूੵϓϥοτϑΥʔϜ
    Ͱ͋Δɻ͢ͳΘͪɺίϯςφ͕ͱͯ΋ଟ͍ɻ
    w͔͠΋ɺ'BTU$POUBJOFSΛ࠾༻͍ͯ͠ΔͳͲɺίϯςφ͸ىಈ͠ଓ͚
    ΔΘ͚Ͱ͸ͳ͍ɻ·ͨɺҰͭ࣌ͷυϝΠϯʹඥͮ͘ίϯςφͷ਺΋ม
    ΘΔɻ
    wͨ͘͞Μ͋ΔͷͰɺूΊΔ໰୊ͱɺ෼͔Γ΍͘͢ݟͤΔ໰୊͕͋Δ

    View Slide

  12. ʮίϯϙʔωϯτ਺ʯͷ໰୊
    wԿ౓͔ର֎ൃද͍ͯ͠Δ௨ΓɺϚωʔδυΫϥ΢υ͸ίϯϙʔωϯτ
    ͕ଟ͘ɺϚΠΫϩαʔϏεతʹ૊Έ߹Θ͍ͬͯ͞Δ
    wʮαΠτ͕ݟ͑ͳ͍ʯͷΑ͏ͳ৔߹ʹɺݪҼͱߟ͑ΒΕΔՕॴͷಛఆ
    ͕೉͘͠ɺΤϥʔτϥοΩϯάʹ՝୊͕͋Δɻ
    wϘτϧωοΫͷಛఆͳͲ΋ಉ༷

    View Slide

  13. গͣͭ͠ɺ֎ງΛຒΊΔ
    ͦΜͳઓ͍ͷաఔΛ࿩͠·͢

    View Slide

  14. ͲΜͳ஋͕औΕΔ͔ʁ

    View Slide

  15. ܭଌͷର৅
    w$16ϝϞϦར༻཰ͷ؂ࢹɾܭଌ
    wωοτϫʔΫͷ؂ࢹɾܭଌ
    w*0΍σΟεΫͷ؂ࢹɾܭଌ
    wϦΫΤετͷτϨʔγϯά
    wίϯςφ಺ϛυϧ΢ΣΞͷϩάʢ͜Ε͸ϩάύʔεͰૉ௚ʹूΊΔʣ

    View Slide

  16. ࢀߟ
    wͦʔ͍ͩ͞ΜͷࢿྉΑΓɺجຊతͳγεςϜʹؔΘΔϝτϦοΫ
    IUUQTPVEBJIBUFOBCMPHDPNFOUSZ

    View Slide

  17. DHSPVQͷTUBUͷར༻
    wDQVDQVBDDUαϒγεςϜ
    wNFNPSZαϒγεςϜ
    w ଞɺCMLJP WͷJP

    wςΩετσʔλͰɺར༻ঢ়گ͕Θ͔Δˠύʔεͯ͠׆༻͢Δ
    DHSPVQࣗମʹ͍ͭͯ͸ʮDHSPVQVE[VSBʯ·ͨ͸

    ʮDHSPVQ5FO'PSXBSEʯͰݕࡧ͢Δͱ೔ຊޠͷղઆ͕ɻɻɻ
    root@compute-1:~# cat /sys/fs/cgroup/cpu/$CONTAINER/cpu.stat
    nr_periods 140373
    nr_throttled 25
    throttled_time 1112602844

    View Slide

  18. ωοτϫʔΫʹؔ͢ΔϝτϦΫε
    wݪଇɺίϯςφ͸ࣗ෼ͷ/FUXPSL/BNFTQBDFʹॴଐ͢Δ
    w/FUXPSL/BNFTQBDFͲ͏͠ʢࢠ͔Β਌ʣ͸ɺWFUIσόΠεΛܦ༝
    ͯ͠௨৴͢Δ͜ͱʹͳΔ
    wͦͷWFUIσόΠεʹؔ͢Δ஋͸ز͔ͭͱΔखஈ͕͋Δ
    wJGUPQͷΑ͏ͳίϚϯυɺQSPDOFUEFWϑΝΠϧ
    wϓϩηε୯ҐͷωοτϫʔΫ৘ใQSPD1*%OFU\EFW UDQ VEQ ^

    View Slide

  19. ':*ίϯςφωοτϫʔΫͷτϙϩδ
    wCSJEHFઃఆͷྫɻ%PDLFSͷσϑΥϧτઃఆ΋͜Εɻ
    FUI
    EPDLFS
    WFUIIPTU
    WFUIHVFTU
    WFUIIPTU
    WFUIHVFTU
    WFUIIPTU
    WFUIHVFTU
    FH
    FH

    ඞཁʹԠ͡/"5ͳͲ
    /FUXPSL
    /BNFTQBDF
    ͻͱͭͷ)PTU
    CSJEHF
    ͜ΕΒΛ؂ࢹ͢Δʂ

    View Slide

  20. *0ΞΧ΢ϯςΟϯά
    wDHSPVQͷCMLJPJP͸ɺσόΠε୯Ґͷܭଌ΍εϩοτϧͳͷͰɺ/'4
    ͷΑ͏ʹσόΠε൪߸͕ৼΒΕͳ͍ϑΝΠϧγεςϜʹ͸೉͍͠
    wҰํɺϓϩηε୯ҐͷSFBEʢ଒ʣɺXSJUFʢ଒ʣͷݺͼग़͠ճ਺΍ɺ
    ಡΈࠐΈॻ͖ࠐΈʹ੒ޭͨ͠όΠτ਺ͷྦྷܭ͸ɺΧʔωϧ͕ूܭͯ͠
    ͍ͯར༻Ͱ͖Δɻ
    wQSPD1*%JPϑΝΠϧܦ༝Ͱ؆୯ʹ֬ೝՄೳ
    wྫ͑͹͜ΕΛίϯςφʹॴଐ͢Δϓϩηε͝ͱʹूܭ͢Δ

    View Slide

  21. ϦΫΤεττϥοΩϯά
    w͋ΔαʔόΛ௨ա͢Δ͝ͱʹಛఆͷϔομʔʹ࣌ؒΛ௥ه͍ͯ͘͠
    wDG9'PSXBSEFE'PSϔομʔ
    w3BDL౳ͷൃ૝ΛਅࣅͯɺϚʔΧʔΛωετ͍ͯ͘͠ͷ͸Ͳ͏͔ʁ
    wίϯϙωϯτ͝ͱͷܦա͕࣌ؒΘ͔ΔΑ͏ʹͳΓɺϘτϧωοΫͳͲ
    ͕Θ͔ΔɻͦΕͦ͜(SBGBOBͳͲͰՄࢹԽͯ͠΋ྑ͍
    wΤϥʔτϥοΩϯάʹ΋Ԡ༻Ͱ͖Δʁ
    IUUQTQZMPOTXFCGSBNFXPSLSFBEUIFEPDTJPFOWSDDPODFQUTIUNM

    View Slide

  22. DGޚࣾͷ෼ࢄτϨʔγϯάج൫
    wؾʹͳΓ·͢
    IUUQUFDIMJGFDPPLQBEDPNFOUSZ

    View Slide

  23. σʔλΛཷΊࠐΉ

    View Slide

  24. େ·͔ͳΞʔΩςΫνϟ
    wूΊΔˠஷΊΔˠՄࢹԽ͢Δ
    wσʔλͷอଘઌ1SPNFUIFVT
    wՄࢹԽ(SBGBOB
    wσʔλΤʔδΣϯτD"EWJTPS΍NUBJMɺࣗ࡞FYQPSUFS

    View Slide

  25. 1SPNFUIFVT
    (SBGBOB
    &YQPSUFS
    &YQPSUFS
    &YQPSUFS
    $POUBJOFS
    $POUBJOFS
    $POUBJOFS
    $POUBJOFS
    $POUBJOFS
    "1*޲͚1SPYZ
    ूܭϝτϦΫεͷάϧʔϓผʹ
    FYQPSUFSΛ༻ҙ
    Ϣʔβ༷μογϡϘʔυ

    ʢ։ൃதʣ
    ӡ༻ΤϯδχΞ

    σΟϨΫλʔͳͲ
    "MFSUNBOBHFS

    View Slide

  26. 1SPNFUIFVTͷ࠾༻
    w(PPHMFͷ#PSHNPOʹӨڹΛड͚ͨ؂ࢹπʔϧ
    w#PSHNPOͷ঺հ͸43&ຊʹ΋͋Δͦ͏
    wΠϯετʔϧʢجຊతʹ͸ϫϯόΠφϦΛ഑උ͢Ε͹0,ʣ΍ɺ

    ӡ༻ʢجຊతʹΦϯϝϞϦʴϩʔΧϧετϨʔδʣ͕༰қ
    wॊೈͳσʔλߏ଄ͱߴػೳͳΫΤϦ
    w"MFSU࿈ܞ͕༰қ
    w(SBGBOBͳͲՄࢹԽπʔϧ΍֤छFYQPSUFS͕ἧ͍ͬͯΔ

    View Slide

  27. 1SPNFUIFVTͱ(SBGBOBͷ࿈ܞ
    w(SBGBOB͸΋͸΍આ໌ͷඞཁ΋ͳ͍΄Ͳͷఆ൪μογϡϘʔυ
    wެࣜʹ΋υΩϡϝϯτ͕͋Δ
    wIUUQTQSPNFUIFVTJPEPDTWJTVBMJ[BUJPOHSBGBOB
    w1SPNFUIFVTͷΫΤϦΛͦͷ·Μ·ॻ͚ͨΓͯ͠ศརײ͕ߴ͍

    View Slide

  28. Մ༻ੑͷߟ͑ํ
    w1SPNFUIFVT͸֤ϊʔυͷFYQPSUFS͔ΒσʔλΛQVMM͍ͯ͘͠ελΠ
    ϧΛͱ͍ͬͯΔʢDG/BHJPT/31&4BB4ܕ؂ࢹʣ
    wͦͷͨΊ"DUJWF"DUJWFߏ੒ΛऔΕ͹ಉ͡σʔλʹͳΓɺ৑௕ߏ੒ͱ
    ͍͏͜ͱʹͳΔɻ
    wSFGIUUQTHJUIVCDPNQSPNFUIFVTQSPNFUIFVTJTTVFT

    View Slide

  29. 4FSWJDF%JTDPWFSZ
    wϊʔυ͔ΒQVMM͢ΔͷͰɺ1SPNFUIFVT͸ϊʔυ͕Ͳ͜ʹ͋Δͷ͔ͱ
    ͍͏৘ใΛ͍࣋ͬͯͳ͍ͱ͍͚ͳ͍ɻ
    w4FSWJDF%JTDPWFSZͷͨΊͷํ๏ࣗମ͸͍͔ͭ͋͘Δɻ
    wϑΝΠϧϕʔεɺ%/4ϕʔεɺ"840QFO4UBDLͳͲ%ZOBNJD*OGSB
    ޲͚ઃఆɺ͋Δ͍͸.BSBUIPO΍$POTVMͱ΋࿈ܞՄೳ

    View Slide

  30. FYQPSUFSͷ֓೦ʹ͍ͭͯ
    wσʔλΛऔΓ͍ͨϊʔυʹɺFYQPSUFSͱݴ͏ܾ·ͬͨϑΥʔϚοτͰ
    σʔλΛు͖ग़͢ਓΛৗற͓ͤͯ͘͞ʢ࣮ଶ͸IUUQ"1*ʣɻ
    w1SPNFUIFVT͸ɺ4FSWJDF%JTDPWFSZΛͯ͠ɺͦͷFYQPSUFSʹΞΫη
    ε͠ɺఆظతʹσʔλΛಘΔ
    wFYQPSUFS͸ɺطʹ࣮૷͞Ε͍ͯΔ΋ͷ΋ͨ͘͞Μ͋Δ͠ɺ

    ࣗ෼Ͱ࣮૷͢Δ͜ͱ΋Մೳɻ

    View Slide

  31. FYQPSUFSͷྫ
    wOPEF@FYQPSUFSl&YQPSUFSGPSNBDIJOFNFUSJDTz
    w$16ɺϝϞϦɺ-"ɺωοτϫʔΫͳͲϚγϯ୯ҐͰ৭ʑͱूܭ
    w(SBGBOBͳͲͰ͍͍ײ͡ʹάϥϑԽͯ͠׆༻
    wD"EWJTPS(PPHMFࣾ಺޲͚ίϯςφ؂ࢹج൫Λϕʔεʹɺ%PDLFSͳ
    ͲʹରԠͨ͠΋ͷɻͪͳΈʹTZTUFNEͳͲͷ؂ࢹͰ΋ศར
    wͦͷଞɺଟ͘ͷ࣮૷͕͋Δɻී௨͸ࣗ࡞͠ͳ͍͍ͯ͘͸ͣ

    IUUQTHJUIVCDPNQSPNFUIFVTEPDTCMPCNBTUFSDPOUFOUEPDTJOTUSVNFOUJOHFYQPSUFSTNE

    View Slide

  32. D"EWJTPSͰ)BDPOJXBͷϝτϦΫεΛऔΔ

    View Slide

  33. D"EWJTPSͰ)BDPOJXBͷϝτϦΫεΛऔΔ

    View Slide

  34. NUBJMIUUQTHJUIVCDPNHPPHMFNUBJM
    wઌड़ͨ͠/HJOYͳͲͷϩάΛύʔε্ͨ͠ͰɺFYQPSUFSʹ΋ͳͬͯ͘
    ΕΔπʔϧ
    w%4-ʹΑΓϧʔϧΛهड़
    wԿ͔طࢹײͷ͋Δྗڧ͞

    DG'MVFOUEͷJO@UBJM

    w/HJOYͷྫˠ
    IUUQTHJUIVCDPNOEJB[HOHJOYQSPNFUIFVTFYQPSUFSCMPCNBTUFSpMFTQSPHTOHJOYNUBJMɹ

    View Slide

  35. ܭଌͷର৅
    w$16ϝϞϦར༻཰ͷ؂ࢹɾܭଌ
    wωοτϫʔΫͷ؂ࢹɾܭଌ
    w*0΍σΟεΫͷ؂ࢹɾܭଌ
    wϦΫΤετͷτϨʔγϯά
    wίϯςφ಺ϛυϧ΢ΣΞͷϩά
    (PPHMFπʔϧ܈ͷΤίγεςϜ͕ศརɻ
    ˡD"EWJTFSͰऔΕΔ
    ˡD"EWJTFSͰ޻෉͢Ε͹औΕΔ
    ˡ·ͩཁݕ౼
    ˡNUBJMͰָͰ͖Δ
    ˡώοʢOHY@NSVCZͳͲͰؤுΕͦ͏ʣ

    View Slide

  36. FYQPSUFSࣗ࡞
    wͱ͸͍͑ɺࣗ࡞΋Ұͭͷख


    (PͳͲͳΒ1SPNFUIFVTެࣜͷϥΠϒϥϦ͕࢖͑Δ


    طଘ࣮૷ׂ͕ͱ๛෋


    ͦ΋ͦ΋ɺຬͨ͢΂͖ϑΥʔϚοτ͕୯७
    wIUUQTQSPNFUIFVTJPEPDTJOTUSVNFOUJOHXSJUJOH@FYQPSUFST

    ΨΠυΛݟͳ͕Βɺ͋ͱطଘ࣮૷ΛோΊͯॻ͘ͱྑͦ͞͏

    View Slide

  37. FYQPSUFSͱ1SPNFUIFVTͷ࿈ܞ
    w4FSWJDF%JTDPWFSZͷதͰ΋$POTVMͱͷ૬ੑ͸ྑ͍ͱ͍͏ࢲݟ
    wFYQPSUFSଆͰɺ$POTVMͷTFSWJDFDIFDLΛ௥ՃˠಛఆͷαʔϏεΛ
    1SPNFUIFVT͔ΒσΟεΧόϦɺ͕ඇৗʹ༰қʹͳ͍ͬͯΔ
    wྫ͑͹Ұ୆ʹෳ਺FYQPSUFSͳͲ΋؆୯ͩ͠ɺFYQPSUFSͷ௥Ճ࡟আɺ
    ࢮ׆؂ࢹ΋$POTVMʹ৐Δ͜ͱͰߟ͑ͳͯ͘ࡁΉ

    View Slide

  38. ͜ΜͳTFSWJDFఆٛΛॻ͚͹
    wεοͱ+0*/͢Δ
    wศར
    {
    "service": {
    "name": "node_exporter",
    "tags": [
    "development"
    ],
    "port": 9100,
    "checks": [
    {
    "id": "ping",
    "tcp": "localhost:9100",
    "interval": "15s",
    "timeout": "1s"
    }
    ]
    }
    }

    View Slide

  39. ͜ͷষ·ͱΊ
    wʮσʔλͷஷΊͲ͜Ζʯʹ1SPNFUIFVTΛਅΜதʹਾ͑Δ
    w1SPNFUIFVTͷΤίγεςϜ͸ڧ͍ɻ
    w͍͟ͱͳΕ͹ཉ͍͠σʔλ͸ࣗ෼ͰऔΓʹߦ͘
    w͜ΕΒͷͭͳ͗ʹ$POTVM͕ศར

    View Slide

  40. ࠓޙ΍Γ͍ͨ͜ͱ

    View Slide

  41. ࠓޙ΍Γ͍ͨ͜ͱ
    wूΊΔͱ͜Ζ·Ͱ͸Կͱ͔ͳΓͦ͏
    wཷΊࠐΜͩσʔλͷ׆༻ʹ͍ͭͯ
    w"MFSUNBOBHFSͷ֓೦ʢࠓ೔͸লུʣ
    wҟৗݕ஌ͷࣗಈԽɺՄࢹԽͳͲ

    View Slide

  42. ҟৗݕ஌ͷࣗಈԽɺՄࢹԽ
    wίϯςφʹؔ͢ΔϝτϦοΫε͸ɺ$.%#ܦ༝Ͱू໿Ͱ͖Δঢ়ଶ
    wϓϩδΣΫτ͝ͱͷσʔλΛूΊͯԿ͔Ͱ͖ͳ͍͔
    1SPKFDUT
    EPNBJO
    $POUBJOFST
    $*% DPOUBJOFSJE

    $POUBJOFST
    $*% DPOUBJOFSJE

    .FUSJDT\EBUBlBBBz ^999
    .FUSJDT\EBUBlCCCz ^:::
    .FUSJDT\EBUBlDDDz ^;;;
    .FUSJDT\EBUBlEEEz ^888

    View Slide

  43. ूΊͨσʔλΛͲ͏͢Δ͔
    w͜Ε͕ɺࠓޙͷ՝୊
    w୯७ʹάϥϑԽͯ͠Έͯ΋ɺͦΕͳΓʹঢ়ଶ͕Θ͔Δؾ΋͢Δ͠ɺٸ
    ʹෛՙ͕ߴ͍ɺ௨৴Ͱ͖ͳ͘ͳͬͯΔɺͳͲ͸໨ࢹͰ͖ͦ͏Ͱ͸͋Δ
    w·ͣ͸ӡ༻ͯ͠Έ͍ͨͱ͜Ζ

    View Slide

  44. ՄࢹԽɾࣗಈ൑ఆͷख๏
    wϓϩδΣΫτʹඥ෇͍ͨσʔλ͸ɺجຊతʹଟ࣍ݩͷσʔλͰ͋Δͷ
    Ͱɺ࣍ݩѹॖΛͯ͠ɺʮ͜Ε͸ਖ਼ৗʯʮ͜Ε͸ো֐ঢ়ଶʯͷΑ͏ʹ෼
    ྨ͕Ͱ͖ΔΜͰ͸ͳ͍͔ͱߟ͍͑ͯΔ͕ɺࠓޙͷ՝୊
    wେྔͷίϯςφϓϩδΣΫτΛɺࣗݾ૊৫ԽϚοϓͳͲͰࣗಈͰ

    ෼ྨͯ͠ɺͦΕΛՄࢹԽͰ͖Δͱ໘നͦ͏ʁ
    w͜ͷลΓʹɺʮ਺ͷ໰୊ʯͷղܾࡦ͕ݟ͑Δؾ͕͍ͯ͠Δ

    View Slide

  45. 5P#F$POUJOVFE

    View Slide

  46. ·ͩ·ͩख୳Γͷ
    ཁૉ΋ଟ͍Ͱ͕͢
    ΍Γ͕͍͕͋Γ·͢ʂ

    View Slide

  47. 13

    View Slide

  48. ϚωʔδυΫϥ΢υΛҰॹʹ࡞Γ·͠ΐ͏
    ࠷৽ͷ࠾༻৘ใΛνΣοΫˠ !QC@SFDSVJU
    QIPUPIUUQTUFDIQFQBCPDPNFOHJOFFSUSBJOJOHWPM

    View Slide