Kubernetes Cluster Migration

Kubernetes Cluster Migration

C3b68d6841dcd343be9685f1cdaf8ac9?s=128

Atsushi Tanaka

March 26, 2020
Tweet

Transcript

  1. ©2020 Wantedly, Inc. Kubernetes Cluster Migration ΫϥελͷҾͬӽ͠ʹΑΔαʔϏεແఀࢭΞοϓάϨʔυ Mar 26, 2020

    Kubernetes Meetup Tokyo #29 - Atsushi Tanaka 1
  2. ©2020 Wantedly, Inc. 2 "HFOEB  $MVTUFSNJHSBUJPOͷඞཁੑ  $MVTUFSNJHSBUJPOͷखॱ

  3. ©2020 Wantedly, Inc. 3 @bgpat / Atsushi Tanaka Infrastructure Engineer

    at Wantedly, Inc. Kubernetes / Docker / Go / JavaScript / Terraform $ whoami
  4. ©2020 Wantedly, Inc. 4 8IBU`T8BOUFEMZ

  5. ©2020 Wantedly, Inc. 8BOUFEMZͱ,VCFSOFUFT w ͢΂ͯͷαʔϏεΛ,VCFSOFUFT্Ͱӡ༻ w "84&$্ʹߏங w &,4͸࢖ΘͣLPQTΛར༻

    w  W 
 ͔Βຊ൪ӡ༻ 5
  6. ©2020 Wantedly, Inc. ·ͣJOQMBDFVQHSBEFΛࢼ͢ αʔϏεμ΢ϯͷՄೳੑ͕͋Ε͹ DMVTUFSNJHSBUJPO 6 ΞοϓάϨʔυઓུ

  7. ©2020 Wantedly, Inc. JOQMBDFVQHSBEF w طଘͷΫϥελΛ
 ͦͷ··࢖͏ w ϊʔυΛॱ൪ʹೖΕସ͑ w

    SPMMJOHVQEBUF 7 ΞοϓάϨʔυํ๏ͷൺֱ DMVTUFSNJHSBUJPO w طଘͷΫϥελ͸࢖Θͣ
 ৽͍͠ΫϥελΛཱͯΔ w ϦιʔεΛ৽Ϋϥελʹίϐʔ w CMVFHSFFOEFQMPZ
  8. ©2020 Wantedly, Inc. ΫϥελΛߏ੒͢ΔϊʔυΛॱ൪ʹೖΕସ͑Δ 8 *OQMBDFVQHSBEF ,VCFSOFUFT$MVTUFS NBTUFSW NBTUFSW NBTUFSW

    OPEFW OPEFW OPEFW 1PE 4FSWJDF $POpH
 .BQ %FQMPZ NFOU *OHSFTT 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE
  9. ©2020 Wantedly, Inc. ΫϥελΛߏ੒͢ΔϊʔυΛॱ൪ʹೖΕସ͑Δ 9 *OQMBDFVQHSBEF ,VCFSOFUFT$MVTUFS NBTUFSW NBTUFSW NBTUFSW

    XPSLFSW XPSLFSW XPSLFSW 4FSWJDF $POpH
 .BQ %FQMPZ NFOU *OHSFTT 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE
  10. ©2020 Wantedly, Inc. ΫϥελΛߏ੒͢ΔϊʔυΛॱ൪ʹೖΕସ͑Δ 10 *OQMBDFVQHSBEF ,VCFSOFUFT$MVTUFS XPSLFSW XPSLFSW XPSLFSW

    NBTUFSW NBTUFSW NBTUFSW 4FSWJDF $POpH
 .BQ %FQMPZ NFOU *OHSFTT 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE
  11. ©2020 Wantedly, Inc. ΫϥελΛߏ੒͢ΔϊʔυΛॱ൪ʹೖΕସ͑Δ 11 *OQMBDFVQHSBEF ,VCFSOFUFT$MVTUFS NBTUFSW NBTUFSW NBTUFSW

    XPSLFSW XPSLFSW XPSLFSW 4FSWJDF $POpH
 .BQ %FQMPZ NFOU *OHSFTT 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE
  12. ©2020 Wantedly, Inc. ΫϥελΛ৽͘͠࡞ͬͯத਎ΛҠ͢ 12 $MVTUFSNJHSBUJPO ,VCFSOFUFT$MVTUFS NBTUFSW NBTUFSW NBTUFSW

    OPEFW OPEFW OPEFW 1PE 4FSWJDF $POpH
 .BQ %FQMPZ NFOU *OHSFTT 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE
  13. ©2020 Wantedly, Inc. ΫϥελΛ৽͘͠࡞ͬͯத਎ΛҠ͢ 13 $MVTUFSNJHSBUJPO 0ME,VCFSOFUFT$MVTUFS NBTUFSW NBTUFSW NBTUFSW

    OPEFW OPEFW OPEFW 1PE 4FSWJDF $POpH
 .BQ %FQMPZ NFOU *OHSFTT 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE /FX,VCFSOFUFT$MVTUFS NBTUFSW NBTUFSW NBTUFSW OPEFW OPEFW OPEFW
  14. ©2020 Wantedly, Inc. ΫϥελΛ৽͘͠࡞ͬͯத਎ΛҠ͢ 14 $MVTUFSNJHSBUJPO 0ME,VCFSOFUFT$MVTUFS NBTUFSW NBTUFSW NBTUFSW

    OPEFW OPEFW OPEFW 1PE 4FSWJDF $POpH
 .BQ %FQMPZ NFOU *OHSFTT 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE /FX,VCFSOFUFT$MVTUFS NBTUFSW NBTUFSW NBTUFSW OPEFW OPEFW OPEFW 1PE 4FSWJDF $POpH
 .BQ %FQMPZ NFOU *OHSFTT 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE
  15. ©2020 Wantedly, Inc. ΫϥελΛ৽͘͠࡞ͬͯத਎ΛҠ͢ 15 $MVTUFSNJHSBUJPO 0ME,VCFSOFUFT$MVTUFS /FX,VCFSOFUFT$MVTUFS NBTUFSW NBTUFSW

    NBTUFSW OPEFW OPEFW OPEFW 1PE 4FSWJDF $POpH
 .BQ %FQMPZ NFOU *OHSFTT 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE 1PE
  16. ©2020 Wantedly, Inc. JOQMBDFVQHSBEF w ؆୯ w ֤Ϋϥ΢υϕϯμʔ͕"1*Λ༻ҙ w ΦϯϓϨͰ΋ߏ੒πʔϧͷػೳ͕࢖͑Δ

    w ޓ׵ੑͷ੍໿͕͋Δ w NJOPSWFSTJPO্͔͛͠ΒΕͳ͍ w ޓ׵ੑͷͳ͍มߋ͸μ΢ϯλΠϜ͕ൃੜ 16 ֤ΞοϓάϨʔυํ๏ͷಛ௃ DMVTUFSNJHSBUJPO w ໘౗ w Ϋϥ΢υ΍πʔϧͷԸܙΛड͚ΒΕͳ͍ w ෳ਺ΫϥελͰͷӡ༻ͷߟྀ͕ඞཁ w ޓ׵ੑͷ੍໿͕ͳ͍ w όʔδϣϯΛඈ͹্ͯ͛͠Δ͜ͱ͕Մೳ w ίΞίϯϙʔωϯτΛೖΕସ͑Δ͜ͱ΋Մೳ
  17. ©2020 Wantedly, Inc. $MVTUFSNJHSBUJPO͕ඞཁͳέʔε աڈʹJOQMBDFVQHSBEF͕Ͱ͖ͳ͔ͬͨྫ  ,VCFSOFUFTͷόά  FUDEͷόʔδϣϯ্͛ 17

  18. ©2020 Wantedly, Inc. ,VCFSOFUFTͷόά NBTUFSOPEFͷϦʔμʔೖΕସ͑࣌ʹ͢΂ͯͷϊʔυ͕/PU3FBEZʹͳΔόά
 IUUQTHJUIVCDPNLVCFSOFUFTLVCFSOFUFTJTTVFT 18 $MVTUFSNJHSBUJPO͕ඞཁͳέʔε

  19. ©2020 Wantedly, Inc. ,VCFSOFUFTͷόά 19 $MVTUFSNJHSBUJPO͕ඞཁͳέʔε NBTUFSOPEF
 MFBEFS NBTUFSOPEF
 OPUMFBEFS

    XPSLFSOPEF 1PE 1PE 1PE XPSLFSOPEF 1PE 1PE 1PE XPSLFSOPEF 1PE 1PE 1PE XPSLFSOPEF 1PE 1PE 1PE NBTUFSOPEFͷϦʔμʔೖΕସ͑࣌ʹ͢΂ͯͷϊʔυ͕/PU3FBEZʹͳΔόά
 IUUQTHJUIVCDPNLVCFSOFUFTLVCFSOFUFTJTTVFT
  20. ©2020 Wantedly, Inc. ,VCFSOFUFTͷόά 20 $MVTUFSNJHSBUJPO͕ඞཁͳέʔε NBTUFSOPEF
 EFMFUFE NBTUFSOPEF
 OPUMFBEFS

    XPSLFSOPEF 1PE 1PE 1PE XPSLFSOPEF 1PE 1PE 1PE XPSLFSOPEF 1PE 1PE 1PE XPSLFSOPEF 1PE 1PE 1PE NBTUFSOPEFͷϦʔμʔೖΕସ͑࣌ʹ͢΂ͯͷϊʔυ͕/PU3FBEZʹͳΔόά
 IUUQTHJUIVCDPNLVCFSOFUFTLVCFSOFUFTJTTVFT
  21. ©2020 Wantedly, Inc. ,VCFSOFUFTͷόά 21 $MVTUFSNJHSBUJPO͕ඞཁͳέʔε NBTUFSOPEF
 EFMFUFE NBTUFSOPEF
 MFBEFS

    XPSLFSOPEF 1PE 1PE 1PE XPSLFSOPEF 1PE 1PE 1PE XPSLFSOPEF 1PE 1PE 1PE XPSLFSOPEF 1PE 1PE 1PE NBTUFSOPEFͷϦʔμʔೖΕସ͑࣌ʹ͢΂ͯͷϊʔυ͕/PU3FBEZʹͳΔόά
 IUUQTHJUIVCDPNLVCFSOFUFTLVCFSOFUFTJTTVFT
  22. ©2020 Wantedly, Inc. ,VCFSOFUFTͷόά 22 $MVTUFSNJHSBUJPO͕ඞཁͳέʔε NBTUFSOPEF
 EFMFUFE NBTUFSOPEF
 MFBEFS

    XPSLFSOPEF
 /PU3FBEZ 1PE 1PE 1PE XPSLFSOPEF
 /PU3FBEZ 1PE 1PE 1PE XPSLFSOPEF
 /PU3FBEZ 1PE 1PE 1PE XPSLFSOPEF
 /PU3FBEZ 1PE 1PE 1PE NBTUFSOPEFͷϦʔμʔೖΕସ͑࣌ʹ͢΂ͯͷϊʔυ͕/PU3FBEZʹͳΔόά
 IUUQTHJUIVCDPNLVCFSOFUFTLVCFSOFUFTJTTVFT
  23. ©2020 Wantedly, Inc. ,VCFSOFUFTͷόά 23 $MVTUFSNJHSBUJPO͕ඞཁͳέʔε NBTUFSOPEF
 EFMFUFE NBTUFSOPEF
 MFBEFS

    XPSLFSOPEF
 /PU3FBEZ 1PE 1PE 1PE XPSLFSOPEF
 /PU3FBEZ 1PE 1PE 1PE XPSLFSOPEF
 /PU3FBEZ 1PE 1PE 1PE XPSLFSOPEF
 /PU3FBEZ 1PE 1PE 1PE × × × × × × × × × NBTUFSOPEFͷϦʔμʔೖΕସ͑࣌ʹ͢΂ͯͷϊʔυ͕/PU3FBEZʹͳΔόά
 IUUQTHJUIVCDPNLVCFSOFUFTLVCFSOFUFTJTTVFT
  24. ©2020 Wantedly, Inc. ,VCFSOFUFTͷόά 24 $MVTUFSNJHSBUJPO͕ඞཁͳέʔε w मਖ਼͸࣍ͷQBUDIWFSTJPOʹCBDLQPSU͞Εͨ w ରԠ͢Δʹ͸ΫϥελͷΞοϓάϨʔυ͕ඞཁ

    w JOQMBDFVQHSBEF͸
 NBTUFSOPEFΛೖΕସ͑Δඞཁ͕͋Δ NBTUFSOPEFͷϦʔμʔೖΕସ͑࣌ʹ͢΂ͯͷϊʔυ͕/PU3FBEZʹͳΔόά
 IUUQTHJUIVCDPNLVCFSOFUFTLVCFSOFUFTJTTVFT
  25. ©2020 Wantedly, Inc. FUDEͷόʔδϣϯ্͛ w WˠW΁ͷΞοϓάϨʔυ,VCFSOFUFT͕αϙʔτ͍ͯ͠ͳ͍ w LPQT͕༻ҙͨ͠FUDEͷNJHSBUJPOQBUI͕͏·͘ಈ͔ͳ͔ͬͨ $MVTUFSNJHSBUJPO͕ඞཁͳέʔε

  26. ©2020 Wantedly, Inc. ͳͥ,VCFSOFUFTΛΞοϓάϨʔυ͢Δͷ͔ w όά΍੬ऑੑͷमਖ਼ w ৽ػೳΛ࢖͍͍ͨ w αϙʔτ͞Ε͍ͯΔόʔδϣϯΛ࢖͏

    26
  27. ©2020 Wantedly, Inc. αϙʔτ͞ΕΔόʔδϣϯ 27 https://kubernetes.io/ja/docs/setup/release/version-skew-policy/

  28. ©2020 Wantedly, Inc. αϙʔτ͞ΕΔόʔδϣϯ 28 w ࠷৽ϦϦʔε͔ΒNJOPSWFSTJPO·Ͱ w ̕ϲ݄ܦͯ͹αϙʔτ͞Εͳ͘ͳΔ 㱺ɹසൟͳΞοϓάϨʔυͷ࣮ࢪ͕ඞཁ

  29. ©2020 Wantedly, Inc. 29 $MVTUFSNJHSBUJPOʹඋ͑Δ ͍ͭDMVTUFSNJHSBUJPO͕ඞཁʹͳΔ͔෼͔Βͳ͍ͷͰ४උΛ͓ͯ͘͠ w Ͳ͏΍Δͷ͔஌Δ w DMVTUFSNJHSBUJPO͠΍͍͢Ϋϥελʹ͢Δ

  30. ©2020 Wantedly, Inc. $MVTUFSNJHSBUJPOͷखॱ 30

  31. ©2020 Wantedly, Inc. $MVTUFSNJHSBUJPOͷखॱ  ৽ΫϥελΛ࡞੒  ৽ΫϥελʹϦετΞ  ৽ΫϥελʹτϥϑΟοΫΛ੾Γସ͑

     چΫϥελ࡟আ 31
  32. ©2020 Wantedly, Inc. LT,VCFSOFUFT'BDUPST 32 https://speakerdeck.com/potsbo/k8s-kubernetes-8-factors

  33. ©2020 Wantedly, Inc. 33 LT,VCFSOFUFT'BDUPST https://speakerdeck.com/potsbo/k8s-kubernetes-8-factors

  34. ©2020 Wantedly, Inc. $MVTUFSNJHSBUJPOͷݪଇ w ΫϥελΛίʔυ؅ཧ ίʔυϕʔεґଘએݴઃఆ෼཭ w ಈ࡞֬ೝͷলྗԽ ୯Ұํ޲Ϗϧυ؀ڥҰக

    w ඇ໊લ؅ཧ w όοΫΞοϓ w ॏෳܽଛ؅ཧ 34
  35. ©2020 Wantedly, Inc. ΫϥελΛίʔυ؅ཧ Ϋϥελͷ࡞੒Λָʹ͍ͨ͠ w DMVTUFSNJHSBUJPO͸ݕূʹ޻਺͕͔͔Γ
 ΫϥελΛ࡞Γ௚͢ճ਺΋૿͑Δ ࠩ෼ͷൺֱΛ؆୯ʹ͍ͨ͠ w

    ৽چΫϥελͷࠩ෼ΛϨϏϡʔ͠΍͘͢ w ݕূ࣌ͷઃఆΛຊ൪ʹ൓ө͠๨ΕΔͷΛ๷ࢭ 㱺ɹΫϥελͷઃఆ͸ίʔυ؅ཧ 㱺ɹΫϥελ࡞੒΋ίʔυԽ 㱺ɹ؀ڥຖͷࠩ෼͸໌ࣔతʹॻ͘ 35
  36. ©2020 Wantedly, Inc. ಈ࡞֬ೝͷলྗԽ JOQMBDFVQHSBEFΑΓ৻ॏͳ֬ೝ͕ඞཁ w Ϋϥ΢υ΍πʔϧͷԸܙ͕ड͚ΒΕͳ͍ w େ͖ͳมߋؚ͕·Ε͍ͯΔέʔε͕ଟ͍ 㱺ɹݕূ؀ڥͱͷࠩҟ͸ग़དྷΔݶΓݮΒ͢

    ຊ൪͚ͩͰى͖Δ໰୊Λચ͍ग़ͤΔΑ͏ʹ͢Δ 㱺ɹΫϥελ΁ͷ൓ө͸ίʔυΛݩʹ͢Δ ҙਤ͠ͳ͍Ϧιʔεͷ࡞੒Λݕ஌Ͱ͖ΔΑ͏ʹ 36
  37. ©2020 Wantedly, Inc. ඇ໊લ؅ཧ ෳ਺Ϋϥελӡ༻Λߟྀ͢Δ w Ϋϥελ໊มߋʹ௥ैͰ͖Δঢ়ଶʹ w Ϋϥελ੾Γସ͑ΛϩʔϧόοΫखஈʹ 㱺ɹಉҰ༻్ͷϦιʔε͕ෳ਺࡞ΒΕΔ͜ͱΛ૝ఆ͢Δ

    w ໋໊Λ޻෉͢Δ 㱺ɹ௚઀໊લͰࢀরͤͣΫογϣϯϨΠϠʔΛڬΉ w Ϣʔβʔʹݟ͑Δ৔ॴ͸$/".&Λ࢖͏ w MBCFM΍UBHͰϦιʔεΛࢦఆ 37
  38. ©2020 Wantedly, Inc. ໋໊Λ޻෉͢Δ ಉҰ༻్ͷϦιʔε͕ෳ਺࡞ΒΕͯ΋িಥ͠ͳ໊͍લ w QSPEVDUJPO 㱺ɹຊ൪؀ڥʹෳ਺ͷΫϥελ͕࡞ΒΕΔ͜ͱΛ૝ఆͰ͖͍ͯͳ͍ w QSPEVDUJPO

    㱺ɹ೔෇Λ෇͚Δ͜ͱͰিಥΛճආ 㱺ɹಉ͡೔ʹෳ਺ͷΫϥελΛӡ༻͢Δ͜ͱ͸ߟ͑ʹ͍͘ 38 ඇ໊લ؅ཧ
  39. ©2020 Wantedly, Inc. ΫογϣϯϨΠϠʔΛڬΉ %BUBEPHʹΑΔ؂ࢹ w ϝτϦΫε΍ΞϥʔτͷઃఆʹΫϥελ໊Λ࢖͍ͬͯͨ w DMVTUFSNJHSBUJPOΛߦ͏ͱ͖ʹஔ͖׵͑Δඞཁ͕͋ͬͨ 㱺ɹࠓޙಉ͡ରԠΛ͠ͳ͍Α͏ʹFOWQSPEVDUJPOͳͲͷλάͰճආ

    39 ඇ໊લ؅ཧ
  40. ©2020 Wantedly, Inc. όοΫΞοϓ ϦετΞʹΑͬͯΫϥελͷத਎ΛҠߦ w ඞཁͳϦιʔε͕͢΂ͯೖͬͨόοΫΞοϓ͕ඞཁ 㱺ɹ͍ͣΕ͔ͷํ๏ͰόοΫΞοϓΛऔ͓ͬͯ͘ 8BOUFEMZͰ͸7FMFSPΛར༻ IUUQTWFMFSPJP

    40
  41. ©2020 Wantedly, Inc. ॏෳܽଛ؅ཧ ෳ਺ΫϥελͰͷӡ༻ʹΑΔ෭࡞༻ΛݮΒ͢ w LT͸ॏෳΛڐ༰͢ΔϦιʔε͔͠૝ఆ͞Ε͍ͯͳ͍ w ॏෳΛڐ༰Ͱ͖ͳ͍Ϧιʔε͸ઐ༻ͷखॱ͕ඞཁ 㱺ɹॏෳ͔ܽଛͷͲͪΒ͔͸ڐ༰Ͱ͖Δߏ੒ʹ

    㱺ɹઐ༻ͷखॱ͕ඞཁͳϦιʔεʹ͸ϥϕϧΛઃఆ 㱺ɹ྆ํͱ΋ڐ༰Ͱ͖ͳ͍Ϧιʔε͸ѻΘͳ͍ 41
  42. ©2020 Wantedly, Inc. $SPO+PC w ఆظతʹόονॲཧΛ࣮ߦͤ͞ΔͨΊʹར༻ w 8BOUFEMZʹ͸ॏෳ࣮ߦ͕ڐ༰͞Εͳ͍KPC͕ଘࡏ͢Δ w ਺ඦͷKPC͕ଘࡏ͢͠΂ͯΛॏෳͯ͠΋͍͍ঢ়ଶʹ͢Δͷ͸ࠔ೉

    㱺ɹܽଛΛڐ༰͢Δ࣮ߦ͞Εͳ͍έʔεΛڐ༰͢Δ چΫϥελ͔Βফ͔ͯ͠Β৽ΫϥελʹϦετΞ͢Δ 42 ॏෳΛڐ༰͠ͳ͍Ϧιʔεͷྫ ॏෳܽଛ؅ཧ
  43. ©2020 Wantedly, Inc. 3%# w ෳ਺ͷঢ়ଶ͕ଘࡏ͢ΔͱҰ؏ੑ͕อͨΕͳ͍ w ྫچΫϥελͱ৽ΫϥελͰೖ͍ͬͯΔσʔλ͕ҧ͏ w μ΢ϯλΠϜ͕͋ΔͱαʔϏεʹӨڹ͕ग़Δ

    㱺ɹ,VCFSOFUFTͰ͸؅ཧ͠ͳ͍ 8BOUFEMZͰ͸"84ͷ3%4Λར༻ 43 ॏෳͱܽଛͷ྆ํΛڐ༰͠ͳ͍Ϧιʔεͷྫ ॏෳܽଛ؅ཧ
  44. ©2020 Wantedly, Inc. ࣮ࡍͷDMVTUFSNJHSBUJPOखॱͷྫ  ৽ΫϥελΛ࡞੒  ॏෳΛڐ༰͢ΔϦιʔεΛ৽ΫϥελʹϦετΞ  ॏෳΛڐ༰͠ͳ͍ϦιʔεΛچΫϥελ͔Β࡟আ

     ॏෳΛڐ༰͠ͳ͍ϦιʔεΛ৽ΫϥελʹϦετΞ  ৽ΫϥελʹτϥϑΟοΫΛ੾Γସ͑  چΫϥελ࡟আ 44
  45. ©2020 Wantedly, Inc. 45

  46. ©2020 Wantedly, Inc. 46

  47. ©2020 Wantedly, Inc. 47 ·ͱΊ  $MVTUFSNJHSBUJPOͷඞཁੑ w جຊ͸JOQMBDFVQHSBEF w

    JOQMBDFVQHSBEFͰμ΢ϯλΠϜ͕ൃੜ͢Δͱ͖ʹ࣮ࢪ  $MVTUFSNJHSBUJPOͷखॱ w όοΫΞοϓ͔ΒϦετΞͯ͠ϦιʔεΛҠߦ w ಉ͡ઃఆͰෳ਺ΫϥελΛӡ༻͢Δ͜ͱΛ૝ఆ͓ͯ͘͠