Upgrade to Pro — share decks privately, control downloads, hide ads and more …

OpenStackクラスタ間マイグレーション事例 Havana to Mitaka / OpenStack Migration Case (Shift from Havana to Mitaka)

OpenStackクラスタ間マイグレーション事例 Havana to Mitaka / OpenStack Migration Case (Shift from Havana to Mitaka)

OpenStack Days Tokyo 2017 で発表した、Havana/Mitaka の並行運用+その間のマイグレーションについて

Shinya Tsunematsu

July 20, 2017
Tweet

More Decks by Shinya Tsunematsu

Other Decks in Technology

Transcript

  1. )BWBOBUP.JUBLB
    ৗদ৳࠸(.01FQBCP *OD
    0QFO4UBDL%BZT5PLZP
    0QFO4UBDLΫϥελؒ

    ϚΠάϨʔγϣϯ

    View Slide

  2. ϓϦϯγύϧΤϯδχΞ
    ৗদ৳࠸!UONU
    ٕज़෦ٕज़ج൫νʔϜ
    IUUQTCMPHUONUJOGP

    View Slide

  3. (.0ϖύϘ
    اۀཧ೦΋ͬͱ͓΋͠Ζ͘Ͱ͖Δ
    ϛογϣϯΠϯλʔωοτͰՄೳੑΛͭͳ͛ΔɺͻΖ͛Δ

    View Slide

  4. View Slide

  5. -0-*101ϚωʔδυΫϥ΢υϦϦʔε
    Ћςετதʂ

    View Slide

  6. ΞδΣϯμ
    w ϓϥΠϕʔτΫϥ΢υಋೖܦҢͱࠓ·Ͱ
    w όʔδϣϯฒߦӡ༻ʹ͍ͭͯ
    w ࣄྫ τϥϒϧγϡʔτ
    w ฒߦɾࣗࣾӡ༻Λܦͯࠓޙ

    View Slide

  7. ϓϥΠϕʔτΫϥ΢υಋೖܦҢ

    View Slide

  8. View Slide

  9. /ZBIJT
    w(.0ϖύϘͷϓϥΠϕʔτΫϥ΢υͷίʔυωʔϜ
    w0QFO4UBDLͰߏங͞ΕͨԾ૝Πϯϑϥج൫
    w֤αʔϏεɾ঎ࡐͷαʔόʔ؀ڥͱͯ͠ར༻த
    w೥ߏ૝݄ΑΓ0QFO4UBDLݕ౼։࢝
    w೥݄άϧʔϓձࣾͷࢧԉΛड͚ɺ)BWBOBελοΫӡ༻։࢝
    w೥݄ࣗࣾߏஙʹͯɺ.JUBLBελοΫͷӡ༻։࢝

    View Slide

  10. ೥݄
    0QFO4UBDLӡ༻೥ΛৼΓฦͬͨ
    ·ͱΊΛൃද
    IUUQTTQFBLFSEFDLDPNUONUQFQBCPT
    QSJWBUFDMPVEOZBIBGUFSUIBU

    View Slide

  11. ೥݄ൃදͷৼΓฦΓ

    View Slide

  12. ৼΓฦΓಋೖܦҢ

    View Slide

  13. ৼΓฦΓޮՌ

    View Slide

  14. ৼΓฦΓ՝୊

    View Slide

  15. ৼΓฦΓ೥ํ਑

    View Slide

  16. ৼΓฦΓࣗࣾͰͷ0QFO4UBDLߏங

    View Slide

  17. ݱࡏͷ/ZBIͷঢ়گ

    View Slide

  18. /ZBICFGPSFBGUFS
    w0QFO4UBDLόʔδϣϯ)BWBOBˠ.JUBLB
    w$JOEFSར༻ແˠ༗ %FMM&.$4DBMF*0

    w/FVUSPOҰ෦ར༻ˠ%734/"5)"

    View Slide

  19. %73
    IUUQTXXXTDIPPMDUDHDPKQDPMVNOTOBLBJOBLBJIUNM

    View Slide

  20. 4/"5)"
    DPNQVUF
    WN
    OFVUSPO OFVUSPO
    LFFQBMJWFE
    .JUBLBҎ߱͸OFVUSPOΛ৑௕Խ͢Δ͜ͱͰɺ4/"5ϧʔλʔΛ෼ࢄ഑ஔ͠
    LFFQBMJWFEͰ৑௕Խ͢Δ͜ͱ͕ग़དྷΔΑ͏ʹͳͬͨ

    View Slide

  21. όʔδϣϯؒϚΠάϨʔγϣϯ

    View Slide

  22. ϚΠάϨʔγϣϯʁ
    wطଘͷ)BWBOB؀ڥΛΞοϓάϨʔυग़དྷͳ͍
    w04͕ҟͳΔ $FOU04ˠ6CVOUV

    w)BWBOBUP.JUBLB͸Ұ౓Ͱ͸ΞοϓάϨʔυෆՄ
    w৽نϥοΫɾαʔόʹͯ.JUBLB؀ڥΛผ్ߏங
    wಉ%$಺ผϥοΫͳͷͰωοτϫʔΫ͸ૄ௨ग़དྷΔ

    View Slide

  23. ؀ڥͷωοτϫʔΫ઀ଓ

    View Slide

  24. ωοτϫʔΫ઀ଓྫ
    WN WN

    OFUXPSL
    7-"/
    %)$1QPPM

    OFUXPSL
    7-"/
    %)$1QPPM
    QPSUSFTFSWF
    /ZBI .JUBLB

    /ZBIDMBTTJD )BWBOB

    -εΠον
    ʢ෺ཧ

    Ͱͭͳ͕͍ͬͯΔ
    .ɾ)
    ؒͰॏෳ͠ͳ͍Α͏
    ʹ*1ઃܭ

    View Slide

  25. ωοτϫʔΫ઀ଓྫ
    wϧʔλʔΛར༻͠ͳ͍ߏ੒
    w. JUBLB
    ɾ) BWBOB
    ؒͰಉ͡7-"/*% αϒωοτΛར༻
    wޓ͍ͷ؀ڥͰ؅ཧ*1͕ॏෳ͠ͳ͍Α͏ʹ؅ཧ
    w)BWBOBSFTFSWFEQPSU %)$1ϨϯδΛޙ͔ΒมߋͰ͖ͳ͍ʜ

    wγϯϓϧͰ෼͔Γ΍͍͢
    wQSPKFDUͷن໛͕େ͖͍ͱ*1͕଍Βͳ͘ͳΔՄೳੑ͕͋Δ

    View Slide

  26. ωοτϫʔΫ઀ଓྫ
    WN
    WN
    ˒UFTUMBO7-"/
    ˒
    ˒DMBTTJDUFTUMBO7-"/
    ˒
    ˒UFTUMBOSPVUFS
    ˒UFTUMBO7-"/
    /ZBIDMBTTJD )BWBOB
    /ZBI .JUBLB

    View Slide

  27. ωοτϫʔΫ઀ଓྫ
    wϧʔλʔΛར༻͢Δߏ੒
    w.ɾ)؀ڥͰͷ؅ཧ*1ॏෳΛؾʹ͠ͳ͘ͱ΋ྑ͍
    wτϙϩδ͸एׯෳࡶʹ͸ͳΔ͕ཧղ͕ग़དྷͳ͍΄ͲͰ͸ͳ͍
    wར༻QPSU਺ͷ૿Ճ͕༧૝͞ΕΔ৔߹ɺผαϒωοτʹͳΔͷͰϨϯδ
    Λ޿͘औΔ͜ͱ͕ग़དྷΔ
    wݱࡏ͸ͪ͜ΒΛར༻͢Δํ਑ʹ͍ͯ͠Δ

    View Slide

  28. ໰୊ύέοτॏෳ໰୊
    WNˠWNSPVUFS͔Βग़Δύέοτͷ43$."$͸%73͕ಠࣗʹ
    ෇༩͢Δ."$
    WNˠWNύέοτͷ%45͕SPVUFSͷ."$ΞυϨε
    w %45͸%73͕ಠࣗʹ෇༩͢Δ."$ʹͯ͠ཉ͍͠
    %73Ͱߏ੒͞ΕͨSPVUFS͸֤DPNQVUFOPEF্ʹଘࡏ͠ɺ.ɾ)ؒ
    ͸-Ͱͭͳ͕͍ͬͯΔͨΊɺ͢΂ͯͷSPVUFSʹύέοτ͕ಧ͘
    WNʹಧ͘ύέοτ͕ॏෳ͠௨৴͕ෆ҆ఆʹͳΔ

    View Slide

  29. ໰୊ύέοτॏෳ໰୊
    w.JUBLBͰͷچϓϩδΣΫτωοτϫʔΫʢલड़7-"/ʣ͸֎෦
    FYUFSOBM
    ωοτϫʔΫͱͯ͠ఆٛ͢Δඞཁ͕͋Δ
    w֎෦ωοτϫʔΫͱͯ͠ઃఆͨ͠৔߹ύέοτͷ%61͸ൃੜ͠ͳ͍
    w಺෦ωοτϫʔΫ֎෦ωοτϫʔΫ΁ͷ௨৴͕4/"5ϧʔλʔΛ
    ܦ༝͢Δ
    wͭ·Γ%73Λ࢖༻͠ͳ͍ͨΊ

    View Slide

  30. 5FSSBGPSN

    View Slide

  31. 5FSSBGPSN
    w.JUBLBΑΓຊ֨ར༻։࢝
    w0QFO4UBDL1SPWJEFS
    wJOTUBODFҎ֎ʹ΋༷ʑͳϦιʔεʹରͯ͠ɺએݴతʹهड़ɾ؅ཧ͕ߦ
    ͑ΔΑ͏ʹͳͬͨ
    wQPSU OFUXPSL WPMVNF౳ʑ
    w؀ڥͷωοτϫʔΫ઀ଓ༻ͷSPVUFS΋࡞੒ग़དྷΔΑ͏ʹͳͬͨ

    View Slide

  32. ໰୊ෆਖ਼ͳϧʔλʔ͕࡞੒͞ΕΔ
    w 4/"5)"ߏ੒ͰSPVUFS࡞੒࣌ʹ[email protected]@JOGPΛࢦఆ͢Δ
    ͱωοτϫʔΫ௨৴͕ग़དྷͳ͍SPVUFS͕࡞੒͞Εͯ͠·͏
    w OFVUSPOϊʔυ্ͰΤϥʔ͕ϧʔϓ͠ϩά͕ංେԽ͢Δ
    w ରॲํ๏ͱͯ͠͸OFVUSPOMBHFOUͷ࠶ىಈͷΈ
    w )PSJ[PO͔ΒͷSPVUFS࡞੒࣌ʹ͸໰୊ʹͳΒͳ͔ͬͨ
    w 5FSSBGPSNΛར༻͢ΔΑ͏ʹͳ͔ͬͯΒ໰୊͕ൃ֮
    w 5'͕[email protected]Λઃఆ͢ΔλΠϛϯά͕ૣ͗͢Δ͜ͱ͕ݪҼ

    View Slide

  33. ໰୊ෆਖ਼ͳϧʔλʔ͕࡞੒͞ΕΔ
    wରॲํ๏ͱͯ͠͸5'ͰͷSPVUFS࡞੒࣌ʹ͸[email protected]͸ࢦఆ
    ͠ͳ͍
    w݄಄ɺCBDLQPSUpYͰࠜຊରԠͱͳΔमਖ਼͕ߦΘΕͨ
    wIUUQTCVHTMBVODIQBEOFUOFVUSPOCVH
    wOFVUSPOϊʔυ͸৑௕ߏ੒͕औΒΕ͍ͯΔҝɺ୆ͣͭϑΣΠϧΦʔ
    όʔͤͯ͞ύοέʔδΛΞοϓσʔτ

    View Slide

  34. ໰୊4/"57331QPSU%08/
    w4/"5)"ͷखಈϑΣΠϧΦʔόʔ࣌ʹൃ֮
    wOFVUSPOϊʔυؒͰLFFQBMJWFEʹΑΔ৑௕Խ͕औΒΕ͍ͯΔ
    w7331Λૹड৴͢ΔQPSU͕ىಈ͍ͯ͠Δͷ͕લఏ͕ͩɺԿނ͔୆Ͱ
    ͦͷQPSU͕μ΢ϯ͍ͯͨ͠
    wQPSU͕μ΢ϯ͍ͯ͠ΔͨΊɺखಈϑΣΠϧΦʔόʔ͕ߦ͑ͳ͍
    w݁ՌΠϯελϯε͔Β֎෦ωοτϫʔΫ΁ͷ௨৴͕ߦ͑ͳ͘ͳΔ

    View Slide

  35. ໰୊4/"57331QPSU%08/
    wࠓͷͱ͜ΖݪҼ͸ෆ໌
    wରԠΛݕ౼த
    wQPSU͕%08/͍ͯ͠Δ͜ͱ͕ݕ஌ग़དྷΔΑ͏ʹ؂ࢹΛߦ͏
    wݕ஌͞ΕͨΒࣗಈతʹىಈ͢Δ

    View Slide

  36. 0DUBWJB -#BB4

    View Slide

  37. 0DUBWJB
    w-#BB4Wͷ࣮૷ 0QFO4UBDLίϯϙʔωϯτͷҰͭ
    w.JUBLB͔Βඪ४Ͱαϙʔτ͞Ε͍ͯΔ
    w֓ཁɺڍಈʹ͍ͭͯ͸ฐࣾϝϯόʔʹΑΔϒϩάهࣄࢀর
    wIUUQCVUZIBUFOBCMPHDPNFOUSZ
    w)BWBOB؀ڥͰ͸֤QSPKFDU΍ཁ݅͝ͱʹϩʔυόϥϯα༻ͷΠϯελϯεΛ࡞
    ੒͍ͯͨ͠
    w͜ΕΛ0QFO4UBDLίϯϙʔωϯτͱͯ͠ղܾ͍ͨ͠

    View Slide

  38. ൺֱ IBQSPYZCBDLFOE

    View Slide

  39. ൺֱ IBQSPYZCBDLFOE

    w'MPBUJOH*1 pQ
    ͸-#ͷ7*1ʹඥ෇͍͍ͯΔ
    wΠϯλʔωοτ͔Βͷύέοτ͸-#Λ௨͠Πϯελϯε΁ϩʔυόϥ
    ϯε͞ΕΔ
    wOFUXPSLOPEF͕γϯάϧϙΠϯτʹͳΓ΍͍͢

    View Slide

  40. ൺֱ 0DUBWJB

    View Slide

  41. ൺֱ 0DUBWJB

    w-#͕IZQFSWJTPS DPNVUFOPEF
    ্ʹ͋Δ
    w-#Λ௨ৗͷΠϯελϯεͱͯ͠ىಈ͢Δ͜ͱͰOFUXPSLOPEF΁ͷτ
    ϥϑΟοΫूதΛආ͚ΕΔ
    wOFUXPSLOPEFͷ410'΋ղফग़དྷΔ

    View Slide

  42. ໰୊0DUBWJBW
    w.JUBLBͷ0DUBWJBόʔδϣϯ͕
    w/FXUPOW 0DBUBW
    wMCBBTMPBECBMBODFSDSFBUF࣌ʹ"NQIPSBΠϯελϯε͕࡞੒͞ΕΔ
    ͕1307*4*0/*/(@45"564͕&3303
    w-#΍QPPMͷ࡟আ͕ࣦഊ͠࡟আग़དྷͳ͍
    w ݫີʹ͸τϥϒϧͰ͸ͳ͍
    1309:1SPUPDPM͕ར༻ग़དྷͳ͍
    w)5514ΛΠϯελϯεͰॲཧ͍ͨ͠৔߹རศੑ͕௿Լ͢Δ

    View Slide

  43. ໰୊0DUBWJBW
    wݱ࣌఺Ͱ࠷৽ͷCΛར༻
    w/FVUSPO-#BB4ͱͷ࿈ܞ͸ఘΊͨ "1*όʔδϣϯ͕߹க͠ͳ͍

    w0DUBWJBΛ؆қʹѻ͑ΔΫϥΠΞϯτΛ࡞੒͠Α͏ͱݕ౼த
    wݱঢ়ͩͱDVSMͰ0DUBWJB"1*Λૢ࡞͢Δ͜ͱͰ-#Λ࡞͍ͬͯΔ
    wͦ΋ͦ΋1309:1SPUPDPMΛ࢖༻͢Δʹ͸0DUBWJB"1*Λ௚઀ୟ͘
    ͔͠ͳ͍

    View Slide

  44. ϚΠάϨʔγϣϯঢ়گ

    View Slide

  45. ϚΠάϨʔγϣϯਐḿ
    w؀ڥؒͷ௨৴͕ߦ͑ΔΑ͏ʹͳͬͨ
    wঢ়ଶ΍σʔλΛอ͍࣋ͯ͠ͳ͍ϩʔϧ͔Βॱ࣍Ҡߦ
    w8FCαʔό΍"1*αʔό
    w"1*αʔό͕.JUBLBʹ͋Γɺσʔλϕʔεαʔό͕)BWBOBʹ͋Δɺ
    ͱ͍͏αʔϏε΋૿͖͑ͯͨ

    View Slide

  46. ϚΠάϨʔγϣϯํ๏
    w.ɾ)ؒͰ௚઀ΠϯελϯεΛҠಈͤ͞Δํ๏͸ࠓͷͱ͜Ζͳ͍
    wͦͷͨΊɺ࠷ॳ͸ঢ়ଶΛ͍࣋ͬͯͳ͍ϩʔϧ͔ΒऔΓ૊ΜͰ͍Δ
    w)BWBOBͷΠϯελϯεΛ(MBODFΠϝʔδԽͯ͠.JUBLBͰͨͯΔɺͱ
    ͍͏πʔϧͷ࡞੒Λݕ౼
    w͜ͷ৔߹͸Πϯελϯεͷఀࢭ͕൐͏
    wσʔλϕʔεαʔόͷҠߦͳͲࠓޙͷ՝୊

    View Slide

  47. ن໛ײ
    w/ZBIDMBTTJD )BWBOB

    wDPNQVUFOPEF
    wJOTUBODF
    w/ZBI .JUBLB

    wDPNQVUFOPEF
    wJOTUBODF

    View Slide

  48. ͦͷଞվળঢ়گ

    View Slide

  49. -JWF.JHSBUJPO CMPDLCBTFE

    wແఀࢭͷΠϯελϯεϚΠάϨʔγϣϯ
    w$JOEFS7PMVNFͷΑ͏ͳڞ༗ετϨʔδΛར༻͠ͳ͍ύλʔϯ
    wΠϯελϯεͷσΟεΫΠϝʔδ΍ϝϞϦ৘ใ͕RFNVΛհͯ͠సૹ
    ͞Ε੾ΓସΘΔ
    wݕূͰ͸੾ΓସΘΓ࣌͸ύέοτ΄ͲͰɺ௨৴ʹେ͖ͳӨڹ͸ݟΒ
    Εͳ͍
    wTTIEͷηογϣϯ΍σʔλϕʔεͷϨϓϦέʔγϣϯ͕੾Εͳ͍

    View Slide

  50. ໰୊ίϯιʔϧϙʔτিಥ
    wϚΠάϨʔγϣϯ࣌ʹΠϯελϯεʹඥ෇͍͍ͯΔ4FSJBMDPOTPMFϙʔ
    τ͕DPNQVUFOPEFؒͰিಥ͢Δ৔߹͕͋Δ
    wϙʔτ͕িಥ͢ΔͱϚΠάϨʔγϣϯʹࣦഊ͢Δ
    wIUUQTCVHTMBVODIQBEOFUOPWBCVHʹͯόάใࠂ
    ͕͞Ε͍ͯΔ

    View Slide

  51. ໰୊ίϯιʔϧϙʔτিಥ
    w/PWBͷઃఆͰΠϯελϯεʹ෷͍ग़͞ΕΔDPOTPMFQPSUͷൣғࢦఆ
    ͕Մೳ
    w֤DPNQVUFOPEFͰൣғ͕ॏෳ͠ͳ͍Α͏ʹগͣͭ͠εϥΠυͤͯ͞
    ઃఆ͢Δ͜ͱͰճආग़དྷͨ
    w/FXUPO͔Β͸pY͞Ε͍ͯΔ

    View Slide

  52. $JOEFS
    w$JOEFS͕ར༻Մೳʹͳͬͨ
    wݱࡏͷόοΫΤϯυ͸%FMM&.$4DBMF*0
    wฐࣾ(JU)VC&OUFSQSJTFͷΠϯελϯε͸ىಈΠϝʔδɾσʔλྖҬ
    ྆ํͱ΋͜ͷ$JOEFS7PMVNFʹ഑ஔ͠Քಇ͍ͤͯ͞Δ
    wσʔλϕʔεͷσʔλͳͲ*014͕ඞཁͳγʔϯͰར༻͍ͯ͘͠༧ఆ

    View Slide

  53. )$* )ZQFS$POWFSHFE*OGSBTUSVDUVSF

    wDPNQVUFOPEF͕4DBMF*0ͷTUPSBHFOPEF΋݉ͶΔ
    w෼ࢄσʔλΛஔ͘σόΠεʹ͸/7.FΛར༻͍ͯ͠Δ
    wIEQBSNͷ؆қ଎౓νΣοΫฏۉ.#TFD.CQT
    w4"5"ͷ(CQTΛ௒͑ΔεϖοΫ
    w4DBMF*0ͷ෼ࢄͰ(CBTFͷετϨʔδ༻ωοτϫʔΫͷํ͕ઌʹଳ
    Ҭ͕଍Βͳ͘ͳΓͦ͏͕ͩɺ·ͩͦ͜·ͰϔϏʔʹ͸࢖ΘΕ͍ͯͳ͍

    View Slide

  54. ·ͱΊ

    View Slide

  55. ࣗલߏஙɾ؀ڥฒྻӡ༻Λܦͯ

    View Slide

  56. 0QFO4UBDLͷίʔυ͕ಡΊͳ͍ͱπϥ͍
    wόάΛ౿Μͩ࣌ɺ৽͍͠ػೳΛ࢖͓͏ͱͨ࣌͠ɺڞʹ
    wޙऀͷ৔߹υΩϡϝϯτ͞Ε͍ͯͳ͍ɺ#MVFQSJOU͚ͩ΋ଟ͍
    wࠓճಛʹ/FVUSPOؚΊωοτϫʔΫͷτϥϒϧγϡʔτʹ࣌ؒΛׂ͍
    ͨ
    wϩάͷௐࠪʹՃ͑ͯ0QFO4UBDLͷίʔυΛಡΉ͕࣌ؒ૿͑ͨ
    wࣗલͰߏஙɾӡ༻͢Δͱ͍͏৔߹ʹ͸֮ޛ͕ඞཁ

    View Slide

  57. ܧଓతʹόʔδϣϯΞοϓ͠ͳ͍ͱπϥ͍
    w঺հͨ͠໰୊΍ػೳͰଟ͔ͬͨʮ৽͍͠όʔδϣϯͩͬͨΒ௚͍ͬͯ
    Δɾ࢖͑ΔʯʹରԠ͍͖͍ͯͨ͠
    wϢʔβ؅ཧ΍ηΩϡϦςΟରԠͳͲ؀ڥΛฒߦӡ༻͢Δखؒ͸͔͔
    Δɻ؀ڥΛूதͯ͠ӡ༻͍ͨ͠ɻ
    w؀ڥؒϚΠάϨʔγϣϯʹ͸ύϫʔ͕ඞཁɻ೥಺͍ͬͺ͍͸ଓ͍ͯ
    ͍͖ͦ͏ͳݟ௨͠ͳͷͰɺຖ౓͸ग़དྷͳ͍ɻ
    wࠓ͋Δ؀ڥΛஈ֊తʹΞοϓάϨʔυ͍ͨ͠

    View Slide

  58. ָ͍͜͠ͱ΋͋Δ
    wlΫϥ΢υͷϓϩόΠμzͱͯ͠ͷ΍Γ͕͍
    wΫϥ΢υͷ࡞Γํֶ͕΂Δ
    wΫϥ΢υͷதͷਓͷؾ͕࣋ͪ෼͔Δ
    w໰୊ղܾͷϨΠϠ͕૿΍ͤΔ
    w4%4 4%/ͳͲιϑτ΢ΣΞͰͷ࣮૷Λ஌Δ͜ͱ͕ग़དྷΔ
    w૯߹ྗ͕ࢼ͞ΕΔ

    View Slide

  59. ࠓޙͷ༧ఆ
    w0QFO4UBDLΞοϓάϨʔυ͠΍͍͢࢓૊Έͮ͘Γ
    wίϯϙʔωϯτͷίϯςφԽ ,PMMB ,VCFSOFUFT

    w4DBMF*0Ҏ֎ͷ4%4Λಋೖ͠શ୆)$*ʹ͢Δ
    wݱࡏͷͱ͜Ζ%3#%Λݕ౼த
    w)$*ʹ൐ͬͨ7.)"
    wΠϯελϯεͷΑΓΑ͍εέδϡʔϦϯάํ๏

    View Slide

  60. ϓϥΠϕʔτΫϥ΢υӡ༻΋
    z΋ͬͱ͓΋͠Ζ͘Ͱ͖Δz
    ࠷৽ͷ࠾༻৘ใΛνΣοΫˠ [email protected]

    View Slide