Kubernetes拡張を利用した自作Autoscalerで実現するストレスフリーな運用の世界

08a2492de1a41c4f836ecf34d9555828?s=47 mayusy
July 22, 2019

 Kubernetes拡張を利用した自作Autoscalerで実現するストレスフリーな運用の世界

08a2492de1a41c4f836ecf34d9555828?s=128

mayusy

July 22, 2019
Tweet

Transcript

  1. ,VCFSOFUFT֦ுΛར༻ͨࣗ͠࡞"VUPTDBMFSͰ ࣮ݱ͢ΔετϨεϑϦʔͳӡ༻ͷੈք .BZV:BNBNPUPc$ZCFSBHFOU *OD $MPVE/BUJWF%BZT5PLZP +VMZ 

  2. c1SPpMF ‣ ࢁຊ ·Ώ(@mayusy) ‣ גࣜձࣾCyberagent ΞυςΫຊ෦(2018.06~) ‣ Software Engineer

    ओʹಈը޿ࠂͷ഑৴ϓϥοτϑΥʔϜ౳Λ։ൃ ‣ Go/ C#/ Kubernetes/ alcohol /PX d $ZCFSBHFOUΞυςΫຊ෦ʹ+PJO d ৽ଔͰ๭4ࣾʹ+PJO  ࣗ࡞"VUPTDBMFS։ൃ
  3. w ฎνʔϜͷΞʔΩςΫνϟ্ɺ͔ͳΓ($1ͷٕज़ʹدͬͨӡ༻ࣄྫʹ ͳΓ·͢ w ($1ͷར༻͸ฎνʔϜʹ͓͚Δ࿩Ͱ͋Γɺ$ZCFSBHFOUʹ͸($1Ҏ֎ ͷํ๏ͰΠϯϑϥΛߏங͍ͯ͠Δ૊৫΋ଘࡏ͠·͢ c͸͡Ίʹ

  4. ‣νʔϜͷϓϩμΫτͷ͝঺հ ‣ࣗ࡞"VUPTDBMFS͕ඞཁʹͳͬͨഎܠ ‣(FNJOJͷ֓ཁͱ࣮૷ ‣࣮ࡍͷӡ༻݁Ռ ‣·ͱΊ c"HFOEB

  5. νʔϜͷϓϩμΫτ ͷ͝঺հ

  6. ಈը޿ࠂͷ ഑৴ϓϥοτϑΥʔϜ ޿ࠂ΁ͷಋઢ ഑৴׬ྃϖʔδ cϓϩμΫτ֓ཁ

  7. ‣$MPVE1VC4VC w ($1Ͱఏڙ͞Ε͍ͯΔϝοηʔδϯάαʔϏε w 5PQJDɿ1VCMJTIFS͔ΒͷϝοηʔδΛड৴ w 4VCTDSJQUJPOɿ4VCTDSJCFS΁ϝοηʔδΛૹ৴

  8. ‣$MPVE#JHUBCMF w($1ͷετϨʔδαʔϏεͷҰͭɽ wεέʔϥϒϧͰߴ͍εϧʔϓοτΛތΔ,74 w/PEF୯ҐͰεέʔϧ͍ͯ͘͠ɽ ,VCFSOFUFTͷ/PEFͱ͸ผ෺  w"VUPTDBMFػೳ͸༻ҙ͞Ε͍ͯͳ͍

  9. Cloud Pub/Sub Publisher Cloud Pub/SubΛ࢖ͬͯඇಉظ ʹϝοηʔδͷॲཧΛߦ͏. Serviceؒͷίϛϡχέʔγϣϯ ͸gRPCͰߦ͏ cΞʔΩςΫνϟ֓ཁ Service

    Subscriber Cloud Bigtable Subscriber͸Bigtableʹσʔλͷ ಡΈॻ͖΋ߦͳ͍ͬͯΔ
  10. ࣗ࡞Autoscaler͕ ඞཁʹͳͬͨഎܠ

  11. ͢Έ·ͤΜɼ͋Γ͕ͱ͏͍͟͝·͢ɽ ͳΜ͔4VCTDSJCFS͕ґଘ͍ͯ͠Δ#JHUBCMFͷ஗ԆͰϝοηʔδ͕ ٧·ͬͯΔΈ͍ͨͰ͢Ͷɽ #JHUBCMFͷ/PEF਺·Ͱ૿΍͠·ͨ͠ɽ #JHUBCMFͷ஗ԆʹΑΓ4VCTDSJCFSͷMBUFODZ্͕͕ͬͨࢫͷΤϥʔ͕ग़Δ

  12. ͢Έ·ͤΜɼ͋Γ͕ͱ͏͍͟͝·͢ɽ εέʔϧͯ͠΋/PEF਺଍Γͯͳ͍Έ͍ͨͳͷͰɼ ߋʹʹ/PEF਺૿΍͠·ͨ͠ɽ #JHUBCMFͷεέʔϧޙ΋4VCTDSJCFSͷMBUFODZ͸ࢭ·Βͣ

  13. #JHUBCMFͷ஗Ԇ΋ऩ·ͬͨͷͰ/PEF਺ΛݮΒ͓͖ͯ͠·ͨ͠ɽ εέʔϧͰ͖ͳ͍͜ͱ͚ͩ͡Όͳͯ͘ɼ /PEF਺͕ଟ͗͢Δࡍʹແବͳίετ͕͔͔ͬͯ͠·͏͜ͱ΋໰୊Ͱ͢Ͷɽ #JHUBCMFͷ஗Ԇ͕ऩଋͨ͠ޙ

  14. HPA Subscriber ᶃpublisherΛ௨ͯ͠Pub/Sub ʹϝοηʔδ͕དྷΔ Subscriber Subscriber ‣֎෦ʹϦΫΤετΛૹ৴͢ΔॲཧΛ൐͏Subscriberʹͯ Publisher c͜ͷͱ͖ϓϩμΫτͰى͖͍ͯͨ໰୊ $MPVE1VC4VC

  15. HPA Subscriber Subscriber Subscriber ᶄSubscriberʹେྔͷ ϝοηʔδ͕ૹ৴͞ΕΔ Publisher $MPVE1VC4VC ‣֎෦ʹϦΫΤετΛૹ৴͢ΔॲཧΛ൐͏Subscriberʹͯ c͜ͷͱ͖ϓϩμΫτͰى͖͍ͯͨ໰୊

  16. ᶅBigtable΁ͷϦΫΤετ ΋େྔʹൃੜ $MPVE#JHUBCMF HPA Subscriber Subscriber Subscriber Publisher $MPVE1VC4VC ‣֎෦ʹϦΫΤετΛૹ৴͢ΔॲཧΛ൐͏Subscriberʹͯ

    c͜ͷͱ͖ϓϩμΫτͰى͖͍ͯͨ໰୊
  17. $MPVE#JHUBCMF ᶆSubscriber͔ΒͷϦΫΤετ૿Ճ ʹΑΓBigtableͷෛՙ͕૿Ճ HPA Subscriber Subscriber Subscriber Publisher $MPVE1VC4VC ‣֎෦ʹϦΫΤετΛૹ৴͢ΔॲཧΛ൐͏Subscriberʹͯ

    c͜ͷͱ͖ϓϩμΫτͰى͖͍ͯͨ໰୊
  18. ᶇSubscriberͷະॲཧͷ ϝοηʔδ͕૿Ճ $MPVE#JHUBCMF HPA Subscriber Subscriber Subscriber Publisher $MPVE1VC4VC ‣֎෦ʹϦΫΤετΛૹ৴͢ΔॲཧΛ൐͏Subscriberʹͯ

    c͜ͷͱ͖ϓϩμΫτͰى͖͍ͯͨ໰୊
  19. ᶇHuman Bigtable Scaler (ਓྗ)ʹΑͬͯ
 NodeΛखಈͰ௥Ճ $MPVE#JHUBCMF HPA Subscriber Subscriber Subscriber

    Publisher $MPVE1VC4VC ‣֎෦ʹϦΫΤετΛૹ৴͢ΔॲཧΛ൐͏Subscriberʹͯ c͜ͷͱ͖ϓϩμΫτͰى͖͍ͯͨ໰୊
  20. ᶇNodeΛखಈͰ௥Ճ $MPVE#JHUBCMF HPA Subscriber Subscriber Subscriber Publisher $MPVE1VC4VC ‣֎෦ʹϦΫΤετΛૹ৴͢ΔॲཧΛ൐͏Subscriberʹͯ c͜ͷͱ͖ϓϩμΫτͰى͖͍ͯͨ໰୊

    ਓྗͷ࡞ۀ͕සൟʹൃੜ͢ΔΑ͏ʹͳͬͯ͠·ͬͨͨΊɼ ΍͸Γ#JHUBCMFͷ"VUPTDBMFS͸ඞཁ
  21. cϓϩάϥϜ͔Β#JHUBCMFΛTDBMF͢Δํ๏ͷެࣜ৘ใ ϓϩάϥϜ্͔Β4DBMF͢Δํ๏͸͋Δ͕ɼBVUPTDBMFػೳ͸ࣗલͰ࣮૷͢Δඞཁ͕͋Δ
 ࢀߟɿIUUQTDMPVEHPPHMFDPNCJHUBCMFEPDTTDBMJOH

  22. w#JHUBCMFͷ$16࢖༻཰ΛτϦΨʔͱͯ͠#JHUBCMFΛBVUPTDBMF͢Δɽ w,VCFSOFUFTͷϦιʔεঢ়گΛτϦΨʔͱͯ͠ɼ#JHUBCMFΛBVUPTDBMF͢Δɽ c#JHUBCMFͷࣗ࡞"VUPTDBMFSͰ࣮ݱ͍ͨ͜͠ͱ ӡ༻ෛՙͷܰݮˍ༨෼ͳϦιʔεͷ࢖༻ʹΑΔແବͳίετͷ࡟ݮ

  23. Bigtableʹ͸Ϧιʔεత ͳ༨༟͕͋Δ $MPVE#JHUBCMF HPA Subscriber Subscriber Subscriber Publisher $MPVE1VC4VC ‣֎෦ʹϦΫΤετΛૹ৴͢ΔॲཧΛ൐͏Subscriberʹͯ

    Subscriberෆ଍ʹΑΓ ϝοηʔδͷॲཧ͕஗Ԇ c೚ҙͷϝτϦΫεىҼͰ1PEΛεέʔϧ͍ͨ͠ CPU/RAMෛՙ͕ߴ͍Θ͚Ͱ͸ͳ͍ͷͰ σϑΥϧτͰ͸Φʔτεέʔϧ͠ͳ͍
  24. Bigtableʹ͸Ϧιʔεత ͳ༨༟͕͋Δ $MPVE#JHUBCMF HPA Subscriber Subscriber Publisher Subscriber $MPVE1VC4VC ‣֎෦ʹϦΫΤετΛૹ৴͢ΔॲཧΛ൐͏Subscriberʹͯ

    Human Pod Autoscaler (ਓྗ)ʹΑͬͯ PodΛεέʔϧ c೚ҙͷϝτϦΫεىҼͰ1PEΛεέʔϧ͍ͨ͠ Subscriber ϦΫΤετ਺౳ΛτϦΨʔ ͱͯ͠TDBMF͍ͨ͠
  25. Bigtableʹ͸Ϧιʔεత ͳ༨༟͕͋Δ $MPVE#JHUBCMF HPA Subscriber Subscriber Publisher Subscriber $MPVE1VC4VC ‣֎෦ʹϦΫΤετΛૹ৴͢ΔॲཧΛ൐͏Subscriberʹͯ

    Human Pod Autoscaler (HPA)ʹΑͬͯ PodΛεέʔϧ c೚ҙͷϝτϦΫεىҼͰ1PEΛεέʔϧ͍ͨ͠ Subscriber ͜͜Ͱ΋ਓྗͷ࡞ۀ͕සൟʹൃੜ͢ΔΑ͏ʹͳͬͯ͠·ͬͨͨΊɼ ೚ҙͷϝτϦΫεͰͷ1PEͷ"VUPTDBMFS͕ඞཁͱͳͬͨ
  26. Geminiͷ ֓ཁͱ։ൃͷৄࡉ

  27. KubernetesͷCRDͱ͍͏ػೳΛར༻ͨ͠Custom Autoscaler ‣ػೳ • ೚ҙͷϝτϦΫεىҼͰͷBigtableͷautoscale • ೚ҙͷϝτϦΫεىҼͰͷPod autoscale c(FNJOJͱ͸

  28. ,VCFSOFUFTͷ"1*ΛΧελϚΠζͰ͖Δػೳͷ͏ͪͷҰͭɽ ‣"1*&YUFOTJPO w $VTUPN3FTPVSDF%FpOJUJPO $VTUPN$POUSPMMFS w "VUIFOUJDBUJPO 5PLFO3FWJFX"1*  w

    "VUIPSJ[BUJPO 4VCKFDU"DDFTT3FWJFX"1*  w "ENJTTJPO$POUSPM c$3%ͷ֓ཁ
  29. w Ϣʔβʔಠࣗͷ3FTPVSDF5ZQFΛఆٛͰ͖Δػೳ w ಠࣗͷϚχϑΣετϑΝΠϧΛѻ͏͜ͱ͕ՄೳʹͳΔɽ c$VTUPN3FTPVSDF%FpOJUJPO

  30. w Ϣʔβʔ͕ಠࣗʹߏஙͨ͠$POUSPMMFSɽ w ಛఆͷλεΫΛࣗಈԽͰ͖Δɽ w $VTUPN3FTPVSDFͱ૊Έ߹Θͤͯɼଞ$POUSPMMFSͱಉ༷ʹ$POUSPM MPPQ 3FDPODJMJBUJPO-PPQ Λ࣮૷͢Δ͜ͱ͕Մೳɽ c$VTUPN$POUSPMMFS

  31. Gemini Resource 1 Gemini … Gemini Resource 2 Gemini Resource

    3 ‣TDBMFઃఆͷऔಘ w$VTUPN$POUSPMMFSͷ$POUSPMMPPQͷதͰܧଓతʹऔಘ w֤(FNJOJ3FTPVSDF͝ͱʹεέʔϧର৅΍NJONBY஋ͷઃఆ͕Մೳ
  32. ‣ϝτϦΫεͷ࣌ܥྻσʔλͷऩू w4UBDLESJWFS.POJUPSJOHͱ࿈ܞ wΞϥʔτϙϦγʔ͕ࢀর͢ΔϝτϦΫεͷ࣌ܥྻσʔλΛऔಘ w ྫɿ#JHUBCMFDQVVTBHF #JHUBCMFͷϝτϦΫε  w ྫɿະॲཧͷϝοηʔδ਺ 1VC4VCͷϝτϦΫε

     wεέʔϧ൑ఆʹ࢖༻͢ΔΞϥʔτϙϦγʔ͸ઃఆϑΝΠϧͰࢦఆՄೳ
  33. ‣ $VTUPNSFTPVSDF΁ͷओͳઃఆ߲໨ #JHUBCMFTDBMFฤ  w ؂ࢹର৅ͷJOTUBODFOBNF DMVTUFSOBNF w /PEF਺ͷNJONBY஋ w

    TDBMFͷτϦΨʔͱ͍ͨ͠ΞϥʔτͷϙϦγʔ*% BQJ7FSTJPOBVUPTDBMFSDZCFSBHFOUDPKQWCFUB LJOEHFNJOJDVTUPNSFTPVSDF໊ NFUBEBUB MBCFMT DPOUSPMMFSUPPMTLTJP OBNFHFNJOJTBNQMF TQFD DMVTUFS/BNFYYYYYYYYYY JOTUBODF/BNFYYYYYYYYYY NBY/PEFT NJO/PEFT QPMJDZ*EYYYYYYYYYYTUBDLESJWFS্ͰϙϦγʔ͝ͱʹৼΒΕΔ*% ઃఆྫ
  34. ‣ $VTUPNSFTPVSDF΁ͷओͳઃఆ߲໨ 1PETDBMFฤ  w ؂ࢹର৅ͷEFQMPZNFOUOBNF w 3FQMJDB਺ͷNJONBY஋ w TDBMFͷτϦΨʔͱ͍ͨ͠ΞϥʔτͷϙϦγʔ*%

    BQJ7FSTJPOBVUPTDBMFSDZCFSBHFOUDPKQWCFUB LJOEHFNJOJDVTUPNSFTPVSDF໊ NFUBEBUB MBCFMT DPOUSPMMFSUPPMTLTJP OBNFHFNJOJTBNQMF TQFD EFQMPZNFOU/BNFYYYYYYYYYY NBY3FQMJDBT NJO3FQMJDBT QPMJDZ*EYYYYYYYYYYTUBDLESJWFS্ͰϙϦγʔ͝ͱʹৼΒΕΔ*% ઃఆྫ
  35. ‣εέʔϧϩδοΫ w աڈͷ࣌ܥྻσʔλ͕ܧଓͯ͠ᮢ஋ ΞϥʔτϙϦγʔʹઃఆ͞Ε ͍ͯΔᮢ஋ Λ௒͑Δ͔൱͔Ͱ൑ఆ w εέʔϧΞ΢τ͸ૣ͘ɼεέʔϧΠϯ͸Ώͬ͘Γͱߦ͏ w εύΠΫ൑ఆϩδοΫɼલճεέʔϧ͔ΒҰఆ࣌ؒ͸εέʔϧ͠ͳ

    ͍Α͏ʹ͢ΔϩδοΫ౳΋ಠࣗͰಋೖ #JHUBCMFΛTDBMFͨ͠ࡍͷ/PEF਺ͷมԽਤ εέʔϧΠϯ εέʔϧΞ΢τ
  36. ‣࣌ܥྻσʔλͷղੳ w ੜͷ࣌ܥྻσʔλͰ͸ϊΠζ͕ଟ͘ɼεέʔϧͷޡ൑ఆ͕ൃੜ͠΍͘͢ͳΔ ͨΊɼҠಈฏۉ 4." Λ͔͚ͯεϜʔδϯάͨ͠σʔλΛར༻ɽ w 4."ͷXJOEPXTJ[F ϓϩοτִؒ աڈԿ෼ͷσʔλΛ൑ఆʹ࢖༻͢Δ͔౳

    ͷύϥϝʔλ͸ࡉ͔͍ௐ੔͕ඞཁɽ εύΠΫ෦෼ͷϊΠζ͕ ܰݮ͞Ε͍ͯΔ
  37. GKE Gemini ᶃϝτϦΫεͷ࣌ܥྻσʔλͱ ొ࿥ࡁΈͷΞϥʔτϙϦγʔΛऔಘ #JHUBCMF ᶄϝτϦΫε͕ϙϦγʔͷᮢ஋Λ Ұఆ࣌ؒ࣋ଓͯ͠௒͑Δ͔൱͔Λ൑ఆ ᶅBigtableͷAPIΛୟ͍ͯ
 Node਺Λߋ৽ c"VUPTDBMF࣌ͷॲཧϑϩʔ

    #JHUBCMFฤ
  38. kube-apiserver ᶃϝτϦΫεͷ࣌ܥྻσʔλͱ ొ࿥ࡁΈͷΞϥʔτϙϦγʔΛऔಘ Node Pod Pod Gemini c"VUPTDBMF࣌ͷॲཧϑϩʔ 1PEฤ

  39. ᶄϝτϦΫε͕ϙϦγʔͷᮢ஋Λ Ұఆ࣌ؒ࣋ଓͯ͠௒͑Δ͔൱͔Λ൑ఆ c"VUPTDBMF࣌ͷॲཧϑϩʔ 1PEฤ Node Pod Pod Gemini kube-apiserver

  40. ᶅscale out/ inͷ൑ఆΛͨ͠৔߹͸ kube-apiserver্ͷdeploymentΛߋ৽ c"VUPTDBMF࣌ͷॲཧϑϩʔ 1PEฤ Node Pod Pod Gemini

    kube-apiserver
  41. ᶆScheduler͕Deploymentͷ৘ใ ʹ߹ΘͤͯPodΛNodeʹׂΓ౰ͯ c"VUPTDBMF࣌ͷॲཧϑϩʔ 1PEฤ Node Pod Pod Gemini kube-apiserver

  42. ᶇPod͕৽ͨʹ࡞ΒΕΔ c"VUPTDBMF࣌ͷॲཧϑϩʔ 1PEฤ Node Pod New Pod Gemini Pod kube-apiserver

  43. w $VTUPN$POUSPMMFS͸LVCFCVJMEFS౳ͷ4%,Λ࢖༻ͤͣϑϧεΫϥονͰߏஙɽ w 4%,ͷ৘ใ͕গͳ͘ɼ·ͩར༻ࣄྫ΋গͳ͍࣌ظʹ࡞ͬͨͨΊɽ ࣮૷౰࣌  w (PͰ࣮૷ɽ w ओʹ࢖༻ͨ͠ϥΠϒϥϦ

    w ࣌ܥྻσʔλ΍ϙϦγʔऔಘɿ4UBDLESJWFS.POJUPSJOH"1*$MJFOU-JCSBSJFT w EFQMPZNFOUߋ৽ɿLVCFSOFUFTDMJFOUHP c։ൃͷৄࡉ
  44. 1PEͷ4DBMFʹؔͯ͠͸ɼ౰ॳҎԼͷ࣮૷ํ๏΋ݕ౼͍ͯͨ͠ɽ ‣HPAͷCustom Metrics • Prometheus΍StackdriverͳͲͷϞχλϦϯάπʔϧ͔Β೚ҙͷϝτϦΫεΛऔಘͯ͠ɺ ᮢ஋ΛઃఆͰ͖Δɻ • ͭ·Γ೚ҙͷϝτϦΫεىҼͰΦʔτεέʔϧͰ͖ΔΑ͏ʹͳΔɻˠGeminiͱಉ͡ػೳ cٕज़બఆ

  45. ‣ԿނCustom MetricsͰ͸ͳ͘CRDΛ࠾༻ͨ͠ͷ͔ • BigtableͷNodeΛεέʔϧ͍ͤͨ͞ཉٻ΋͋ͬͨͷͰɺPodͷεέʔϧτϦΨʔ Λ૿΍ͤΔ͚ͩͰ͸ෆे෼ɽ • ୯७ʹCRDΛ࢖ͬͯΈ͔ͨͬͨɾCRDͷӡ༻ࣄྫΛ࡞Γ͔ͨͬͨɽ

  46. ࣮ࡍͷӡ༻݁Ռ

  47. 4DBMFPVU ͷᮢ஋ ˞ॾࣄ৘ʹΑΓ1PEͷTDBMFػೳͷํͷӡ༻݁Ռͷσʔλ͕͓ݟͤͰ͖ͳ͍ͷͰɼ
 ࠓճ͸CJHUBCMFͷTDBMFͷํͷӡ༻݁ՌͷΈͷ͝঺հͱͳΓ·͢ɽ

  48. None
  49. None
  50. Human ** Scalerͷग़ಈ͸ ௥Ճ Pod ௥Ճ

  51. Human ** Scalerͷग़ಈ͸ ௥Ճ Pod ௥Ճ ΄΅θϩʹ

  52. ‣ ྑ͔ͬͨ఺ w εύΠΫ൑ఆ౳ɼಠࣗͷεέʔϧ൑ఆج४ͷಋೖ͕Մೳɽ w εέʔϧτϦΨʔΛॊೈʹ௥ՃͰ͖Δɽ w ྫɿؔ࿈͢ΔଞαʔϏεͷϦΫΤετ૿ՃΛࣗαʔϏεͷTDBMFPVUτϦΨʔʹ͢Δɽ ‣ ۤ࿑ͨ͠఺

    w τϦΨʔΛ૿΍͢͜ͱͰɼ$POUSPMMFSͷεέʔϧϩδοΫ͕ෳࡶʹͳΔɽ w ҆ఆͨ͠εέʔϧϩδοΫʹ͢ΔͨΊʹௐ੔ͱվળΛ܁Γฦ͢ඞཁ͕͋ΔͨΊɼ҆ఆՔಈ Ͱ͖ΔϨϕϧʹͳΔ·Ͱʹ͸खؒͱ͕͔͔࣌ؒΔɽ cࣗ࡞"VUPTDBMFSͷ։ൃΛऴ͑ͯ
  53. ·ͱΊ

  54. • ӡ༻ෆՄܰݮͱίετ࡟ݮͷͨΊʹɼCRDΛ࢖ͬͯPod΍BigtableΛscaleͰ͖ ΔAutoscalerΛࣗ࡞ͨ͠ɽ • ͜Ε·ͰखಈͰߦ͍͍͔ͬͯͨͭ͘ͷӡ༻͕΄΅ࣗಈԽͰ͖ΔΑ͏ʹͳͬͨɽ • AutoscalerΛࣗ࡞͢ΔͱεέʔϧϩδοΫ΍τϦΨʔʹॊೈੑΛ࣋ͨͤΒΕΔ ҰํͰɼ҆ఆՔಈ·Ͱʹ࣌ؒͱख͕͔͔ؒΔɽ • ࠓޙOSSԽ౳΋ࢹ໺ʹ͍ΕͯܧଓతͳվળΛଓ͚͍͖͍ͯͨɽ

    c·ͱΊ
  55. Thank you!