Upgrade to Pro — share decks privately, control downloads, hide ads and more …

わりとゴツいKubernetesハンズオン そのあとに

わりとゴツいKubernetesハンズオン そのあとに

ふくばねてす node-2

A9f8382f6cdd535370c723eba233a8f7?s=128

Keita Mohri

July 11, 2019
Tweet

Transcript

  1. ΘΓͱΰπ͍,VCFSOFUFTϋϯζΦϯ ͦͷ͋ͱʹ 
 ;͘͹Ͷͯ͢OPEF ໟརܒଠ

  2. ࣗݾ঺հ ‣ ໟརܒଠ  'VTJD$P -UE  ൃҊऀϓϩμΫτΦʔφʔ LVCFSOFUFT࢖ͬͯͳ͍ 

     ,VCFSOFUFT͸׬શʹཧղ͍ͯ͠Δ 2
  3. 3 ΘΓͱΰπ͍,VCFSOFUFTϋϯζΦϯ

  4. ΘΓͱΰπ͍΍ͭ ‣ ΘΓͱόζΓ·ͨ͠ʂ͋Γ͕ͱ͏͍͟͝·͢ 4 https://qiita.com/Kta-M/items/ce475c0063d3d3f36d5d

  5. ΘΓͱΰπ͍΍ͭ ‣ ͜Μͳ໨ඪͰ΍͍ͬͯ·ͨ͠ 5

  6. ΘΓͱΰπ͍΍ͭ ‣ ओཁͳཁૉͷઆ໌͸͍ͯ͠Δ͸ͣ  ,VCFSOFUFTͷ֓ཁ  αϯϓϧΞϓϦͷσϓϩΠ  ֤छϦιʔεͷઆ໌ 

    &,4 FLTDUM ͰͷΫϥελߏங  μογϡϘʔυ  ϩάऩू  )FMN  ؂ࢹ 1SPNFUIFVT (SBGBOB 6
  7. ΘΓͱΰπ͍΍ͭ ‣ Ͱ΋ɺطଘͷ%PDLFSΠϝʔδ͔͠࢖ͬͯͳ͍ʜ Kubernetes Cluster Manifest File Container Registry ׬શʹཧղͨ͠ൣғ

  8. ΘΓͱΰπ͍΍ͭ ‣ ͓෼͔Γ͍͚ͨͩͨͩΖ͏͔ʁ 8 Ͱ͖ΔΑ͏ʹͳΔͱ͸ݴͬͯͳ͍

  9. ΦϦδφϧͷΞϓϦΛߏங͢Δʹ͸ ‣ ϩʔΧϧͷ։ൃϑϩʔΛ࡞Βͳ͖Ό 9 Kubernetes Cluster Manifest File Application Source

    Code Dockerfile Container Registry 1. ίʔυΛฤू 3.σϓϩΠ 2. Docker ImageΛBuild
  10. ΦϦδφϧͷΞϓϦΛߏங͢Δʹ͸ ‣ $*$%؀ڥ΋੔͑ͳ͖Ό 10 Kubernetes Cluster Manifest File Application Source

    Code Dockerfile Container Registry 1. ίʔυΛฤू 2. Git Push 3. Docker ImageΛBuild 4. Docker ImageΛPush 5.σϓϩΠ
  11. Ұؾʹ໘౗͘͘͞ͳ͖ͬͯͨ 11

  12. ؆୯ʹͰ͖ΔΜͰ͢ ͦ͏ɺ4LBGGPMEͳΒͶ 12

  13. 13 4LBGGPMEͱ͸

  14. 4LBGGPMEͱ͸ ‣ https://skaffold.dev/ ‣ (PPHMFۘ੡ͷ,VCFSOFUFTͷ։ൃࢧԉπʔϧ ‣ ΫϥελʹΞϓϦέʔγϣϯΛσϓϩΠ͢Δ·Ͱͷ
 Ұ࿈ͷ࡞ۀΛ ϩʔΧϧϦϞʔτ໰Θͣ ࣗಈԽͯ͘͠ΕΔ

     %PDLFSΠϝʔδͷCVJME  %PDLFSΠϝʔδͷߏ଄ςετ  %PDLFSΠϝʔδ΁ͷλά෇͚  ίϯςφϨδετϦʹ%PDLFSΠϝʔδΛ1VTI  ,VCFSOFUFTΫϥελʹ%FQMPZ 14
  15. 4LBGGPMEͱ͸ ‣ ֤ϑΣʔζ͸ಠཱ͍ͯͯ͠ɺ͍Ζ͍Ζͳ૊Έ߹Θ͕ͤՄೳ 15

  16. ͱΓ͋͑ͣ΍ͬͯΈΑ͏ 16

  17. 17 4LBGGPMEΛ
 ϩʔΧϧ։ൃͰ࢖ͬͯΈΑ͏

  18. ͜ΕΛ΍Γ͍ͨ 18 Kubernetes Cluster Manifest File Application Source Code Dockerfile

    Container Registry 1. ίʔυΛฤू 3.σϓϩΠ 2. Docker ImageΛBuild
  19. ϩʔΧϧͷ
 ,VCFSOFUFT؀ڥΛ४උ͢Δ 19

  20. ,VCFSOFUFT༗ޮԽ 20 1SFGFSFODFTը໘ ,VCFSOFUFTλϒΛબ୒ νΣοΫʂ "QQMZ ॳճ͸༗ޮʹͳΔ·Ͱ
 ෼΄Ͳ͔͔Γ·͢

  21. ,VCFSOFUFT༗ޮԽ 21 Ҏલ࢖ͬͨ͜ͱ͋Δਓ͸Ұ୴Ϧηοτ͓ͯ͘͠ͱ͍͍͔΋ 3FTFUλϒΛબ୒ Ϧηοτ

  22. LVCFDUMͷDPOpHઃఆ 22 # kubectlͰૢ࡞͢ΔKubernetesΫϥελʔΛdocker-for-desktopʹ $ kubectl config use-context docker-for-desktop #

    ͪΌΜͱઃఆ͞Ε͍ͯΔ͔֬ೝ $ kubectl config current-context docker-for-desktop
  23. ಈ࡞֬ೝͱԼ४උ 23 # ಈ࡞֬ೝɻͳΜ͔͍Ζ͍Ζಈ͍ͯΔ(આ໌͸͋ͱͰ) $ kubectl get pods --namespace=kube-system NAMESPACE

    NAME READY STATUS RESTARTS AGE kube-system etcd-docker-for-desktop 1/1 Running 0 1m kube-system kube-apiserver-docker-for-desktop 1/1 Running 0 1m kube-system kube-controller-manager-docker-for-desktop 1/1 Running 0 1m kube-system kube-dns-86f4d74b45-xb4qh 3/3 Running 0 2m kube-system kube-proxy-8r45p 1/1 Running 0 2m kube-system kube-scheduler-docker-for-desktop 1/1 Running 0 1m # docker-for-desktopͰingress͕࢖͑ΔΑ͏Լ४උɻ͋·Γؾʹ͠ͳͯ͘OK $ kubectl create namespace ingress-nginx $ cat << EOF > kustomization.yaml apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization namespace: ingress-nginx bases: - github.com/kubernetes/ingress-nginx/deploy/cluster-wide - github.com/kubernetes/ingress-nginx/deploy/cloud-generic EOF $ kubectl apply -k .
  24. αϯϓϧΞϓϦέʔγϣϯΛ
 ४උ͢Δ 24

  25. αϯϓϧΞϓϦέʔγϣϯऔಘ 25 # kubernetes/examples͔ΒguestbookΞϓϦΛऔͬͯ͘Δ $ git clone git@github.com:kubernetes/examples.git $ mkdir

    skaffold_example $ cp -R examples/guestbook/* skaffold_example $ cd skaffold_example # ෆཁͳϑΝΠϧΛ࡟আ $ rm -rf all-in-one $ rm -rf legacy # PCͷෛ୲ΛԼ͛ΔͨΊͪΐͬͱௐ੔ & ΠϝʔδͷऔಘઌΛมߋ $ vi examples/guestbook/frontend-deployment.yaml 10ߦ໨ replicas: 3 <- ͜ΕΛ1ʹมߋ 19ߦ໨ image: gcr.io/google-samples/gb-frontend:v4 <- ͜ΕΛ ${dockerhubͷϢʔβʔ໊}/skaffold- example-frontendʹมߋ $ vi examples/guestbook/redis-slave-deployment.yaml 11ߦ໨ replicas: 2 <- ͜ΕΛ1ʹมߋ 21ߦ໨ image: gcr.io/google_samples/gb-redisslave:v1 <- ͜ΕΛ ${dockerhubͷϢʔβʔ໊}/skaffold- example-redis-slaveʹมߋ
  26. JOHSFTTͷϚχϑΣετΛ࡞੒ 26 $ cat << 'EOT' >./guestbook-ingress.yaml apiVersion: extensions/v1beta1 kind:

    Ingress metadata: name: guestbook-ingress spec: rules: - http: paths: - path: / backend: serviceName: frontend servicePort: 80 EOT
  27. 4LBGGPME ΠϯετʔϧηοτΞοϓ 27

  28. 4LBGGPMEͷΠϯετʔϧ ‣ https://skaffold.dev/docs/getting-started/#installing-skaffold ‣ NBDͳΒCSFXͰೖΔ  CSFXJOTUBMMTLBGGPME 28

  29. 4LBGGPMEͷઃఆϑΝΠϧΛ࡞੒ 29 apiVersion: skaffold/v1beta11 kind: Config # DockerΠϝʔδͷϏϧυʹؔ͢Δઃఆ build: artifacts:

    # Ϗϧυ͢ΔΠϝʔδ - image: ${dockerhubͷϢʔβʔ໊}/skaffold-example-frontend context: php-redisɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹ # ιʔεϑΝΠϧ͕͋Δύεͷࢦఆ - image: ${dockerhubͷϢʔβʔ໊}/skaffold-example-redis-slave context: redis-slave local: # ϩʔΧϧͰΠϝʔδΛϏϧυ͢Δͱ͖ͷઃఆ push: false # Ϗϧυͨ͠ΠϝʔδΛpush͢Δ͔ # σϓϩΠʹؔ͢Δઃఆ deploy: kubectl: # σϓϩΠର৅ͷϚχϑΣετϑΝΠϧͷࢦఆ manifests: - frontend-*.yaml - redis-master-*.yaml - redis-slave-*.yaml - guestbook-ingress.yaml ͦͷଞύϥϝʔλʹ͍ͭͯ͸ https://skaffold.dev/docs/references/yaml/Λࢀর skaffold.yaml
  30. TLBGGPMEEFW 30

  31. TLBGGPMEEFW 31 $ skaffold dev Generating tags... - mohri1219/skaffold-example-frontend ->

    mohri1219/skaffold-example-frontend:76a6cd0-dirty - mohri1219/skaffold-example-redis-slave -> mohri1219/skaffold-example-redis-slave:76a6cd0 Tags generated in 49.700092ms Starting build... Found [docker-for-desktop] context, using local docker daemon. Building [mohri1219/skaffold-example-frontend]... Sending build context to Docker daemon 7.68kB Step 1/8 : FROM php:5-apache ---> 24c791995c1e (தུ) Starting deploy... kubectl client version: 1.14 deployment.apps/frontend created service/frontend created ingress.extensions/guestbook-ingress created deployment.apps/redis-master created service/redis-master created deployment.apps/redis-slave created service/redis-slave created (தུ) Watching for changes every 1s... [frontend-7df8f8b7f8-mscxk php-redis] AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 10.1.1.154. Set the 'ServerName' directive globally to suppress this message ΠϝʔδͷCVJME σϓϩΠ ϑΝΠϧมߋ଴ػϩάग़ྗ
  32. Ͱ͖ͨʂ 32 IUUQMPDBMIPTU 1$ͷঢ়گʹΑͬͯ͸͏·͘ΞΫηεͰ͖ͳ͔ͬͨΓ͢Δ͔΋ IUUQͩͱ͏·͍͘͘৔߹΋͋Δͱ͔

  33. ͓΋ΉΖʹ1)1ͷϑΝΠϧΛม͑ͯΈΔ 33 # ద౰ʹλΠτϧΛม͑ͯΈΔ 11ߦ໨ <h2>Guestbook!!</h2> <- "!!"Λ௥Ճͯ͠ΈΔ Generating tags...

    - mohri1219/skaffold-example-frontend -> mohri1219/skaffold-example-frontend:76a6cd0-dirty Tags generated in 62.953504ms Starting build... Found [docker-for-desktop] context, using local docker daemon. Building [mohri1219/skaffold-example-frontend]... Sending build context to Docker daemon 7.68kB Step 1/8 : FROM php:5-apache ---> 24c791995c1e Step 2/8 : RUN pear channel-discover pear.nrk.io ---> Using cache ---> bd9823d8109d Step 3/8 : RUN pear install nrk/Predis ---> Using cache ---> 390dd33a211f ϑΝΠϧͷมߋΛࣗಈݕ஌ͯ͠ߋ৽ͯ͘͠ΕΔʂ php-redis/index.html
  34. มΘͬͨʂ 34

  35. $USM $Ͱऴྃ͢Δ 35 Pruning images... untagged image mohri1219/skaffold-example-frontend: 3eba4d84484d6648b4a0907db8f472e5df4439d9ba4ca012a5b96f95af4dd09b untagged

    image mohri1219/skaffold-example-frontend:c4971ec-dirty untagged image mohri1219/skaffold-example- frontend@sha256:c2f8eb06b6a9d0624c1f8ea6cba4dfaefa450f11d99c90a209617618b0687730 deleted image sha256:3eba4d84484d6648b4a0907db8f472e5df4439d9ba4ca012a5b96f95af4dd09b deleted image sha256:42794d3d5de90049ae108429e1536b747b145918b9972edd62e958b58b09c1ec WARN[0290] builder cleanup: pruning images: Error response from daemon: conflict: unable to delete 48d1ae3a57c7 (cannot be forced) - image is being used by running container 0b0107ee518b Cleaning up... deployment.apps "frontend" deleted service "frontend" deleted ingress.extensions "guestbook-ingress" deleted deployment.apps "redis-master" deleted service "redis-master" deleted deployment.apps "redis-slave" deleted service "redis-slave" deleted Cleanup complete in 3.70313407s ͖Ε͍ʹย෇͚ͯ͘ΕΔʂ Πϝʔδͷ࡟আ Ϋϥελ͔Β࡟আ
  36. TLBGGPMESVO 36

  37. TLBGGPMESVO 37 $ skaffold run Generating tags... - mohri1219/skaffold-example-frontend ->

    mohri1219/skaffold-example-frontend:76a6cd0-dirty - mohri1219/skaffold-example-redis-slave -> mohri1219/skaffold-example-redis-slave:76a6cd0 Tags generated in 48.802686ms Starting build... Found [docker-for-desktop] context, using local docker daemon. Building [mohri1219/skaffold-example-frontend]... Sending build context to Docker daemon 7.68kB Step 1/8 : FROM php:5-apache ---> 24c791995c1e (தུ) Starting deploy... kubectl client version: 1.14 deployment.apps/frontend created service/frontend created ingress.extensions/guestbook-ingress created deployment.apps/redis-master created service/redis-master created deployment.apps/redis-slave created service/redis-slave created Deploy complete in 1.763257066s ΠϝʔδͷCVJME σϓϩΠ ୯ൃͷσϓϩΠɻ׬ྃͯ͠΋Ϋϥελʹ࢒Γଓ͚Δɻ
  38. TLBGGPMEͰςετ΋ͯ͠ΈΔ 38

  39. DPOUBJOFSTUSVDUVSFUFTU ‣ https://github.com/GoogleContainerTools/container-structure-test ‣ ίϯςφΠϝʔδͷߏ଄Λςετ  Πϝʔδ಺ͷίϚϯυͷग़ྗΛνΣοΫ  ϑΝΠϧγεςϜͷϝλσʔλͱ಺༰Λݕূ 

    FUD 39
  40. ςετϑΝΠϧΛઃఆ 40 schemaVersion: 2.0.0 # ֤छϑΝΠϧͷଘࡏ֬ೝ fileExistenceTests: - name: 'php

    file' path: '/var/www/html/guestbook.php' shouldExist: true - name: 'js file' path: '/var/www/html/controllers.js' shouldExist: true - name: 'html file' path: '/var/www/html/index.html' shouldExist: true ͦͷଞͷςετ߲໨ʹ͍ͭͯ͸ https://github.com/GoogleContainerTools/container-structure-test/blob/master/README.mdΛࢀর structure-test/frontend.yaml
  41. 4LBGGPMEͷઃఆϑΝΠϧΛߋ৽ 41 apiVersion: skaffold/v1beta11 kind: Config build: (ུ) # ςετͷઃఆ

    test: test: - image: ${dockerhubͷϢʔβʔ໊}/skaffold-example-frontend structureTests: - ./structure-test/frontend.yaml deploy: (ུ) skaffold.yaml
  42. TLBGGPMESVO 42 $ skaffold run (தུ) Starting test... ====================================== ======

    Test file: frontend.yaml ====== ====================================== === RUN: File Existence Test: php file --- PASS === RUN: File Existence Test: js file --- PASS === RUN: File Existence Test: html file --- PASS ======================================= =============== RESULTS =============== ======================================= Passes: 3 Failures: 0 Total tests: 3 PASS Test complete in 1.455478421s ΋ͪΖΜςετ͕௨Βͳ͔ͬͨΒσϓϩΠ͞Εͳ͍
  43. ϩʔΧϧͰ༡Ϳͷ͸͜͜·Ͱ✋ 43

  44. ,VCFSOFUFTແޮԽ 44 1SFGFSFODFTը໘ ,VCFSOFUFTλϒΛબ୒ νΣοΫΛ֎͢ "QQMZ 3FTFUλϒΛબ୒ ҰԠϦηοτ

  45. 45 4LBGGPMEΛ
 $*$%޲͚ʹ࢖ͬͯΈΑ͏ʂ

  46. ͜ΕΛ΍Γ͍ͨ 46 Kubernetes Cluster Manifest File Application Source Code Dockerfile

    Container Registry 1. ίʔυΛฤू 2. Git Push 3. Docker ImageΛBuild 4. Docker ImageΛPush 5.σϓϩΠ ׬੒൛͸ͪ͜Βhttps://github.com/Kta-M/skaffold_example
  47. ΍Δ͜ͱ ‣ (JU)VCͷϦϙδτϦΛ࡞੒ লུ  ‣ %PDLFS)VCͷϦϙδτϦΛ࡞੒  \EPDLFSIVCͷϢʔβʔ໊^TLBGGPMEFYBNQMFGSPOUFOE 

    \EPDLFSIVCͷϢʔβʔ໊^TLBGGPMEFYBNQMFSFEJTTMBWF ‣ &,4ͰΫϥελΛ࡞੒ ‣ 4LBGGPMEͷઃఆϑΝΠϧΛௐ੔ ‣ $JSDMF$*ͷઃఆ ‣ $JSDMF$*ͷDPOpHΛ࡞੒ ‣ (JUIVCʹ1VTI 47
  48. %PDLFS)VCͷϦϙδτϦΛ࡞੒ 48

  49. &,4ͰΫϥελΛ࡞੒ ‣ ʮΘΓͱΰπ͍ϋϯζΦϯʯͰ΍ͬͨ΍ͭ 49 $ eksctl create cluster \ --name

    eksctl-handson \ --region ap-northeast-1 \ --nodes 3 \ --nodes-min 3 \ --nodes-max 3 \ --node-type t2.medium \ --ssh-public-key <ΩʔϖΞ໊> ˞ߏங׬ྃ·Ͱ෼͙Β͍͔͔Γ·͢ʜ ˞BQOPSUIFBTUC͕બ୒Ͱ͖Δݹ͍"84ΞΧ΢ϯτ͸ɺ";ࢦఆ΋ඞཁ
  50. 4LBGGPMEͷઃఆϑΝΠϧΛௐ੔ ‣ ϩʔΧϧͱڍಈΛม͍͑ͨͱ͜Ζ  ΠϝʔδΛίϯςφϨδετϦʹ1VTI͢Δ  JOHSFTTͷϚχϑΣετϑΝΠϧ͸࢖Θͳ͍ ‣ QSPpMFΛ࢖͓͏ 

    ༩͑ΒΕͨQSPpMF໊ʹԠͯ͡ઃఆΛΦʔόʔϥΠυͯ͘͠ΕΔػೳ 50
  51. 4LBGGPMEͷઃఆϑΝΠϧΛௐ੔ 51 apiVersion: skaffold/v1beta11 kind: Config (தུ) # ҎԼΛ௥Ճ profiles:

    - name: prd build: local: push: true # Ϗϧυͨ͠ΠϝʔδΛίϯςφϨδετϦʹPush͢Δ deploy: kubectl: manifests: # ingressͷϑΝΠϧΛআ֎ - frontend-*.yaml - redis-master-*.yaml - redis-slave-*.yaml TLBGGPMESVOQSPpMFQSEͱ͢Ε͹ɺQSEͷઃఆͰΦʔόʔϥΠυͯ͘͠ΕΔ skaffold.yml
  52. $JSDMF$*ͷઃఆ ‣ (JUIVCͷϦϙδτϦΛొ࿥ 52

  53. $JSDMF$*ͷઃఆ 53

  54. $JSDMF$*ͷઃఆ ‣ ϓϩδΣΫτ؀ڥม਺Λઃఆ 54 "84@%&'"6-5@3&(*0/ $645&3@/".& %0$,&3@)6#@148 %0$,&3@)6#@643 BQOPSUIFBTU FLTDUMIBOETPO

    %PDLFS)VCͷύεϫʔυΛCBTFΤϯίʔυͨ͠΋ͷ %PDLFS)VCͷϢʔβʔ໊
  55. $JSDMF$*ͷઃఆ ‣ "84ͷೝূ৘ใΛઃఆ 55 &,4ΫϥελΛཱͯΔͷʹ࢖ͬͨ*".Ϣʔβʔͱಉ͡΋ͷΛઃఆ͢Δ
 PSCΛ࢖͑ͱݴΘΕͯΔ͚ͲɺόʔδϣϯͰ͔͠࢖͑ͳ͍ʜɻ͸ϩʔΧϧͰͷಈ࡞֬ೝ͕Ͱ͖ͳ͍ͷͰͪ͜ΒͰ

  56. $JSDMF$*ͷDPOpHΛ࡞੒ 56 version: 2 jobs: build: docker: - image: docker:18.09

    steps: - run: name: set shell and install tools command: (ུ) - checkout - setup_remote_docker - run: name: install kubectl command: (ུ) - run: (kubectlͷcontextʹEKSΫϥελΛઃఆ͢ΔͨΊͷ΋Ζ΋Ζ) - run: name: install aws-iam-authenticator command: (ུ) - run: name: login to dockerhub command: echo $DOCKER_HUB_PSW | base64 -d | docker login -u $DOCKER_HUB_USR --password-stdin - run: name: install skaffold command: (ུ) - run: name: skaffold run command: skaffold run --profile prd workflows: version: 2 workflow: jobs: - build ԿΛ͍ͯ͠Δ͔ͷઆ໌ͷͨΊɺ֤छΠϯετʔϧ΋$JSDMF$*Ͱ΍͍ͬͯΔ ຊདྷ͸΋Ζ΋ΖΠϯετʔϧͨ͠%PDLFS*NBHFΛ࡞Δ΄͏͕͍͍ͱࢥ͏ .circleci/config.yml
  57. (JUIVCʹ1VTI͢Δͱʜ 57

  58. $ kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

    service/frontend LoadBalancer 10.100.61.11 xxxxxx.ap-northeast-1.elb.amazonaws.com 80:31673/TCP 8s service/kubernetes ClusterIP 10.100.0.1 <none> 443/TCP 23m service/redis-master ClusterIP 10.100.137.217 <none> 6379/TCP 7s service/redis-slave ClusterIP 10.100.217.57 <none> 6379/TCP 7s Ͱ͖ͨʂ 58 http://xxxxxx.ap-northeast-1.elb.amazonaws.comʹΞΫηεʂ
  59. 59 ·ͱΊ

  60. ·ͱΊ ‣ ʮΘΓͱΰπ͍΍ͭʯ͔ΒҰาਐΜͰɺ
 ,VCFSOFUFTΛ࢖ͬͨ։ൃͷ۩ମతͳΠϝʔδ͕
 ௫ΊͨͷͰ͋Ε͹޾͍Ͱ͢ ‣ ,VCFSOFUFTপʹ଍Λ౿ΈೖΕͨ͹͔Γ  ࠓճ঺հ͖͠Εͳ͔ͬͨ༻ޠɺΤίγεςϜͳͲ·ͩ๲େʹʜ ‣

    Զͨͪͷਅͷઓ͍͸͜Ε͔Βͩʂ 60
  61. 61