Upgrade to Pro — share decks privately, control downloads, hide ads and more …

HelmとService Brokerで始める検証環境自動構築 / Helm and Service Broker

Teppei Fukuda
October 16, 2018

HelmとService Brokerで始める検証環境自動構築 / Helm and Service Broker

第39回 PaaS勉強会での発表資料です。
HelmとService Brokerを使ってKubernetes上でPR単位などで環境を自動構築する話をしました。

Teppei Fukuda

October 16, 2018
Tweet

More Decks by Teppei Fukuda

Other Decks in Technology

Transcript

  1. ࣗݾ঺հ • Teppei Fukuda (@knqyf263) • ߪಡອըࡶࢽ • δϟϯϓ •

    αϯσʔ • ϚΨδϯ • Ϡϯάδϟϯϓ • ϠϯάϚΨδϯ • δϟϯϓSQ • ผ࡭গ೥ϚΨδϯ • ίϛοΫDAYSɾεϚϗΞϓϦଟ਺
  2. • KubernetesͷύοέʔδϚωʔδϟʔ • apt/yum/homebrewͷΑ͏ͳ΋ͷ • deb, rpmʹ૬౰͢Δ΋ͷ͸Chartͱ ݺ͹ΕΔ • ༗໊ͳΞϓϦέʔγϣϯͷChart͸

    ഑෍͞Ε͍ͯΔ • helm install͢Δ͚ͩͰk8s্ʹల։ • e.g. helm install jenkins https://github.com/helm/charts/tree/master/stable/ Helmͱ͸ʁ https://helm.sh/
  3. Chartͱ͸ʁ $ tree wordpress wordpress !"" Chart.yaml … !"" templates

    # !"" NOTES.txt # !"" _helpers.tpl # !"" deployment.yaml # !"" externaldb-secrets.yaml # !"" ingress.yaml # !"" pvc.yaml # !"" secrets.yaml # !"" svc.yaml … # %"" tls-secrets.yaml !"" values-production.yaml %"" values.yaml • Kubernetestͷ
 Ϧιʔεͷू߹ • ࡶʹݴ͑͹ςϯϓϨʔ τͱม਺ͷू߹ ςϯϓϨʔτ ʢ%FQMPZNFOUͱ͔ *OHSFTTͱ͔ʣ ม਺ఆٛ
  4. ςϯϓϨʔτ apiVersion: extensions/v1beta1 kind: Ingress metadata: ... spec: rules: -

    host: {{ .Values.ingress.hostname }} http: paths: - path: / backend: serviceName: {{ template "fullname" . }} servicePort: {{ .Values.service.port }} ීஈॻ͍ͯΔYAMLΛςϯϓϨʔτԽ͢Δ͚ͩ ม਺ ม਺
  5. ී௨ͷYAML apiVersion: extensions/v1beta1 kind: Ingress metadata: ... spec: rules: -

    host: www.example.com http: paths: - path: / backend: serviceName: test-svc servicePort: 80 ͜Ε͕
  6. ςϯϓϨʔτԽͨ͠YAML apiVersion: extensions/v1beta1 kind: Ingress metadata: ... spec: rules: -

    host: {{ .Values.ingress.hostname }} http: paths: - path: / backend: serviceName: {{ template "fullname" . }} servicePort: {{ .Values.service.port }} ͜͏
  7. ม਺ఆٛ $ cat values.yaml service: port: 80 ingress: hostname: www.example.com

    ͋ͱ͸ values.yaml ʹม਺Λఆٛ͢Δ͚ͩ γϯϓϧ Πϯσϯτ͸υοτͰܨ͙ ʢJOHSFTTIPTUOBNF
  8. ෳ਺ͷϦιʔεΛ·ͱΊΒΕΔ $ tree wordpress wordpress !"" Chart.yaml … !"" templates

    # !"" NOTES.txt # !"" _helpers.tpl # !"" deployment.yaml # !"" externaldb-secrets.yaml # !"" ingress.yaml # !"" pvc.yaml # !"" secrets.yaml # !"" svc.yaml … # %"" tls-secrets.yaml !"" values-production.yaml %"" values.yaml Service༻ͷYAML Deployment༻ͷYAML Ingress༻ͷYAML helm installͰશͯk8s্ʹల։͞ΕΔ
  9. ґଘؔ܎΋هड़Մೳ $ cat sentry/requirements.yaml dependencies: - name: postgresql version: 0.18.0

    repository: https://kubernetes-charts.storage.googleapis.com/ - name: redis version: 3.8.1 repository: https://kubernetes-charts.storage.googleapis.com/ sentryͷChartʹ͸PostgreSQLͱRedis͕ඞཁ
  10. ChartΛࣗ࡞͢Δ $ tree test-chart test-chart !"" Chart.yaml !"" charts !""

    requirements.lock !"" requirements.yaml !"" templates # !"" NOTES.txt # !"" _helpers.tpl # !"" api # # !"" deployment.yaml # # !"" ingress.yaml # # %"" svc.yaml # %"" ui # !"" deployment.yaml # !"" ingress.yaml # %"" svc.yaml %"" values.yaml • ྫɿDBͱΩϟογϡαʔόΛ
 ඞཁͱ͢ΔγϯάϧϖʔδΞ ϓϦέʔγϣϯ • UI: ੩తϑΝΠϧΛ഑෍͢Δίϯςφ
 
 ʢຊ൪͸CDNʹஔ͘ͷͰݕূ؀ڥͷΈʣ • API: APIαʔόΛ࣮૷ͨ͠ίϯςφ • DB: PostgresQLʢطଘͷChartΛར༻ʣ • ΩϟογϡɿRedisʢطଘͷChartΛར༻ʣ 3FEJTͱ1PTUHSF42-Λهड़ ʢલड़ʣ "1*ؔ࿈ 6*ؔ࿈ ม਺
  11. APIαʔό apiVersion: extensions/v1beta1 kind: Deployment ... spec: template: spec: ...

    containers: - name: {{ .Chart.Name }}-api image: "{{ .Values.api.image.repository }}:{{ .Values.api.image.tag }}" ... env: - name: POSTGRES_HOST value: {{ template "postgresql.fullname" . }} - name: REDIS_HOST value: {{ template "redis.fullname" . }} DB৘ใ౳͸؀ڥม਺Ͱ౉ͤΔΑ͏ʹ͓ͯ͘͠ (౉͠ํ͸ConfigMap/Secretܦ༝Ͱ΋ԿͰ΋ྑ͍ʣ %#αʔό৘ใ Ωϟογϡαʔό৘ใ
  12. UIαʔό apiVersion: extensions/v1beta1 kind: Deployment ... spec: template: spec: ...

    containers: - name: {{ .Chart.Name }}-ui ... env: - name: API_HOST value: {{ .Values.api.ingress.hostname }} APIαʔόͷϗετ໊͸؀ڥม਺Ͱ౉͢ "1*αʔό৘ใ
  13. UIαʔό {{- if .Values.test.enabled -}} apiVersion: extensions/v1beta1 kind: Deployment ...

    spec: template: spec: ... containers: - name: {{ .Chart.Name }}-ui image: "{{ .Values.ui.image.repository }}:{{ .Values.ui.image.tag }}" ... env: - name: API_HOST value: {{ .Values.api.ingress.hostname }} {{- end -}} ςετ࣌ͷΈ༗ޮʹ͢Δ ৚݅෼ذ༻ͷม਺Λ ࣗ෼Ͱఆ͓ٛͯ͘͠
  14. ม਺ఆٛ $ cat values.yaml test enabled: true api image: tag:

    latest ingress: hostname: api.example.com ui: image: tag: latest ingress: Hostname: ui.example.com Ϧιʔεؒͷܨ͗໨Λશͯม਺Խ͢Δ ʢͦͯ͠؀ڥม਺౳Ͱ౉ͤΔΑ͏ʹ͢Δʣ
  15. ॏཁͳ͜ͱͳͷͰ੔ཧ • Ϧιʔεؒͷܨ͗໨Λશͯม਺Խ͢Δ • UIαʔό <=> APIαʔό • APIαʔό <=>

    DBαʔό • APIαʔό <=> Ωϟογϡαʔό, etc. • ม਺Խ͢Δ͜ͱͰಉ͡Chart͔Βෳ਺ੜ੒͠΍͘͢ͳΔ • ݕূ࣌ͷΈඞཁͳϦιʔε͸৚݅෼ذ͢Δ
  16. test-chart΋ෳ਺ల։Մೳ 6* "1* 1PTUHSF42- 3FEJT test-chart 6* "1* 1PTUHSF42- 3FEJT

    6* "1* 1PTUHSF42- 3FEJT test2 test1 helm install --name test1 test-chart helm install --name test2 test-chart
  17. test-chart΋ෳ਺ల։Մೳ test-chart 6* "1* 1PTUHSF42- 3FEJT helm install --set api.ingress.hostname=test1-api.example.com

    --set ui.ingress.hostname=test1-ui.example.com --name test1 test-chart 6* "1* 1PTUHSF42- 3FEJT test1-ui.example.com test1-api.example.com 6* "1* 1PTUHSF42- 3FEJT helm install --set api.ingress.hostname=test2-api.example.com --set ui.ingress.hostname=test2-ui.example.com --name test2 test-chart test2-ui.example.com test2-api.example.com %/4Ϩίʔυ΋ࣗಈੜ੒ ʢ؀ڥʹΑͬͯ͸ࣄલઃఆ͕ඞཁʣ
  18. 6* "1* 1PTUHSF42- 3FEJT pr1-api.example.com helm install --set api.ingress.hostname=pr1-api.example.com --set

    ui.ingress.hostname=pr1-ui.example.com --name pr1 test-chart pr1-ui.example.com PR࡞੒ʢ#1ʣ Job࣮ߦ Kubernetes PR୯ҐͰͷݕূ؀ڥͷࣗಈߏங͕Մೳʂ
  19. Service Broker಺Ͱ΋෼୲ Service Broker AWS Service Broker GCP Service Broker

    ͦΕͧΕͷϦιʔεΛ୲౰ ࢦࣔΛड͚ ద੾ͳ૬खʹґཔ
  20. SQSΛ࡞Γ͍ͨ৔߹ apiVersion: servicecatalog.k8s.io/v1beta1 kind: ServiceInstance metadata: name: sqs-test spec: clusterServiceClassExternalName:

    sqs clusterServicePlanExternalName: standard ୹͍YAMLΛॻ͚ͩ͘ Ϧιʔεͷ͜ͱΛ4FSWJDF*OTUBODFͱݺͿ 424Λࢦఆ
  21. Secret $ kubectl get secret sqs-test-binding -o yaml apiVersion: v1

    data: DEAD_LETTER_QUEUE_ARN: "" DEAD_LETTER_QUEUE_NAME: "" DEAD_LETTER_QUEUE_URL: "" QUEUE_ARN: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX QUEUE_NAME: YYYYYYYYYYYYYYYYYYYYYYYYYYYYYY QUEUE_URL: ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ ... kind: Secret type: Opaque ੜ੒͞ΕͨSQSʹؔ͢Δ৘ใ͕อଘ͞Ε͍ͯΔ
  22. Status • Status͸CLIͰ֬ೝ͢Δ͜ͱ͕ग़དྷΔ • Provisining, Ready, Deprovisioning, etc. $ kubectl

    plugin svcat get instances NAME NAMESPACE CLASS PLAN STATUS +----------+-----------+-------+------------+--------------+ sqs-test default s3 custom Ready $ kubectl plugin svcat get binding NAME NAMESPACE INSTANCE STATUS +------------------+-----------+-----------+------------+ sqs-test-binding default sqs-test Provisining
  23. APIαʔό apiVersion: apps/v1beta2 kind: Deployment metadata: name: api template: spec:

    containers: ... env: - name: RDS_HOSTNAME valueFrom: secretKeyRef: name: rds-binding key: ENDPOINT_ADDRESS - name: QUEUE_NAME valueFrom: secretKeyRef: name: sqs-binding key: QUEUE_NAME Secretܦ༝ͰDB৘ใ౳Λ౉͢ ※Πϝʔδ
  24. HelmͰAWSϦιʔε΋ల։ 6* "1* 6* "1* 3%4 4FSWJDF*OTUBODF
 
 4FSWJDF#JOEJOH test-chart

    424 4FSWJDF*OTUBODF
 
 4FSWJDF#JOEJOH 3%4 424 AWS Secret Secret Kubernetes
  25. 1. S3ͷόέοτ͕ফ͑ͳ͍ $ aws s3 ls 2018-10-12 20:44:25 aws-service-broker-s3-AAAAA-AAA-s3bucket-AAAAA 2018-10-15

    17:58:44 aws-service-broker-s3-BBBBB-BBB-s3bucket-BBBBB 2018-10-15 17:59:12 aws-service-broker-s3-CCCCC-CCC-s3bucket-CCCCC 2018-10-15 18:12:53 aws-service-broker-s3-DDDDD-DDD-s3bucket-DDDDD 2018-10-16 17:58:39 aws-service-broker-s3-EEEEE-EEE-s3bucket-EEEEE S3ͷServiceInstanceΛফͯ͠΋όέοτ͕࢒Δ aws-servicebrokerͷ໰୊
  26. 2. SNSͷ࠶ૹϙϦγʔ͕ઃఆͰ͖ͳ͍ $ aws sns list-subscriptions { "Subscriptions": [ {

    "SubscriptionArn": "PendingConfirmation", "Owner": "XXXXXXXXXXXX", "Protocol": "https", "Endpoint": "https://test.example.com", "TopicArn": "arn:aws:sns:ap-northeast-1:XXXXXXXXXX:aws-service-broker-sns" } ] } ͏·͘࠶ૹ͞ΕͣPeindingConfirmationʹͳͬͯ͠·͏ aws-servicebrokerͷ໰୊
  27. 3. ද͕ࣔͣΕͯΔ $ kubectl plugin svcat get classes NAME NAMESPACE

    DESCRIPTION +------------------+-----------+----------------------------------------------+ dh-emr AWS Service Broker - Amazon EMR dh-dynamodb AWS Service Broker - Amazon DynamoDB dh-rdsmariadb AWS Service Broker - Amazon RDS for MariaDB dh-rekognition AWS Service Broker - Amazon Rekognition dh-athena AWS Service Broker - Amazon Athena. dh-sqs AWS Servicebroker - Amazon SQS dh-kms AWS Service Broker - KMS Key dh-rdspostgresql AWS Service Broker - Amazon RDS for ... SQS͚͓͔͍ͩ͠ aws-servicebrokerͷ໰୊
  28. 4. Service Brokerʹ༩͑Δݖݶ͕ڧ͍ { "Sid": "ServiceClassPermissions", "Action": [ "athena:*", "dynamodb:*",

    "kms:*", "elasticache:*", "elasticmapreduce:*", "kinesis:*", "rds:*", "redshift:*", "route53:*", "s3:*", "sns:*", "sns:*", "sqs:*", "ec2:*", "iam:*", "lambda:*" ], "Resource": [ "*" ], "Effect": "Allow" } IAMΠϯελϯεϩʔϧͰ༩͑ΔͱଞͷPod΋ڧ͍ݖݶΛ࣋ͬͯ͠·͏ AWSݻ༗ͷ࿩
  29. $ kubectl get pods -n aws-sb -o yaml apiVersion: v1

    items: - apiVersion: v1 kind: Pod metadata: annotations: iam.amazonaws.com/role: awssb_role kiamΛ࢖͏ Pod୯ҐͰIAMϩʔϧΛׂΓ౰ͯΔ 1PEઐ༻ͷ *".ϩʔϧ https://github.com/uswitch/kiam
  30. 5. annotation͕ઃఆͰ͖ͳ͍ $ cat values.yaml image: awsservicebroker/aws-servicebroker:beta imagePullPolicy: Always authenticate:

    true tls: cert: key: deployClusterServiceBroker: true aws: region: us-east-1 bucket: awsservicebroker key: templates/latest s3region: us-east-1 tablename: awssb accesskeyid: "" secretkey: "" targetaccountid: "" targetrolename: "" vpcid: "" brokerconfig: verbosity: 10 brokerid: awsservicebroker prescribeoverrides: true aws-servicebrokerͷ໰୊ kiam͸annotationͰIAMϩʔϧΛ੍ޚ͍ͯ͠Δ
  31. initContainers: - name: init-s3-binding image: k8s-kubectl command: ['sh', '-c', 'until

    kubectl get secrets s3-binding; do echo waiting for s3-binding; sleep 5; done;'] initContainersΛ࢖͏ Secret͕ੜ੒͞ΕΔ·ͰPodͷੜ੒Λ଴ػ
 ʢղܾ͢Δͱ͸ݴ͑ਏ͍ʣ https://kubernetes.io/docs/concepts/workloads/pods/init-containers/
  32. 7. ServiceCatalogෆ҆ఆ໰୊ kubectl get pods -n catalog NAME READY STATUS

    RESTARTS AGE catalog-catalog-apiserver-86d695b7dc-d78xf 0/2 CrashLoopBackOff 359 11d catalog-catalog-controller-manager-64f69dd964-lzs7c 0/1 CrashLoopBackOff 206 11d test-chart 6* "1* 3%4 424 Service Catalog 4FSWJDF$BUBMPH͕ ࢮ͵ͱԿ΋Ͱ͖ͳ͘ͳΔ
  33. Service CatalogΛ࠶Πϯετʔϧͨ͠৔߹ͷฐ֐ $ kubectl plugin svcat get brokers NAME NAMESPACE

    URL STATUS +-------------------+-----------+----------------------------------------------------+--------+ test-chart 6* "1* 3%4 424 Service Catalog 4FSWJDF$BUBMPHʹొ࿥͍ͯͨ͠ 4FSWJDF#SPLFS΋ফ͑Δ 4FSWJDF#JOEJOH΋ফ͑Δ େࢂࣄ
  34. 8. Deprovisionʹࣦഊͨ͋͠ͱServiceInstance͕࢒Γଓ͚Δ $ kubectl plugin svcat get instances NAME NAMESPACE

    CLASS PLAN STATUS +---------------------+-----------+--------+--------------+----------------------+ test-s3 default dh-s3 custom DeprovisionCallFailed test-chart Service Catalog %FQSPWJTJPOGBJMFE IFMNEFMFUF ґཔ YAMLࣗମ͸΋͏ফ͍͑ͯΔͷͰग़དྷΔ͜ͱ͕ͳ͍ 6* "1* 3%4 424
  35. ServiceBroker·ͱΊ • ServiceBrokerΛ࢖͏͜ͱͰଞͷϦιʔεͱಉ༷ʹΫϥ΢υαʔϏ ε΋؅ཧͰ͖ͯඇৗʹศར • Helm + ServiceBrokerͰ؀ڥߏஙͷࣗಈԽ͕େ෯ʹਪ͠ਐΊΒΕΔ • ҰํͰ·ͩ໰୊΋ଟ͘ൃల్্ͳײ͡

    • ಛʹaws-servicebroker͸·ͩ·ͩʢAzure΍GCP͸҆ఆͯͦ͠͏ʣ • ࣗ෼ͰίʔυΛ௚͢ؾ͕֓͋ΔਓͳΒΦεεϝ • ࠓ͙҆͢ఆͯ͠࢖͍͍ͨਓʹ͸Φεεϝ͠ͳ͍
  36. Open Service Broker API Kubernetes Service Catalog AWS Service Broker

    GCP Service Broker 0QFO4FSWJDF#SPLFS"1* ͱͯ͠ఆٛ͞Ε͍ͯΔ
  37. Open Service Broker for PostgreSQL • PostgreSQLΛϓϩϏδϣχϯά͢ΔService Broker • ษڧͷͨΊʹࣗ࡞ͯ͠Έͨ

    • https://github.com/knqyf263/osbpsql • ڵຯ͕͋Δਓ͕͍Ε͹͍͔ͭͦͷ࿩΋ • ຊ౰͸͜ͷลΓͷ࿩΋͔ͨͬͨ͠
  38. ࢀߟαΠτ • k8s͕ಋೖ͢ΔService Brokerͷ࢓૊Έͱ͸ • http://jaco.udcp.info/entry/k8s-service-broker • CFͷศརػೳΛଞͷ؀ڥͰ΋ɻOpen Service Broker

    • https://www.slideshare.net/jacopen/cfopen-service-broker • Open Service Broker for ͘͞ΒͷΫϥ΢υͰKubernetes + Service Catalogग़དྷΔΑ͏ʹ ͳΓ·ͨ͠ • http://febc-yamamoto.hatenablog.jp/entry/2018/03/10/090229 • Open Service Broker APIΛ࢖ͬͯCloud FoundryͱKubernetesͰService BrokerΛ૬ޓ ӡ༻͢Δ • https://blog.ik.am/entries/497