Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Kubernetes on AWS at Chatwork

Ryo Sakamoto
May 28, 2018
1.3k

Kubernetes on AWS at Chatwork

AWS Summit 2018 スタートアップ特設エリアにおける発表内容で、ChatWorkにおけるKubernetesの運用話です。

Ryo Sakamoto

May 28, 2018
Tweet

Transcript

  1. νϟοτϫʔΫʹ͓͚Δ
 Kubernetes on AWS / Kubernetes on AWS at ChatWork

    SRE Ryo Sakamoto
  2. © ChatWork ▸ ೔ຊൃϏδωενϟοτ ▸ λεΫ؅ཧ΍ϏσΦ௨࿩͕Մೳ ▸ ಋೖاۀ174,000ࣾҎ্ʢ※2018೥4݄຤೔࣌఺ʣ

  3. © ChatWork ΞδΣϯμ ▸ Kubernetesͷ؀ڥ΍ಈ͍͍ͯΔΞϓϦ ▸ KubernetesͰར༻͍ͯ͠Δπʔϧ ▸ Kubernetesͷ؂ࢹɺϩΪϯά ▸

    Kubernetesͷversion up ▸ ·ͱΊ
  4. © ChatWork Kubernetesͷར༻ ▸ ϝοηʔδॲཧ෦෼ͷϦϓϨΠε(2016೥12݄) ▸ backen appΛkubernetes Ͱಈ͔͢ ▸

    ࡉ͔͍࿩͸ࡢ೥ͷAWS summitͰ…
  5. © ChatWork Kubernetesͷ؀ڥ ▸ ؀ڥ AWS ▸ AWSͳͲطଘͷࢿݯΛར༻͔ͨͬͨͨ͠Ί ▸ ߏஙπʔϧ

    kube-aws ▸ https://github.com/kubernetes-incubator/kube-aws ▸ ϝΠϯϝϯςφ͸mumoshu (chatwork kubernetes ސ໰) ▸ cloudformation(ͱcloud-init)Ͱ·ΔͬͱߏஙͰ͖Δ
  6. © ChatWork KubernetesͰಈ͍͍ͯΔ΋ͷ ▸ backend ▸ ࡢ೥ͷAWS summitͰmessage backendΛϦϓϨΠεͨ͠࿩ ▸

    ͜ͷϓϩδΣΫτҎ߱(webhook, oauthͳͲ)͸͢΂ͯkubernetes ▸ ϊʔυ਺͸m4.2xlarge * 10୆ఔ౓ ▸ CD؀ڥ(concourse) ▸ spot instance ͳnodepoolΛར༻
  7. © ChatWork KubernetesͰར༻͍ͯ͠Δπʔϧ(1) ▸ cluster-autoscaler ▸ podͷauto scaleͰ͸ͳ͘ɺnodeͷauto scaleͯ͘͠ΕΔ ▸

    schedulerΛ؂ࢹͯ͠ɺϦιʔε͕଍Γͳ͍pod͕͍ΔͱASGΛૢ࡞ ▸ nodeͷ୆਺࡟ݮʹେ͖͘ߩݙ ▸ σϓϩΠ࣌ͷpodͷೖସ͑ͳͲͰҰ࣌తʹϊʔυ͕଍Γͳ͘ͳΔͱ͖ ΋͞ΒͬͱରԠͯ͘͠ΕΔ
  8. © ChatWork cluster-autoscalerͷಈ͖(scale out) controller nodepool api-server scheduler cluster- autoscaler

    pod (1) watch (2) 
 “fails to be scheduled due to insufficient” (4)scale out (3) 
 set-desired-capacity ൑ྫ
  9. © ChatWork cluster-autoscalerͷಈ͖(scale in) controller nodepool api-server scheduler cluster- autoscaler

    pod (1) watch
 (apiܦ༝) nodeͷ࢖༻཰௿ (3) 
 set-desired-capacity ൑ྫ a b a b nodeͷ࢖༻཰௿ a b a b (4) scale in (2) evict
  10. © ChatWork KubernetesͰར༻͍ͯ͠Δπʔϧ(2) ▸ kube2iam ▸ podຖʹroleͷ෇༩ ▸ ௨ৗ͸ΠϯελϯεͷϩʔϧΛར༻ ▸

    ෆཁͳpolicy͕෇͘ & ଍Γͳ͍΋ͷ͸APIKEYΛ΋ͨͤΔ͜ͱʹͳΔ ▸ secret͸base64ͳ͚ͩ ▸ एׯෆ҆ఆͰɺkiamʹஔ͖׵͑༧ఆ
  11. © ChatWork kube2iam ▸ annotationʹroleΛهࡌ ▸ role͸workerͷroleΛ৴པ͓ͯ͘͠ ▸ worker͕asuumeͰ͖ΔΑ͏ʹ͓ͯ͘͠ ▸

    pod͕ɺAWSͷAPIΛར༻͠Α͏ͱ͢ΔͱɺmetadataʹΞΫηε͢Δ ▸ metadata΁ͷΞΫηεΛiptablesͰkube2iamʹసૹ ▸ kube2iam͕annotationͷroleͷΫϨσϯγϟϧΛൃߦ
  12. © ChatWork kube2iam app kube2iam 1. credentialͷൃߦ(ec2-metadata) 2. iptablesͰkube2iamͷpodʹϦΫΤετ͕సૹ 3.

    credentialͷൃߦ pod ൑ྫ
  13. © ChatWork Kubernetesͷ؂ࢹ ▸ datadog only ▸ daemonsetͰ഑ஔ ▸ not

    k8sͳ؀ڥͷ؂ࢹͱ౷Ұ͍ͨ͠ & prometheusͷ؅ཧΛͨ͘͠ͳ͍ ▸ prometheusͷΑ͏ʹΤϯυϙΠϯτΛੜ΍͢ͷͰ͸ͳ͘ɺ֤ϗετͷ statsdʹૹ৴ ▸ version 6Λར༻ ▸ v5ͰϝτϦΫε͕͚͍ܽͯͨ(ϝτϦΫεᷓΕ)͕ɺv6Ͱ͚ܽͳ͘ͳͬͨ
  14. © ChatWork datadogͷlive container monitoring

  15. © ChatWork KubernetesͷϩΪϯά ▸ fluentd + stackdriver ▸ fluentdΛdaemonsetͰ഑ஔ ▸

    ֤ίϯςφ͸ϗετͷಛఆͷ৔ॴʹstdoutΛు͖ग़͍ͯ͠Δ ▸ audit-log΋fluentdͰstackdriverʹૹ৴ ▸ S3Ͱ΋Α͔͕ͬͨɺKubernetesΛಋೖͨ࣌͠ʹ͸Athena͸ग़ͨ͹͔Γ ▸ stackdriver + bigquery΋Ұ෦ͷϩάͰಋೖ
  16. © ChatWork Kubernetesͷversion up ▸ kube-awsͰ؅ཧ͍ͯ͠ΔҎ্ɺϚωʔδυͳversion up͸Ͱ͖ͳ͍ ▸ version up΍kubernetes

    ౷߹Λߦͬͨ ▸ version up 1.7 -> 1.8, 1.5 -> 1.8 && 1.8ԽͷλΠϛϯάͰΫϥελ౷߹ ▸ νϟοτϫʔΫͰ͸·ͩingress͸ར༻Ͱ͖͍ͯͳͯ͘ɺELB + NodePort ▸ ͳͷͰɺversion up͸ELBʹ৽چ྆ํΛͿΒԼ͛ͯɺݹ͍ํΛޙୀ ▸ ࠓͷΞϓϦέʔγϣϯͱͯ͠͸onlineͰversion up׬ྃ
  17. © ChatWork version up old k8s pod ൑ྫ app chatwork

    web NodePort
  18. © ChatWork version up old k8s pod ൑ྫ app app

    chatwork web NodePort NodePort new k8s
  19. © ChatWork version up pod ൑ྫ app chatwork web NodePort

    new k8s
  20. © ChatWork KubernetesͰࠓޙ΍Γ͍ͨ(1) ▸ EKS ▸ kube-awsʹ૊Έࠐ·ΕΔ༧ఆ ▸ ΍ͬͺΓϩʔϦϯάΞοϓσʔτ͍ͨ͠ ▸

    service mesh ▸ envoyͷಋೖ ▸ istio, linkerd ▸ grpc loadbalancer -> envoy, nginx-ingress-controller
  21. © ChatWork KubernetesͰࠓޙ΍Γ͍ͨ(2) ▸ prometheusͷಋೖ ▸ hpa͸σϑΥϧτͰ͸cpu͔͠ͳ͍ͷͰ͔ͭʹ͍͘ ▸ cpuͰ͸ͳ͘kafkaͷeventͷ٧·Γ۩߹Ͱscale in/out͍ͨ͠

    ▸ datadogͰ΋apiΛ࢖ͬͯͰ͖Δ͚Ͳɺdatadogͱͷ઀ଓෆ҆ ▸ ϓϥοτϑΥʔϜԽ ▸ openFaaSͳͲ
  22. © ChatWork EKS΁ͷظ଴ ▸ preview൛Λར༻͍͍ͤͯͨͩͨ͞ ▸ workerͷ௥Ճ͕͕͕͕͕….configmapܦ༝Ͱొ࿥͢Δɺͱ͍͏ํ๏ͩͬͨ ▸ ͜Εͩͱkubernetesͷ֎ͰɺΫϥελߏங͕ด͡ͳ͍ ▸

    AWSͷϦιʔε׆༻(IAMɺVPC)ͳͲظ଴ ▸ fargateͰnodeͦͷ΋ͷΛҙࣝ͠ͳ͍ͷ͸͍͍͕ɺlogging΍؂ࢹ͸…
  23. © ChatWork ·ͱΊ ▸ νϟοτϫʔΫͷKubernetes؀ڥʹ͍ͭͯͷ࿩ ▸ ͍Ζ͍Ζ΍Γ͍ͨ͜ͱ͸͋Δ ▸ EKSʹ΋ظ଴ ▸

    controll plane͕Ϛωʔδυ͞ΕΔ҆৺ײ
  24. © ChatWork ΤϯδχΞืूத http://corp.chatwork.com/ja/recruit/ ▸ ओମੑΛ࣋ͪɺࣗΒߦಈͰ͖Δ ▸ ଞऀΛೝΊɺଚॏͰ͖Δ ▸ ৘ใΛूΊɺڞ༗Ͱ͖Δ

    ͱ͍͏ํΛ׻ܴ͠·͢ʂ