Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Kubernetes on AWS at Chatwork

Ryo Sakamoto
May 28, 2018
1.4k

Kubernetes on AWS at Chatwork

AWS Summit 2018 スタートアップ特設エリアにおける発表内容で、ChatWorkにおけるKubernetesの運用話です。

Ryo Sakamoto

May 28, 2018
Tweet

Transcript

  1. © ChatWork Kubernetesͷ؀ڥ ▸ ؀ڥ AWS ▸ AWSͳͲطଘͷࢿݯΛར༻͔ͨͬͨͨ͠Ί ▸ ߏஙπʔϧ

    kube-aws ▸ https://github.com/kubernetes-incubator/kube-aws ▸ ϝΠϯϝϯςφ͸mumoshu (chatwork kubernetes ސ໰) ▸ cloudformation(ͱcloud-init)Ͱ·ΔͬͱߏஙͰ͖Δ
  2. © ChatWork KubernetesͰಈ͍͍ͯΔ΋ͷ ▸ backend ▸ ࡢ೥ͷAWS summitͰmessage backendΛϦϓϨΠεͨ͠࿩ ▸

    ͜ͷϓϩδΣΫτҎ߱(webhook, oauthͳͲ)͸͢΂ͯkubernetes ▸ ϊʔυ਺͸m4.2xlarge * 10୆ఔ౓ ▸ CD؀ڥ(concourse) ▸ spot instance ͳnodepoolΛར༻
  3. © ChatWork KubernetesͰར༻͍ͯ͠Δπʔϧ(1) ▸ cluster-autoscaler ▸ podͷauto scaleͰ͸ͳ͘ɺnodeͷauto scaleͯ͘͠ΕΔ ▸

    schedulerΛ؂ࢹͯ͠ɺϦιʔε͕଍Γͳ͍pod͕͍ΔͱASGΛૢ࡞ ▸ nodeͷ୆਺࡟ݮʹେ͖͘ߩݙ ▸ σϓϩΠ࣌ͷpodͷೖସ͑ͳͲͰҰ࣌తʹϊʔυ͕଍Γͳ͘ͳΔͱ͖ ΋͞ΒͬͱରԠͯ͘͠ΕΔ
  4. © ChatWork cluster-autoscalerͷಈ͖(scale out) controller nodepool api-server scheduler cluster- autoscaler

    pod (1) watch (2) 
 “fails to be scheduled due to insufficient” (4)scale out (3) 
 set-desired-capacity ൑ྫ
  5. © ChatWork cluster-autoscalerͷಈ͖(scale in) controller nodepool api-server scheduler cluster- autoscaler

    pod (1) watch
 (apiܦ༝) nodeͷ࢖༻཰௿ (3) 
 set-desired-capacity ൑ྫ a b a b nodeͷ࢖༻཰௿ a b a b (4) scale in (2) evict
  6. © ChatWork KubernetesͰར༻͍ͯ͠Δπʔϧ(2) ▸ kube2iam ▸ podຖʹroleͷ෇༩ ▸ ௨ৗ͸ΠϯελϯεͷϩʔϧΛར༻ ▸

    ෆཁͳpolicy͕෇͘ & ଍Γͳ͍΋ͷ͸APIKEYΛ΋ͨͤΔ͜ͱʹͳΔ ▸ secret͸base64ͳ͚ͩ ▸ एׯෆ҆ఆͰɺkiamʹஔ͖׵͑༧ఆ
  7. © ChatWork kube2iam ▸ annotationʹroleΛهࡌ ▸ role͸workerͷroleΛ৴པ͓ͯ͘͠ ▸ worker͕asuumeͰ͖ΔΑ͏ʹ͓ͯ͘͠ ▸

    pod͕ɺAWSͷAPIΛར༻͠Α͏ͱ͢ΔͱɺmetadataʹΞΫηε͢Δ ▸ metadata΁ͷΞΫηεΛiptablesͰkube2iamʹసૹ ▸ kube2iam͕annotationͷroleͷΫϨσϯγϟϧΛൃߦ
  8. © ChatWork Kubernetesͷ؂ࢹ ▸ datadog only ▸ daemonsetͰ഑ஔ ▸ not

    k8sͳ؀ڥͷ؂ࢹͱ౷Ұ͍ͨ͠ & prometheusͷ؅ཧΛͨ͘͠ͳ͍ ▸ prometheusͷΑ͏ʹΤϯυϙΠϯτΛੜ΍͢ͷͰ͸ͳ͘ɺ֤ϗετͷ statsdʹૹ৴ ▸ version 6Λར༻ ▸ v5ͰϝτϦΫε͕͚͍ܽͯͨ(ϝτϦΫεᷓΕ)͕ɺv6Ͱ͚ܽͳ͘ͳͬͨ
  9. © ChatWork KubernetesͷϩΪϯά ▸ fluentd + stackdriver ▸ fluentdΛdaemonsetͰ഑ஔ ▸

    ֤ίϯςφ͸ϗετͷಛఆͷ৔ॴʹstdoutΛు͖ग़͍ͯ͠Δ ▸ audit-log΋fluentdͰstackdriverʹૹ৴ ▸ S3Ͱ΋Α͔͕ͬͨɺKubernetesΛಋೖͨ࣌͠ʹ͸Athena͸ग़ͨ͹͔Γ ▸ stackdriver + bigquery΋Ұ෦ͷϩάͰಋೖ
  10. © ChatWork Kubernetesͷversion up ▸ kube-awsͰ؅ཧ͍ͯ͠ΔҎ্ɺϚωʔδυͳversion up͸Ͱ͖ͳ͍ ▸ version up΍kubernetes

    ౷߹Λߦͬͨ ▸ version up 1.7 -> 1.8, 1.5 -> 1.8 && 1.8ԽͷλΠϛϯάͰΫϥελ౷߹ ▸ νϟοτϫʔΫͰ͸·ͩingress͸ར༻Ͱ͖͍ͯͳͯ͘ɺELB + NodePort ▸ ͳͷͰɺversion up͸ELBʹ৽چ྆ํΛͿΒԼ͛ͯɺݹ͍ํΛޙୀ ▸ ࠓͷΞϓϦέʔγϣϯͱͯ͠͸onlineͰversion up׬ྃ
  11. © ChatWork version up old k8s pod ൑ྫ app app

    chatwork web NodePort NodePort new k8s
  12. © ChatWork KubernetesͰࠓޙ΍Γ͍ͨ(1) ▸ EKS ▸ kube-awsʹ૊Έࠐ·ΕΔ༧ఆ ▸ ΍ͬͺΓϩʔϦϯάΞοϓσʔτ͍ͨ͠ ▸

    service mesh ▸ envoyͷಋೖ ▸ istio, linkerd ▸ grpc loadbalancer -> envoy, nginx-ingress-controller