Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Securing Kubernetes Clusters on AWS (Japanese)

Securing Kubernetes Clusters on AWS (Japanese)

Kubernetes Meetup Tokyo #7
https://k8sjp.connpass.com/event/67092

8e045bf747ca7a90b1d955dc30217271?s=128

KUOKA Yusuke

October 13, 2017
Tweet

Transcript

  1. Securing Kubernetes Clusters on AWS @mumoshu

  2. K8S on AWSͰӡ༻͍ͯ͠ΔαʔϏε͕߈ܸ͞Εͯ΋҆৺ ͳঢ়ଶʹ͍ͨ͠

  3. ߈ܸͬͯ۩ମతʹ͸? ྫ͑͹ɺWebΞϓϦʹରͯ͠ - ո͛͠ͳΫΤϦετϦϯά - ո͛͠ͳHTTP POSTύϥϝʔλ - ruby΍shellεχϖοτΈ͍ͨͳͷ͕ࠞͬͯ͟Δ ϦΫΤετΛ౤͛ΒΕΔ

  4. ߈ܸͬͯ۩ମతʹ͸? • ΞϓϦʹίʔυΠϯδΣΫγϣϯ੬ऑੑ͕͋ͬͯɺಛఆͷϦ ΫΤετΛૹΔͱ೚ҙͷ໋ྩΛ࣮ߦ͞ΕͨΓɺͻͲ͍ͱ͖͸ όοΫυΞΛ࢓ࠐ·ΕΔ • ʮΞϓϦͷಈ࡞؀ڥ͔ΒΞΫηεͰ͖Δ΋ͷ͢΂ͯʯʹΞΫ ηε͞Εͯ͠·͏ • !

  5. ߈ܸର৅ͷ͜ͱ

  6. ͓͞Β͍: K8SͰΞϓϦ͸Ͳ͜Ͱಈ͘? K8SʹσϓϩΠͨ͠ΞϓϦ͸ʮίϯςφʯͰಈ͘ɻ

  7. ͓͞Β͍: ެ։͞ΕͨΞϓϦ͸Ͳ͏͍͏ܦ࿏ͰΞΫηε͞ΕΔ? -> LB (ELB, ALB, NLB) -> EC2Πϯελϯε (ServiceͷNodePortܦ༝)

    -> Pod (kube-proxy/iptables͕ServiceͷclusterIPΛpodIPʹม ׵) —> ίϯςφ (containerPortܦ༝) —> ΞϓϦέʔγϣϯϓϩηε ͱ͍͏ܦ࿏ͰΞΫηε͞ΕΔɻ
  8. ίϯςφʹೖ͍ͬͯΔ΋ͷ • ίϚϯυ • bash, curl, etc. • ϛυϧ΢ΣΞ •

    apache, nginx, etc. • ΞϓϦέʔγϣϯ • όΠφϦ(Golang)ɺεΫϦϓτ(ruby, python, etc.)
  9. ίϯςφʹೖ͍ͬͯΔ΋ͷ • ࣮ߦ؀ڥ • python, rubyϥϯλΠϜ, ruby gems, JDK, …

    • ڞ༗ϥΠϒϥϦ • glibc, libssl
  10. ίϯςφʹೖ͍ͬͯΔ΋ͷ • Ͳ͜ʹ੬ऑੑ͕͋Δ͔Θ͔Βͳ͍Ͱ͢ΑͶ… • ͔ͩΒͦ͜ɺͰ͖Δ͚ͩK8SϨϕϧͰ๷ޚ͍ͨ͠

  11. ྫ͑͹ɺόοΫυΞΛ࢓ࠐ·ΕͨΒԿΛ͞Εͯ͠·͏͔ • ίϯςφʹଘࡏ͢Δ೚ҙͷίϚϯυΛ࣮ߦͰ͖Δ • ྫ: ίϯςφʹcurl͕͋Ε͹೚ҙͷhttp(s)ϦΫΤετΛ஗ΕΔ

  12. ೚ҙͷίϚϯυΛ࣮ߦͰ͖Δͱ… • KubernetesͳΒͰ͸ͷ΋ͷ • K8S API, Kubelet API(ඇެ։), Podʹcurlͱ͔ͰΞΫηε •

    ίϯςφΦʔέετϨʔγϣϯγεςϜશൠʹ͋ͯ͸·Δ΋ ͷ • ଞͷίϯςφʹΞΫηε • localhost΍ϩʔΧϧϘϦϡʔϜܦ༝
  13. ରࡦ • όοΫυΞ࢓ࠐΊͳ͍Α͏ʹ͢Δ • όοΫυΞ࢓ࠐ·Εͨͱ͖ͷӨڹΛ࠷খݶʹ͢Δ

  14. ۩ମతʹͲ͏ରࡦ͢Δ͔ • kube2iam • KubernetesͷNetwork Policies • RBAC • TLS

    Bootstrapping • ϊʔυϓʔϧɺΠϯελϯεϓʔϧ • coreos/clair
  15. όοΫυΞ࢓ࠐΊͳ͍Α͏ʹ͢Δ (1) ੬ऑͳΞϓϦΛ࡞Βͳ͍ ! • ಛʹޠΕΔ͜ͱͳ͍ɾɾɾ • ϑϨʔϜϫʔΫ࢖͍·͠ΐ • ݴޠॲཧܥͷ੬ऑੑ͕ݟ͔ͭͬͨΒΞοϓσʔτ͠·͠ΐ͏

    ʢGolang, RubyͳͲͷϨϕϧͰʣ • ϢʔβೖྗͷαχςΟνΣοΫ • ੬ऑͳίϚϯυ΍ϥΠϒϥϦΛؚΊͳ͍
  16. όοΫυΞ࢓ࠐΊͳ͍Α͏ʹ͢Δ (2) CoreOSͷClairͰίϯςφΠϝʔδ੬ऑੑνΣοΫ ྫ: ੬ऑͳΠϝʔδ͕σϓϩΠ͞Εͳ͍Α͏ʹɺCIͰDocker Build & Pushͷޙʹ clair-scanner --clair

    https://myclaier.example.com --ip $YOUR_LOCAL_IP $IMAGE ͦͷ͋ͱK8SʹσϓϩΠ͢ΔΑ͏ʹ͢Δɻ ಉछͷπʔϧʹfuture-architect/vuls΋͋Γ·͢Ͷɻ
  17. ࢓ࠐ·Εͨͱ͖ͷӨڹΛ࠷খݶʹ͢Δ kube2iam kube2iam: EC2ϝλσʔλαʔϏε(http://169.254.169.254)Λ ஔ͖׵͑ΔϦόʔεϓϩΩγ • EC2Πϯελϯεʹ͸ͦͦ͜͜ڧྗͳIAMϙϦγʔ͕͍ͭͯ Δ • ΞϓϦ͔ΒͦΕΛར༻Ͱ͖ͳ͍Α͏ʹ͢Δ

    • ௨ৗ͸EC2ϝλσʔλαʔϏεʹcurl͢ΔͱɺΠϯελϯε
  18. Network Policies Podؒͷ௨৴ΛϧʔϧϕʔεͰڐՄɾېࢭ͢ΔͨΊͷ࢓૊Έ • PodΛෆཁͳΠϯλʔωοτ্ͷαʔϏεͱ௨৴ͤͨ͘͞ͳ ͍ • Pod΁ೖΔ: Ingress •

    Pod͔Βग़Δ: Egress (K8S 1.8͔Β࢖͑Δ) • EC2ϝλσʔλαʔϏε(169.254.169.254)͸σϑΥϧτϒ ϩοΫͰ͍͍ͷͰ͸
  19. RBAC • K8S APIΛPod΍ϢʔβຖʹΞΫηείϯτϩʔϧ͢Δػೳ • K8S APIΛݺ΂ͳ͍PodʹόοΫυΞ࢓ࠐ·Εͯ΋҆৺ • K8S 1.8͔ΒGAʹͳΓ·ͨ͠Ͷ

  20. TLS Bootstrapping • K8S API Server͕Kubelet΁ͷ௨৴ͷೝূɾ҉߸Խʹ࢖͏TLS ൿີ伴Λࣗಈੜ੒͢Δ • ྫ: kube-aws:

    伴͕શKubelet=WorkerͰڞ௨ɻ1 Workerͷ伴 ͕࿙ΕΔͱଞͷWorkerʹ΋Өڹ͕… • TLS BootstrappingΛ࢖͏ͱɺWorkerผͷ伴ʹͳΔɻ΋Εͯ ΋ͦͷWorker͚ͩʹӨڹ
  21. RotateKubeletClientCertificate • K8S 1.8Ͱalphaʹͳͬͨ • TLS BootstrappingͰɺ伴Λఆظతʹ(਺ϲ݄ɻͳ͕͍)ʹೖΕ ସ͑Δػೳ • 伴͕࿙Εͯ΋ظݶ੾ΕʢೖΕସ͑ʣޙͰ͋Ε͹Өڹͳ͠

  22. ϊʔυϓʔϧɺΠϯελϯεϓʔϧ • GKE, kops, kube-awsͳͲʹ͋ΔɺNodeͷάϧʔϓʢlabel΍ αʔόεϖοΫɺGPUͷ༗ແͳͲ͕ҟͳΔʣΛ࢖͍෼͚Δػ ೳ • ྫ: ઀ଓՄೳͳݸਓ৘ใΛؚΉDBΠϯελϯεผʹϓʔϧΛ

    Θ͚Δ • ΍Γ͗͢ΔͱϦιʔεޮ཰͕Լ͕Δʀ
  23. ͓·͚: K8S APIͷೝূ໰୊ • ΤϯδχΞ͕K8S APIʹΞΫηεͰ͖ΔτʔΫϯΛπΠʔτ͠ ͯ͠·ͬͨ • ΤϯδχΞ͕K8S APIʹΞΫηεͰ͖Δൿີ伴Λ͓࿙Βͯ͠͠

    ͠·ͬͨ • !
  24. ରࡦ: ωοτϫʔΫతʹ K8S API Endpoint (AWSͷ৔߹ଟ͘͸ELB)͸ • ࣾ಺ωοτϫʔΫ(IP੍ݶɺͰ͖Ε͹VPN) • VPC಺ͷϓϥΠϕʔτωοτϫʔΫ(ྫ:

    VPC಺ͷCIγεςϜ͔ ΒK8S APIΛར༻͢Δ) ͔ΒͷΈΞΫηεͰ͖ΔΑ͏ʹ͠·͠ΐ͏ɻ
  25. ରࡦ: K8SΫϥελʹGitHubϩάΠϯ Using guard, you can log into your Kubernetes

    cluster using your Github or Google authentication token https://github.com/appscode/guard • ͨͩ͠ɺGitHubͷOAuthτʔΫϯΛK8S APIτʔΫϯ͕ΘΓ ʹ࢖͏ • τʔΫϯͷ༗ޮظݶ͕௕͍ͱͦΕ͸ͦΕͰةͳ͍͔΋ • GoogleϩάΠϯ΋Ͱ͖Δ
  26. ରࡦ: K8SΫϥελʹAWS IAMͰϩάΠϯ kubernetes-aws-authenticator A tool for using AWS IAM

    credentials to authenticate to a Kubernetes cluster. https://github.com/heptiolabs/kubernetes-aws-authenticator • ͨͩ͠ɺAWSΫϨσϯγϟϧͷ༗ޮظݶ͕̍࣌ؒͳͷͰɺ࿙ ΕͨΒ̍࣌ؒ͸࢖ΘΕͯ͠·͏Մೳੑ • ωοτϫʔΫతͳରࡦ΋߹Θͤͯ
  27. TL;DR; K8S on AWSΛηΩϡΞʹ͍ͨ͠ͳΒɺ࠷௿ݶ • kube2iam • Network Policies •

    coreos/clair • RBAC Ͱ͖Ε͹ • ϊʔυϓʔϧɺTLS Bootstrapping