Securing Kubernetes Clusters on AWS (Japanese)

by KUOKA Yusuke

Slide 1

Slide 1 text

Securing Kubernetes Clusters on AWS @mumoshu

Slide 2

Slide 2 text

K8S on AWSͰӡ༻͍ͯ͠ΔαʔϏε͕߈ܸ͞Εͯ΋҆৺ ͳঢ়ଶʹ͍ͨ͠

Slide 3

Slide 3 text

߈ܸͬͯ۩ମతʹ͸? ྫ͑͹ɺWebΞϓϦʹରͯ͠ - ո͛͠ͳΫΤϦετϦϯά - ո͛͠ͳHTTP POSTύϥϝʔλ - ruby΍shellεχϖοτΈ͍ͨͳͷ͕ࠞͬͯ͟Δ ϦΫΤετΛ౤͛ΒΕΔ

Slide 4

Slide 4 text

߈ܸͬͯ۩ମతʹ͸? • ΞϓϦʹίʔυΠϯδΣΫγϣϯ੬ऑੑ͕͋ͬͯɺಛఆͷϦ ΫΤετΛૹΔͱ೚ҙͷ໋ྩΛ࣮ߦ͞ΕͨΓɺͻͲ͍ͱ͖͸ όοΫυΞΛ࢓ࠐ·ΕΔ • ʮΞϓϦͷಈ࡞؀ڥ͔ΒΞΫηεͰ͖Δ΋ͷ͢΂ͯʯʹΞΫ ηε͞Εͯ͠·͏ • !

Slide 5

Slide 5 text

߈ܸର৅ͷ͜ͱ

Slide 6

Slide 6 text

͓͞Β͍: K8SͰΞϓϦ͸Ͳ͜Ͱಈ͘? K8SʹσϓϩΠͨ͠ΞϓϦ͸ʮίϯςφʯͰಈ͘ɻ

Slide 7

Slide 7 text

͓͞Β͍: ެ։͞ΕͨΞϓϦ͸Ͳ͏͍͏ܦ࿏ͰΞΫηε͞ΕΔ? -> LB (ELB, ALB, NLB) -> EC2Πϯελϯε (ServiceͷNodePortܦ༝) -> Pod (kube-proxy/iptables͕ServiceͷclusterIPΛpodIPʹม ׵) —> ίϯςφ (containerPortܦ༝) —> ΞϓϦέʔγϣϯϓϩηε ͱ͍͏ܦ࿏ͰΞΫηε͞ΕΔɻ

Slide 8

Slide 8 text

ίϯςφʹೖ͍ͬͯΔ΋ͷ • ίϚϯυ • bash, curl, etc. • ϛυϧ΢ΣΞ • apache, nginx, etc. • ΞϓϦέʔγϣϯ • όΠφϦ(Golang)ɺεΫϦϓτ(ruby, python, etc.)

Slide 9

Slide 9 text

ίϯςφʹೖ͍ͬͯΔ΋ͷ • ࣮ߦ؀ڥ • python, rubyϥϯλΠϜ, ruby gems, JDK, … • ڞ༗ϥΠϒϥϦ • glibc, libssl

Slide 10

Slide 10 text

ίϯςφʹೖ͍ͬͯΔ΋ͷ • Ͳ͜ʹ੬ऑੑ͕͋Δ͔Θ͔Βͳ͍Ͱ͢ΑͶ… • ͔ͩΒͦ͜ɺͰ͖Δ͚ͩK8SϨϕϧͰ๷ޚ͍ͨ͠

Slide 11

Slide 11 text

ྫ͑͹ɺόοΫυΞΛ࢓ࠐ·ΕͨΒԿΛ͞Εͯ͠·͏͔ • ίϯςφʹଘࡏ͢Δ೚ҙͷίϚϯυΛ࣮ߦͰ͖Δ • ྫ: ίϯςφʹcurl͕͋Ε͹೚ҙͷhttp(s)ϦΫΤετΛ஗ΕΔ

Slide 12

Slide 12 text

೚ҙͷίϚϯυΛ࣮ߦͰ͖Δͱ… • KubernetesͳΒͰ͸ͷ΋ͷ • K8S API, Kubelet API(ඇެ։), Podʹcurlͱ͔ͰΞΫηε • ίϯςφΦʔέετϨʔγϣϯγεςϜશൠʹ͋ͯ͸·Δ΋ ͷ • ଞͷίϯςφʹΞΫηε • localhost΍ϩʔΧϧϘϦϡʔϜܦ༝

Slide 13

Slide 13 text

ରࡦ • όοΫυΞ࢓ࠐΊͳ͍Α͏ʹ͢Δ • όοΫυΞ࢓ࠐ·Εͨͱ͖ͷӨڹΛ࠷খݶʹ͢Δ

Slide 14

Slide 14 text

۩ମతʹͲ͏ରࡦ͢Δ͔ • kube2iam • KubernetesͷNetwork Policies • RBAC • TLS Bootstrapping • ϊʔυϓʔϧɺΠϯελϯεϓʔϧ • coreos/clair

Slide 15

Slide 15 text

όοΫυΞ࢓ࠐΊͳ͍Α͏ʹ͢Δ (1) ੬ऑͳΞϓϦΛ࡞Βͳ͍ ! • ಛʹޠΕΔ͜ͱͳ͍ɾɾɾ • ϑϨʔϜϫʔΫ࢖͍·͠ΐ • ݴޠॲཧܥͷ੬ऑੑ͕ݟ͔ͭͬͨΒΞοϓσʔτ͠·͠ΐ͏ ʢGolang, RubyͳͲͷϨϕϧͰʣ • ϢʔβೖྗͷαχςΟνΣοΫ • ੬ऑͳίϚϯυ΍ϥΠϒϥϦΛؚΊͳ͍

Slide 16

Slide 16 text

όοΫυΞ࢓ࠐΊͳ͍Α͏ʹ͢Δ (2) CoreOSͷClairͰίϯςφΠϝʔδ੬ऑੑνΣοΫ ྫ: ੬ऑͳΠϝʔδ͕σϓϩΠ͞Εͳ͍Α͏ʹɺCIͰDocker Build & Pushͷޙʹ clair-scanner --clair https://myclaier.example.com --ip $YOUR_LOCAL_IP $IMAGE ͦͷ͋ͱK8SʹσϓϩΠ͢ΔΑ͏ʹ͢Δɻ ಉछͷπʔϧʹfuture-architect/vuls΋͋Γ·͢Ͷɻ

Slide 17

Slide 17 text

࢓ࠐ·Εͨͱ͖ͷӨڹΛ࠷খݶʹ͢Δ kube2iam kube2iam: EC2ϝλσʔλαʔϏε(http://169.254.169.254)Λ ஔ͖׵͑ΔϦόʔεϓϩΩγ • EC2Πϯελϯεʹ͸ͦͦ͜͜ڧྗͳIAMϙϦγʔ͕͍ͭͯ Δ • ΞϓϦ͔ΒͦΕΛར༻Ͱ͖ͳ͍Α͏ʹ͢Δ • ௨ৗ͸EC2ϝλσʔλαʔϏεʹcurl͢ΔͱɺΠϯελϯε

Slide 18

Slide 18 text

Network Policies Podؒͷ௨৴ΛϧʔϧϕʔεͰڐՄɾېࢭ͢ΔͨΊͷ࢓૊Έ • PodΛෆཁͳΠϯλʔωοτ্ͷαʔϏεͱ௨৴ͤͨ͘͞ͳ ͍ • Pod΁ೖΔ: Ingress • Pod͔Βग़Δ: Egress (K8S 1.8͔Β࢖͑Δ) • EC2ϝλσʔλαʔϏε(169.254.169.254)͸σϑΥϧτϒ ϩοΫͰ͍͍ͷͰ͸

Slide 19

Slide 19 text

RBAC • K8S APIΛPod΍ϢʔβຖʹΞΫηείϯτϩʔϧ͢Δػೳ • K8S APIΛݺ΂ͳ͍PodʹόοΫυΞ࢓ࠐ·Εͯ΋҆৺ • K8S 1.8͔ΒGAʹͳΓ·ͨ͠Ͷ

Slide 20

Slide 20 text

TLS Bootstrapping • K8S API Server͕Kubelet΁ͷ௨৴ͷೝূɾ҉߸Խʹ࢖͏TLS ൿີ伴Λࣗಈੜ੒͢Δ • ྫ: kube-aws: 伴͕શKubelet=WorkerͰڞ௨ɻ1 Workerͷ伴 ͕࿙ΕΔͱଞͷWorkerʹ΋Өڹ͕… • TLS BootstrappingΛ࢖͏ͱɺWorkerผͷ伴ʹͳΔɻ΋Εͯ ΋ͦͷWorker͚ͩʹӨڹ

Slide 21

Slide 21 text

RotateKubeletClientCertiﬁcate • K8S 1.8Ͱalphaʹͳͬͨ • TLS BootstrappingͰɺ伴Λఆظతʹ(਺ϲ݄ɻͳ͕͍)ʹೖΕ ସ͑Δػೳ • 伴͕࿙Εͯ΋ظݶ੾ΕʢೖΕସ͑ʣޙͰ͋Ε͹Өڹͳ͠

Slide 22

Slide 22 text

ϊʔυϓʔϧɺΠϯελϯεϓʔϧ • GKE, kops, kube-awsͳͲʹ͋ΔɺNodeͷάϧʔϓʢlabel΍ αʔόεϖοΫɺGPUͷ༗ແͳͲ͕ҟͳΔʣΛ࢖͍෼͚Δػ ೳ • ྫ: ઀ଓՄೳͳݸਓ৘ใΛؚΉDBΠϯελϯεผʹϓʔϧΛ Θ͚Δ • ΍Γ͗͢ΔͱϦιʔεޮ཰͕Լ͕Δʀ

Slide 23

Slide 23 text

͓·͚: K8S APIͷೝূ໰୊ • ΤϯδχΞ͕K8S APIʹΞΫηεͰ͖ΔτʔΫϯΛπΠʔτ͠ ͯ͠·ͬͨ • ΤϯδχΞ͕K8S APIʹΞΫηεͰ͖Δൿີ伴Λ͓࿙Βͯ͠͠ ͠·ͬͨ • !

Slide 24

Slide 24 text

ରࡦ: ωοτϫʔΫతʹ K8S API Endpoint (AWSͷ৔߹ଟ͘͸ELB)͸ • ࣾ಺ωοτϫʔΫ(IP੍ݶɺͰ͖Ε͹VPN) • VPC಺ͷϓϥΠϕʔτωοτϫʔΫ(ྫ: VPC಺ͷCIγεςϜ͔ ΒK8S APIΛར༻͢Δ) ͔ΒͷΈΞΫηεͰ͖ΔΑ͏ʹ͠·͠ΐ͏ɻ

Slide 25

Slide 25 text

ରࡦ: K8SΫϥελʹGitHubϩάΠϯ Using guard, you can log into your Kubernetes cluster using your Github or Google authentication token https://github.com/appscode/guard • ͨͩ͠ɺGitHubͷOAuthτʔΫϯΛK8S APIτʔΫϯ͕ΘΓ ʹ࢖͏ • τʔΫϯͷ༗ޮظݶ͕௕͍ͱͦΕ͸ͦΕͰةͳ͍͔΋ • GoogleϩάΠϯ΋Ͱ͖Δ

Slide 26

Slide 26 text

ରࡦ: K8SΫϥελʹAWS IAMͰϩάΠϯ kubernetes-aws-authenticator A tool for using AWS IAM credentials to authenticate to a Kubernetes cluster. https://github.com/heptiolabs/kubernetes-aws-authenticator • ͨͩ͠ɺAWSΫϨσϯγϟϧͷ༗ޮظݶ͕̍࣌ؒͳͷͰɺ࿙ ΕͨΒ̍࣌ؒ͸࢖ΘΕͯ͠·͏Մೳੑ • ωοτϫʔΫతͳରࡦ΋߹Θͤͯ

Slide 27

Slide 27 text

TL;DR; K8S on AWSΛηΩϡΞʹ͍ͨ͠ͳΒɺ࠷௿ݶ • kube2iam • Network Policies • coreos/clair • RBAC Ͱ͖Ε͹ • ϊʔυϓʔϧɺTLS Bootstrapping