Share
Embed
Transcript
29 slides
-
ຊͷൃද༰ • Πϯϑϥ৽ϓϩδΣΫτ Neco ͷհ • ͳͥCNIϓϥάΠϯΛࣗ࡞ͨ͠ͷ͔ʁ • ࣗ࡞CNIϓϥάΠϯ coil
ͷհ • CNIϓϥάΠϯ։ൃͰಘΒΕͨݟ • ·ͱΊ 2 -
Πϯϑϥ৽ϓϩδΣΫτNecoͱ • Kubernetes Λಋೖͯ͠ ͷΠϯϑϥ Λ৽͢ΔϓϩδΣΫτ • ͱ •
kintoneGaroon, OfficeͳͲͷαʔϏεΛSaaSͱͯ͠ఏڙ • ಋೖاۀ2.5ສࣾɺϢʔβʔ100ສਓ • 2011ϦϦʔε • VMϕʔεͷΞʔΩςΫνϟ • σʔληϯλʔΛआΓͯɺ1,000Ҏ্ͷαʔόʔΛࣗલͰӡ༻ 4 -
Πϯϑϥ৽ϓϩδΣΫτNecoͱ • త • ϝϯςφϯείετͷେ෯ݮ (NoOpsΛࢦ͢) • εέʔϥϏϦςΟͷ্ • αʔόʔͷूੑ্
• ΞϓϦέʔγϣϯ։ൃνʔϜ͕σϓϩΠɾӡ༻ʹࢀՃ • ΄ͱΜͲͷՌΛOSSͱͯ͠ެ։ 5 -
NecoͷΞʔΩςΫνϟ Kubernetes CoreOS Node LB Prometheus squid CoreOS Node CoreOS
Node CoreOS Node CoreOS Node Boot Server CKE sabakan CoreDNS 1,000نͷαʔόʔ ઍʙͷΞϓϦέʔγϣϯίϯςφ 5ͷϒʔταʔόʔ Kubernetesͷ σϓϩΠ ཧ ܧଓతσϦόϦʔ app app MySQL Elastic search 6 CoreOS Node neco updater Ubuntu Argo CD Rook -
NecoΛࢧ͑ΔιϑτΣΞͨͪ • sabakan • αʔόʔػࡐͷϥΠϑαΠΫϧཧͱϓϩϏδϣχϯάͷࣗಈԽΛ͓͜ͳ͏ɻ • BIOSͷઃఆɺOSͷωοτϒʔτɺσΟεΫ҉߸Խ֤छγεςϜιϑτΣΞͷ ηοτΞοϓΛࣗಈతʹ͓͜ͳ͏ɻ • CKE
(Cybozu Container Engine) • KubernetesΫϥελͷࣗಈߏஙɾӡ༻Λ͓͜ͳ͏πʔϧɻ • sabakan͕ߏஙͨ͠αʔόʔʹKubernetesΛࣗಈతʹσϓϩΠ͢Δɻ • ΤϥʔΛࣗಈम෮ͨ͠ΓɺނোػࡐΛΫϥελ͔ΒऔΓ֎͢ͳͲͷӡ༻ΛࣗಈԽɻ • neco-updater • ΠϯϑϥͷܧଓతσϦόϦʔπʔϧɻ • GitHubͷϦϦʔεใΛνΣοΫ͠ɺCKEsabakanΛ͡Ίͱ͢Δ֤छιϑτ ΣΞͷσϓϩΠɺCoreOSΠϝʔδͷߋ৽ͳͲΛࣗಈతʹ࣮ࢪ͢Δɻ 7 -
NecoͷωοτϫʔΫߏ Rack1 Rack2 Rack3 • CLOSΞʔΩςΫνϟ • ϑϥοτͳL3ωοτϫʔΫ • ֤ϊʔυʹಉҰϗοϓͰ౸ୡՄೳ
• East-WestτϥϑΟοΫͷ૿େʹର͠ ͯεέʔϧՄೳ • BGPʹΑΔϧʔςΟϯά • AS per Rack • ECMPʹΑΔܦ࿏Խ • BFDʹΑΔߴͳܦ࿏ऩଋ • ৄ͘͠ϒϩάͰ • https://blog.cybozu.io/entry/2018/1 1/01/113000 10 -
CNIϓϥάΠϯͷબఆ • σʔληϯλʔωοτϫʔΫͱ߹ΘͤͯɺKubernetesͷωοτϫʔΫʹBGP Λ࠾༻ͯ͠ޮతʹϧʔςΟϯάΛ͓͜ͳ͍͍ͨɻ • Calico • ։ൃ͕׆ൃͰػೳ๛ɻ࠾༻࣮ଟ͍ɻ • BGP
SpeakerΛแ͍ͯ͠Δ͜ͱɺେنΫϥελͰܦ࿏͕૿େ͢Δ ͜ͱͳͲ͕ݒ೦ɻ • Romana • ػೳతʹཁ݅Λຬ͍ͨͯ͠Δɻ • etcd v3ະରԠɻ։ൃ͕׆ൃͰͳ͘࠷৽ͷKubernetesʹैͰ͖͍ͯͳ͍ɻ 11 -
ϓϥάΠϯͷΈ߹Θͤ • ίϯςφωοτϫʔΫͰղܾ͖͢՝͕ଟ͋Δɻ • IPΞυϨεཧ (IPAM) • ωοτϫʔΫଓੑ • ωοτϫʔΫϙϦγʔ
• ଳҬ੍ݶ • ͯ͢ͷΛ1ͭͷϓϥάΠϯͰղܾ͢Δඞཁͳ͍ɻ • ෳͷϓϥάΠϯΛΈ߹ΘͤΔྫ: Canal • IPAMCNIϓϥάΠϯͷඪ४ػೳΛར༻ • ωοτϫʔΫଓੑ flannel Λར༻ • ωοτϫʔΫϙϦγʔ Calico Λར༻ 12 -
Ͳ͜Λࣗ࡞͢Δඞཁ͕͋Δͷ͔ʁ • ωοτϫʔΫϙϦγʔ • ࣮͕େมɻಠࣗੑΛग़͢ඞཁͳ͍ɻ CalicoCiliumͳͲΛར༻ • IPΞυϨεཧ (IPAM)
• ϧʔςΟϯάςʔϒϧͷ૿େΛආ͚ΔͨΊͷΈ͕ඞཁɻ • PodʹάϩʔόϧIPΞυϨεΛׂΓͯΔػೳཉ͍͠ɻ ࣗ࡞ • ωοτϫʔΫଓ • ϊʔυͷ֎ͱͷܦ࿏ަɺDCͷϧʔςΟϯάιϑτΣΞʹͤΔ ϊʔυͷϧʔςΟϯά෦Λࣗ࡞ 13 -
coilͱ • αΠϘζ͕։ൃ͢ΔCNIϓϥάΠϯ • OSSͱͯ͠ެ։ • https://github.com/cybozu-go/coil • ಛ •
CNI Spec v0.3.1 (Kubernetes v1.13)ʹରԠ • GoݴޠͰ࣮ɺόοΫΤϯυʹ etcd Λ࠾༻ • Linux only, Kubernetes only, IPv4 only • IPAMͱϊʔυͷϧʔςΟϯάػೳͷΈΛఏڙ • ϧʔςΟϯάιϑτΣΞΛแ͠ͳ͍ • େنΫϥελͰͷར༻Λߟྀ 15 -
େنΫϥελͰར༻͢ΔͨΊʹ • ΦʔόϨΠωοτϫʔΫར༻͠ͳ͍ • VXLANͳͲͷΦʔόʔϨΠωοτϫʔΫεϧʔϓοτ͕མͪΔɻ ୯७ͳϧʔςΟϯάʹΑΓωοτϫʔΫͷଓੑΛཱ֬ • Linux BridgeΛར༻͠ͳ͍
• Linux BridgeΛ͏ͱCPU༻͕૿͑ͯ͠·͏ɻ ϧʔςΟϯάςʔϒϧʹvethͷϧʔτΛՃ͢Δɻ • etcd API v3ͷΈʹରԠ • etcd API v2ɺଟͷΫϥΠΞϯτ͔Βଓ͞Εͨ࣌ʹύϑΥʔϚϯε͕མͪΔɻ • ܦ࿏ͷ૿େΛ͙ • Pod͝ͱʹܦ࿏Λࠂ͢Δͱܦ࿏͕૿େͯ͠͠·͏ɻ ΞυϨεϒϩοΫͱ͍͏ΈʹΑΓɺαϒωοτ͝ͱͷܦ࿏Λࠂɻ 16 -
Node coilͷߏཁૉ • etcd • ׂΓͯͨΞυϨεใΛཧ • coil-controller • k8s্ͷDeployment
• ΘΕͳ͘ͳͬͨΞυϨεϒϩοΫͷղ์ • coil: • CNIϓϥάΠϯຊମ • PodͷvethͷՃɾআɺIPΞυϨεׂΓͯɺ ϧʔςΟϯάઃఆͳͲΛ࣮ࢪ • coil-node: k8s্ͷDaemonSet • coild: • ϊʔυ͝ͱͷΞυϨεཧϧʔςΟϯάͷઃఆΛ࣮ࢪ • coil-installer: • coilઃఆϑΝΠϧͷΠϯετʔϧΛ࣮ࢪ 17 <DeamonSet> coil-node <CNI> coil etcd House Keeping <Deployment> coil-controller coild coil installer conf ηοτΞοϓ -
ΞυϨεϒϩοΫ (Inspired by Romana) • coilͰΞυϨεϒϩοΫͱ͍͏ΈΛ औΓೖΕɺαϒωοτ୯Ґ(ྫ: /28)Ͱܦ࿏ Λࠂ͢Δ͜ͱͰϧʔςΟϯάςʔϒϧͷ ංେԽΛճආ͍ͯ͠Δɻ
18 Node2 Pod 10.0.1.16/32 Node1 Pod 10.0.1.0/32 Pod 10.0.1.17/32 Pod 10.0.1.1/32 BGP Router 10.0.1.0/28 -> Node1 10.0.1.16/28 -> Node2 αϒωοτ୯ҐͰ ܦ࿏Λࠂ PodͷΞυϨεϨϯδ: 10.0.1.0/24 ΞυϨεϒϩοΫ: 10.0.1.0/28 ΞυϨεϒϩοΫ: 10.0.1.16/28 ΞυϨεϒϩοΫ: 10.0.1.32/28 ΞυϨεϒϩοΫ: 10.0.1.48/28 ΞυϨεϒϩοΫ: 10.0.1.64/28 ΞυϨεϒϩοΫ 10.0.1.0/28 ΞυϨεϒϩοΫ 10.0.1.16/28 ɿ ϊʔυ͝ͱʹ ΞυϨεϒϩοΫ ΛׂΓͯ ׂΓͯՄೳͳ ΞυϨεΛ ϒϩοΫͱ͍͏ ୯Ґʹׂ PodͷΞυϨε ΞυϨεϒ ϩοΫ͔Βׂ Γͯ -
coilͷॲཧͷྲྀΕ 1. kubelet ͕ࢦࣔΛड͚ͯ Pod Λ࡞ 2. CNIϓϥάΠϯ coil Λ࣮ߦ
3. coil ͔Β coild ʹIPΞυϨεΛཁٻ 4. etcd ͔ΒΞυϨεϒϩοΫΛׂΓͯΔ 5. ܦ࿏ΛϧʔςΟϯάςʔϒϧʹॻ͖ग़͢ 6. ϧʔςΟάςʔϒϧΛಡΈࠐΈܦ࿏Λࠂ 7. coild ͔Β coil ʹIPΞυϨεΛฦ͢ 8. Pod ͷ netns ʹ veth ͷϖΞΛ࡞͠ɺIPΞυ ϨεͷׂΓͯͱϧʔτͷઃఆΛ͓͜ͳ͏ Node <DeamonSet> coild <CNI> coil kubelet Pod etcd routing table BGP Speaker eth0 veth BGP Router ᶃ ᶃ ᶄ ᶅ ᶆ ᶇ ᶈ ᶈ ᶊ ᶉ 19 -
boot-taint • CNIϓϥάΠϯ͕ηοτΞοϓ͞Ε͍ͯͳ͍ϊʔυ ʹPod͕εέδϡʔϧ͞ΕΔͱࠔΔɻ • kubeletͷىಈΦϓγϣϯʹ —register-with-taints Λ ࢦఆͯ͠ɺىಈޙͷϊʔυʹPodΛεέδϡʔ ϧͰ͖ͳ͍Α͏ʹ͓ͯ͘͠ɻ
• coil ͷηοτΞοϓ͕ྃͨ͠Β taints Λআ͠ɺ PodΛεέδϡʔϦϯάՄೳʹ͢Δɻ 20 taints/tolerations Kubernetesͷػೳɻ ϊʔυʹtaintsΛ༩͢ Δ͜ͱͰɺPodͷεέ δϡʔϦϯά࣮ߦΛ ېࢭ͢Δ͜ͱ͕Ͱ͖Δɻ ಛఆͷtolerations͕ ༩͞ΕͨPodͷΈεέ δϡʔϦϯάՄೳͱͳ Δɻ -
KubernetesͷใΛಘΔʹʁ • CNIϓϥάΠϯKubernetesઐ༻ͷͷͰͳ͍ͨΊɺPodͷใΛऔಘ͢Δ ༷ఆΊΒΕ͍ͯͳ͍ɻ • https://github.com/containernetworking/cni/issues/606 • ݱঢ়Kubernetes͔ΒCNIϓϥάΠϯΛݺͼग़͢ࡍʹɺCNI_ARGS Ͱ K8S_POD_NAME
K8S_POD_NAMESPACE ͳͲͷใΛ͍ͯ͠Δɻ • ໌֬ʹ༷ͱͯ͠ఆ·͍ͬͯΔΘ͚Ͱͳ͍ͷͰ࣮ΛಡΈղ͘ඞཁ͕͋ͬͨɻ (dockershimcontainerdͳͲɺίϯςφϥϯλΠϜ͝ͱʹͦΕͧΕ࣮ͯ͠Δ) 23 -
Kubernetes 1.13ͰPodؒ௨৴͕Ͱ͖ͳ͍ • KubernetesΛv1.13ʹΞοϓάϨʔυͨ͠ΒɺPodؒͷ௨৴͕Ͱ͖ͳ͘ͳͬͨɻ • kube-proxyͷIPVSϞʔυͷ࣮ͰɺLinuxͷΧʔωϧύϥϝʔλ(sysctl)ͷઃఆ͕ มߋ͞Ε͍ͯͨ (ϦϦʔεϊʔτʹهड़ͳ͠) • net.ipv4.conf.all.arp_ignore:
0 -> 1 • net.ipv4.conf.all.arp_announce: 0 -> 2 • ͍͔ͭ͘ͷCNIϓϥάΠϯʹӨڹ͕ग़͍ͯΔ • https://github.com/kubernetes/kubernetes/issues/71555 25 -
ͱղܾํ๏ Node Pod ens3 eth0 veth 192.168.1.11 IP: 10.0.1.1/32 GW:
192.168.1.11 ARP: 192.168.1.11 Node Pod ens3 eth0 veth 192.168.1.11 IP: 10.0.1.1/32 GW: 169.254.1.1 169.254.1.1/32 arp_ignore=0 veth͕ens3ͷMACΞυϨεΛฦ͢ arp_ignore=1 veth͕ens3ͷMACΞυϨεΛฦ͞ͳ͍ Pod͔ΒNodeʹͭͳ͕Βͳ͍ʂ vethʹϦϯΫϩʔΧϧΞυϨεΛׂΓ ͯɺPodͷσϑΥϧτήʔτΣΠ Λ͜ͷΞυϨεʹมߋͨ͠ɻ मਖ਼લ मਖ਼ޙ 26 -
·ͱΊͱࠓޙ • ·ͱΊ • େنͳKubernetesΫϥελͰͷར༻Λߟྀͨ͠CLOSΞʔΩςΫνϟ͚ CNIϓϥάΠϯ coil Λ։ൃͨ͠ • coil
ͷ࣮γϯϓϧͰಡΈ͍͢ͱࢥ͏ͷͰɺCNIϓϥάΠϯΛֶͼ͍ͨਓ ʹΦεεϝ • ࠓޙ • େنΫϥελʹద༻ͯ҆͠ఆੑΛݕূ • νϡʔτϦΞϧͳͲΛ༻ҙͯ͠ར༻͘͢͠ 28 -
We are hiring! • NecoϓϩδΣΫτͷ࠾༻ใ • https://cybozu.co.jp/company/job/recruitment/list/neco_project.html • εΩϧνΣοΫγʔτ •
https://gist.github.com/ymmt2005/bd92296166e52d1beba9df8ac516a9db • NecoϓϩδΣΫτͰʹ͚ͭΒΕΔεΩϧΛհ • ଟ༷ͳಇ͖ํ • શϦϞʔτϫʔΫɺि20࣌ؒۈɺଞࣾͱͷ݉ۀͳͲɺ͍Ζ͍Ζͳಇ͖ํ Λ͍ͯ͠Δϝϯόʔ͕ॴଐ 29