Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up
for free
coil.pdf
Akihiro Ikezoe
February 19, 2019
Technology
4
2.3k
coil.pdf
Akihiro Ikezoe
February 19, 2019
Tweet
Share
More Decks by Akihiro Ikezoe
See All by Akihiro Ikezoe
zoetrope
7
5.2k
zoetrope
7
3.1k
zoetrope
4
450
zoetrope
3
12k
zoetrope
10
1.5k
Other Decks in Technology
See All in Technology
kaga
0
190
yuuturn
1
130
y0hgi
1
370
line_developers
PRO
3
480
hecateball
1
12k
humank
0
210
saoritakita
0
340
hacker2202
0
750
awsbelaraby
4
1.2k
hsano
0
120
ama_ch
0
3.3k
sasakendayo
2
320
Featured
See All Featured
bryan
30
3.3k
maltzj
500
36k
aarron
258
36k
sachag
446
36k
jacobian
255
20k
malarkey
393
60k
marktimemedia
6
330
kneath
219
15k
davidbonilla
69
3.5k
bryan
100
11k
smashingmag
229
18k
danielanewman
1
470
Transcript
େنKubernetesΫϥελ͚ʹ CNIϓϥάΠϯΛࣗ࡞ͨ͠ αΠϘζגࣜձࣾ ఴ ໌ 1 Kubernetes Meetup Tokyo #16
2019/02/19
ຊͷൃද༰ • Πϯϑϥ৽ϓϩδΣΫτ Neco ͷհ • ͳͥCNIϓϥάΠϯΛࣗ࡞ͨ͠ͷ͔ʁ • ࣗ࡞CNIϓϥάΠϯ coil
ͷհ • CNIϓϥάΠϯ։ൃͰಘΒΕͨݟ • ·ͱΊ 2
Πϯϑϥ৽ϓϩδΣΫτ Neco 3
Πϯϑϥ৽ϓϩδΣΫτNecoͱ • Kubernetes Λಋೖͯ͠ ͷΠϯϑϥ Λ৽͢ΔϓϩδΣΫτ • ͱ •
kintoneGaroon, OfficeͳͲͷαʔϏεΛSaaSͱͯ͠ఏڙ • ಋೖاۀ2.5ສࣾɺϢʔβʔ100ສਓ • 2011ϦϦʔε • VMϕʔεͷΞʔΩςΫνϟ • σʔληϯλʔΛआΓͯɺ1,000Ҏ্ͷαʔόʔΛࣗલͰӡ༻ 4
Πϯϑϥ৽ϓϩδΣΫτNecoͱ • త • ϝϯςφϯείετͷେ෯ݮ (NoOpsΛࢦ͢) • εέʔϥϏϦςΟͷ্ • αʔόʔͷूੑ্
• ΞϓϦέʔγϣϯ։ൃνʔϜ͕σϓϩΠɾӡ༻ʹࢀՃ • ΄ͱΜͲͷՌΛOSSͱͯ͠ެ։ 5
NecoͷΞʔΩςΫνϟ Kubernetes CoreOS Node LB Prometheus squid CoreOS Node CoreOS
Node CoreOS Node CoreOS Node Boot Server CKE sabakan CoreDNS 1,000نͷαʔόʔ ઍʙͷΞϓϦέʔγϣϯίϯςφ 5ͷϒʔταʔόʔ Kubernetesͷ σϓϩΠ ཧ ܧଓతσϦόϦʔ app app MySQL Elastic search 6 CoreOS Node neco updater Ubuntu Argo CD Rook
NecoΛࢧ͑ΔιϑτΣΞͨͪ • sabakan • αʔόʔػࡐͷϥΠϑαΠΫϧཧͱϓϩϏδϣχϯάͷࣗಈԽΛ͓͜ͳ͏ɻ • BIOSͷઃఆɺOSͷωοτϒʔτɺσΟεΫ҉߸Խ֤छγεςϜιϑτΣΞͷ ηοτΞοϓΛࣗಈతʹ͓͜ͳ͏ɻ • CKE
(Cybozu Container Engine) • KubernetesΫϥελͷࣗಈߏஙɾӡ༻Λ͓͜ͳ͏πʔϧɻ • sabakan͕ߏஙͨ͠αʔόʔʹKubernetesΛࣗಈతʹσϓϩΠ͢Δɻ • ΤϥʔΛࣗಈम෮ͨ͠ΓɺނোػࡐΛΫϥελ͔ΒऔΓ֎͢ͳͲͷӡ༻ΛࣗಈԽɻ • neco-updater • ΠϯϑϥͷܧଓతσϦόϦʔπʔϧɻ • GitHubͷϦϦʔεใΛνΣοΫ͠ɺCKEsabakanΛ͡Ίͱ͢Δ֤छιϑτ ΣΞͷσϓϩΠɺCoreOSΠϝʔδͷߋ৽ͳͲΛࣗಈతʹ࣮ࢪ͢Δɻ 7
ͳͥCNIϓϥάΠϯΛ ࣗ࡞ͨ͠ͷ͔ʁ 8
ͳͥCNIϓϥάΠϯΛࣗ࡞ͨ͠ͷ͔ʁ • NecoϓϩδΣΫτͷωοτϫʔΫߏʹϚονͨ͠ωοτϫʔΫϓ ϥάΠϯ͕ඞཁ • طଘͷϓϥάΠϯΛબఆ͕ͨ͠ɺશʹཁ݅ʹϚον͢Δͷ͕ͳ ͔ͬͨɻ • CNIϓϥάΠϯෳΛΈ߹Θͤͯར༻͢Δ͜ͱ͕Մೳɻඞཁͳ෦ ͚ͩࣗ࡞͠Α͏ɻ
9
NecoͷωοτϫʔΫߏ Rack1 Rack2 Rack3 • CLOSΞʔΩςΫνϟ • ϑϥοτͳL3ωοτϫʔΫ • ֤ϊʔυʹಉҰϗοϓͰ౸ୡՄೳ
• East-WestτϥϑΟοΫͷ૿େʹର͠ ͯεέʔϧՄೳ • BGPʹΑΔϧʔςΟϯά • AS per Rack • ECMPʹΑΔܦ࿏Խ • BFDʹΑΔߴͳܦ࿏ऩଋ • ৄ͘͠ϒϩάͰ • https://blog.cybozu.io/entry/2018/1 1/01/113000 10
CNIϓϥάΠϯͷબఆ • σʔληϯλʔωοτϫʔΫͱ߹ΘͤͯɺKubernetesͷωοτϫʔΫʹBGP Λ࠾༻ͯ͠ޮతʹϧʔςΟϯάΛ͓͜ͳ͍͍ͨɻ • Calico • ։ൃ͕׆ൃͰػೳ๛ɻ࠾༻࣮ଟ͍ɻ • BGP
SpeakerΛแ͍ͯ͠Δ͜ͱɺେنΫϥελͰܦ࿏͕૿େ͢Δ ͜ͱͳͲ͕ݒ೦ɻ • Romana • ػೳతʹཁ݅Λຬ͍ͨͯ͠Δɻ • etcd v3ະରԠɻ։ൃ͕׆ൃͰͳ͘࠷৽ͷKubernetesʹैͰ͖͍ͯͳ͍ɻ 11
ϓϥάΠϯͷΈ߹Θͤ • ίϯςφωοτϫʔΫͰղܾ͖͢՝͕ଟ͋Δɻ • IPΞυϨεཧ (IPAM) • ωοτϫʔΫଓੑ • ωοτϫʔΫϙϦγʔ
• ଳҬ੍ݶ • ͯ͢ͷΛ1ͭͷϓϥάΠϯͰղܾ͢Δඞཁͳ͍ɻ • ෳͷϓϥάΠϯΛΈ߹ΘͤΔྫ: Canal • IPAMCNIϓϥάΠϯͷඪ४ػೳΛར༻ • ωοτϫʔΫଓੑ flannel Λར༻ • ωοτϫʔΫϙϦγʔ Calico Λར༻ 12
Ͳ͜Λࣗ࡞͢Δඞཁ͕͋Δͷ͔ʁ • ωοτϫʔΫϙϦγʔ • ࣮͕େมɻಠࣗੑΛग़͢ඞཁͳ͍ɻ CalicoCiliumͳͲΛར༻ • IPΞυϨεཧ (IPAM)
• ϧʔςΟϯάςʔϒϧͷ૿େΛආ͚ΔͨΊͷΈ͕ඞཁɻ • PodʹάϩʔόϧIPΞυϨεΛׂΓͯΔػೳཉ͍͠ɻ ࣗ࡞ • ωοτϫʔΫଓ • ϊʔυͷ֎ͱͷܦ࿏ަɺDCͷϧʔςΟϯάιϑτΣΞʹͤΔ ϊʔυͷϧʔςΟϯά෦Λࣗ࡞ 13
ࣗ࡞CNIϓϥάΠϯ coil 14
coilͱ • αΠϘζ͕։ൃ͢ΔCNIϓϥάΠϯ • OSSͱͯ͠ެ։ • https://github.com/cybozu-go/coil • ಛ •
CNI Spec v0.3.1 (Kubernetes v1.13)ʹରԠ • GoݴޠͰ࣮ɺόοΫΤϯυʹ etcd Λ࠾༻ • Linux only, Kubernetes only, IPv4 only • IPAMͱϊʔυͷϧʔςΟϯάػೳͷΈΛఏڙ • ϧʔςΟϯάιϑτΣΞΛแ͠ͳ͍ • େنΫϥελͰͷར༻Λߟྀ 15
େنΫϥελͰར༻͢ΔͨΊʹ • ΦʔόϨΠωοτϫʔΫར༻͠ͳ͍ • VXLANͳͲͷΦʔόʔϨΠωοτϫʔΫεϧʔϓοτ͕མͪΔɻ ୯७ͳϧʔςΟϯάʹΑΓωοτϫʔΫͷଓੑΛཱ֬ • Linux BridgeΛར༻͠ͳ͍
• Linux BridgeΛ͏ͱCPU༻͕૿͑ͯ͠·͏ɻ ϧʔςΟϯάςʔϒϧʹvethͷϧʔτΛՃ͢Δɻ • etcd API v3ͷΈʹରԠ • etcd API v2ɺଟͷΫϥΠΞϯτ͔Βଓ͞Εͨ࣌ʹύϑΥʔϚϯε͕མͪΔɻ • ܦ࿏ͷ૿େΛ͙ • Pod͝ͱʹܦ࿏Λࠂ͢Δͱܦ࿏͕૿େͯ͠͠·͏ɻ ΞυϨεϒϩοΫͱ͍͏ΈʹΑΓɺαϒωοτ͝ͱͷܦ࿏Λࠂɻ 16
Node coilͷߏཁૉ • etcd • ׂΓͯͨΞυϨεใΛཧ • coil-controller • k8s্ͷDeployment
• ΘΕͳ͘ͳͬͨΞυϨεϒϩοΫͷղ์ • coil: • CNIϓϥάΠϯຊମ • PodͷvethͷՃɾআɺIPΞυϨεׂΓͯɺ ϧʔςΟϯάઃఆͳͲΛ࣮ࢪ • coil-node: k8s্ͷDaemonSet • coild: • ϊʔυ͝ͱͷΞυϨεཧϧʔςΟϯάͷઃఆΛ࣮ࢪ • coil-installer: • coilઃఆϑΝΠϧͷΠϯετʔϧΛ࣮ࢪ 17 <DeamonSet> coil-node <CNI> coil etcd House Keeping <Deployment> coil-controller coild coil installer conf ηοτΞοϓ
ΞυϨεϒϩοΫ (Inspired by Romana) • coilͰΞυϨεϒϩοΫͱ͍͏ΈΛ औΓೖΕɺαϒωοτ୯Ґ(ྫ: /28)Ͱܦ࿏ Λࠂ͢Δ͜ͱͰϧʔςΟϯάςʔϒϧͷ ංେԽΛճආ͍ͯ͠Δɻ
18 Node2 Pod 10.0.1.16/32 Node1 Pod 10.0.1.0/32 Pod 10.0.1.17/32 Pod 10.0.1.1/32 BGP Router 10.0.1.0/28 -> Node1 10.0.1.16/28 -> Node2 αϒωοτ୯ҐͰ ܦ࿏Λࠂ PodͷΞυϨεϨϯδ: 10.0.1.0/24 ΞυϨεϒϩοΫ: 10.0.1.0/28 ΞυϨεϒϩοΫ: 10.0.1.16/28 ΞυϨεϒϩοΫ: 10.0.1.32/28 ΞυϨεϒϩοΫ: 10.0.1.48/28 ΞυϨεϒϩοΫ: 10.0.1.64/28 ΞυϨεϒϩοΫ 10.0.1.0/28 ΞυϨεϒϩοΫ 10.0.1.16/28 ɿ ϊʔυ͝ͱʹ ΞυϨεϒϩοΫ ΛׂΓͯ ׂΓͯՄೳͳ ΞυϨεΛ ϒϩοΫͱ͍͏ ୯Ґʹׂ PodͷΞυϨε ΞυϨεϒ ϩοΫ͔Βׂ Γͯ
coilͷॲཧͷྲྀΕ 1. kubelet ͕ࢦࣔΛड͚ͯ Pod Λ࡞ 2. CNIϓϥάΠϯ coil Λ࣮ߦ
3. coil ͔Β coild ʹIPΞυϨεΛཁٻ 4. etcd ͔ΒΞυϨεϒϩοΫΛׂΓͯΔ 5. ܦ࿏ΛϧʔςΟϯάςʔϒϧʹॻ͖ग़͢ 6. ϧʔςΟάςʔϒϧΛಡΈࠐΈܦ࿏Λࠂ 7. coild ͔Β coil ʹIPΞυϨεΛฦ͢ 8. Pod ͷ netns ʹ veth ͷϖΞΛ࡞͠ɺIPΞυ ϨεͷׂΓͯͱϧʔτͷઃఆΛ͓͜ͳ͏ Node <DeamonSet> coild <CNI> coil kubelet Pod etcd routing table BGP Speaker eth0 veth BGP Router ᶃ ᶃ ᶄ ᶅ ᶆ ᶇ ᶈ ᶈ ᶊ ᶉ 19
boot-taint • CNIϓϥάΠϯ͕ηοτΞοϓ͞Ε͍ͯͳ͍ϊʔυ ʹPod͕εέδϡʔϧ͞ΕΔͱࠔΔɻ • kubeletͷىಈΦϓγϣϯʹ —register-with-taints Λ ࢦఆͯ͠ɺىಈޙͷϊʔυʹPodΛεέδϡʔ ϧͰ͖ͳ͍Α͏ʹ͓ͯ͘͠ɻ
• coil ͷηοτΞοϓ͕ྃͨ͠Β taints Λআ͠ɺ PodΛεέδϡʔϦϯάՄೳʹ͢Δɻ 20 taints/tolerations Kubernetesͷػೳɻ ϊʔυʹtaintsΛ༩͢ Δ͜ͱͰɺPodͷεέ δϡʔϦϯά࣮ߦΛ ېࢭ͢Δ͜ͱ͕Ͱ͖Δɻ ಛఆͷtolerations͕ ༩͞ΕͨPodͷΈεέ δϡʔϦϯάՄೳͱͳ Δɻ
CNIϓϥάΠϯ։ൃͰ ಘΒΕͨݟ 21
։ൃσόοά͕େมͰʁ • NecoϓϩδΣΫτͰɺσʔληϯλʔͷωοτϫʔΫߏΛؙ͝ͱιϑτ ΣΞͰԾԽͨ͠ڥΛ༻ҙ͓ͯ͠ΓɺखݩͰ؆୯ʹಈ࡞֬ೝΛ͓͜ͳ͏͜ͱ ͕Ͱ͖Δɻ • γϯϓϧͳL3ωοτϫʔΫͳͷͰௐ͍ࠪ͢͠ɻ nsenterͱtcpdump͕͋ΕɺେͷௐࠪͰ͖Δɻ 22
KubernetesͷใΛಘΔʹʁ • CNIϓϥάΠϯKubernetesઐ༻ͷͷͰͳ͍ͨΊɺPodͷใΛऔಘ͢Δ ༷ఆΊΒΕ͍ͯͳ͍ɻ • https://github.com/containernetworking/cni/issues/606 • ݱঢ়Kubernetes͔ΒCNIϓϥάΠϯΛݺͼग़͢ࡍʹɺCNI_ARGS Ͱ K8S_POD_NAME
K8S_POD_NAMESPACE ͳͲͷใΛ͍ͯ͠Δɻ • ໌֬ʹ༷ͱͯ͠ఆ·͍ͬͯΔΘ͚Ͱͳ͍ͷͰ࣮ΛಡΈղ͘ඞཁ͕͋ͬͨɻ (dockershimcontainerdͳͲɺίϯςφϥϯλΠϜ͝ͱʹͦΕͧΕ࣮ͯ͠Δ) 23
CNIϓϥάΠϯ͕ಡΈࠐ·Εͳ͍ • Kubernetes/etc/cni/net.dʹ͓͍ͯ͋ΔઃఆΛݩʹϓϥάΠϯΛ࣮ߦ͢Δɻ • ઃఆϑΝΠϧͷߋ৽ޙʹΞϓϦέʔγϣϯͷPodΛ࡞ͨ͠ͱ͜Ζɺઃఆͨ͠ ϓϥάΠϯ͕ར༻͞Εͳ͍͜ͱ͕͋Δɻ • kubeletͰɺ5ඵ͝ͱʹίϯςφϥϯλΠϜͷεςʔλεߋ৽Λߦͳ͍ͬͯΔɻ ͜ͷͱ͖ʹCNIϓϥάΠϯಡΈࠐΜͰ͍Δɻ 24
Kubernetes 1.13ͰPodؒ௨৴͕Ͱ͖ͳ͍ • KubernetesΛv1.13ʹΞοϓάϨʔυͨ͠ΒɺPodؒͷ௨৴͕Ͱ͖ͳ͘ͳͬͨɻ • kube-proxyͷIPVSϞʔυͷ࣮ͰɺLinuxͷΧʔωϧύϥϝʔλ(sysctl)ͷઃఆ͕ มߋ͞Ε͍ͯͨ (ϦϦʔεϊʔτʹهड़ͳ͠) • net.ipv4.conf.all.arp_ignore:
0 -> 1 • net.ipv4.conf.all.arp_announce: 0 -> 2 • ͍͔ͭ͘ͷCNIϓϥάΠϯʹӨڹ͕ग़͍ͯΔ • https://github.com/kubernetes/kubernetes/issues/71555 25
ͱղܾํ๏ Node Pod ens3 eth0 veth 192.168.1.11 IP: 10.0.1.1/32 GW:
192.168.1.11 ARP: 192.168.1.11 Node Pod ens3 eth0 veth 192.168.1.11 IP: 10.0.1.1/32 GW: 169.254.1.1 169.254.1.1/32 arp_ignore=0 veth͕ens3ͷMACΞυϨεΛฦ͢ arp_ignore=1 veth͕ens3ͷMACΞυϨεΛฦ͞ͳ͍ Pod͔ΒNodeʹͭͳ͕Βͳ͍ʂ vethʹϦϯΫϩʔΧϧΞυϨεΛׂΓ ͯɺPodͷσϑΥϧτήʔτΣΠ Λ͜ͷΞυϨεʹมߋͨ͠ɻ मਖ਼લ मਖ਼ޙ 26
·ͱΊͱࠓޙ 27
·ͱΊͱࠓޙ • ·ͱΊ • େنͳKubernetesΫϥελͰͷར༻Λߟྀͨ͠CLOSΞʔΩςΫνϟ͚ CNIϓϥάΠϯ coil Λ։ൃͨ͠ • coil
ͷ࣮γϯϓϧͰಡΈ͍͢ͱࢥ͏ͷͰɺCNIϓϥάΠϯΛֶͼ͍ͨਓ ʹΦεεϝ • ࠓޙ • େنΫϥελʹద༻ͯ҆͠ఆੑΛݕূ • νϡʔτϦΞϧͳͲΛ༻ҙͯ͠ར༻͘͢͠ 28
We are hiring! • NecoϓϩδΣΫτͷ࠾༻ใ • https://cybozu.co.jp/company/job/recruitment/list/neco_project.html • εΩϧνΣοΫγʔτ •
https://gist.github.com/ymmt2005/bd92296166e52d1beba9df8ac516a9db • NecoϓϩδΣΫτͰʹ͚ͭΒΕΔεΩϧΛհ • ଟ༷ͳಇ͖ํ • શϦϞʔτϫʔΫɺि20࣌ؒۈɺଞࣾͱͷ݉ۀͳͲɺ͍Ζ͍Ζͳಇ͖ํ Λ͍ͯ͠Δϝϯόʔ͕ॴଐ 29