Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up for free
coil.pdf
Akihiro Ikezoe
February 19, 2019
Technology
4
2.7k
coil.pdf
Akihiro Ikezoe
February 19, 2019
Tweet
Share
More Decks by Akihiro Ikezoe
See All by Akihiro Ikezoe
Kubernetes Admission Webhook Deep Dive
zoetrope
7
740
Kubernetesオペレータのアンチパターン&ベストプラクティス
zoetrope
11
3.5k
Production-Ready Kubernetesに至るまでの3年間とこれから
zoetrope
4
630
オンプレKubernetesでMySQLクラスタの運用を自動化するためにOperatorを自作している話
zoetrope
5
1.8k
サイボウズを支える技術~インフラ刷新プロジェクトNecoを中心に紹介~
zoetrope
0
440
Kuebernetesクラスタのマルチテナンシーベストプラクティス
zoetrope
8
6.1k
クラウドネイティブなチームづくり
zoetrope
7
3.5k
Open Policy Agent / Gatekeeper 勉強会
zoetrope
5
2.2k
Kubernetesクラスタの自動管理システムのつくりかた
zoetrope
3
15k
Other Decks in Technology
See All in Technology
lt53
98_justdoit
0
110
Optimizing your Swift code
kateinoigakukun
0
1.3k
Astroで始める爆速個人サイト開発
takanorip
12
8.5k
メドレー エンジニア採用資料/ Medley Engineer Guide
medley
3
5k
WebLogic Server for OCI 概要
oracle4engineer
PRO
3
860
スクラム導入して変わったチーム、組織のありかた
yumechi
0
120
日経電子版だけじゃない! 日経の新規Webメディアの開発 - NIKKEI Tech Talk #3
sztm
0
150
証明書って何だっけ? 〜AWSの中間CA移行に備える〜
minorun365
3
2k
Kaggleシミュレーションコンペの動向
nagiss
0
240
ECテックカンファレンス2023 EC事業部のモバイル開発2023
tatsumi0000
0
180
IoT から見る AWS re:invent 2022 ― AWSのIoTの歴史を添えて/Point of view the AWS re:invent 2022 with IoT - with a history of IoT in AWS
ma2shita
0
230
地方自治体業務あるある ーアナログ最適化編-
y150saya
1
110
Featured
See All Featured
Bootstrapping a Software Product
garrettdimon
299
110k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
270
12k
5 minutes of I Can Smell Your CMS
philhawksworth
198
18k
Robots, Beer and Maslow
schacon
154
7.3k
Support Driven Design
roundedbygravity
88
8.9k
Making the Leap to Tech Lead
cromwellryan
117
7.7k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
239
19k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
44
14k
What the flash - Photography Introduction
edds
64
10k
Facilitating Awesome Meetings
lara
33
4.6k
How STYLIGHT went responsive
nonsquared
89
4.2k
10 Git Anti Patterns You Should be Aware of
lemiorhan
643
54k
Transcript
େنKubernetesΫϥελ͚ʹ CNIϓϥάΠϯΛࣗ࡞ͨ͠ αΠϘζגࣜձࣾ ఴ ໌ 1 Kubernetes Meetup Tokyo #16
2019/02/19
ຊͷൃද༰ • Πϯϑϥ৽ϓϩδΣΫτ Neco ͷհ • ͳͥCNIϓϥάΠϯΛࣗ࡞ͨ͠ͷ͔ʁ • ࣗ࡞CNIϓϥάΠϯ coil
ͷհ • CNIϓϥάΠϯ։ൃͰಘΒΕͨݟ • ·ͱΊ 2
Πϯϑϥ৽ϓϩδΣΫτ Neco 3
Πϯϑϥ৽ϓϩδΣΫτNecoͱ • Kubernetes Λಋೖͯ͠ ͷΠϯϑϥ Λ৽͢ΔϓϩδΣΫτ • ͱ •
kintoneGaroon, OfficeͳͲͷαʔϏεΛSaaSͱͯ͠ఏڙ • ಋೖاۀ2.5ສࣾɺϢʔβʔ100ສਓ • 2011ϦϦʔε • VMϕʔεͷΞʔΩςΫνϟ • σʔληϯλʔΛआΓͯɺ1,000Ҏ্ͷαʔόʔΛࣗલͰӡ༻ 4
Πϯϑϥ৽ϓϩδΣΫτNecoͱ • త • ϝϯςφϯείετͷେ෯ݮ (NoOpsΛࢦ͢) • εέʔϥϏϦςΟͷ্ • αʔόʔͷूੑ্
• ΞϓϦέʔγϣϯ։ൃνʔϜ͕σϓϩΠɾӡ༻ʹࢀՃ • ΄ͱΜͲͷՌΛOSSͱͯ͠ެ։ 5
NecoͷΞʔΩςΫνϟ Kubernetes CoreOS Node LB Prometheus squid CoreOS Node CoreOS
Node CoreOS Node CoreOS Node Boot Server CKE sabakan CoreDNS 1,000نͷαʔόʔ ઍʙͷΞϓϦέʔγϣϯίϯςφ 5ͷϒʔταʔόʔ Kubernetesͷ σϓϩΠ ཧ ܧଓతσϦόϦʔ app app MySQL Elastic search 6 CoreOS Node neco updater Ubuntu Argo CD Rook
NecoΛࢧ͑ΔιϑτΣΞͨͪ • sabakan • αʔόʔػࡐͷϥΠϑαΠΫϧཧͱϓϩϏδϣχϯάͷࣗಈԽΛ͓͜ͳ͏ɻ • BIOSͷઃఆɺOSͷωοτϒʔτɺσΟεΫ҉߸Խ֤छγεςϜιϑτΣΞͷ ηοτΞοϓΛࣗಈతʹ͓͜ͳ͏ɻ • CKE
(Cybozu Container Engine) • KubernetesΫϥελͷࣗಈߏஙɾӡ༻Λ͓͜ͳ͏πʔϧɻ • sabakan͕ߏஙͨ͠αʔόʔʹKubernetesΛࣗಈతʹσϓϩΠ͢Δɻ • ΤϥʔΛࣗಈम෮ͨ͠ΓɺނোػࡐΛΫϥελ͔ΒऔΓ֎͢ͳͲͷӡ༻ΛࣗಈԽɻ • neco-updater • ΠϯϑϥͷܧଓతσϦόϦʔπʔϧɻ • GitHubͷϦϦʔεใΛνΣοΫ͠ɺCKEsabakanΛ͡Ίͱ͢Δ֤छιϑτ ΣΞͷσϓϩΠɺCoreOSΠϝʔδͷߋ৽ͳͲΛࣗಈతʹ࣮ࢪ͢Δɻ 7
ͳͥCNIϓϥάΠϯΛ ࣗ࡞ͨ͠ͷ͔ʁ 8
ͳͥCNIϓϥάΠϯΛࣗ࡞ͨ͠ͷ͔ʁ • NecoϓϩδΣΫτͷωοτϫʔΫߏʹϚονͨ͠ωοτϫʔΫϓ ϥάΠϯ͕ඞཁ • طଘͷϓϥάΠϯΛબఆ͕ͨ͠ɺશʹཁ݅ʹϚον͢Δͷ͕ͳ ͔ͬͨɻ • CNIϓϥάΠϯෳΛΈ߹Θͤͯར༻͢Δ͜ͱ͕Մೳɻඞཁͳ෦ ͚ͩࣗ࡞͠Α͏ɻ
9
NecoͷωοτϫʔΫߏ Rack1 Rack2 Rack3 • CLOSΞʔΩςΫνϟ • ϑϥοτͳL3ωοτϫʔΫ • ֤ϊʔυʹಉҰϗοϓͰ౸ୡՄೳ
• East-WestτϥϑΟοΫͷ૿େʹର͠ ͯεέʔϧՄೳ • BGPʹΑΔϧʔςΟϯά • AS per Rack • ECMPʹΑΔܦ࿏Խ • BFDʹΑΔߴͳܦ࿏ऩଋ • ৄ͘͠ϒϩάͰ • https://blog.cybozu.io/entry/2018/1 1/01/113000 10
CNIϓϥάΠϯͷબఆ • σʔληϯλʔωοτϫʔΫͱ߹ΘͤͯɺKubernetesͷωοτϫʔΫʹBGP Λ࠾༻ͯ͠ޮతʹϧʔςΟϯάΛ͓͜ͳ͍͍ͨɻ • Calico • ։ൃ͕׆ൃͰػೳ๛ɻ࠾༻࣮ଟ͍ɻ • BGP
SpeakerΛแ͍ͯ͠Δ͜ͱɺେنΫϥελͰܦ࿏͕૿େ͢Δ ͜ͱͳͲ͕ݒ೦ɻ • Romana • ػೳతʹཁ݅Λຬ͍ͨͯ͠Δɻ • etcd v3ະରԠɻ։ൃ͕׆ൃͰͳ͘࠷৽ͷKubernetesʹैͰ͖͍ͯͳ͍ɻ 11
ϓϥάΠϯͷΈ߹Θͤ • ίϯςφωοτϫʔΫͰղܾ͖͢՝͕ଟ͋Δɻ • IPΞυϨεཧ (IPAM) • ωοτϫʔΫଓੑ • ωοτϫʔΫϙϦγʔ
• ଳҬ੍ݶ • ͯ͢ͷΛ1ͭͷϓϥάΠϯͰղܾ͢Δඞཁͳ͍ɻ • ෳͷϓϥάΠϯΛΈ߹ΘͤΔྫ: Canal • IPAMCNIϓϥάΠϯͷඪ४ػೳΛར༻ • ωοτϫʔΫଓੑ flannel Λར༻ • ωοτϫʔΫϙϦγʔ Calico Λར༻ 12
Ͳ͜Λࣗ࡞͢Δඞཁ͕͋Δͷ͔ʁ • ωοτϫʔΫϙϦγʔ • ࣮͕େมɻಠࣗੑΛग़͢ඞཁͳ͍ɻ CalicoCiliumͳͲΛར༻ • IPΞυϨεཧ (IPAM)
• ϧʔςΟϯάςʔϒϧͷ૿େΛආ͚ΔͨΊͷΈ͕ඞཁɻ • PodʹάϩʔόϧIPΞυϨεΛׂΓͯΔػೳཉ͍͠ɻ ࣗ࡞ • ωοτϫʔΫଓ • ϊʔυͷ֎ͱͷܦ࿏ަɺDCͷϧʔςΟϯάιϑτΣΞʹͤΔ ϊʔυͷϧʔςΟϯά෦Λࣗ࡞ 13
ࣗ࡞CNIϓϥάΠϯ coil 14
coilͱ • αΠϘζ͕։ൃ͢ΔCNIϓϥάΠϯ • OSSͱͯ͠ެ։ • https://github.com/cybozu-go/coil • ಛ •
CNI Spec v0.3.1 (Kubernetes v1.13)ʹରԠ • GoݴޠͰ࣮ɺόοΫΤϯυʹ etcd Λ࠾༻ • Linux only, Kubernetes only, IPv4 only • IPAMͱϊʔυͷϧʔςΟϯάػೳͷΈΛఏڙ • ϧʔςΟϯάιϑτΣΞΛแ͠ͳ͍ • େنΫϥελͰͷར༻Λߟྀ 15
େنΫϥελͰར༻͢ΔͨΊʹ • ΦʔόϨΠωοτϫʔΫར༻͠ͳ͍ • VXLANͳͲͷΦʔόʔϨΠωοτϫʔΫεϧʔϓοτ͕མͪΔɻ ୯७ͳϧʔςΟϯάʹΑΓωοτϫʔΫͷଓੑΛཱ֬ • Linux BridgeΛར༻͠ͳ͍
• Linux BridgeΛ͏ͱCPU༻͕૿͑ͯ͠·͏ɻ ϧʔςΟϯάςʔϒϧʹvethͷϧʔτΛՃ͢Δɻ • etcd API v3ͷΈʹରԠ • etcd API v2ɺଟͷΫϥΠΞϯτ͔Βଓ͞Εͨ࣌ʹύϑΥʔϚϯε͕མͪΔɻ • ܦ࿏ͷ૿େΛ͙ • Pod͝ͱʹܦ࿏Λࠂ͢Δͱܦ࿏͕૿େͯ͠͠·͏ɻ ΞυϨεϒϩοΫͱ͍͏ΈʹΑΓɺαϒωοτ͝ͱͷܦ࿏Λࠂɻ 16
Node coilͷߏཁૉ • etcd • ׂΓͯͨΞυϨεใΛཧ • coil-controller • k8s্ͷDeployment
• ΘΕͳ͘ͳͬͨΞυϨεϒϩοΫͷղ์ • coil: • CNIϓϥάΠϯຊମ • PodͷvethͷՃɾআɺIPΞυϨεׂΓͯɺ ϧʔςΟϯάઃఆͳͲΛ࣮ࢪ • coil-node: k8s্ͷDaemonSet • coild: • ϊʔυ͝ͱͷΞυϨεཧϧʔςΟϯάͷઃఆΛ࣮ࢪ • coil-installer: • coilઃఆϑΝΠϧͷΠϯετʔϧΛ࣮ࢪ 17 <DeamonSet> coil-node <CNI> coil etcd House Keeping <Deployment> coil-controller coild coil installer conf ηοτΞοϓ
ΞυϨεϒϩοΫ (Inspired by Romana) • coilͰΞυϨεϒϩοΫͱ͍͏ΈΛ औΓೖΕɺαϒωοτ୯Ґ(ྫ: /28)Ͱܦ࿏ Λࠂ͢Δ͜ͱͰϧʔςΟϯάςʔϒϧͷ ංେԽΛճආ͍ͯ͠Δɻ
18 Node2 Pod 10.0.1.16/32 Node1 Pod 10.0.1.0/32 Pod 10.0.1.17/32 Pod 10.0.1.1/32 BGP Router 10.0.1.0/28 -> Node1 10.0.1.16/28 -> Node2 αϒωοτ୯ҐͰ ܦ࿏Λࠂ PodͷΞυϨεϨϯδ: 10.0.1.0/24 ΞυϨεϒϩοΫ: 10.0.1.0/28 ΞυϨεϒϩοΫ: 10.0.1.16/28 ΞυϨεϒϩοΫ: 10.0.1.32/28 ΞυϨεϒϩοΫ: 10.0.1.48/28 ΞυϨεϒϩοΫ: 10.0.1.64/28 ΞυϨεϒϩοΫ 10.0.1.0/28 ΞυϨεϒϩοΫ 10.0.1.16/28 ɿ ϊʔυ͝ͱʹ ΞυϨεϒϩοΫ ΛׂΓͯ ׂΓͯՄೳͳ ΞυϨεΛ ϒϩοΫͱ͍͏ ୯Ґʹׂ PodͷΞυϨε ΞυϨεϒ ϩοΫ͔Βׂ Γͯ
coilͷॲཧͷྲྀΕ 1. kubelet ͕ࢦࣔΛड͚ͯ Pod Λ࡞ 2. CNIϓϥάΠϯ coil Λ࣮ߦ
3. coil ͔Β coild ʹIPΞυϨεΛཁٻ 4. etcd ͔ΒΞυϨεϒϩοΫΛׂΓͯΔ 5. ܦ࿏ΛϧʔςΟϯάςʔϒϧʹॻ͖ग़͢ 6. ϧʔςΟάςʔϒϧΛಡΈࠐΈܦ࿏Λࠂ 7. coild ͔Β coil ʹIPΞυϨεΛฦ͢ 8. Pod ͷ netns ʹ veth ͷϖΞΛ࡞͠ɺIPΞυ ϨεͷׂΓͯͱϧʔτͷઃఆΛ͓͜ͳ͏ Node <DeamonSet> coild <CNI> coil kubelet Pod etcd routing table BGP Speaker eth0 veth BGP Router ᶃ ᶃ ᶄ ᶅ ᶆ ᶇ ᶈ ᶈ ᶊ ᶉ 19
boot-taint • CNIϓϥάΠϯ͕ηοτΞοϓ͞Ε͍ͯͳ͍ϊʔυ ʹPod͕εέδϡʔϧ͞ΕΔͱࠔΔɻ • kubeletͷىಈΦϓγϣϯʹ —register-with-taints Λ ࢦఆͯ͠ɺىಈޙͷϊʔυʹPodΛεέδϡʔ ϧͰ͖ͳ͍Α͏ʹ͓ͯ͘͠ɻ
• coil ͷηοτΞοϓ͕ྃͨ͠Β taints Λআ͠ɺ PodΛεέδϡʔϦϯάՄೳʹ͢Δɻ 20 taints/tolerations Kubernetesͷػೳɻ ϊʔυʹtaintsΛ༩͢ Δ͜ͱͰɺPodͷεέ δϡʔϦϯά࣮ߦΛ ېࢭ͢Δ͜ͱ͕Ͱ͖Δɻ ಛఆͷtolerations͕ ༩͞ΕͨPodͷΈεέ δϡʔϦϯάՄೳͱͳ Δɻ
CNIϓϥάΠϯ։ൃͰ ಘΒΕͨݟ 21
։ൃσόοά͕େมͰʁ • NecoϓϩδΣΫτͰɺσʔληϯλʔͷωοτϫʔΫߏΛؙ͝ͱιϑτ ΣΞͰԾԽͨ͠ڥΛ༻ҙ͓ͯ͠ΓɺखݩͰ؆୯ʹಈ࡞֬ೝΛ͓͜ͳ͏͜ͱ ͕Ͱ͖Δɻ • γϯϓϧͳL3ωοτϫʔΫͳͷͰௐ͍ࠪ͢͠ɻ nsenterͱtcpdump͕͋ΕɺେͷௐࠪͰ͖Δɻ 22
KubernetesͷใΛಘΔʹʁ • CNIϓϥάΠϯKubernetesઐ༻ͷͷͰͳ͍ͨΊɺPodͷใΛऔಘ͢Δ ༷ఆΊΒΕ͍ͯͳ͍ɻ • https://github.com/containernetworking/cni/issues/606 • ݱঢ়Kubernetes͔ΒCNIϓϥάΠϯΛݺͼग़͢ࡍʹɺCNI_ARGS Ͱ K8S_POD_NAME
K8S_POD_NAMESPACE ͳͲͷใΛ͍ͯ͠Δɻ • ໌֬ʹ༷ͱͯ͠ఆ·͍ͬͯΔΘ͚Ͱͳ͍ͷͰ࣮ΛಡΈղ͘ඞཁ͕͋ͬͨɻ (dockershimcontainerdͳͲɺίϯςφϥϯλΠϜ͝ͱʹͦΕͧΕ࣮ͯ͠Δ) 23
CNIϓϥάΠϯ͕ಡΈࠐ·Εͳ͍ • Kubernetes/etc/cni/net.dʹ͓͍ͯ͋ΔઃఆΛݩʹϓϥάΠϯΛ࣮ߦ͢Δɻ • ઃఆϑΝΠϧͷߋ৽ޙʹΞϓϦέʔγϣϯͷPodΛ࡞ͨ͠ͱ͜Ζɺઃఆͨ͠ ϓϥάΠϯ͕ར༻͞Εͳ͍͜ͱ͕͋Δɻ • kubeletͰɺ5ඵ͝ͱʹίϯςφϥϯλΠϜͷεςʔλεߋ৽Λߦͳ͍ͬͯΔɻ ͜ͷͱ͖ʹCNIϓϥάΠϯಡΈࠐΜͰ͍Δɻ 24
Kubernetes 1.13ͰPodؒ௨৴͕Ͱ͖ͳ͍ • KubernetesΛv1.13ʹΞοϓάϨʔυͨ͠ΒɺPodؒͷ௨৴͕Ͱ͖ͳ͘ͳͬͨɻ • kube-proxyͷIPVSϞʔυͷ࣮ͰɺLinuxͷΧʔωϧύϥϝʔλ(sysctl)ͷઃఆ͕ มߋ͞Ε͍ͯͨ (ϦϦʔεϊʔτʹهड़ͳ͠) • net.ipv4.conf.all.arp_ignore:
0 -> 1 • net.ipv4.conf.all.arp_announce: 0 -> 2 • ͍͔ͭ͘ͷCNIϓϥάΠϯʹӨڹ͕ग़͍ͯΔ • https://github.com/kubernetes/kubernetes/issues/71555 25
ͱղܾํ๏ Node Pod ens3 eth0 veth 192.168.1.11 IP: 10.0.1.1/32 GW:
192.168.1.11 ARP: 192.168.1.11 Node Pod ens3 eth0 veth 192.168.1.11 IP: 10.0.1.1/32 GW: 169.254.1.1 169.254.1.1/32 arp_ignore=0 veth͕ens3ͷMACΞυϨεΛฦ͢ arp_ignore=1 veth͕ens3ͷMACΞυϨεΛฦ͞ͳ͍ Pod͔ΒNodeʹͭͳ͕Βͳ͍ʂ vethʹϦϯΫϩʔΧϧΞυϨεΛׂΓ ͯɺPodͷσϑΥϧτήʔτΣΠ Λ͜ͷΞυϨεʹมߋͨ͠ɻ मਖ਼લ मਖ਼ޙ 26
·ͱΊͱࠓޙ 27
·ͱΊͱࠓޙ • ·ͱΊ • େنͳKubernetesΫϥελͰͷར༻Λߟྀͨ͠CLOSΞʔΩςΫνϟ͚ CNIϓϥάΠϯ coil Λ։ൃͨ͠ • coil
ͷ࣮γϯϓϧͰಡΈ͍͢ͱࢥ͏ͷͰɺCNIϓϥάΠϯΛֶͼ͍ͨਓ ʹΦεεϝ • ࠓޙ • େنΫϥελʹద༻ͯ҆͠ఆੑΛݕূ • νϡʔτϦΞϧͳͲΛ༻ҙͯ͠ར༻͘͢͠ 28
We are hiring! • NecoϓϩδΣΫτͷ࠾༻ใ • https://cybozu.co.jp/company/job/recruitment/list/neco_project.html • εΩϧνΣοΫγʔτ •
https://gist.github.com/ymmt2005/bd92296166e52d1beba9df8ac516a9db • NecoϓϩδΣΫτͰʹ͚ͭΒΕΔεΩϧΛհ • ଟ༷ͳಇ͖ํ • શϦϞʔτϫʔΫɺि20࣌ؒۈɺଞࣾͱͷ݉ۀͳͲɺ͍Ζ͍Ζͳಇ͖ํ Λ͍ͯ͠Δϝϯόʔ͕ॴଐ 29