Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Kubernetes the hard way revisited 〜ゼロダウンタイムクラスタ...

Kubernetes the hard way revisited 〜ゼロダウンタイムクラスタ移行から学ぶクラスタ構築の仕組み〜

CloudNative Days Tokyo 2023 での登壇資料です

More Decks by LINEヤフーTech (LY Corporation Tech)

Other Decks in Technology

Transcript

  1. ˜-:$PSQPSBUJPO • ޒ൓ా ਖ਼ଠ࿠ ˏTHPUBOE • -*/&Ϡϑʔגࣜձࣾ ιϑτ΢ΣΞΤϯδχΞ • ϓϥΠϕʔτΫϥ΢υ7FSEBͷ,BB4։ൃʹैࣄ

    • ೥-*/&גࣜձࣾ ౰࣌ ʹೖࣾɺҎདྷݱ৬ • ,VCFDPO /"Ͱొஃ • “Leveraging Cluster-API for Production-Ready Multi-Regional Infrastructures”  ࣗݾ঺հ
  2. ˜-:$PSQPSBUJPO • ύϒϦοΫΫϥ΢υ&,4 (,& ",4 • اۀ޲͚੡඼3FE)BU 0QFOTIJGU 7.XBSF5BO[V •

    ΦϯϓϨϛε,VCFBEN ,VCFTQSBZ LPQT LJOE .JOJLVCF • ͦͷଞ,VCFSFOFUFT UIF )BSE 8BZ  ,VCFSOFUFTσΟετϦϏϡʔγϣϯ
  3. ˜-:$PSQPSBUJPO • 7FSEB0QFOTUBDL ϕʔεͷࣾ಺Ϋϥ΢υ • 7FSEB ,VCFSOFUFT4FSWJDF 7,4 • ॳظ͸3BODIFSΛGPMLͯ͠։ൃ

    • ݱࡏ $MVTUFS"1* ,VCFBEN ΁ͱҠߦத  7FSEBͰͷ,VCFSOFUFTσΟετϦϏϡʔγϣϯ ˞ຊεϥΠυͰ3BODIFSͱݴٴ͢Δࡍ͸ɺ'PSLͨ͠΋ͷΛࢦ͠·͢ɻΞοϓετϦʔϜͱҰகੑ͸อূ͠·ͤΜ
  4. ˜-:$PSQPSBUJPO • ӡ༻ऀࢹ఺ • ෳ਺ͷσΟετϦϏϡʔγϣϯΛ؅ཧ͢Δͷ͸ਏ͍ • ΫϥελͷҠߦΛਐΊ͍ͨ • ར༻ऀࢹ఺ •

    طଘͷΫϥελΛ࢖͍ଓ͚͍ͨ • ΫϥελΛ࡞Γ௚ͨ͘͠ͳ͍ • ໨ඪطଘͷΫϥελΛಈ͔ͨ͠··Ҡߦ͢Δ  3BODIFS͔Β$MVTUFS"1*,VCFBEN΁ͷҠߦ
  5. ˜-:$PSQPSBUJPO • ֶश༻σΟετϦϏϡʔγϣϯ • ֤ίϯϙʔωϯτΛखಈͰઃఆɾߏங • FUDE • LVCFBQJTFSWFS •

    LVCFDPOUSPMMFSNBOBHFS • LVCFTDIFEVMFS • LVCFMFU • LVCFQSPYZ  ,VCFSOFUFTUIFIBSEXBZͱ͸ https://github.com/kelseyhightower/kubernetes-the-hard-way
  6. ˜-:$PSQPSBUJPO • Ϋϥ΢υϦιʔε࡞੒ • $POUSPM1MBOF༻7. • ,VCFSOFUFT"1*༻ *1ΞυϨε • 8PSLFS༻7.

    • ʜ • ຊՈ͸($1͕ͩɺ೿ੜ൛΋͋Δ • "84 • MJCWJSULWN • ʜ  ,5)8෮शɿϦιʔεͷ࡞੒ IP address for apiserver VM VM worker-* (VM) https://github.com/kelseyhightower/kubernetes-the-hard-way/blob/master/docs/03-compute-resources.md VM VM controller-* (VM)
  7. ˜-:$PSQPSBUJPO • 5-4ɺΩʔϖΞͷੜ੒ • $"ূ໌ॻɾൿີݤ • αʔόʔূ໌ॻɾൿີݤ • ΫϥΠΞϯτূ໌ॻɾൿີݤ •

    4FSWJDF"DDPVOU༻ΩʔϖΞ • ֤ϊʔυʹ഑෍  ,5)8෮शɿূ໌ॻΩʔϖΞͷੜ੒ɾ഑෍ https://github.com/kelseyhightower/kubernetes-the-hard-way/blob/master/docs/04-certificate-authority.md controller-* (VM) worker-* (VM)
  8. ˜-:$PSQPSBUJPO • ,VCFDPOGJHͷੜ੒ • ҎԼΛؚΉ • $"ূ໌ॻ • ΫϥΠΞϯτূ໌ॻɾൿີݤ •

    "1*4FSWFSͷΤϯυϙΠϯτ  ,5)8෮शɿ,VCFDPOGJHͷੜ੒ɾ഑෍ https://github.com/kelseyhightower/kubernetes-the-hard-way/blob/master/docs/05-kubernetes-configuration-files.md
  9. ˜-:$PSQPSBUJPO • &UDE όΠφϦͷ഑ஔ • ઃఆϑΝΠϧʢTZTUFNEʣͷੜ੒ • $" 5-4ূ໌ॻ •

    ॳظͷϝϯόͷΞυϨε • ҉߸Խ • ʜ • &UDEαʔϏεͷىಈ  ,5)8෮शɿ&UDE Ϋϥελͷߏங controller-1 (VM) controller-2 (VM) controller-0 (VM) https://github.com/kelseyhightower/kubernetes-the-hard-way/blob/master/docs/07-bootstrapping-etcd.md
  10. ˜-:$PSQPSBUJPO • όΠφϦͷ഑ஔ • LVCFBQJTFSWFS • LVCFDPOUSPMMFSNBOBHFS • LVCFTDIFEVMFS •

    ઃఆϑΝΠϧʢTZTUFNEʣͷੜ੒ • $" 5-4ূ໌ॻɾൿີݤ • 1&. • LVCFDPOGJH • ΤϯυϙΠϯτ • FUDEˡˠFUDE • LVCFBQJTFSWFSˠFUDE • LVCFTDIFEVMFSˠLVCFBQJTFSWFS • LVCFDPOUSPMMFSNBOBHFSˠLVCFBQJTFSWFS • ֤αʔϏεͷىಈ  ,5)8෮शɿ$POUSPM1MBOFͷىಈ https://github.com/kelseyhightower/kubernetes-the-hard-way/blob/master/docs/08-bootstrapping-kubernetes-controllers.md controller-* (VM)
  11. ˜-:$PSQPSBUJPO • ্ཱ͕ͪͬͨLVCFBQJTFSWFSʹରͯ͠ -PBE#BMBODFSΛ઀ଓ • ࣄલ֬อͨ͠ *1ΞυϨεΛ෇༩ • ,VCFDPOGJHʹࢦఆͨ͠΋ͷ 

    ,5)8෮शɿ-#ͷ઀ଓ https://github.com/kelseyhightower/kubernetes-the-hard-way/blob/master/docs/08-bootstrapping-kubernetes-controllers.md ce2(VM) ce3(VM) ce1(VM) LoadBalancer
  12. ˜-:$PSQPSBUJPO • ϗετͷઃఆ • TXBQPGG • ίϯϙʔωϯτͷΠϯετʔϧ • DPOUBJOFSE $/*QMVHJO

    ,VCFMFU LVCFQSPYZ • ઃఆϑΝΠϧੜ੒ • 4ZTUFNE 6OJUͷىಈ  ϫʔΧʔϊʔυͷߏஙɾ௥Ճ Worker1 (VM) Worker2 (VM) Worker3 (VM) hSps://github.com/kelseyhightower/kubernetes-the-hard-way/blob/master/docs/09-bootstrapping-kubernetes-workers.md
  13. ˜-:$PSQPSBUJPO • ҟͳΔઃఆͷίϯϙʔωϯτ΁ߋ৽ • 3BODIFS SLF ˠ,VCFBEN • ϊʔυΛ̍ͭͣͭೖΕସ͑ •

    ࠷ऴతʹશͯͷϊʔυɾίϯϙʔωϯτΛߋ৽  ઓུɿϊʔυΛͭͣͭೖΕସ͑Δ
  14. ˜-:$PSQPSBUJPO • "1*4FSWFSʹରͯ͠-#Λ࡞੒ • ,VCFSOFUFT UIF IBSEXBZͱಉ͡ʂ • ৽ߏ੒ʹ߹ΘͤͯυϝΠϯ໊Λ෇༩ •

    NZDMVTUFSFYBNQMFDPN • ˠ઀ଓΤϥʔ  -#Λ࡞Ε͹ղܾʁ LB worker چߏ੒ʢ3BODIFSʣվ mycluster.example.com
  15. ˜-:$PSQPSBUJPO • Ϋϥελͷ࡞੒΍ϊʔυͷ௥Ճ༻ʹ༻ҙ͞Ε͍ͯΔೝূํࣜͷͨΊͷτʔΫϯ • BCDEFGBCDEFGͷΑ͏ͳܗࣜ • CPPUTUSBQLVCFSOFUFTJPUPLFO ܕͷ4FDSFUͱͯ͠؅ཧ • ҎԼͷϑϥάΛઃఆͯ͠༗ޮԽ

    • LVCFBQJTFSWFSFOBCMFCPPUTUSBQUPLFOBVUI • LVCFDPOUSPMMFSNBOBHFS DPOUSPMMFST UPLFODMFBOFS CPPUTUSBQTJHOFS  #PPUTUSBQτʔΫϯͱ͸ https://kubernetes.io/docs/reference/access-authn-authz/bootstrap-tokens/
  16. ˜-:$PSQPSBUJPO • ,VCFMFU͸ΫϥΠΞϯτূ໌ॻΛ༻͍ͯ"1*4FSWFSͱN5-4Ͱ௨৴ • #PPUTUSBQUPLFO͸ىಈ࣌ͷΈ࢖༻ • ΫϥΠΞϯτূ໌ॻ͸$43"1*Λ༻͍ͯLVCFMFUࣗ਎Ͱੜ੒Մೳ • $43 $FSUJGJDBUF

    4JHOJOH 3FRVFTU • ,5)8Ͱ͸खಈͰੜ੒ • ैདྷΫϥελͰ͸༗ޮʹͳ͍ͬͯͳ͔ͬͨ • ,VCFBEN޲͚ʹ༗ޮԽ • LVCFDPOUSPMMFSNBOBHFSDMVTUFSTJHOJOHLFZGJMF DMVTUFSTJHOJOHDFSUGJMF  $43ͱ,VCFMFUͷ5-4#PPUTUSBQQJOH https://kubernetes.io/docs/reference/access-authn-authz/kubelet-tls-bootstrapping/
  17. ˜-:$PSQPSBUJPO • ,VCFBEN KPJO༻ͷઃఆϑΝΠϧ४උ • apiserver endpoint • CA ハッシュ

    • Bootstrapトークン • ͦͷଞඞཁͳௐ੔ • RBAC • Certificate Signing Request API • 設定⽤ConfigMap • …  ·ͱΊɿ8PSLFS/PEF௥Ճʹඞཁͳमਖ਼
  18. ˜-:$PSQPSBUJPO • ҟͳΔઃఆͷίϯϙʔωϯτ΁ߋ৽ • 3BODIFS SLF ˠ,VCFBEN • ϊʔυΛ̍ͭͣͭೖΕସ͑ •

    ࠷ऴతʹશͯͷϊʔυɾίϯϙʔωϯτΛߋ৽  ઓུɿϊʔυΛͭͣͭೖΕସ͑Δ ࠶ܝ
  19. ˜-:$PSQPSBUJPO • FUDE͸LVCFMFU TUBUJDQPEͱ࣮ͯ͠ߦ • )PTU/FUXPSLϞʔυ • ,VCFSOFUFT"1*ܦ༝Ͱ৘ใऔಘ • -BCFMͰFUDE

    1PETΛࣝผ • DPNQPOFOUFUDE • UJFSDPOUSPMQMBOF • ΤϯυϙΠϯτ͸1PEͷ"OOPUBUJPOʹ֨ೲ • kubeadm.kubernetes.io/etcd.advertise-client-urls  ैདྷͷ&UDEϝϯόʔ؅ཧ control-1 (IP: 10.0.0.1) containerd kubelet 通常Pod etcd Pod listen at 10.0.0.1:2379
  20. ˜-:$PSQPSBUJPO • ैདྷʢ3BODIFSʣ͸%PDLFSίϯςφͱͯ͠ FUDEΛ࣮ߦ • )PTUOFUXPSLϞʔυ • FUDE%PDLFSίϯςφ͸3BODIFS͕؅ཧ • ,VCFSOFUFT

    "1*͔Β͸FUDEίϯςφ͸ ૢ࡞औಘͰ͖ͳ͍  ैདྷͷ&UDEϝϯόʔ؅ཧ ؅ཧ control-1 (IP: 10.0.0.1) containerd kubelet 通常Pod Etcd コンテナ Docker listen at 10.0.0.1:2379
  21. ˜-:$PSQPSBUJPO ෇࿥ɿσΟετϦϏϡʔγϣϯຖͷҧ͍ ,VCFBEN ,5)8 3BODIFS3,& ,VCFMFU࣮ߦ؀ڥ 4ZTUFNE 6OJU౳ QBDLBHF౳ 4ZTUFNE

    6OJU TFMGCJOBSZ %PDLFSίϯςφ BQJTFSWFS౳࣮ߦ؀ڥ LVCFMFU TUBUJD QPE 4ZTUFNE 6OJU TFMGCJOBSZ %PDLFSίϯςφ LVCF QSPYZ࣮ߦ؀ڥ %BFNPOTFU %BFNPOTFU %PDLFSίϯςφ BQJTFSWFSˠFUDE௨৴ MPDBMIPTUͷΈ શϝϯόʔྻڍ MPDBMIPTUͷΈ ,VCFMFUˠBQJTFSWFS௨৴ -#౳ܦ༝ -# *1 "EESFTT ܦ༝ ௚઀ ,VCFMFU༻ূ໌ॻੜ੒ खಈੜ੒ ίϯτϩʔϥͰੜ੒