Upgrade to Pro — share decks privately, control downloads, hide ads and more …

自分だけの仮想クラスタを高速かつ効率的に作る kubefork

Kazuki Obata
February 25, 2025

自分だけの仮想クラスタを高速かつ効率的に作る kubefork

Wantedly Tech Night 〜ウォンテッドリーで内製している開発用ツールの紹介〜

https://wantedly.connpass.com/event/341010/

Kazuki Obata

February 25, 2025
Tweet

More Decks by Kazuki Obata

Other Decks in Technology

Transcript

  1. © 2025 Wantedly, Inc. ࣗݾ঺հ ڊി ࿨थ (Obata Kazuki) @donkomura_

    Wantedly, Inc. (2024/09 ~ ) ΠϯϑϥΤϯδχΞ #k8s #ϘϧμϦϯά🧗
  2. © 2025 Wantedly, Inc. ໨࣍ • Ϟνϕʔγϣϯ • ՝୊ͱ΍Γ͍ͨ͜ͱ •

    Ͳ͏΍࣮ͬͯݱ͢Δ͔ ◦ Wantedly ։ൃͷ৔߹ ◦ ΞΠσΟΞͱ޻෉ • ·ͱΊ
  3. © 2025 Wantedly, Inc. Ϟνϕʔγϣϯɿ։ൃதͷϓϨϏϡʔ • ؾܰʹϓϨϏϡʔͰ͖Δͱศར ◦ ຊ൪ͱಉ͡Α͏ͳ؀ڥͰಈ࡞֬ೝ͠ͳ͕Β։ൃͰ͖Δ ◦

    ϨϏϡʔ͠΍͍͢ Πϝʔδɿhttps://github.com/marketplace/actions/deploy-pr-preview ։ൃதʹ ࣅͨΑ͏ͳ͜ ͱ͕͍ͨ͠
  4. © 2025 Wantedly, Inc. ੔ཧɿݱঢ়ͷ։ൃϓϩηεͷτϨʔυΦϑ ͍͍ͱ͜औΓΛ͍ͨ͠ ݸਓͷ։ൃαʔόʔʢओʹϩʔΧϧʣ - ✅؀ڥ͕ଞͷӨڹʹΑͬͯԚ͞Εͳ͍ -

    ✅Docker ͳͲͰຊ൪ʹ͍ۙ؀ڥ͸࣮ݱͰ͖Δ - ❌Πϯϑϥ෦෼͢΂ͯΛ࠶ݱͰ͖ΔΘ͚Ͱ͸ ͳ͍ - ❌ґଘͨ͠αʔϏε͕͋Δͱ։ൃ͕೉͍͠ ڞ༗ͷ։ൃαʔόʔʢओʹϦϞʔτʣ - ✅ຊ൪ʹ͍ۙ؀ڥͰ։ൃͰ͖Δ - ❌ଞͷਓͷ࡞ۀ͔ΒӨڹΛड͚Δɾ༩͑Δ - ❌҆ఆͨ͠؀ڥΛอͪʹ͍͘ - յΕΔ͜ͱ͕͋Δ - ❌໰୊ͷಛఆ͕೉͘͠ͳΔ - σϓϩΠ͕িಥ͢Δ - ϩά͕ࠞࡏ͢Δ
  5. © 2025 Wantedly, Inc. Ͳ͏࣮ݱ͢ΔɿWantedly ։ൃͷ৔߹ લఏ • ϚΠΫϩαʔϏεʢΞʔΩςΫνϟʣ •

    Πϯϑϥ͸ Kubernetes ◦ 1Ϋϥελ60ݸͷϚΠΫϩαʔϏεΛӡ༻ • ։ൃऀ͸໿50ਓ
  6. © 2025 Wantedly, Inc. ޻෉ • deployment ͱ service ͚ͩมߋ͢Δ

    • ϧʔςΟϯά ◦ Ծ૝Ϋϥελ΁ͷΞΫηε৘ใΛ HTTP Header, gRPC Metadata Ͱ఻ൖ
  7. © 2025 Wantedly, Inc. ʮมߋ͍ͨ͠෦෼͚ͩʯઐ༻ • ΞϓϦέʔγϣϯ։ൃऀʹΑΔมߋ ◦ Stateless ͳϦιʔε͚ͩΛίϐʔ͢Ε͹े෼

    ◦ Kubernetes ͩͱ Deployment • Deployment ͷίϐʔʹىҼ͢Δมߋ ◦ ίϐʔͨ͠ Deployment ʹ޲͚ͯϦΫΤετΛྲྀ͍ͨ͠ ◦ Kubernetes Ͱ͸ Service Λ࢖ͬͯτϥϑΟοΫΛసૹ͢Δ
  8. © 2025 Wantedly, Inc. ʮมߋ͍ͨ͠෦෼͚ͩʯઐ༻ • deployment ͱ service ͷΈίϐʔ͢Δ

    ◦ มߋ͍ͨ͠ϚΠΫϩαʔϏεͷ Deployment ͱ Service Λίϐʔ ◦ มߋ͠ͳ͍ϚΠΫϩαʔϏε͸ڞ༻ͷ΋ͷΛ࢖͏Α͏ʹϧʔςΟϯά
  9. © 2025 Wantedly, Inc. • มߋ෦෼ʢService, DeploymentʣͷΈΛίϐʔ ◦ Ծ૝ΫϥελʹΞΫηε͢Δ৔߹ ▪

    ଞͷϚΠΫϩαʔϏε͔ΒͷϦΫΤετΛ͜ΕΒʹྲྀ͍ͨ͠ ▪ ϓϨϏϡʔͰ͖Δঢ়ଶʹ͍ͨ͠ Ͳ͏΍ͬͯίϐʔͨ͠ Service, Deployment ʹ ϦΫΤετΛྲྀ͔͢ ͜͜·Ͱ A B B
  10. © 2025 Wantedly, Inc. Istio ͷϦιʔεΛ࢖ͬͨϧʔςΟϯά • VirtualService ◦ ϚΠΫϩαʔϏε಺ͷαʔϏεؒ௨৴ͷϧʔςΟϯάΛઃఆ͢Δ

    resource ◦ header ͷ஋ΛجʹϦΫΤετͷϧʔςΟϯάઌΛมߋ͢Δ apiVersion: networking.istio.io/v1beta1 kind: VirtualService spec: hosts: - wantedly-x http: - match: - headers: x-fork-identifer: exact: donkomura route: - destination: host: kube-fork-donkomura x-fork-identifier
 ͱ͍͏ header Λݟͯ ࣗ෼ઐ༻ͷ host ʹ ϧʔςΟϯά͢Δ ίϐʔࡁΈͷ host
  11. © 2025 Wantedly, Inc. fork ʹ͓͚Δ VirtualService ӡ༻ͷ՝୊ • 1ͭͷ

    VirtualService ʹԾ૝Ϋϥελ͝ͱͷઃఆ͕ඞཁ • ྫ: 2ਓ͕ಉ͡αʔϏεΛมߋ͍ͨ͠ͱ͖ ◦ Ұͭͷ Virtual Service ʹ৚݅Λ·ͱΊΔ ◦ x-fork-identifer: cluster-A ʹϚον͢ΔͳΒ ▪ service-A ΁ϧʔςΟϯά ◦ x-fork-identifer: cluster-B ʹϚον͢ΔͳΒ ▪ service-B ΁ϧʔςΟϯ ◦ ͦΕҎ֎ͳΒ service-X ΁ϧʔςΟϯά
  12. © 2025 Wantedly, Inc. fork ʹ͓͚Δ VirtualService ӡ༻ͷ՝୊ • 1ͭͷ

    VirtualService ʹԾ૝Ϋϥελ͝ͱͷઃఆ͕ඞཁ • ྫ: 2ਓ͕ಉ͡αʔϏεΛมߋ͍ͨ͠ͱ͖ ◦ Ұͭͷ Virtual Service ʹ৚݅Λ·ͱΊΔ ◦ x-fork-identifer: cluster-A ʹϚον͢ΔͳΒ ▪ service-A ΁ϧʔςΟϯά ◦ x-fork-identifer: cluster-B ʹϚον͢ΔͳΒ ▪ service-B ΁ϧʔςΟϯ ◦ ͦΕҎ֎ͳΒ service-X ΁ϧʔςΟϯά ධՁॱΛ੍ޚͰ͖ͳ͍ ίϯϑϦΫτ͢Δ
  13. © 2025 Wantedly, Inc. VSConfig • VSConfig ◦ Ծ૝ΫϥελͷϧʔςΟϯάઃఆΛهड़͢ΔͨΊͷಠࣗϦιʔε ◦

    ࣮૷͸ΧελϜίϯτϩʔϥʔ ▪ ઃఆΛू໿ͯ͠ VirtualService Λੜ੒͢Δ ◦ VirtualService ͷઃఆΛෳ਺ਓ͕৮Βͳͯ͘ྑ͍ ▪ ίϯϑϦΫτ͕ى͖ͳ͍ ◦ յΕͯ΋ࣗಈͰݩʹ໭Δ VirtualService ΁ͷҟͳΔมߋΛू໿͢Δ apiVersion: vsconfig.k8s.wantedly.com/v1beta1 kind: VSConfig spec: headerName: x-fork-identifer headerValue: donkomura host: # ϧʔςΟϯάର৅ͷ Service wantedly-x service: # ϦΫΤετΛྲྀ͢ઌͷ Service kube-fork-donkomura
  14. © 2025 Wantedly, Inc. ։ൃதʹԾ૝ΫϥελʹϦΫΤετΛඈ͹͍ͨ͠ • ͜͜·Ͱ ◦ ಛఆͷ header

    ͷ͍ͭͨϦΫΤετ͸Ծ૝ΫϥελʹϧʔςΟϯά͞ΕΔ ◦ ϧʔςΟϯά͸ VirtualService ͰߦΘΕ͍ͯΔ ◦ VirtualService ಺ͷݸʑͷϧʔςΟϯάઃఆ͸ VSConfig ʹهड़͞Ε͍ͯΔ ͜ΕͰे෼͔ʁ
  15. © 2025 Wantedly, Inc. ϚΠΫϩαʔϏεʹ͓͚ΔϦΫΤετॲཧͷྲྀΕ A B C E ௨ৗͷϦΫΤετ

    req B E A B C E req ؒ઀తʹϦΫΤετΛड͚ͯ ڍಈ͕มΘΔ ίϐʔͨ͠ Deployment Ծ૝Ϋϥελ΁ͷ ϦΫΤετ
  16. © 2025 Wantedly, Inc. ʮԾ૝Ϋϥελ΁ͷϦΫΤετʯͱ͍͏ίϯςΩετ B E A B C

    E req Ծ૝Ϋϥελ΁ͷ ϦΫΤετ A B C E ௨ৗͷϦΫΤετ req ϦΫΤετΛ ίϐʔͨ͠ deploymen ΁޲͚͍ͨ
  17. © 2025 Wantedly, Inc. ʮԾ૝Ϋϥελ΁ͷϦΫΤετʯͱ͍͏ίϯςΩετ B E A B C

    E req A B C E req ϦΫΤετΛ ίϐʔͨ͠ deploymen ΁޲͚͍ͨ Ծ૝Ϋϥελ΁ͷ ϦΫΤετ ௨ৗͷϦΫΤετ
  18. © 2025 Wantedly, Inc. ʮԾ૝Ϋϥελ΁ͷϦΫΤετʯͱ͍͏ίϯςΩετ B E A B C

    E req A B C E req ϦΫΤετΛ ίϐʔͨ͠ deploymen ΁޲͚͍ͨ Ծ૝Ϋϥελ΁ͷ ϦΫΤετ ௨ৗͷϦΫΤετ ʮԾ૝Ϋϥελ΁ͷΞΫηεͰ͋Δʯ ͱ͍͏৘ใʢίϯςΩετʣΛ ޙଓͷϚΠΫϩαʔϏε΁ ఻ൖͤ͞Δඞཁ͕͋Δ
  19. © 2025 Wantedly, Inc. Ծ૝ΫϥελͷϦΫΤετͰ͋Δ͜ͱΛ఻ൖ͢Δ • ͜͜·Ͱʢ࠶ܝʣ ◦ ಛఆͷ header

    ͷ͍ͭͨϦΫΤετ͸Ծ૝ΫϥελʹϧʔςΟϯά͞ΕΔ ◦ ϧʔςΟϯά͸ VirtualService ͰߦΘΕ͍ͯΔ ◦ VirtualService ͷ؅ཧ͸ VSConfig ͕ߦ͍ͬͯΔ Ծ૝Ϋϥελͷ ID Λ࣋ͭ header Λ͚ͭͯ ϚΠΫϩαʔϏε಺Ͱ࢖͏ɾ఻ൖͤ͞Δ → ϦΫΤετ͕ίϐʔͨ͠ Deployment ΁ → Ծ૝తͳΫϥελʹΞΫηε͢Δମݧ
  20. © 2025 Wantedly, Inc. • ಛఆͷ ID ͷ෇͍ͨϦΫΤετ (e.g. header)

    Λड͚Δ ◦ ड͚ͨϚΠΫϩαʔϏε͸ผͷϚΠΫϩαʔϏε΁ϦΫΤετ ◦ ͦͷࡍʹಉ͡ ID Λ͚ͭͯϦΫΤετΛൃߦ͢Δ Ծ૝ΫϥελͷIDΛ఻ൖ͢Δ A B C E req id: hoge E req id: hoge
  21. © 2025 Wantedly, Inc. • ಛఆͷ ID ͷ෇͍ͨϦΫΤετ (e.g. header)

    Λड͚Δ ◦ ड͚ͨϚΠΫϩαʔϏε͸ผͷϚΠΫϩαʔϏε΁ϦΫΤετ ◦ ͦͷࡍʹಉ͡ ID Λ͚ͭͯϦΫΤετΛൃߦ͢Δ Ծ૝ΫϥελͷIDΛ఻ൖ͢Δ req id: hoge req id: hoge req id: hoge header Λ఻ൖ͞ ͓ͤͯ͘ A B C E req id: hoge E
  22. © 2025 Wantedly, Inc. Context propagation • ʮԾ૝ΫϥελAʹདྷͨϦΫΤετͰ͋Δʯͱ͍͏ίϯςΩετΛ ϚΠΫϩαʔϏεʹ఻ൖ (propagate)

    ͢Δ ◦ ྫ ▪ X-Fork-Identifer: fork-a ͷΑ͏ͳ header ෇͖Ͱ request ͷॲཧத ▪ ผͷ microservice ʹ௨৴Λߦ͏৔߹͸ඞͣ X-Fork-Identifer: fork-a Λ෇͚Δ • servicex ͱ͍͏ϚΠΫϩαʔϏεڞ௨ϥΠϒϥϦͰ࣮ݱ ◦ wrap ͨ͠ΫϥΠΞϯτΛ࢖͏ ◦ ଞ΁ͷϦΫΤετ࣌ʹಉ͡ϔομɾϝλσʔλΛ࢖͏ ◦ ΋͏গ͠஌Γ͍ͨ > Wantedly Engineering Handbook
  23. © 2025 Wantedly, Inc. ·ͱΊ • kubefork ͸Ծ૝ΫϥελΛ࡞੒͢Δ࢓૊Έ ◦ ։ൃऀ͕ࣗ෼ઐ༻ͷΫϥελΛ͍࣋ͬͯΔ͔ͷΑ͏ͳମݧΛఏڙ͢Δ

    • ΠϯϑϥͱΞϓϦέʔγϣϯͷٕज़Λ૊Έ߹Θͤͯ Ͱ͖͍ͯΔ ◦ Kubernetes (VirtualService, VSConfig etc.), Istio ◦ servicex