Copy Kubernetes Clusters Really Fast

©2021 Wantedly, Inc.
Kubernetes Cluster Λ Copy
ޙΖଆͷ Microservice ͷ Review ؆୯ʹ
CloudNative Days Spring 2021 ONLINE
Mar 11, 2020 - Shimpei Otsubo @potsbo

View Slide

Shimpei Otsubo @potsbo
Developer eXperience Squad
Wantedly, Inc.
Kubernetes
Dev Tools
CI / CD
AuthN / Z
Productivity Dvorak

View Slide

Developer eXperience Squad ͱ͸ʁ
ྑ͍։ൃମݧ͕ྑ͍ϓϩμΫτͱձࣾΛ࡞Δ
։ൃମݧͷͨΊͳΒԿͰ΋͢Δ
։ൃ࣌ؒͷ30%ఔ౓ͰϓϩμΫτ։ൃʹճΔ
Backend, Frontend, Infrastructure, Mobile ͳͲ෼໺Λ໰Θͳ͍
੔උͨ͠ج൫ͷυοάϑʔσΟϯά / ʮ։ൃऀ͕ຊ౰ʹٻΊ͍ͯΔ΋ͷ͸ͳʹ͔?ʯΛߟ͑Δ
֎෦ʹ͸ٕज़ج൫νʔϜͱ঺հ͢Δ͜ͱ΋ଟ͍
ߴ଎͔ͭ҆શͳ deploy ͸ϓϩμΫτͱ૊৫Λ੒௕ͤ͞Δ
ٕज़౤ࢿͱ໨ͷલͷϓϩμΫτͷ༏ઌॱҐΛߟ͑ΔλΠϛϯάΛ࣋ͪଓ͚Δ
ίʔυΛॻ͔ͳ͍ղܾࡦ΋ࢹ໺ʹೖΕΔ
10೥ޙͷ։ൃͷৗࣝΛ࡞͍͚ͬͯΔνʔϜʹ͍ͨ͠

View Slide

TL;DR
Istio + Ambassador Λ࢖ͬͯ
Ծ૝తͳ Kubernetes ͷ copy Λ࣮ݱ
͜ΕʹΑͬͯ։ൃମݧ͕ΨϥοͱมΘΔ

View Slide

ߴ଎Ͱޮ཰తͳΫϥελ copy ͕͍ͨ͠
Copy on Write ͰԾ૝ΫϥελΛ࡞Δ
Ԡ༻ͱಘΒΕͨମݧ
WHY
HOW
WHAT

View Slide

WHY
Microservice ͷґଘղܾΛ͠ͳͯ͘ྑ͍
ߴ଎Ͱޮ཰తͳΫϥελ copy ͕͍ͨ͠
Preview URL Λ࡞ΕΔ
Photo by Jørgen Håland on Unsplash

View Slide

ϙνϙν͔ͯ֬͠Ί͍ͨ͜ͱ͸Α͋͘Δ
UI มߋͳΒ΋ͪΖΜ
ΞϧΰϦζϜͷมߋ΋࣮σʔλͰݟͯΈ͍ͨ

View Slide

Preview Deployment ͸ੲ͔ΒΑ͋͘Δ
https://vercel.com/docs/git/vercel-for-github
Vercel, Heroku, Netlify
νʔϜશһͰಉ͡΋ͷΛݟΕΔ
ຊ൪ʹ͍ۙঢ়ଶ
ϫϯΫϦοΫͰ֬ೝ
࠷ۙͷ PaaS ʹ͸ී௨ʹ͋Δػೳ
ىಈํ๏Λ஌Βͳ͍͍ͯ͘
ΞϓϦέʔγϣϯͷશମ૾͢Β஌Βͳ͍͍ͯ͘
։ൃ build Ͱ͔͠ى͖ͳ͍໰୊ʹϋϚͬͨΓ͠ͳ͍
ԶͷPCͰ͸ಈͧ͘ʁͱ͔ى͜Βͳ͍
࣮σʔλΛ࢖͑ͨΓ͢Δ
Kubernetes ্Ͱ࣮ݱ͢Δ OSS ΋͋Δ

View Slide

Microservice Ͱ͸؆୯Ͱ͸ͳ͍
ཪଆͷ Microservice ʹͲ͏΍ͬͯ౸ୡ͢Δ͔ʁ
௚઀ཪଆʹܨ͍Ͱ΋ integration ͨ͠ঢ়ଶͰͷ test ͕Ͱ͖ͳ͍
Wantedly Ͱ͸΋ͱ΋ͱ͜ͷχʔζʹ͸ڞ༗؀ڥʹ deploy ͢Δ͜ͱͰ࿫͍ͬͯͨ
Ұ൪ front ʹཱͭ service ͳΒ؆୯ ޙΖଆʹཱͭ service ʹ͸؆୯ʹ౸ୡͰ͖ͳ͍
ޙΖଆ͚͕ͩҧ͏ front ͷ microservice ΛͲ͏΍ͬͯ࡞Γग़͔͢ʁ
͜͜ͷ Routing Ͳ͏΍Δͷʁ

View Slide

Microservice Ͱ͸؆୯Ͱ͸ͳ͍
ඞཁͳ෼͚ͩ copy ͠Α͏ͱ͢Δͱͦͷґଘղܾ͕ͦ΋ͦ΋େม
Microservice ͷ interface ͷઌΛҙࣝͨ͘͠ͳ͍
Wantedly Ͱ͸΋ͱ΋ͱ͜ͷχʔζʹ͸ڞ༗؀ڥʹ deploy ͢Δ͜ͱͰ࿫͍ͬͯͨ
͜Ε͕ࣗ໌ʹͳΔΑ͏ͳ architecture Λ໨ࢦ͢΂͖ͱ͍͏࿦΋͋Δ

View Slide

Kubernetes Cluster Λ copy ͢Ε͹ ok!
ґଘؔ܎͸উखʹ͢΂ͯຬͨ͞ΕΔ
ཪଆ͚ͩҧ͏ঢ়ଶΛ׬શʹ࠶ݱ
Wantedly ͷ৔߹͸શ microservice 140ݸ͕ 1 cluster ʹଘࡏ
Microservice Ͱ΋ Preview ͕Ͱ͖ΔΑ͏ʹͳΔ
https://speakerdeck.com/morux2/istio-integrated-telepresence
https://event.cloudnativedays.jp/cndt2020/talks/17
͜ͷลͷϞνϕʔγϣϯ͸લճৄ͘͠

View Slide

Kubernetes Cluster Λ copy ͢Ε͹ ok!
ຊ౰ʹͦΜͳ͜ͱͰ͖Δͷ͔
Wantedly ͷ৔߹͸શ microservice 140ݸ͕ 1 cluster ʹଘࡏ

View Slide

Context Propagation
Service Mesh
Gateway
Header Λ఻ൖ
Header Ͱ Routing
Host Λ Header ʹม׵
HOW Copy on Write Ͱ Virtual Copy
Photo by Alina Grubnyak on Unsplash

View Slide

Ծ૝తͳΫϥελ copy πʔϧ kubefork Λ࡞ͬͨ
ڞ༗Ͱ͖Δͱ͜Ζ͸ڞ༗
Copy on Write
͢΂ͯ Copy ͢Δͱ N * D
ඞཁͳͱ͜Ζ͚ͩͩͱ N + D
ޮ཰త
ҟͳΔ෦෼͚ͩஔ͖׵͑
N: microservice ਺, D: ։ൃऀ਺
4 microservice ߏ੒ʹ2ͭͷ copy Λੜ੒ͨ͠৔߹
શ෦࣮ copy ͩͱ node ଴͕ͪൃੜ
ߴ଎
OSS ʹ͍͚ͨ͠Ͳؒʹ߹ͬͯͳ͍
Deployment ͱ service ͷΈ

View Slide

Terminology
Ծ૝తʹ copy ͞ΕͨΫϥελ
Virtual Cluster
Virtual Cluster Λಛఆ͢Δ id
Fork Identifier
Ұͭͷ fork identifier Λ࣋ͭ
Shared ͳ cluster Λ΋ͱʹ
Fork ͢Δ
Virtual Cluster Λੜ੒͢Δ

View Slide

Wildcard DNS Ͱ upstream ΛଋͶΔ
Gateway
Request Λద੾ͳ copy ઌ΁ૹΔ
Service Mesh
ߏ੒ཁૉ
Context Propagation
fork identifier Λ header ͱͯ͠఻ൖ
Ͳͷ Virtual Cluster ༻ͷ request?
ମײ͸৽͍͠ host ΁ͷΞΫηε

View Slide

Context Propagation
ʮͲͷ Virtual Cluster ʹ޲͚ΒΕͨ΋ͷ͔ʯΛRequest ͝ͱʹ؅ཧ
e.g. X-Fork-Identifier: forked-a
ࣾ಺ڞ௨ϥΠϒϥϦ servicex Ͱ࣮ݱ
Feature Flag ͳͲԠ༻͸৭ʑ
X-Feature-Flags: {"name": "new_feature?": "enabled": true}
e.g. ޙΖଆͷ microservice ͷ new-feature Λ༗ޮʹͨ͠ঢ়ଶ͕ݟ͍ͨͱ͖ʹ
HTTP header Λ఻ൖ͢Δ͜ͱͰ request ͝ͱͷॲཧΛ؆୯ʹ௥ՃͰ͖ΔΑ͏ʹͳΔ
ಛఆͷ Prefix Λ࣋ͭ header / metadata ΛࣗಈͰ఻ൖ
ޙड़
Ͳͷ Microservice ؒ௨৴Ͱ΋ʮѼઌClusterʯ͕໌ࣔతʹͳΔ
https://www.wantedly.com/companies/wantedly/post_articles/311135

View Slide

༨ஊ: Metadata ͷ࢖͍͗͢ʹ஫ҙ
఻ൖ͍ͨ͠΋ͷ͸͍ͬͺ͍͋Δ
UserId, Country, Language ͳͲ͸ޙΖଆͷ server Ͱཉ͘͠ͳΔ͜ͱ΋Α͋͘Δ
಺෦޲͚: wantedly/dev#412
్தܦ࿏ͷ͢΂ͯͷ schema ʹੜ΍͍ͯ͘͠Έ͍ͨͳͷ͸ਏ͍
৴༻Ͱ͖Δ஋͔Θ͔Βͳ͍
Schema ʹݱΕʹ͘͘ͳΔ
Debug ໨త͸ڐ༰
ʮProto ͷ௨Γʹ request ͨ͠Β header ͕଍Γͣʹ஄͔ΕΔʯͱ͔͕ىͬͯ͜͠·͏
৴༻͍ͨ͠৔߹͸֎෦͔Β inject Ͱ͖ͳ͍࢓૊Έ͕ඞཁ
ʮX-Current-User-Id Λ΋ͱʹೝՄͯ͠Α͍ʯͱ͍͏ policy ͸͔ͳΓ৻ॏʹ
Metadata ΋ proto ʹ serialize ͢Δͱ͍͏ΞΠσΞ΋͋Δ(ະ࣮૷)
ຊ൪Ͱ͸੾Ε͹ྑ͍͠ɺabuse ͞Εͯ΋க໋తͰ͸ͳ͍

View Slide

Service Mesh
ࠓ͍Δ Virtual Cluster ʹ߹Θͤͨ routing
Header ʹԠͯ͡ request ઌΛม͑Δ
e.g. X-Fork-Identifier: forked-a
֎͔ΒݟΔͱ࣮࣭ copy
ڞ༗͍ͯ͠Δ microservice ΋ෳ਺͋ΔΑ͏ʹݟ͑Δ
ͬͪ͜ͷ path Λ௨͍ͬͯΔΑ͏ʹײ͡Δ
ͨͩ͠Ӆṭ͕ॏཁ
Service mesh Λҙࣝ͢Δͱ copy ʹײ͡ͳ͘ͳΔ
͋͑ͯʮcopy ͕Ͱ͖Δʯ఻͑ͨ΄͏͕࢖ΘΕΔ
࢓૊Έ͔Βઆ໌͢Δͱ೉͍͠ͱײ͡ΒΕͯ͠·͏

View Slide

Gateway
Host ʹ߹Θͤͯ header Λ෇༩
Wildcard DNS ͕޲͍ͨ proxy
ʮCopy ͞ΕͯΔײʯʹ͸௒ॏཁ
fork-a.example.com Έ͍ͨͳ URL ͕ൃߦ͞ΕΔ
֎͔ΒݟΔͱʮcopy ͞Ε࣮ͨଶʯ
ʮURL ͕ൃߦͰ͖Δʯ͜ͱ͕ϝϯλϧϞσϧߏஙʹྑ͍
Wantedly Ͱ΋͜ͷػೳ͕Ͱ͖Δ·Ͱܟԕ͞Ε͕ͪͩͬͨ
Routing ͷ೉͠͞ΛӅṭ
͜͜ʹ઀ଓ͍ͯ͠Δؾ࣋ͪʹͳΕΔ
URL ͕͋Δͱڞ༗͕Ͱ͖Δ
ࣗ෼ͷมߋΛଞͷਓʹ৮ͬͯ΋Β͑Δ
Mobile ͔ΒମݧΛ͔֬ΊΔ

View Slide

2020೥9݄࣌఺Ͱ͸ Chrome Extension ͷΈͷରԠͩͬͨ
ࠓͲ͜ʹܨ͍Ͱ͍Δͷ͔ʁ͕෼͔ΓͮΒ͔ͬͨ
https://speakerdeck.com/morux2/istio-integrated-telepresence?slide=44
Gateway
Host ʹ߹Θͤͯ header Λ෇༩
Wildcard DNS ͕޲͍ͨ proxy
ʮCopy ͞ΕͯΔײʯʹ͸௒ॏཁ
fork-a.example.com Έ͍ͨͳ URL ͕ൃߦ͞ΕΔ
֎͔ΒݟΔͱʮcopy ͞Ε࣮ͨଶʯ
ʮURL ͕ൃߦͰ͖Δʯ͜ͱ͕ϝϯλϧϞσϧߏஙʹྑ͍
Wantedly Ͱ΋͜ͷػೳ͕Ͱ͖Δ·Ͱܟԕ͞Ε͕ͪͩͬͨ
Routing ͷ೉͠͞ΛӅṭ
URL ͕͋Δͱڞ༗͕Ͱ͖Δ
ࣗ෼ͷมߋΛଞͷਓʹ৮ͬͯ΋Β͑Δ
Mobile ͔ΒମݧΛ͔֬ΊΔ
Chrome Ҏ֎ͰରԠͰ͖ͳ͔ͬͨ

View Slide

Context Propagation
Service Mesh
Gateway
Header Λ఻ൖ
Header Ͱ Routing
HOW Copy on Write Ͱ Virtual Copy

View Slide

࣮૷
apiVersion: vsconfig.k8s.wantedly.com/v1beta1

kind: Fork

metadata:

name: wantedly-some-identifier

namespace: wantedly

spec:

identifier: some-identifier

manager: fork-proxy-ambassador/default

services:

selector:

matchLabels:

role: web

deployments:

selector:

matchLabels:

newrelic: "true"

template:

metadata:

labels:

app: some-identifier

role: fork

annotations:

wantedly.com/deploy-target: "false"

spec:

containers:

- image: example:tag

name: wantedly
CRD Ͱ atomic ͳॲཧʹݟͤΔ

View Slide

࣮૷

kind: Fork

metadata:


namespace: wantedly

spec:



services:

selector:

matchLabels:

role: web

deployments:

selector:

matchLabels:

newrelic: "true"

template:

metadata:

labels:


role: fork

annotations:


spec:

containers:


name: wantedly
some-identifer ͱ͍͏ id Ͱ cluster ൃߦΛґཔ

View Slide

࣮૷

kind: Fork

metadata:


namespace: wantedly

spec:



services:

selector:

matchLabels:

role: web

deployments:

selector:

matchLabels:

newrelic: "true"

template:

metadata:

labels:


role: fork

annotations:


spec:

containers:


name: wantedly
Namespce ಺Ͱ copy ͢Δ΂͖ service Λࢦఆ

View Slide

࣮૷

kind: Fork

metadata:


namespace: wantedly

spec:



services:

selector:

matchLabels:

role: web

deployments:

selector:

matchLabels:

newrelic: "true"

template:

metadata:

labels:


role: fork

annotations:


spec:

containers:


name: wantedly
ͦͷ service ͔Β route ͞ΕΔ deploy Λ list ͯ͠ߋʹ੍ݶ

View Slide

࣮૷

kind: Fork

metadata:


namespace: wantedly

spec:



services:

selector:

matchLabels:

role: web

deployments:

selector:

matchLabels:

newrelic: "true"

template:

metadata:

labels:


role: fork

annotations:


spec:

containers:


name: wantedly
ͦͷ service ͔Β route ͞ΕΔ deploy Λ list ͯ͠ߋʹ੍ݶ
pod template ͷ patch Λ copy ʹ౰ͯΔ

View Slide

࣮૷
CRDͰදݱ
Gateway Service Mesh
Copy ੜ੒
CRD͔ΒશͯΛੜ੒

View Slide

࣮૷
apiVersion: getambassador.io/v2

kind: Mapping

metadata:

labels:

fork.k8s.wantedly.com/manager: default

name: some-dev-wantedly-com-some-identifier

namespace: fork-proxy-ambassador

spec:

add_request_headers:

x-fork-identifier: some-identifier

ambassador_id:

- fork-proxy-ambassador

host: some-identifier.some-dev.wantedly.com

prefix: /

service: https://some-dev.wantedly.com
Gateway ͱͯ͠ͷ Mapping ੜ੒
఻ൖ͢Δ header Λ௥Ճ
ಛఆͷ host ͱͯ͠ request ͕དྷͨΒ
ਖ਼͍͠ upstream ʹ઀ଓ

kind: ForkManager

metadata:

name: default

namespace: fork-proxy-ambassador

spec:

upstreams:

- host: "some-dev.wantedly.com"

- host: "another-dev.wantedly.com"

- host: "yet-another-dev.wantedly.com"

- host: …
Cluster ΁ͷೖΓޱΛ༧Ί list ͓ͯ͘͠
Upstreams * Virtual Cluster ͷ਺ͷ mapping ͕ੜ੒͞ΕΔ
ಉ͜͡ͱ͕Ͱ͖Ε͹ Ambassador ͡Όͳͯ͘΋͍͍

View Slide

WHAT
Telepresence ͠์୊
Copy ͞Εͨ virtual cluster Ͱ͸ԿΛ΍ͬͯ΋୭ʹ΋໎࿭Λ͔͚ͳ͍
Feature Flag ͝ͱʹ؆୯ʹ URL Λൃߦ
Pull Request Preview URL
ͲΜͳʹޙΖଆͷ Microservice Ͱ΋ ok
Pull Request Λ࡞ͬͨΒͦΕ͕ Preview Ͱ͖Δ URL ͕ൃߦͰ͖Δ
new-feature-released.dev.example.com Έ͍ͨͳ URL ͕؆୯ʹ࡞ΕΔ
Virtual cluster ಺ͷ͢΂ͯͷ request Λ local ʹૹͬͯ΋େৎ෉
͔͠΋௥Ճͷܭࢉࢿݯ͕Ұ੾͍Βͳ͍
Photo by SpaceX on Unsplash
Ԡ༻ͱಘΒΕͨମݧ

View Slide

Telepresence ͠์୊
Virtual cluster Ͱ͸ԿͰ΋͋Γ
ࣗ෼ઐ༻ͷ؀ڥͩͱײ͡ΒΕΔ
Telepresence ͯ͠େৎ෉
ڞ༗ΫϥελͰ࢖͏ͱଞͷਓͷ request ΋ٵ͍ࠐΉ
όάͬͨ΋ͷΛىಈͯ͠΋େৎ෉
https://speakerdeck.com/morux2/istio-integrated-telepresence?slide=33
Virtual ͳΒࣗ෼ͷ request ͚ͩͳͷͰ҆શ
͜ͷ༻్ʹ͍ͭͯ͸લճͷൃද͕ৄ͍͠

View Slide

αʔόʔͷ೚ҙͷڍಈΛ override Ͱ͖Δج൫Λ͍࣋ͬͯͨ
શͯͷ AB test
໌ࣔతʹ annotate ͨ͠΋ͷ
Rails ͷ ೚ҙͷHelper ͷ೚ҙͷ method
X-Feature-Flags: {"name": "new_feature?": "enabled": true}
Header ΛૹΓ͚ͭΔ͚ͩͰྑ͍
఻ൖ΋͞Ε͍ͯΔͷͰޙΖଆͷ microservice ͷڍಈ΋ม͑ΒΕΔ
Assets ͚ͩखݩͷ webpack-dev-server Λ࢖͏
༗༻ͳ Debug πʔϧ
AB test ͷಛఆͷ variant Ͱදࣔ͢Δ
ϦϦʔεલͷػೳΛ৮Δ
Host / Header ม׵ج൫ͱͯ͠ͷ Gateway Λ༗ޮར༻
https://speakerdeck.com/potsbo/wrap-every-method-with-just-one-line
؆୯ʹ Header Λ௥ՃͰ͖Δ Chrome Extension Λ࡞͍ͬͯͨ

View Slide

Gateway ͱ integrate
some.wantedly.com
some-dev.wantedly.com
new-feature-released.some-dev.wantedly.com
ຊ൪
։ൃ
NEW!!
Host / Header ม׵ج൫ͱͯ͠ͷ Gateway Λ༗ޮར༻

kind: Fork

metadata:

name: new-feature-enabled

namespace: wantedly

spec:

identifier: new-feature-enabled


gatewayOptions:

addRequestHeaders:

x-feature-flags: '{"name": "new_feature?": "enabled": true}'
Feature Flag Header Λૠೖ
Resourceফඅ͕օແ
Virtual Cluster ͸෷͍ग़͞ΕΔ
͜ͷͨΊ URL ΋෷͍ग़͞ΕΔ
͔͠͠৽نͷ pod ͸Ұ੾ͳ͍

View Slide

Microservice Ͱ΋ Preview Ͱ͖Δ
ͲΜͳʹޙΖଆͷ Microservice Ͱ΋࣮ࡍͷ UI ͔Β࣮֬ʹ౸ୡͰ͖Δ
※·ͩ׬੒ͯ͠ͳ͍ͷͰΠϝʔδͰ͢…
σβΠφͱͷ Review ΋εϜʔζʹͳΔ
Microservice ؔ܎ͳͯ͘΋ศར
GitHub ͷ webhook ʹ߹Θͤͯ fork Λ࡞Δ͚ͩ

View Slide

GitHub ͷ webhook ʹ߹Θͤͯ fork Λ࡞Δ͚ͩ

View Slide

࠷ޙʹ༨ஊ
ج൫͸ੵΈ্͛
Context Propagation ͸ࣾ಺ڞ௨ϥΠϒϥϦ͕͔͋ͬͨΒ࡞Εͨ
఻ൖ͸΋ͱ΋ͱ͸ Feature Flag ༻ͷج൫ͩͬͨ΋ͷΛ࢖͍·Θͨ͠
Istio / Ambassador ͕े෼ݕূ͞ΕͯಋೖࡁΈͩͬͨ
ݹ͍αʔϏε΋શͯ Kubernetes ʹҠߦ͖͍ͬͯͨ͠
Deployment Λ Copy ͢Δ CRD ͕࡞੒ࡁΈͩͬͨ
શ෦ Terraform ͔ͩͬͨΒূ໌ॻ/DNSΛҰؾʹ૿΍ͤͨ
…

View Slide

Context Propagation
Service Mesh
Gateway
Header Λ఻ൖ
Header Ͱ Routing
Կ͔ͱ͋Δͱศརͳ3ͭͷج൫

View Slide

ʮCluster Copy ͱ͍͏ந৅Խʯ͕͋Δͱ
։ൃ΋ϨϏϡʔ΋QA΋ָʹͳΔ
·ͱΊ
ج൫ΛੵΈ্͍͛ͯ͜͏

View Slide

Copy Kubernetes Clusters Really Fast

Copy Kubernetes Clusters Really Fast

Shimpei Otsubo

More Decks by Shimpei Otsubo

Other Decks in Technology

Featured

Transcript