Slide 1
Slide 1 text
Takeshi Yoneda, Software Engineer, Tetrate.io
Kubernetes Meetup Tokyo #40
WasmͰ͕ΔEnvoyͱIstioͷੈք
Slide 2
Slide 2 text
• Takeshi Yoneda (Ϛελέ) / Twitter, Github: @mathetake
• Software Engineer at Tetrate.io
• “Paid” OSS dev: Envoy, Istio, Proxy-Wasm, TinyGo
• C++ committer of Proxy-Wasm project
• Creator of Go SDK for Proxy-Wasm
• Contributor of V8
whoami
Slide 3
Slide 3 text
Software Design 20213݄߸ʹࠓͷʹ͍ͭͯͷهࣄΛدߘ͠·ͨ͠ʂ
Slide 4
Slide 4 text
1. Introduction to WebAssembly
2. Background: Envoy’s extensibility
3. Proxy-Wasm: WebAssembly For Proxies
4. Proxy-Wasm in Istio
5. Challenges and Future
Agenda
Slide 5
Slide 5 text
1. Introduction to WebAssembly
Slide 6
Slide 6 text
Wasm = CNCF’s tech to watch in 2021
Slide 7
Slide 7 text
• Stack-basedͳԾϚγϯͱͦͷ༷
• ݩʑϒϥβ(JS)ͷߴԽ͕త
• asm.js -> WebAssembly(Wasm)ͱਐԽ
• ༷ΛಡΊ͔Δ͕Ұݴ “host” ͷཁٻ͕ͳ͍
• Portable, Platform-agnostic, Open-ended
• Run at near-native speed: ΊͬͪΌ͍(※࣮ʹΑΔ)
• Security: ελοΫ͕ϓϩάϥϜ͔Βݟ͑ͳ͍ͱ͔ͦ͏͍͏ͷ
WebAssembly 101
Slide 8
Slide 8 text
• ༷ʑͳݴޠ͔ΒίϯύΠϧՄೳ: C, C++, Rust, Go(TinyGo), AssemblyScript
• ॳjsͷΈࠐΈ͕લఏ: ͦΕͧΕͷݴޠ͕ಠࣗͷ “glue.js”Λ࣋ͭ
• ίϯύΠϥڞ௨ͷ“Platform”λʔήοτ͕ͳ͍(͍, jsͳΜ͚ͩͲ͞, Έ͍ͨͳ)
• VMͱͯ͠༏ल&ίϯύΠϥج൫͋Δͷʹମͳ͍
• ϒϥβͷ֎Ͱ͍͍ͨ
WebAssembly 101
Slide 9
Slide 9 text
• Wasm <-> Host OSͷ͠ํ(ΠϯλʔϑΣΠε)=SystemcallΛඪ४Խ͠·͠ΐ͏
• WasmͷίϯύΠϥͷ“platform”λʔήοτʹ͠Α͏
• WASI (WebAssembly System Interface)ͷొ
WebAssembly 101
https://hacks.mozilla.org/2019/03/standardizing-wasi-a-webassembly-system-interface/
Slide 10
Slide 10 text
• WASIΛ࣮ͨ͠ϥϯλΠϜ͕ొ
• WAVM, Wasmtime, Wasmer, Lucet, ʑ
• “ී௨ͷϓϩηε”ͱಉ͡Α͏ʹWasm͕ಈ͘Α͏ʹͳΔ
• KrustletͳͲͷ৺తϓϩδΣΫτͷొ
Wasm gets out of web browsers
Slide 11
Slide 11 text
• WASI = Wasmͷ༷ʹ͋Δ “host function” Λsystem call͚ʹඪ४Խͨ͠ͷ
• ݁ہABI (Wasm <-> Hostͷ͠ํ)ܾ͑͞ΊΕͳΜͰͰ͖Δ
• ҙͷΞϓϦͷதͰVMΛಈָ͔͍ͯ͜͠͠ͱ͕Ͱ͖ͦ͏ʂ
• WASIʹݶΒ༷ͣʑͳABI͕ొ
• Blockchain༻ͷABI
• k8s֦ு༻ͷABI
• Proxyαʔό༻ͷABI(Proxy-Wasm)
Wasm gets out of web browsers
WASI ABI
Linux / Darwin / Windows / …
Wasm Virtual Machine
Slide 12
Slide 12 text
• WASI = Wasmͷ༷ʹ͋Δ “host function” Λsystem call͚ʹඪ४Խͨ͠ͷ
• ݁ہABI (Wasm <-> Hostͷ͠ํ)ܾ͑͞ΊΕͳΜͰͰ͖Δ
• ҙͷΞϓϦͷதͰVMΛಈָ͔͍ͯ͜͠͠ͱ͕Ͱ͖ͦ͏ʂ
• WASIʹݶΒ༷ͣʑͳABI͕ొ
• Blockchain༻ͷABI
• k8s֦ு༻ͷABI
• Proxyαʔό༻ͷABI(Proxy-Wasm)
Wasm gets out of web browsers
Proxy-Wasm ABI
Envoy / Nginx / ATS / …
Wasm Virtual Machine
Slide 13
Slide 13 text
2. Background: Envoy’s extensibility
Slide 14
Slide 14 text
What is Envoy?
Slide 15
Slide 15 text
• “Cloud-native high-performance edge/middle/service proxy”
• CNCF Graduated Project, Github Star: 16,000+
• Written in C++
• αʔϏεϝογϡͷData planeͱͯ͠།Ұແೋͷଘࡏ
What is Envoy?
Slide 16
Slide 16 text
• ϓϩΩγαʔόʔͳͷͰuse case͕ແݶ
• EnvoyʹવϓϥάΠϯ(֦ு)ػߏ͕ଘࡏ
• ֦ுͷྫ
• ࣾಠࣗͷAuthn/Authz ػೳΛೖΕ͍ͨ
• ಛघͳϓϩτίϧΛѻ͍͍ͨ
• MySQL, Redis, DynamoDB, etc.
Envoy’s extensibility
Slide 17
Slide 17 text
Envoy’s extensibility
Slide 18
Slide 18 text
• Envoy੩తϦϯΫ͕લఏ
• C++Ͱॻ͔ͳ͍ͱ͍͚ͳ͍
• มߋөͷͨΊʹશ࠶ىಈ͕ඞཁ
• StableͳABI͕ܾ·͍ͬͯͳ͍
• ֦ுػೳΛՃ͢Δʹ
1. UpstreamϚʔδ͢Δ —> ಛघ(private)ͳϢʔεέʔεෆՄ
2. ಠࣗϏϧυΛ͢Δ —> upstreamͷै͕େม(෦ͷAPI͕unstable)
Envoy’s extensibility
Slide 19
Slide 19 text
• ղܾ͍ͨ͠՝ҎԼͷ௨Γ
• ಈతʹϓϥάΠϯͷload͕Ͱ͖Δ
• ෳݴޠͷαϙʔτ
• EnvoyࣗମͷಠࣗϏϧυඞཁͳ͍
• Security-awareͳ༷
Envoy’s extensibility
Slide 20
Slide 20 text
• ͜ΕΒͷཁIstioͷνʔϜ͔Β্͕͍ͬͯͨ
• ~v1.4ͷMixerγεςϜͷෛ࠴
• latencyIstioࣗମͷෳࡶੑ
• MixerࣗମΛEnvoyʹຒΊࠐΈ͍ͨ
• ॊೈͳ֦ுػߏ͕Envoyʹཉ͍͠
Istio Mixer: past
Slide 21
Slide 21 text
• ղܾ͍ͨ͠՝ҎԼͷ௨Γ
• ಈతʹϓϥάΠϯͷload͕Ͱ͖Δ
• ෳݴޠͷαϙʔτ
• EnvoyࣗମͷಠࣗϏϧυඞཁͳ͍
• Security-awareͳ༷
Envoy’s extensibility
Slide 22
Slide 22 text
• ղܾ͍ͨ͠՝ҎԼͷ௨Γ
• ಈతʹϓϥάΠϯͷload͕Ͱ͖Δ
• ෳݴޠͷαϙʔτ
• EnvoyࣗମͷಠࣗϏϧυඞཁͳ͍
• Security-awareͳ༷
Envoy’s extensibility
Slide 23
Slide 23 text
• ղܾ͍ͨ͠՝ҎԼͷ௨Γ
• ಈతʹϓϥάΠϯͷload͕Ͱ͖Δ
• ෳݴޠͷαϙʔτ
• EnvoyࣗମͷಠࣗϏϧυඞཁͳ͍
• Security-awareͳ༷
Envoy’s extensibility
Slide 24
Slide 24 text
• ղܾ͍ͨ͠՝ҎԼͷ௨Γ
• ಈతʹϓϥάΠϯͷload͕Ͱ͖Δ
• ෳݴޠͷαϙʔτ
• EnvoyࣗମͷಠࣗϏϧυඞཁͳ͍
• Security-awareͳ༷
Envoy’s extensibility
Slide 25
Slide 25 text
• ղܾ͍ͨ͠՝ҎԼͷ௨Γ
• ಈతʹϓϥάΠϯͷload͕Ͱ͖Δ
• ෳݴޠͷαϙʔτ
• EnvoyࣗମͷಠࣗϏϧυඞཁͳ͍
• Security-awareͳ༷
Envoy’s extensibility
Slide 26
Slide 26 text
No content
Slide 27
Slide 27 text
• ϓϥάΠϯ = WasmͷόΠφϦͱͯ͠Envoyʹ
• Envoyͷ“தͰ” WasmͷVMΛಈ͔͢
• Envoy <-> Wasmͷ͠ํ(ABI)Λstableʹ͠Α͏
• ͯ͢ͷ՝ΛΫϦΞ
• ҙͷݴޠ, ಈతload, ηΩϡΞ, stableͳABI
Extending Envoy with Wasm
ಠࣗͷABI
Slide 28
Slide 28 text
• ͦͦϓϩΩγαʔόͷ֦ு༻ABIීวతͳͷͳͣ
• ྫ: Http Header/Body/Trailerʹରͯ͠○○͢Δ, tcpίωΫγϣϯʹରͯ͠xx͢Δ
• Envoy͔ΒΓͤΔͷͰʁ
=> Proxy-Wasmͱ͍͏Envoy͔Βಠཱͨ͠ϓϩδΣΫτʹ
Extending Envoy with Wasm
Proxy-Wasm
Slide 29
Slide 29 text
3. Proxy-Wasm: WebAssembly For Proxies
Slide 30
Slide 30 text
• https://github.com/proxy-wasm
• ϓϩΩγαʔόͷ֦ுػߏͷͨΊͷABIͱSDKͷ։ൃͷͨΊͷϓϩδΣΫτ
• ։ൃ࣮࣭తʹEnvoyίϛϡχςΟ(ݱঢ়GoogleͱTetrate)
• LinkerdͳͲ΄͔ͷϓϩΩγք۾ࢀೖͷؾ
Proxy-Wasm: WebAssembly for Proxies
Slide 31
Slide 31 text
• ̐ͭͷݴޠͷSDK͕͋Γ, ؆୯ʹcompatibleͳWasmͷίϯύΠϧ͕Մೳ
• C++, Rust, Go, AssemblyScript
• Hostͷެ࣮ࣜC++ͷΈ
• https://github.com/proxy-wasm/proxy-wasm-cpp-host
• VMͱͯ͠ Wasmtime, V8, WAVM͕ݱঢ়ར༻Մೳ
• Envoy, Apache Traffic Server(PoC)͜ͷެ࣮ࣜΛϥΠϒϥϦͱͯ͠͏
Proxy-Wasm: WebAssembly for Proxies
Slide 32
Slide 32 text
• Spec: https://github.com/proxy-wasm/spec
• WasmͷϓϩάϥϜͱϓϩΩγαʔόͷABI(͠ํ)ΛఆΊͨͷ
Proxy-Wasm Spec
Slide 33
Slide 33 text
Proxy-Wasm Spec: Example 1.
ϩΪϯάͷͨΊʹ “Wasm͔Β”ݺͿؔ
Slide 34
Slide 34 text
Proxy-Wasm Implementation: Example 1.
proxy_logͷ࣮ in ϗετ
https://github.com/proxy-wasm/proxy-wasm-cpp-host/blob/master/src/exports.cc#L854-L864
Slide 35
Slide 35 text
Proxy-Wasm Implementation: Example 1.
proxy_logͷ࣮Λhostʹظ
https://github.com/proxy-wasm/proxy-wasm-rust-sdk/blob/master/src/hostcalls.rs#L20-L32
Slide 36
Slide 36 text
Proxy-Wasm Spec: Example 2.
ίωΫγϣϯཱ֬࣌, ϗετ͔ΒݺΕΔ “Wasm”ͷؔ
Tcp data͕౸ண࣌, ϗετ͔ΒݺΕΔ “Wasm”ͷؔ
Slide 37
Slide 37 text
Proxy-Wasm Spec: Example 2.
http request header౸ண࣌, ϗετ͔ΒݺΕΔ “Wasm”ͷؔ
http request body͕౸ண࣌, ϗετ͔ΒݺΕΔ “Wasm”ͷؔ
Slide 38
Slide 38 text
Proxy-Wasm Implementation: Example 2.
SDKͰ࣮ & export
https://github.com/tetratelabs/proxy-wasm-go-sdk/blob/main/proxywasm/abi_l7.go#L21-L40
Slide 39
Slide 39 text
Proxy-Wasm Implementation: Example 2.
EnvoyͷEventϧʔϓ
https://github.com/envoyproxy/envoy/blob/master/source/extensions/common/wasm/context.cc#L1577-L1587
https://github.com/proxy-wasm/proxy-wasm-cpp-host/blob/master/src/context.cc#L308-L319
Slide 40
Slide 40 text
Proxy-Wasm Implementation: Example 2.
EnvoyͷEventϧʔϓ
https://github.com/envoyproxy/envoy/blob/master/source/extensions/common/wasm/context.cc#L1577-L1587
https://github.com/proxy-wasm/proxy-wasm-cpp-host/blob/master/src/context.cc#L308-L319
proxy-wasm-cpp-hostͷ
onResponseHeadersΛݺͿ
Slide 41
Slide 41 text
Proxy-Wasm Implementation: Example 2.
EnvoyͷEventϧʔϓ
https://github.com/envoyproxy/envoy/blob/master/source/extensions/common/wasm/context.cc#L1577-L1587
https://github.com/proxy-wasm/proxy-wasm-cpp-host/blob/master/src/context.cc#L308-L319
proxy-wasm-cpp-hostͷ
onResponseHeadersΛݺͿ
“Wasm”ͷproxy_on_request_headers
Slide 42
Slide 42 text
Proxy-Wasm in Envoy: strace-ish log
WasmͷVM<->Envoy HTTP filterͷձϩά
Slide 43
Slide 43 text
Proxy-Wasm in Envoy: strace-ish log
WasmͷVM<->Envoy Network filterͷձϩά
Slide 44
Slide 44 text
• 1 VM / (Plugin, Worker Thread)
• Nativeͷ֦ுͷதͰproxy-wasm-
cpp-hostΛ༻
• 1ͷؒ “envoy-wasm”ͱ͍͏
forkઌͰ։ൃ
• 2020/10ʹupstreammerge
• v1.17ͰॳͷstableϦϦʔε
Proxy-Wasm in Envoy: Impl. model
Slide 45
Slide 45 text
Proxy-Wasm in Envoy: released in v1.17
Slide 46
Slide 46 text
4. Proxy-Wasm In Istio
Slide 47
Slide 47 text
• Proxy-WasmʹΑΓMixer͕Envoyʹ
Έࠐ·Εͨ
• γϯϓϧ͔ͭLatencyͷݮগ
• ϚΠΫϩαʔϏεΛࢧ͑ΔIstio͕ࣗϞ
ϊϦεʹͳͬͨʹ
Istio v1.5~
Before
After
Slide 48
Slide 48 text
• github.com/istio/proxy ʹProxy-Wasm SDKΛ༻͍ͨIstioಠ֦ࣗுͷ࣮
Istio: official plugins in Proxy-Wasm
Slide 49
Slide 49 text
• چTelemetryػೳProxy-WasmϓϥάΠϯͱͯ͠࠶࣮͞Εͨ (stats plugin)
Istio: official plugins in Proxy-Wasm
https://github.com/istio/proxy/blob/master/extensions/stats/plugin.h
Slide 50
Slide 50 text
• ࣮͜ΕΒͷϓϥάΠϯਖ਼֬ʹσϑΥϧτͰ“Wasm”Ͱಈ͍͍ͯͳ͍
• Proxy-WasmͷABIΛ༻͍͍ͯΔ͕NativeʹCompile & Envoyʹstatic link
• NullVmͱզʑ͕ݺΜͰ͍ΔΈ
• EnvoyͷWasm Extension͕·ͩalphaεςʔλεͳͨΊ
• ҰํͰɺWasmͱͯ͠ίϯύΠϧՄೳͳΑ͏ʹίʔυ͕ॻ͔Ε͍ͯΔ
• ͍ۙকདྷ(v1.10~)σϑΥϧτͰWasm VMͰϓϥάΠϯ͕ಈ͘Α͏ʹͳΔͣ
Istio: official plugins in Proxy-Wasm
Slide 51
Slide 51 text
• ݱঢ়Wasm plugin༻ͷ
API(CRD)ଘࡏ͠ͳ͍
• EnvoyFilterͰؤுΔ͔͠ͳ͍
How to deploy plugins in Istio: API
Slide 52
Slide 52 text
• ݱঢ়Wasm plugin༻ͷ
API(CRD)ଘࡏ͠ͳ͍
• EnvoyFilterͰؤுΔ͔͠ͳ͍
• Wasm-SIGͰ1st classͳ
APIΛࡦఆத
How to deploy plugins in Istio: API
Slide 53
Slide 53 text
• ݱঢ়Istio (Istio-agent), http(s)Ͱserving͞ΕͨWasmόΠφϦͷΈLoadՄ
• OCI-imageͱͯ͠WasmͷbinaryΛOCI registryʹ֨ೲ͢ΔྲྀΕ͕͋Δ
• https://github.com/engineerd/wasm-to-oci
• https://github.com/solo-io/wasm-image-spec
• Proxy-Wasm༻ͷOCI-image specΛIstio Wasm-SIGͰࡦఆத: ΄΅ݻ·ͬͨ
• ͍ۙকདྷOCI registryʹpush͞ΕͨWasmΛIstio͕αϙʔτ͢ΔΑ͏ʹͳΔ
• (࣮ͷTODOͰ͢…)
How to deploy plugins in Istio: Delivery
Slide 54
Slide 54 text
Future of Workflow of Istio Wasm plugins
Build Push
OCI-registries
k apply -f
wasm.yaml
Image: hoge.com/my-plugin:v1.10
config: …
Slide 55
Slide 55 text
5. Challenges and Future
Slide 56
Slide 56 text
• ҙͷϓϩάϥϜ͔ΒͲ͏ͷΑ͏ʹϗετΛकΔ͔?
• ProxyαʔόϛογϣϯΫϦςΟΧϧͳιϑτΣΞ
• ςετॻ͍ͯॻ͍ͯॻ͖͖Εͳ͍
• Wasmࣗମ͕ηΩϡΞͱ͍͑ɺಛఆͷύεͰΫϥογϡ͢Δ͜ͱ͋ΓಘΔ
Challenges in Proxy-Wasm
Slide 57
Slide 57 text
• ύϑΥʔϚϯεͷ
• GC͖ͷݴޠProxy-Wasm͚ͷGCΞϧΰϦζϜΛ։ൃ͠ͳ͍ͱ͍͚
ͳ͍?
• Nativeʹൺ͓ͯΑͦ50%͍ͱ͍͏ϕϯνϚʔΫ
• EnvoyશମͷॲཧൺͰແࢹͰ͖ΔఔͳͷͰ? ͱ͍͏͋Δ
Challenges in Proxy-Wasm
Slide 58
Slide 58 text
• RuntimeͲΕબྑ͍ͷ͔
• ࣮ߦϝϞϦ༻ྔͷτϨʔυΦϑ
Challenges in Proxy-Wasm
Slide 59
Slide 59 text
• αϙʔτݴޠ, ͦ͜·Ͱଟ͘ͳ͍Α͏ͳ….?
• WasmͷΤίγεςϜࣗମ͕·ͩະख़
Challenges in Proxy-Wasm
Slide 60
Slide 60 text
• ABI͕Envoyͷ࣮ʹ͍ͩͿد͍ͬͯΔ(ͨΓલͱ͍͑ͨΓલ…)
• 1VM͕ෳͷϦΫΤετΛࡹ͘ͱ͍͏ઃܭ
• V8 / WAVMͷoverhead͕എܠ
• 1VM = 1 requestͱ͍͏Ϟσϧͷ߹ʹෆཁͳઃఆ͕͍͔ͭ͋͘Δ
Challenges in Proxy-Wasm
Slide 61
Slide 61 text
• ςετͱσόά͕͔ͳΓਏ͍
• ςετͷͨΊʹEnvoyΛಈ͔͢…?
• Go SDKͰEnvoyͷΤϛϡϨʔλΛGoͰॻ͍ͯ, nativeίϯύΠϧ
Ͱςετ: ݶք͕͋Δ
• WasmଆʹdebugͷͨΊͷ༷͕ͳ͍ͷͰ, ֤ݴޠ͝ͱʹ৭ʑ
• LLVMϕʔεͷݴޠͳΒDWARF͕όΠφϦʹೖͬͯΔͷͰ৭ʑͰ͖
Δ͕Proxy-Wasm HostଆͰະ࣮(Ջ͕͋ͬͨΒΔ)
• Stack trace͕औΓͮΒ͍(Ϣʔβʔۭ͔ؒΒstackݟΕͳ͍ͷͰ)
Challenges in Proxy-Wasm
Slide 62
Slide 62 text
• WASIͷؒೖΓ…?
• ࣮ࡍProxy-Wasm͋ΔछͷWASIͷ֦ுͰ͋Δ
• Pre-Proposal phaseͱͯ͠WASIͷProposalʹ͍Δ
• https://github.com/WebAssembly/WASI/blob/master/docs/
Proposals.md
Future of Proxy-Wasm
Slide 63
Slide 63 text
• Deployํ๏ͷσϑΝΫτελϯμʔυͷཱ֬
• IstioͰͷ࣮͜Ε͔Β
• υΩϡϝϯςʔγϣϯdeveloper toolingपΓͷ֦ॆ
Future of Proxy-Wasm
Slide 64
Slide 64 text
• ·ͩ·ͩEarly days (Join us!): server-side WasmͷޭྫͱͳΓ͍ͨ
Future of Proxy-Wasm
Slide 65
Slide 65 text
Future of Cloud Native Wasm
2015 2019/03 2020/03
WASIͷొ
WASMͷొ
Proxy-Wasmͷొ
2013/03
asm.jsͷొ
2020/04
Krustletͷొ
202x/yy
????
Slide 66
Slide 66 text
• WasmηΩϡΞͰportableͳόΠφϦϑΥʔϚοτ+ԾϚγϯͷ༷
• CNCF͕༧ଌ͢ΔΑ͏ʹ, ༷ʑͳ໘Ͱར༻͕૿͖͑ͯͦ͏
• Cloud Native Wasm DayͷτʔΫΛݟΔͱಈ͕ݟ͑ͯ͘Δ?
• Proxy-WasmʹΑΓEnvoy/Istio͕WasmͰ֦ுͰ͖ΔΑ͏ͳੈք͕طʹདྷ͍ͯΔ
• Isito͚ͩͰͳ͘, େاۀ͕طʹproductionͰಈ͔͍ͯ͠Δ
• ·ͩ·ͩൃల్্Ͱ͕͢, 2021தʹ͔ͳΓख़͢ΔݟࠐΈ
·ͱΊ
Slide 67
Slide 67 text
We are hiring! https://www.tetrate.io/careers/
Work Anytime and Anywhere + Unlimited paid time off