WIT - the language melting pot

A new age of FFI.
Compile go and python code to a single wasm binary and distrubute it on web.

https://github.com/yue4u/lab/tree/81743f4/projects/tools/iter-tpl

Transcript

1. WIT - the language melting pot Compile go and python code to a single wasm binary and distrubute it on web. A new age of FFI
2. Steps 1. Compile go and python code to a a single wasm binary 2. Distrubute it on web
3. Complete!
4. 1. Compile go and python code to a single wasm binary -> gowasm / py2wasm / emscripten + merge them somehow? -> use WIT ✅ WIT (Wasm Interface Type) The WebAssembly Component Model https://component-model.bytecodealliance.org/design/wit.html
5. WIT 101 Can use string , list , Record , enum across langugaes!! Interface World Packages Platforms WASI A ABI contract with (higher-level types, interfaces).
6. Interface An interface is a named set of types and functions, enclosed in braces and introduced with the interface keyword: use types.{dimension, point}; interface types { type dimension = u32; record point { x: dimension, y: dimension, } } interface canvas { type canvas-id = u64; draw-line: func(canvas: canvas-id, from: point, to: point, thickness: dimension); }
7. World Roughly, a world describes the contract of a component. yue’s understanding: World is imports.d.ts + exports.d.ts of the component boundary. Interfaces are not exposed without world. Files can be mixed like golang. interface printer { print: func(text: string); } interface error-reporter { report-error: func(error-message: string); } world multi-function-device { // The component implements the `printer` interface export printer; // The component implements the `scan` function export scan: func() -> list<u8>; // The component needs to be supplied with an `error-reporter` import error-reporter; }
8. Packages A package is a set of WIT files containing a related set of interfaces and worlds. Example: a http package = proxy interfaces + server Platforms Runtime: host env WASI The WebAssembly System Interface (since 2019?)
9. Get started what we are building? -> iter-tpl export python’s builtin itertools.combinations , itertools.permutations to generate list export go’s builtin text/template to do formating + = list(itertools.combinations([1, 2, 3], 2)) # [(1, 2), (1, 3), (2, 3)] input := [1,2] tpl, _ := template.New("test").Parse("{{index .list 1}} -> {{index .list 2}}") _ = tpl.Execute(os.Stdout, input) // 1 -> 2 1 -> 2 1 -> 3 2 -> 3
10. Define WIT Generate types and bindings package iter-tpl:iter@0.0.1; interface tools { // TODO: use variant to support u32? // TODO: tuple is better? combinations: func(iterable: list<string>, r: u32) -> list<list<string>>; permutations: func(iterable: list<string>, r: u32) -> list<list<string>>; } world iter { export tools; } pip install componentize-py componentize-py --wit-path wit --world iter bindings .
11. Building python’s itertools Impl the interface from typing import List from wit_world import exports from itertools import combinations, permutations def make_list(iterable): return list(map(list, list(iterable))) class Tools(exports.Tools): def combinations(self, iterable: List[str], r: int) -> List[List[str]]: return make_list(combinations(iterable, r)) def permutations(self, iterable: List[str], r: int) -> List[List[str]]: return make_list(permutations(iterable, r))
12. Building itertools’s wasm Tesing via wasmtime 🎉! File size componentize-py \ --wit-path wit/iter.wit \ --world iter \ componentize \ app \ -o iter.wasm wasmtime run --invoke 'combinations(["1", "2", "3"], 2)' iter.wasm # [["1", "2"], ["1", "3"], ["2", "3"]] wasmtime run --invoke 'permutations(["1", "2", "3"], 2)' iter.wasm # [["1", "2"], ["1", "3"], ["2", "1"], ["2", "3"], ["3", "1"], ["3", "2"]] # minify wasm-tools strip src/iter/iter.wasm -o src/iter/iter.min.wasm du -sh src/iter/*.wasm # 18M src/iter/iter.min.wasm # 39M src/iter/iter.wasm
13. Building golang’s text/template Define WIT install deps package iter-tpl:tpl@0.0.1; interface template { execute: func(tpl: string, array: list<string>) -> result<string, string>; } world tpl { include wasi:cli/imports@0.2.0; export template; } pacman -S tinygo cargo install wkg # wasm-pkg-tools go get -tool go.bytecodealliance.org/cmd/wit-bindgen-go
14. generate and impl go’s bindings 1. build wasm only with types 2. generate related code wkg wit build go tool wit-bindgen-go generate --world tpl --out internal ./iter-tpl:tpl@0.0.1.wasm
15. package main import ( "bytes" "text/template" "go.bytecodealliance.org/cm" tpl "yue.cat/internal/iter-tpl/tpl/template" ) type TplResult = cm.Result[string, string, string] func init() { tpl.Exports.Execute = func(tpl string, array cm.List[string]) (result TplResult) { t, err := template.New("tpl").Parse(tpl) if err != nil { return cm.Err[TplResult](err.Error()) } var buf bytes.Buffer if err = t.Execute(&buf, array); err != nil { return cm.Err[TplResult](err.Error()) } return cm.OK[TplResult](buf.String()) } }
16. Build wasm with tinygo Tesing via wasmtime 🎉! tinygo build -target=wasip2 \ -o tpl.wasm \ --wit-package iter-tpl:tpl@0.0.1.wasm \ --wit-world tpl main.go du -sh src/tpl/*.wasm # 12K src/tpl/iter-tpl:tpl@0.0.1.wasm # 536K src/tpl/tpl.min.wasm # 1.3M src/tpl/tpl.wasm wasmtime run --invoke 'execute("{{ $1 }}", ["1", "2"])' tpl.wasm # Ok("1")
17. Merging two wasm Define WIT package iter-tpl:main@0.0.1; interface iter-tpl { combinations: func(iterable: list<string>, r: u32) -> list<list<string>>; permutations: func(iterable: list<string>, r: u32) -> list<list<string>>; template: func(tpl: string, array: list<string>) -> result<string, string>; } world main { import iter-tpl:iter/tools@0.0.1; import iter-tpl:tpl/template@0.0.1; export iter-tpl; }
18. impl WIT with js Build incomplete wasm import { combinations, permutations } from 'iter-tpl:iter/tools@0.0.1'; import { execute } from 'iter-tpl:tpl/template@0.0.1'; export const iterTpl = { combinations, permutations, template: execute }; pnpm add @bytecodealliance/jco --global jco componentize \ iter-tpl.mjs \ --wit wit/ \ --world-name main \ --out iter-tpl.incomplete.wasm \ --disable=all
19. Plug python and go wasm (magic) WebAssembly Compositions (WAC) Tesing via wasmtime 🎉! cargo install wac-cli wac plug -o iter-tpl.wasm --plug ../iter/iter.wasm --plug ../tpl/tpl.wasm iter-tpl.incomplete.wasm du -sh src/iter-tpl/*.wasm # 12M src/iter-tpl/iter-tpl.incomplete.wasm # 30M src/iter-tpl/iter-tpl.min.wasm # 51M src/iter-tpl/iter-tpl.wasm wasmtime run --invoke 'combinations(["1", "2", "3"], 2)' iter-tpl.wasm wasmtime run --invoke 'permutations(["1", "2", "3"], 2)' iter-tpl.wasm wasmtime run --invoke 'template("{{ $1 }}", ["a", "2"])' iter-tpl.wasm
20. transpile for browser Tesing in frontend 🎉! Wrap in worker import Worker from "./iter-tpl.worker?worker"; jco transpile iter-tpl.min.wasm -o js import { iterTpl } from "./js/iter-tpl.min"; iterTpl.combinations(["1", "2", "3"], 2) // [..] iterTpl.permutations(["1", "2", "3"], 2) // [..] iterTpl.template("{{ $0 }} -> {{ $1 }}",["1", "2", "3"]) // "1 -> 2" self.onmessage = async (e) => { const [tpl, items] = e.data as [string, string[][]]; const { iterTpl } = await import("./js/iter-tpl.min"); const ret = iterTpl.template(tpl, items); self.postMessage(ret); };
21. Demo Next Run in Android webview 🎉 pnpm tauri android init adb shell uname -m -> aarch64 pnpm tauri android build --target aarch64 --apk true keytool -genkey -v -keystore ~/upload-keystore.jks -keyalg RSA -keysize 2048 - validity 10000 -alias upload adb install src-tauri/gen/android/app/build/outputs/apk/universal/release/app- universal-release.apk https://lab.yue.coffee/tools/iter-tpl
22. Run in k8s FROM nginx WORKDIR /app COPY dist /usr/share/nginx/html apiVersion: apps/v1 kind: Deployment metadata: # ... spec: # ... spec: containers: - name: wit image: registry.yue-home.local/wit:0.0.1 ports: - containerPort: 80 resources: limits: cpu: "0.5" memory: 100Mi requests: cpu: "0.5" memory: 100Mi
23. To wrap it all up sugoi… Anything that can be compile via WIT will be a part of frontend Anything that can be compile via WIT can run inside wasm/wasi’s secruity sandbox To explore Run as k8s wasm workload Can wasmtime itself compile to android app?
24. Project: ZF https://github.com/yue4u/zf godot + wasmtime + nushell
25. Fin.