Calling Rust from Go, without cgo @ GothamGo 2017

AKA "Building your own FFI"

https://blog.filippo.io/rustgo

Transcript

1. From Go to Rust without cgo Filippo Valsorda
2. cgo Foreign function interface (FFI). Call C code from Go.
3. cgo Foreign function interface (FFI). Call C code from Go. • Code generator tool • FFI internals to make C happy • Safety features to make Go less unhappy • Utility libraries and types
4. From Go to Rust without cgo Filippo Valsorda DIY FFI
5. FFI: starting kit A way to: • Compile the foreign code together with the Go code • Jump to the foreign code by function name • Share arguments, return values and memory • Bonus: packaging and redistribution
6. Foreign code
7. #![no_std] #[no_mangle] pub extern fn multiply_two(a: u64) -> u64 { return a * 2; } [package] name = "multiplication" version = "0.0.1" [lib] crate-type = ["staticlib"]
8. Compiling the foreign code together
9. $GOROOT/pkg/tool/darwin_amd64/compile \ -o $WORK/.../rustgo.a [...] -pack ./main.go $GOROOT/pkg/tool/darwin_amd64/link \ -o $WORK/.../a.out -buildmode=exe [...] \ -linkmode=internal $WORK/.../rustgo.a mv $WORK/.../a.out rustgo go build -x (package main)
10. .go .go .go package .go package cmd/compile cmd/compile .o .o .a .o .o .a binary cmd/link symbol "x" symbol "y" symbol "foo" symbol "main" ...
11. Internal linking cmd/link Go object files darwin_amd64 binary linux_amd64 binary linux_arm binary
12. External linking cmd/link Go object files platform binary ld foreign code static and dynamic libraries
13. go build -ldflags='-linkmode=external \ -extldflags=.../libmultiplication.a' External linking the Rust static library
14. $GOROOT/pkg/tool/darwin_amd64/compile \ -o $WORK/.../multiplication.a [...] -pack ./mul.go mv $WORK/.../multiplication.a \ $GOPATH/pkg/darwin_amd64/.../multiplication.a go build -i -x (package multiplication)
15. .go .go .go package .go package cmd/compile cmd/compile .o .o .a .o .o .a binary cmd/link symbol "x" symbol "y" symbol "foo" symbol "main" ...
16. .o .o .a symbol "x" symbol "y" symbol "foo" symbol "main" ... $GOPATH/pkg/.../
18. ar xv .../libmultiplication.a # the Rust library Stuffing .a files go tool pack r \ $GOPATH/pkg/darwin_amd64/.../multiplication.a *.o
19. .o .o .a .o .o the Rust static library .o .o .o .o ar cmd/pack .o .o .a .o .o $GOPATH/pkg/.../
20. ld -r -o .../libmultiplication.o --gc-sections \ -u multiply_two .../libmultiplication.a Stuffing .a files go tool pack r \ $GOPATH/pkg/darwin_amd64/.../multiplication.a \ .../libmultiplication.o
21. .o .o .a .o .o the Rust static library ld -r -u cmd/pack .o .o .a .o .o .o $GOPATH/pkg/.../
22. Compiling the foreign code together What we did: put the Rust code with the multiply_two function inside the Go package How we did it: ld -r -u and cmd/pack
23. Calling the foreign symbol
24. package multiplication func MultiplyByTwo(n uint64) uint64 .go TEXT ·MultiplyByTwo(SB), 0, $16384-8 RET .s
26. //go:cgo_import_static multiply_two //go:cgo_import_dynamic multiply_two //go:linkname multiply_two multiply_two var multiply_two uintptr var _multiply_two = &multiply_two .go TEXT MultiplyByTwo(SB), 0, $16384-8 MOVQ ·_multiply_two(SB), AX JMP AX RET .s
27. Calling the foreign symbol What we did: got the linker to put a pointer to the Rust function in _multiply_two How we did it: //go:cgo_import_xxx
28. Args, returns and
 calling conventions
30. a := multiplyByTwo(0x5555555555555555) func multiplyByTwo(a uint64) uint64 { return a * 2 } (lldb) disassemble -n main.main
31. movabsq $0x5555555555555555, %rax movq %rax, (%rsp) callq 0x104f360 ; main.multiplyByTwo at main.go:9 PUSH the argument and CALL
32. xxx`main.multiplyByTwo: movq %gs:0x8a0, %rcx leaq -0x8(%rsp), %rax cmpq 0x10(%rcx), %rax jbe 0x104f3e1 ; <+129> at main.go:9 [...] Stack size check preamble
33. subq $0x88, %rsp [...] addq $0x88, %rsp retq Stack pointer
35. subq $0x88, %rsp [...] addq $0x88, %rsp retq Stack pointer
36. movq %rbp, 0x80(%rsp) leaq 0x80(%rsp), %rbp [...] movq 0x80(%rsp), %rbp Frame pointer
37. movq 0x90(%rsp), %rax Argument
38. movq %rax, 0x98(%rsp) Return value
39. The C calling convention Or "sysv64" on x86_64. • Arguments: RDI, RSI, RDX, RCX, R8, and R9 • Return value: RAX •Stack alignment: 16 bytes Default for Rust extern functions.
40. TEXT ·MultiplyByTwo(SB), 0, $2048-16 MOVQ n+0(FP), DI // Load the argument before messing with SP MOVQ SP, BX // Save SP in a callee-saved registry ADDQ $2048, SP // Rollback SP to reuse this function's frame ANDQ $~15, SP // Align the stack to 16-bytes MOVQ ·_multiply_two(SB), AX CALL AX MOVQ BX, SP // Restore SP MOVQ AX, ret+8(FP) // Place the return value on the stack RET Trampoline
41. package main import "./multiplication" func main() { println(multiplication.MultiplyByTwo(21)) }
42. TEXT ·MultiplyByTwo(SB), 0, $2048-16 MOVQ n+0(FP), DI // Load the argument before messing with SP MOVQ SP, BX // Save SP in a callee-saved registry ADDQ $2048, SP // Rollback SP to reuse this function's frame ANDQ $~15, SP // Align the stack to 16-bytes MOVQ ·_multiply_two(SB), AX CALL AX MOVQ BX, SP // Restore SP MOVQ AX, ret+8(FP) // Place the return value on the stack RET
43. #[no_mangle] pub extern fn multiply_two(a: u64) -> u64 { return a * 2; }
44. $ make install cargo build --release ld -r -u _multiply_two go tool asm go tool compile go tool pack cp multiplication/multiplication.a $GOPATH/pkg/... $ go build . $ ./rustgo 4
45. Go small stacks Each goroutine has its own stack. When it runs out, the preamble calls morestack. We don't, so we have to fit in a fixed stack size. (This is not ok.) TEXT ·MultiplyByTwo(SB), 0, $2048-16
46. Args, returns and calling conventions What we did: made a trampoline from the Go calling convention to the C one. How we did it: sacrilegious assembly.
47. Bonus: distribution
48. //go:binary-only-package package multiplication import _ "unsafe" //go:cgo_import_static multiply_two //go:cgo_import_dynamic multiply_two //go:linkname multiply_two multiply_two var multiply_two uintptr var _multiply_two = &multiply_two func MultiplyByTwo(n uint64) uint64
49. Thank you! Filippo Valsorda — @FiloSottile https://blog.filippo.io/rustgo