Pro Yearly is on sale from $80 to $50! »

Calling Rust from Go, without cgo @ GothamGo 2017

Calling Rust from Go, without cgo @ GothamGo 2017

AKA "Building your own FFI"

https://blog.filippo.io/rustgo

9fdab9d005b82612cadbfe699b541f83?s=128

Filippo Valsorda

October 05, 2017
Tweet

Transcript

  1. From Go to Rust without cgo Filippo Valsorda

  2. cgo Foreign function interface (FFI). Call C code from Go.

  3. cgo Foreign function interface (FFI). Call C code from Go.

    • Code generator tool • FFI internals to make C happy • Safety features to make Go less unhappy • Utility libraries and types
  4. From Go to Rust without cgo Filippo Valsorda DIY FFI

  5. FFI: starting kit A way to: • Compile the foreign

    code together with the Go code • Jump to the foreign code by function name • Share arguments, return values and memory • Bonus: packaging and redistribution
  6. Foreign code

  7. #![no_std] #[no_mangle] pub extern fn multiply_two(a: u64) -> u64 {

    return a * 2; } [package] name = "multiplication" version = "0.0.1" [lib] crate-type = ["staticlib"]
  8. Compiling the foreign code together

  9. $GOROOT/pkg/tool/darwin_amd64/compile \ -o $WORK/.../rustgo.a [...] -pack ./main.go $GOROOT/pkg/tool/darwin_amd64/link \ -o

    $WORK/.../a.out -buildmode=exe [...] \ -linkmode=internal $WORK/.../rustgo.a mv $WORK/.../a.out rustgo go build -x (package main)
  10. .go .go .go package .go package cmd/compile cmd/compile .o .o

    .a .o .o .a binary cmd/link symbol "x" symbol "y" symbol "foo" symbol "main" ...
  11. Internal linking cmd/link Go object files darwin_amd64 binary linux_amd64 binary

    linux_arm binary
  12. External linking cmd/link Go object files platform binary ld foreign

    code static and dynamic libraries
  13. go build -ldflags='-linkmode=external \ -extldflags=.../libmultiplication.a' External linking the Rust static

    library
  14. $GOROOT/pkg/tool/darwin_amd64/compile \ -o $WORK/.../multiplication.a [...] -pack ./mul.go mv $WORK/.../multiplication.a \

    $GOPATH/pkg/darwin_amd64/.../multiplication.a go build -i -x (package multiplication)
  15. .go .go .go package .go package cmd/compile cmd/compile .o .o

    .a .o .o .a binary cmd/link symbol "x" symbol "y" symbol "foo" symbol "main" ...
  16. .o .o .a symbol "x" symbol "y" symbol "foo" symbol

    "main" ... $GOPATH/pkg/.../
  17. None
  18. ar xv .../libmultiplication.a # the Rust library Stuffing .a files

    go tool pack r \ $GOPATH/pkg/darwin_amd64/.../multiplication.a *.o
  19. .o .o .a .o .o the Rust static library .o

    .o .o .o ar cmd/pack .o .o .a .o .o $GOPATH/pkg/.../
  20. ld -r -o .../libmultiplication.o --gc-sections \ -u multiply_two .../libmultiplication.a Stuffing

    .a files go tool pack r \ $GOPATH/pkg/darwin_amd64/.../multiplication.a \ .../libmultiplication.o
  21. .o .o .a .o .o the Rust static library ld

    -r -u cmd/pack .o .o .a .o .o .o $GOPATH/pkg/.../
  22. Compiling the foreign code together What we did: put the

    Rust code with the multiply_two function inside the Go package How we did it: ld -r -u and cmd/pack
  23. Calling the foreign symbol

  24. package multiplication func MultiplyByTwo(n uint64) uint64 .go TEXT ·MultiplyByTwo(SB), 0,

    $16384-8 RET .s
  25. None
  26. //go:cgo_import_static multiply_two //go:cgo_import_dynamic multiply_two //go:linkname multiply_two multiply_two var multiply_two uintptr

    var _multiply_two = &multiply_two .go TEXT MultiplyByTwo(SB), 0, $16384-8 MOVQ ·_multiply_two(SB), AX JMP AX RET .s
  27. Calling the foreign symbol What we did: got the linker

    to put a pointer to the Rust function in _multiply_two How we did it: //go:cgo_import_xxx
  28. Args, returns and
 calling conventions

  29. None
  30. a := multiplyByTwo(0x5555555555555555) func multiplyByTwo(a uint64) uint64 { return a

    * 2 } (lldb) disassemble -n main.main
  31. movabsq $0x5555555555555555, %rax movq %rax, (%rsp) callq 0x104f360 ; main.multiplyByTwo

    at main.go:9 PUSH the argument and CALL
  32. xxx`main.multiplyByTwo: movq %gs:0x8a0, %rcx leaq -0x8(%rsp), %rax cmpq 0x10(%rcx), %rax

    jbe 0x104f3e1 ; <+129> at main.go:9 [...] Stack size check preamble
  33. subq $0x88, %rsp [...] addq $0x88, %rsp retq Stack pointer

  34. None
  35. subq $0x88, %rsp [...] addq $0x88, %rsp retq Stack pointer

  36. movq %rbp, 0x80(%rsp) leaq 0x80(%rsp), %rbp [...] movq 0x80(%rsp), %rbp

    Frame pointer
  37. movq 0x90(%rsp), %rax Argument

  38. movq %rax, 0x98(%rsp) Return value

  39. The C calling convention Or "sysv64" on x86_64. • Arguments:

    RDI, RSI, RDX, RCX, R8, and R9 • Return value: RAX •Stack alignment: 16 bytes Default for Rust extern functions.
  40. TEXT ·MultiplyByTwo(SB), 0, $2048-16 MOVQ n+0(FP), DI // Load the

    argument before messing with SP MOVQ SP, BX // Save SP in a callee-saved registry ADDQ $2048, SP // Rollback SP to reuse this function's frame ANDQ $~15, SP // Align the stack to 16-bytes MOVQ ·_multiply_two(SB), AX CALL AX MOVQ BX, SP // Restore SP MOVQ AX, ret+8(FP) // Place the return value on the stack RET Trampoline
  41. package main import "./multiplication" func main() { println(multiplication.MultiplyByTwo(21)) }

  42. TEXT ·MultiplyByTwo(SB), 0, $2048-16 MOVQ n+0(FP), DI // Load the

    argument before messing with SP MOVQ SP, BX // Save SP in a callee-saved registry ADDQ $2048, SP // Rollback SP to reuse this function's frame ANDQ $~15, SP // Align the stack to 16-bytes MOVQ ·_multiply_two(SB), AX CALL AX MOVQ BX, SP // Restore SP MOVQ AX, ret+8(FP) // Place the return value on the stack RET
  43. #[no_mangle] pub extern fn multiply_two(a: u64) -> u64 { return

    a * 2; }
  44. $ make install cargo build --release ld -r -u _multiply_two

    go tool asm go tool compile go tool pack cp multiplication/multiplication.a $GOPATH/pkg/... $ go build . $ ./rustgo 4
  45. Go small stacks Each goroutine has its own stack. When

    it runs out, the preamble calls morestack. We don't, so we have to fit in a fixed stack size. (This is not ok.) TEXT ·MultiplyByTwo(SB), 0, $2048-16
  46. Args, returns and calling conventions What we did: made a

    trampoline from the Go calling convention to the C one. How we did it: sacrilegious assembly.
  47. Bonus: distribution

  48. //go:binary-only-package package multiplication import _ "unsafe" //go:cgo_import_static multiply_two //go:cgo_import_dynamic multiply_two

    //go:linkname multiply_two multiply_two var multiply_two uintptr var _multiply_two = &multiply_two func MultiplyByTwo(n uint64) uint64
  49. Thank you! Filippo Valsorda — @FiloSottile https://blog.filippo.io/rustgo