$30 off During Our Annual Pro Sale. View Details »

Calling Rust from Go, without cgo @ GothamGo 2017

Calling Rust from Go, without cgo @ GothamGo 2017

AKA "Building your own FFI"

https://blog.filippo.io/rustgo

Filippo Valsorda

October 05, 2017
Tweet

More Decks by Filippo Valsorda

Other Decks in Programming

Transcript

  1. From Go to Rust without cgo Filippo Valsorda

  2. cgo Foreign function interface (FFI). Call C code from Go.

  3. cgo Foreign function interface (FFI). Call C code from Go.

    • Code generator tool • FFI internals to make C happy • Safety features to make Go less unhappy • Utility libraries and types
  4. From Go to Rust without cgo Filippo Valsorda DIY FFI

  5. FFI: starting kit A way to: • Compile the foreign

    code together with the Go code • Jump to the foreign code by function name • Share arguments, return values and memory • Bonus: packaging and redistribution
  6. Foreign code

  7. #![no_std] #[no_mangle] pub extern fn multiply_two(a: u64) -> u64 {

    return a * 2; } [package] name = "multiplication" version = "0.0.1" [lib] crate-type = ["staticlib"]
  8. Compiling the foreign code together

  9. $GOROOT/pkg/tool/darwin_amd64/compile \ -o $WORK/.../rustgo.a [...] -pack ./main.go $GOROOT/pkg/tool/darwin_amd64/link \ -o

    $WORK/.../a.out -buildmode=exe [...] \ -linkmode=internal $WORK/.../rustgo.a mv $WORK/.../a.out rustgo go build -x (package main)
  10. .go .go .go package .go package cmd/compile cmd/compile .o .o

    .a .o .o .a binary cmd/link symbol "x" symbol "y" symbol "foo" symbol "main" ...
  11. Internal linking cmd/link Go object files darwin_amd64 binary linux_amd64 binary

    linux_arm binary
  12. External linking cmd/link Go object files platform binary ld foreign

    code static and dynamic libraries
  13. go build -ldflags='-linkmode=external \ -extldflags=.../libmultiplication.a' External linking the Rust static

    library
  14. $GOROOT/pkg/tool/darwin_amd64/compile \ -o $WORK/.../multiplication.a [...] -pack ./mul.go mv $WORK/.../multiplication.a \

    $GOPATH/pkg/darwin_amd64/.../multiplication.a go build -i -x (package multiplication)
  15. .go .go .go package .go package cmd/compile cmd/compile .o .o

    .a .o .o .a binary cmd/link symbol "x" symbol "y" symbol "foo" symbol "main" ...
  16. .o .o .a symbol "x" symbol "y" symbol "foo" symbol

    "main" ... $GOPATH/pkg/.../
  17. None
  18. ar xv .../libmultiplication.a # the Rust library Stuffing .a files

    go tool pack r \ $GOPATH/pkg/darwin_amd64/.../multiplication.a *.o
  19. .o .o .a .o .o the Rust static library .o

    .o .o .o ar cmd/pack .o .o .a .o .o $GOPATH/pkg/.../
  20. ld -r -o .../libmultiplication.o --gc-sections \ -u multiply_two .../libmultiplication.a Stuffing

    .a files go tool pack r \ $GOPATH/pkg/darwin_amd64/.../multiplication.a \ .../libmultiplication.o
  21. .o .o .a .o .o the Rust static library ld

    -r -u cmd/pack .o .o .a .o .o .o $GOPATH/pkg/.../
  22. Compiling the foreign code together What we did: put the

    Rust code with the multiply_two function inside the Go package How we did it: ld -r -u and cmd/pack
  23. Calling the foreign symbol

  24. package multiplication func MultiplyByTwo(n uint64) uint64 .go TEXT ·MultiplyByTwo(SB), 0,

    $16384-8 RET .s
  25. None
  26. //go:cgo_import_static multiply_two //go:cgo_import_dynamic multiply_two //go:linkname multiply_two multiply_two var multiply_two uintptr

    var _multiply_two = &multiply_two .go TEXT MultiplyByTwo(SB), 0, $16384-8 MOVQ ·_multiply_two(SB), AX JMP AX RET .s
  27. Calling the foreign symbol What we did: got the linker

    to put a pointer to the Rust function in _multiply_two How we did it: //go:cgo_import_xxx
  28. Args, returns and
 calling conventions

  29. None
  30. a := multiplyByTwo(0x5555555555555555) func multiplyByTwo(a uint64) uint64 { return a

    * 2 } (lldb) disassemble -n main.main
  31. movabsq $0x5555555555555555, %rax movq %rax, (%rsp) callq 0x104f360 ; main.multiplyByTwo

    at main.go:9 PUSH the argument and CALL
  32. xxx`main.multiplyByTwo: movq %gs:0x8a0, %rcx leaq -0x8(%rsp), %rax cmpq 0x10(%rcx), %rax

    jbe 0x104f3e1 ; <+129> at main.go:9 [...] Stack size check preamble
  33. subq $0x88, %rsp [...] addq $0x88, %rsp retq Stack pointer

  34. None
  35. subq $0x88, %rsp [...] addq $0x88, %rsp retq Stack pointer

  36. movq %rbp, 0x80(%rsp) leaq 0x80(%rsp), %rbp [...] movq 0x80(%rsp), %rbp

    Frame pointer
  37. movq 0x90(%rsp), %rax Argument

  38. movq %rax, 0x98(%rsp) Return value

  39. The C calling convention Or "sysv64" on x86_64. • Arguments:

    RDI, RSI, RDX, RCX, R8, and R9 • Return value: RAX •Stack alignment: 16 bytes Default for Rust extern functions.
  40. TEXT ·MultiplyByTwo(SB), 0, $2048-16 MOVQ n+0(FP), DI // Load the

    argument before messing with SP MOVQ SP, BX // Save SP in a callee-saved registry ADDQ $2048, SP // Rollback SP to reuse this function's frame ANDQ $~15, SP // Align the stack to 16-bytes MOVQ ·_multiply_two(SB), AX CALL AX MOVQ BX, SP // Restore SP MOVQ AX, ret+8(FP) // Place the return value on the stack RET Trampoline
  41. package main import "./multiplication" func main() { println(multiplication.MultiplyByTwo(21)) }

  42. TEXT ·MultiplyByTwo(SB), 0, $2048-16 MOVQ n+0(FP), DI // Load the

    argument before messing with SP MOVQ SP, BX // Save SP in a callee-saved registry ADDQ $2048, SP // Rollback SP to reuse this function's frame ANDQ $~15, SP // Align the stack to 16-bytes MOVQ ·_multiply_two(SB), AX CALL AX MOVQ BX, SP // Restore SP MOVQ AX, ret+8(FP) // Place the return value on the stack RET
  43. #[no_mangle] pub extern fn multiply_two(a: u64) -> u64 { return

    a * 2; }
  44. $ make install cargo build --release ld -r -u _multiply_two

    go tool asm go tool compile go tool pack cp multiplication/multiplication.a $GOPATH/pkg/... $ go build . $ ./rustgo 4
  45. Go small stacks Each goroutine has its own stack. When

    it runs out, the preamble calls morestack. We don't, so we have to fit in a fixed stack size. (This is not ok.) TEXT ·MultiplyByTwo(SB), 0, $2048-16
  46. Args, returns and calling conventions What we did: made a

    trampoline from the Go calling convention to the C one. How we did it: sacrilegious assembly.
  47. Bonus: distribution

  48. //go:binary-only-package package multiplication import _ "unsafe" //go:cgo_import_static multiply_two //go:cgo_import_dynamic multiply_two

    //go:linkname multiply_two multiply_two var multiply_two uintptr var _multiply_two = &multiply_two func MultiplyByTwo(n uint64) uint64
  49. Thank you! Filippo Valsorda — @FiloSottile https://blog.filippo.io/rustgo