Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Calling Rust from Go, without cgo @ GothamGo 2017

Calling Rust from Go, without cgo @ GothamGo 2017

AKA "Building your own FFI"

https://blog.filippo.io/rustgo

Filippo Valsorda

October 05, 2017
Tweet

More Decks by Filippo Valsorda

Other Decks in Programming

Transcript

  1. From Go to Rust
    without cgo
    Filippo Valsorda

    View Slide

  2. cgo
    Foreign function interface (FFI). Call C code from Go.

    View Slide

  3. cgo
    Foreign function interface (FFI). Call C code from Go.
    • Code generator tool
    • FFI internals to make C happy
    • Safety features to make Go less unhappy
    • Utility libraries and types

    View Slide

  4. From Go to Rust
    without cgo
    Filippo Valsorda
    DIY FFI

    View Slide

  5. FFI: starting kit
    A way to:
    • Compile the foreign code together with the Go code
    • Jump to the foreign code by function name
    • Share arguments, return values and memory
    • Bonus: packaging and redistribution

    View Slide

  6. Foreign code

    View Slide

  7. #![no_std]
    #[no_mangle]
    pub extern fn multiply_two(a: u64) -> u64 {
    return a * 2;
    }
    [package]
    name = "multiplication"
    version = "0.0.1"
    [lib]
    crate-type = ["staticlib"]

    View Slide

  8. Compiling the foreign
    code together

    View Slide

  9. $GOROOT/pkg/tool/darwin_amd64/compile \
    -o $WORK/.../rustgo.a [...] -pack ./main.go
    $GOROOT/pkg/tool/darwin_amd64/link \
    -o $WORK/.../a.out -buildmode=exe [...] \
    -linkmode=internal $WORK/.../rustgo.a
    mv $WORK/.../a.out rustgo
    go build -x (package main)

    View Slide

  10. .go
    .go
    .go
    package
    .go
    package
    cmd/compile
    cmd/compile
    .o
    .o
    .a
    .o
    .o
    .a binary
    cmd/link
    symbol "x"
    symbol "y"
    symbol "foo"
    symbol "main"
    ...

    View Slide

  11. Internal linking
    cmd/link
    Go
    object files darwin_amd64 binary
    linux_amd64 binary
    linux_arm binary

    View Slide

  12. External linking
    cmd/link
    Go object files platform binary
    ld
    foreign code
    static and dynamic
    libraries

    View Slide

  13. go build -ldflags='-linkmode=external \
    -extldflags=.../libmultiplication.a'
    External linking
    the Rust static library

    View Slide

  14. $GOROOT/pkg/tool/darwin_amd64/compile \
    -o $WORK/.../multiplication.a [...] -pack ./mul.go
    mv $WORK/.../multiplication.a \
    $GOPATH/pkg/darwin_amd64/.../multiplication.a
    go build -i -x (package multiplication)

    View Slide

  15. .go
    .go
    .go
    package
    .go
    package
    cmd/compile
    cmd/compile
    .o
    .o
    .a
    .o
    .o
    .a binary
    cmd/link
    symbol "x"
    symbol "y"
    symbol "foo"
    symbol "main"
    ...

    View Slide

  16. .o
    .o
    .a
    symbol "x"
    symbol "y"
    symbol "foo"
    symbol "main"
    ...
    $GOPATH/pkg/.../

    View Slide

  17. View Slide

  18. ar xv .../libmultiplication.a # the Rust library
    Stuffing .a files
    go tool pack r \
    $GOPATH/pkg/darwin_amd64/.../multiplication.a *.o

    View Slide

  19. .o
    .o
    .a
    .o
    .o
    the Rust static library
    .o
    .o
    .o
    .o
    ar cmd/pack
    .o
    .o
    .a
    .o
    .o
    $GOPATH/pkg/.../

    View Slide

  20. ld -r -o .../libmultiplication.o --gc-sections \
    -u multiply_two .../libmultiplication.a
    Stuffing .a files
    go tool pack r \
    $GOPATH/pkg/darwin_amd64/.../multiplication.a \
    .../libmultiplication.o

    View Slide

  21. .o
    .o
    .a
    .o
    .o
    the Rust static library
    ld -r -u cmd/pack
    .o
    .o
    .a
    .o
    .o
    .o
    $GOPATH/pkg/.../

    View Slide

  22. Compiling the foreign code together
    What we did: put the Rust code with the
    multiply_two function inside the Go package
    How we did it: ld -r -u and cmd/pack

    View Slide

  23. Calling the foreign
    symbol

    View Slide

  24. package multiplication
    func MultiplyByTwo(n uint64) uint64
    .go
    TEXT ·MultiplyByTwo(SB), 0, $16384-8
    RET
    .s

    View Slide

  25. View Slide

  26. //go:cgo_import_static multiply_two
    //go:cgo_import_dynamic multiply_two
    //go:linkname multiply_two multiply_two
    var multiply_two uintptr
    var _multiply_two = &multiply_two
    .go
    TEXT MultiplyByTwo(SB), 0, $16384-8
    MOVQ ·_multiply_two(SB), AX
    JMP AX
    RET
    .s

    View Slide

  27. Calling the foreign symbol
    What we did: got the linker to put a pointer to the
    Rust function in _multiply_two
    How we did it: //go:cgo_import_xxx

    View Slide

  28. Args, returns and

    calling conventions

    View Slide

  29. View Slide

  30. a := multiplyByTwo(0x5555555555555555)
    func multiplyByTwo(a uint64) uint64 {
    return a * 2
    }
    (lldb) disassemble -n main.main

    View Slide

  31. movabsq $0x5555555555555555, %rax
    movq %rax, (%rsp)
    callq 0x104f360 ; main.multiplyByTwo at main.go:9
    PUSH the argument and CALL

    View Slide

  32. xxx`main.multiplyByTwo:
    movq %gs:0x8a0, %rcx
    leaq -0x8(%rsp), %rax
    cmpq 0x10(%rcx), %rax
    jbe 0x104f3e1 ; <+129> at main.go:9
    [...]
    Stack size check preamble

    View Slide

  33. subq $0x88, %rsp
    [...]
    addq $0x88, %rsp
    retq
    Stack pointer

    View Slide

  34. View Slide

  35. subq $0x88, %rsp
    [...]
    addq $0x88, %rsp
    retq
    Stack pointer

    View Slide

  36. movq %rbp, 0x80(%rsp)
    leaq 0x80(%rsp), %rbp
    [...]
    movq 0x80(%rsp), %rbp
    Frame pointer

    View Slide

  37. movq 0x90(%rsp), %rax
    Argument

    View Slide

  38. movq %rax, 0x98(%rsp)
    Return value

    View Slide

  39. The C calling convention
    Or "sysv64" on x86_64.
    • Arguments: RDI, RSI, RDX, RCX, R8, and R9
    • Return value: RAX
    •Stack alignment: 16 bytes
    Default for Rust extern functions.

    View Slide

  40. TEXT ·MultiplyByTwo(SB), 0, $2048-16
    MOVQ n+0(FP), DI // Load the argument before messing with SP
    MOVQ SP, BX // Save SP in a callee-saved registry
    ADDQ $2048, SP // Rollback SP to reuse this function's frame
    ANDQ $~15, SP // Align the stack to 16-bytes
    MOVQ ·_multiply_two(SB), AX
    CALL AX
    MOVQ BX, SP // Restore SP
    MOVQ AX, ret+8(FP) // Place the return value on the stack
    RET
    Trampoline

    View Slide

  41. package main
    import "./multiplication"
    func main() {
    println(multiplication.MultiplyByTwo(21))
    }

    View Slide

  42. TEXT ·MultiplyByTwo(SB), 0, $2048-16
    MOVQ n+0(FP), DI // Load the argument before messing with SP
    MOVQ SP, BX // Save SP in a callee-saved registry
    ADDQ $2048, SP // Rollback SP to reuse this function's frame
    ANDQ $~15, SP // Align the stack to 16-bytes
    MOVQ ·_multiply_two(SB), AX
    CALL AX
    MOVQ BX, SP // Restore SP
    MOVQ AX, ret+8(FP) // Place the return value on the stack
    RET

    View Slide

  43. #[no_mangle]
    pub extern fn multiply_two(a: u64) -> u64 {
    return a * 2;
    }

    View Slide

  44. $ make install
    cargo build --release
    ld -r -u _multiply_two
    go tool asm
    go tool compile
    go tool pack
    cp multiplication/multiplication.a $GOPATH/pkg/...
    $ go build .
    $ ./rustgo
    4

    View Slide

  45. Go small stacks
    Each goroutine has its own stack.
    When it runs out, the preamble calls morestack.
    We don't, so we have to fit in a fixed stack size.
    (This is not ok.)
    TEXT ·MultiplyByTwo(SB), 0, $2048-16

    View Slide

  46. Args, returns and calling conventions
    What we did: made a trampoline from the Go
    calling convention to the C one.
    How we did it: sacrilegious assembly.

    View Slide

  47. Bonus: distribution

    View Slide

  48. //go:binary-only-package
    package multiplication
    import _ "unsafe"
    //go:cgo_import_static multiply_two
    //go:cgo_import_dynamic multiply_two
    //go:linkname multiply_two multiply_two
    var multiply_two uintptr
    var _multiply_two = &multiply_two
    func MultiplyByTwo(n uint64) uint64

    View Slide

  49. Thank you!
    Filippo Valsorda — @FiloSottile
    https://blog.filippo.io/rustgo

    View Slide