Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Calling Rust from Go, without cgo @ GothamGo 2017

Calling Rust from Go, without cgo @ GothamGo 2017

AKA "Building your own FFI"

https://blog.filippo.io/rustgo

Filippo Valsorda

October 05, 2017
Tweet

More Decks by Filippo Valsorda

Other Decks in Programming

Transcript

  1. From Go to Rust
    without cgo
    Filippo Valsorda

    View full-size slide

  2. cgo
    Foreign function interface (FFI). Call C code from Go.

    View full-size slide

  3. cgo
    Foreign function interface (FFI). Call C code from Go.
    • Code generator tool
    • FFI internals to make C happy
    • Safety features to make Go less unhappy
    • Utility libraries and types

    View full-size slide

  4. From Go to Rust
    without cgo
    Filippo Valsorda
    DIY FFI

    View full-size slide

  5. FFI: starting kit
    A way to:
    • Compile the foreign code together with the Go code
    • Jump to the foreign code by function name
    • Share arguments, return values and memory
    • Bonus: packaging and redistribution

    View full-size slide

  6. Foreign code

    View full-size slide

  7. #![no_std]
    #[no_mangle]
    pub extern fn multiply_two(a: u64) -> u64 {
    return a * 2;
    }
    [package]
    name = "multiplication"
    version = "0.0.1"
    [lib]
    crate-type = ["staticlib"]

    View full-size slide

  8. Compiling the foreign
    code together

    View full-size slide

  9. $GOROOT/pkg/tool/darwin_amd64/compile \
    -o $WORK/.../rustgo.a [...] -pack ./main.go
    $GOROOT/pkg/tool/darwin_amd64/link \
    -o $WORK/.../a.out -buildmode=exe [...] \
    -linkmode=internal $WORK/.../rustgo.a
    mv $WORK/.../a.out rustgo
    go build -x (package main)

    View full-size slide

  10. .go
    .go
    .go
    package
    .go
    package
    cmd/compile
    cmd/compile
    .o
    .o
    .a
    .o
    .o
    .a binary
    cmd/link
    symbol "x"
    symbol "y"
    symbol "foo"
    symbol "main"
    ...

    View full-size slide

  11. Internal linking
    cmd/link
    Go
    object files darwin_amd64 binary
    linux_amd64 binary
    linux_arm binary

    View full-size slide

  12. External linking
    cmd/link
    Go object files platform binary
    ld
    foreign code
    static and dynamic
    libraries

    View full-size slide

  13. go build -ldflags='-linkmode=external \
    -extldflags=.../libmultiplication.a'
    External linking
    the Rust static library

    View full-size slide

  14. $GOROOT/pkg/tool/darwin_amd64/compile \
    -o $WORK/.../multiplication.a [...] -pack ./mul.go
    mv $WORK/.../multiplication.a \
    $GOPATH/pkg/darwin_amd64/.../multiplication.a
    go build -i -x (package multiplication)

    View full-size slide

  15. .go
    .go
    .go
    package
    .go
    package
    cmd/compile
    cmd/compile
    .o
    .o
    .a
    .o
    .o
    .a binary
    cmd/link
    symbol "x"
    symbol "y"
    symbol "foo"
    symbol "main"
    ...

    View full-size slide

  16. .o
    .o
    .a
    symbol "x"
    symbol "y"
    symbol "foo"
    symbol "main"
    ...
    $GOPATH/pkg/.../

    View full-size slide

  17. ar xv .../libmultiplication.a # the Rust library
    Stuffing .a files
    go tool pack r \
    $GOPATH/pkg/darwin_amd64/.../multiplication.a *.o

    View full-size slide

  18. .o
    .o
    .a
    .o
    .o
    the Rust static library
    .o
    .o
    .o
    .o
    ar cmd/pack
    .o
    .o
    .a
    .o
    .o
    $GOPATH/pkg/.../

    View full-size slide

  19. ld -r -o .../libmultiplication.o --gc-sections \
    -u multiply_two .../libmultiplication.a
    Stuffing .a files
    go tool pack r \
    $GOPATH/pkg/darwin_amd64/.../multiplication.a \
    .../libmultiplication.o

    View full-size slide

  20. .o
    .o
    .a
    .o
    .o
    the Rust static library
    ld -r -u cmd/pack
    .o
    .o
    .a
    .o
    .o
    .o
    $GOPATH/pkg/.../

    View full-size slide

  21. Compiling the foreign code together
    What we did: put the Rust code with the
    multiply_two function inside the Go package
    How we did it: ld -r -u and cmd/pack

    View full-size slide

  22. Calling the foreign
    symbol

    View full-size slide

  23. package multiplication
    func MultiplyByTwo(n uint64) uint64
    .go
    TEXT ·MultiplyByTwo(SB), 0, $16384-8
    RET
    .s

    View full-size slide

  24. //go:cgo_import_static multiply_two
    //go:cgo_import_dynamic multiply_two
    //go:linkname multiply_two multiply_two
    var multiply_two uintptr
    var _multiply_two = &multiply_two
    .go
    TEXT MultiplyByTwo(SB), 0, $16384-8
    MOVQ ·_multiply_two(SB), AX
    JMP AX
    RET
    .s

    View full-size slide

  25. Calling the foreign symbol
    What we did: got the linker to put a pointer to the
    Rust function in _multiply_two
    How we did it: //go:cgo_import_xxx

    View full-size slide

  26. Args, returns and

    calling conventions

    View full-size slide

  27. a := multiplyByTwo(0x5555555555555555)
    func multiplyByTwo(a uint64) uint64 {
    return a * 2
    }
    (lldb) disassemble -n main.main

    View full-size slide

  28. movabsq $0x5555555555555555, %rax
    movq %rax, (%rsp)
    callq 0x104f360 ; main.multiplyByTwo at main.go:9
    PUSH the argument and CALL

    View full-size slide

  29. xxx`main.multiplyByTwo:
    movq %gs:0x8a0, %rcx
    leaq -0x8(%rsp), %rax
    cmpq 0x10(%rcx), %rax
    jbe 0x104f3e1 ; <+129> at main.go:9
    [...]
    Stack size check preamble

    View full-size slide

  30. subq $0x88, %rsp
    [...]
    addq $0x88, %rsp
    retq
    Stack pointer

    View full-size slide

  31. subq $0x88, %rsp
    [...]
    addq $0x88, %rsp
    retq
    Stack pointer

    View full-size slide

  32. movq %rbp, 0x80(%rsp)
    leaq 0x80(%rsp), %rbp
    [...]
    movq 0x80(%rsp), %rbp
    Frame pointer

    View full-size slide

  33. movq 0x90(%rsp), %rax
    Argument

    View full-size slide

  34. movq %rax, 0x98(%rsp)
    Return value

    View full-size slide

  35. The C calling convention
    Or "sysv64" on x86_64.
    • Arguments: RDI, RSI, RDX, RCX, R8, and R9
    • Return value: RAX
    •Stack alignment: 16 bytes
    Default for Rust extern functions.

    View full-size slide

  36. TEXT ·MultiplyByTwo(SB), 0, $2048-16
    MOVQ n+0(FP), DI // Load the argument before messing with SP
    MOVQ SP, BX // Save SP in a callee-saved registry
    ADDQ $2048, SP // Rollback SP to reuse this function's frame
    ANDQ $~15, SP // Align the stack to 16-bytes
    MOVQ ·_multiply_two(SB), AX
    CALL AX
    MOVQ BX, SP // Restore SP
    MOVQ AX, ret+8(FP) // Place the return value on the stack
    RET
    Trampoline

    View full-size slide

  37. package main
    import "./multiplication"
    func main() {
    println(multiplication.MultiplyByTwo(21))
    }

    View full-size slide

  38. TEXT ·MultiplyByTwo(SB), 0, $2048-16
    MOVQ n+0(FP), DI // Load the argument before messing with SP
    MOVQ SP, BX // Save SP in a callee-saved registry
    ADDQ $2048, SP // Rollback SP to reuse this function's frame
    ANDQ $~15, SP // Align the stack to 16-bytes
    MOVQ ·_multiply_two(SB), AX
    CALL AX
    MOVQ BX, SP // Restore SP
    MOVQ AX, ret+8(FP) // Place the return value on the stack
    RET

    View full-size slide

  39. #[no_mangle]
    pub extern fn multiply_two(a: u64) -> u64 {
    return a * 2;
    }

    View full-size slide

  40. $ make install
    cargo build --release
    ld -r -u _multiply_two
    go tool asm
    go tool compile
    go tool pack
    cp multiplication/multiplication.a $GOPATH/pkg/...
    $ go build .
    $ ./rustgo
    4

    View full-size slide

  41. Go small stacks
    Each goroutine has its own stack.
    When it runs out, the preamble calls morestack.
    We don't, so we have to fit in a fixed stack size.
    (This is not ok.)
    TEXT ·MultiplyByTwo(SB), 0, $2048-16

    View full-size slide

  42. Args, returns and calling conventions
    What we did: made a trampoline from the Go
    calling convention to the C one.
    How we did it: sacrilegious assembly.

    View full-size slide

  43. Bonus: distribution

    View full-size slide

  44. //go:binary-only-package
    package multiplication
    import _ "unsafe"
    //go:cgo_import_static multiply_two
    //go:cgo_import_dynamic multiply_two
    //go:linkname multiply_two multiply_two
    var multiply_two uintptr
    var _multiply_two = &multiply_two
    func MultiplyByTwo(n uint64) uint64

    View full-size slide

  45. Thank you!
    Filippo Valsorda — @FiloSottile
    https://blog.filippo.io/rustgo

    View full-size slide