Calling Rust from Go, without cgo @ GothamGo 2017

Slide 1

Slide 1 text

From Go to Rust without cgo Filippo Valsorda

Slide 2

Slide 2 text

cgo Foreign function interface (FFI). Call C code from Go.

Slide 3

Slide 3 text

cgo Foreign function interface (FFI). Call C code from Go. • Code generator tool • FFI internals to make C happy • Safety features to make Go less unhappy • Utility libraries and types

Slide 4

Slide 4 text

From Go to Rust without cgo Filippo Valsorda DIY FFI

Slide 5

Slide 5 text

FFI: starting kit A way to: • Compile the foreign code together with the Go code • Jump to the foreign code by function name • Share arguments, return values and memory • Bonus: packaging and redistribution

Slide 6

Slide 6 text

Foreign code

Slide 7

Slide 7 text

#![no_std] #[no_mangle] pub extern fn multiply_two(a: u64) -> u64 { return a * 2; } [package] name = "multiplication" version = "0.0.1" [lib] crate-type = ["staticlib"]

Slide 8

Slide 8 text

Compiling the foreign code together

Slide 9

Slide 9 text

$GOROOT/pkg/tool/darwin_amd64/compile \ -o $WORK/.../rustgo.a [...] -pack ./main.go $GOROOT/pkg/tool/darwin_amd64/link \ -o $WORK/.../a.out -buildmode=exe [...] \ -linkmode=internal $WORK/.../rustgo.a mv $WORK/.../a.out rustgo go build -x (package main)

Slide 10

Slide 10 text

.go .go .go package .go package cmd/compile cmd/compile .o .o .a .o .o .a binary cmd/link symbol "x" symbol "y" symbol "foo" symbol "main" ...

Slide 11

Slide 11 text

Internal linking cmd/link Go object ﬁles darwin_amd64 binary linux_amd64 binary linux_arm binary

Slide 12

Slide 12 text

External linking cmd/link Go object ﬁles platform binary ld foreign code static and dynamic libraries

Slide 13

Slide 13 text

go build -ldflags='-linkmode=external \ -extldflags=.../libmultiplication.a' External linking the Rust static library

Slide 14

Slide 14 text

$GOROOT/pkg/tool/darwin_amd64/compile \ -o $WORK/.../multiplication.a [...] -pack ./mul.go mv $WORK/.../multiplication.a \ $GOPATH/pkg/darwin_amd64/.../multiplication.a go build -i -x (package multiplication)

Slide 15

Slide 15 text

.go .go .go package .go package cmd/compile cmd/compile .o .o .a .o .o .a binary cmd/link symbol "x" symbol "y" symbol "foo" symbol "main" ...

Slide 16

Slide 16 text

.o .o .a symbol "x" symbol "y" symbol "foo" symbol "main" ... $GOPATH/pkg/.../

Slide 17

Slide 17 text

No content

Slide 18

Slide 18 text

ar xv .../libmultiplication.a # the Rust library Stuﬃng .a ﬁles go tool pack r \ $GOPATH/pkg/darwin_amd64/.../multiplication.a *.o

Slide 19

Slide 19 text

.o .o .a .o .o the Rust static library .o .o .o .o ar cmd/pack .o .o .a .o .o $GOPATH/pkg/.../

Slide 20

Slide 20 text

ld -r -o .../libmultiplication.o --gc-sections \ -u multiply_two .../libmultiplication.a Stuﬃng .a ﬁles go tool pack r \ $GOPATH/pkg/darwin_amd64/.../multiplication.a \ .../libmultiplication.o

Slide 21

Slide 21 text

.o .o .a .o .o the Rust static library ld -r -u cmd/pack .o .o .a .o .o .o $GOPATH/pkg/.../

Slide 22

Slide 22 text

Compiling the foreign code together What we did: put the Rust code with the multiply_two function inside the Go package How we did it: ld -r -u and cmd/pack

Slide 23

Slide 23 text

Calling the foreign symbol

Slide 24

Slide 24 text

package multiplication func MultiplyByTwo(n uint64) uint64 .go TEXT ·MultiplyByTwo(SB), 0, $16384-8 RET .s

Slide 25

Slide 25 text

No content

Slide 26

Slide 26 text

//go:cgo_import_static multiply_two //go:cgo_import_dynamic multiply_two //go:linkname multiply_two multiply_two var multiply_two uintptr var _multiply_two = &multiply_two .go TEXT MultiplyByTwo(SB), 0, $16384-8 MOVQ ·_multiply_two(SB), AX JMP AX RET .s

Slide 27

Slide 27 text

Calling the foreign symbol What we did: got the linker to put a pointer to the Rust function in _multiply_two How we did it: //go:cgo_import_xxx

Slide 28

Slide 28 text

Args, returns and  calling conventions

Slide 29

Slide 29 text

No content

Slide 30

Slide 30 text

a := multiplyByTwo(0x5555555555555555) func multiplyByTwo(a uint64) uint64 { return a * 2 } (lldb) disassemble -n main.main

Slide 31

Slide 31 text

movabsq $0x5555555555555555, %rax movq %rax, (%rsp) callq 0x104f360 ; main.multiplyByTwo at main.go:9 PUSH the argument and CALL

Slide 32

Slide 32 text

xxx`main.multiplyByTwo: movq %gs:0x8a0, %rcx leaq -0x8(%rsp), %rax cmpq 0x10(%rcx), %rax jbe 0x104f3e1 ; <+129> at main.go:9 [...] Stack size check preamble

Slide 33

Slide 33 text

subq $0x88, %rsp [...] addq $0x88, %rsp retq Stack pointer

Slide 34

Slide 34 text

No content

Slide 35

Slide 35 text

subq $0x88, %rsp [...] addq $0x88, %rsp retq Stack pointer

Slide 36

Slide 36 text

movq %rbp, 0x80(%rsp) leaq 0x80(%rsp), %rbp [...] movq 0x80(%rsp), %rbp Frame pointer

Slide 37

Slide 37 text

movq 0x90(%rsp), %rax Argument

Slide 38

Slide 38 text

movq %rax, 0x98(%rsp) Return value

Slide 39

Slide 39 text

The C calling convention Or "sysv64" on x86_64. • Arguments: RDI, RSI, RDX, RCX, R8, and R9 • Return value: RAX •Stack alignment: 16 bytes Default for Rust extern functions.

Slide 40

Slide 40 text

TEXT ·MultiplyByTwo(SB), 0, $2048-16 MOVQ n+0(FP), DI // Load the argument before messing with SP MOVQ SP, BX // Save SP in a callee-saved registry ADDQ $2048, SP // Rollback SP to reuse this function's frame ANDQ $~15, SP // Align the stack to 16-bytes MOVQ ·_multiply_two(SB), AX CALL AX MOVQ BX, SP // Restore SP MOVQ AX, ret+8(FP) // Place the return value on the stack RET Trampoline

Slide 41

Slide 41 text

package main import "./multiplication" func main() { println(multiplication.MultiplyByTwo(21)) }

Slide 42

Slide 42 text

Slide 43

Slide 43 text

#[no_mangle] pub extern fn multiply_two(a: u64) -> u64 { return a * 2; }

Slide 44

Slide 44 text

$ make install cargo build --release ld -r -u _multiply_two go tool asm go tool compile go tool pack cp multiplication/multiplication.a $GOPATH/pkg/... $ go build . $ ./rustgo 4

Slide 45

Slide 45 text

Go small stacks Each goroutine has its own stack. When it runs out, the preamble calls morestack. We don't, so we have to ﬁt in a ﬁxed stack size. (This is not ok.) TEXT ·MultiplyByTwo(SB), 0, $2048-16

Slide 46

Slide 46 text

Args, returns and calling conventions What we did: made a trampoline from the Go calling convention to the C one. How we did it: sacrilegious assembly.

Slide 47

Slide 47 text

Bonus: distribution

Slide 48

Slide 48 text

//go:binary-only-package package multiplication import _ "unsafe" //go:cgo_import_static multiply_two //go:cgo_import_dynamic multiply_two //go:linkname multiply_two multiply_two var multiply_two uintptr var _multiply_two = &multiply_two func MultiplyByTwo(n uint64) uint64

Slide 49

Slide 49 text

Thank you! Filippo Valsorda — @FiloSottile https://blog.ﬁlippo.io/rustgo