聊聊 Cgo 的二三事

Transcript

1. 聊聊 Cgo 的⼆三事 David Chou @ Golang Taipei / Crescendo Lab CC-BY-SA-3.0-TW
2. @ Crescendo Lab @ Golang Taipei Co-organizer Software engineer, DevOps, and Gopher david74.chou @ gmail david74.chou @ facebook david74chou @ telegram
3. What is Cgo? Cgo lets Go packages call C code
4. package main /* #include <stdlib.h> */ import "C" import "fmt" func Random() int { r := C.random() // r is C.long return int(r) } func Seed(i int) { C.srandom(C.uint(i)) } func main() { Seed(0) for i := 0; i < 10; i++ { fmt.Printf("rand(%d): %d\n", i, Random()) } } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 /* #include <stdlib.h> */ import "C" package main 1 2 3 4 5 6 import "fmt" 7 8 func Random() int { 9 r := C.random() // r is C.long 10 return int(r) 11 } 12 13 func Seed(i int) { 14 C.srandom(C.uint(i)) 15 } 16 17 func main() { 18 Seed(0) 19 20 for i := 0; i < 10; i++ { 21 fmt.Printf("rand(%d): %d\n", i, Random()) 22 } 23 } 24 r := C.random() // r is C.long return int(r) C.srandom(C.uint(i)) package main 1 2 /* 3 #include <stdlib.h> 4 */ 5 import "C" 6 import "fmt" 7 8 func Random() int { 9 10 11 } 12 13 func Seed(i int) { 14 15 } 16 17 func main() { 18 Seed(0) 19 20 for i := 0; i < 10; i++ { 21 fmt.Printf("rand(%d): %d\n", i, Random()) 22 } 23 } 24
5. // #cgo CFLAGS: -DPNG_DEBUG=1 // #cgo amd64 386 CFLAGS: -DX86=1 // #cgo LDFLAGS: -lpng // #include <png.h> import "C" Cgo Preamble CFLAGS / CPPFLAGS / LDFLAGS Add build constraints
6. // #cgo pkg-config: png // #cgo CFLAGS: -DPNG_DEBUG=1 // #include <png.h> import "C" Cgo Preamble Use pkg-config tool
7. // #cgo LDFLAGS: -L${SRCDIR}/libs -lfoo import "C" Cgo Preamble Use pre-compiled library in the package
8. package native /* #cgo CFLAGS: -I${SRCDIR}/includes/ #cgo windows LDFLAGS: ${SRCDIR}/libs/winlibz.a #cgo linux LDFLAGS: ${SRCDIR}/libs/linuxlibz.a #cgo darwin LDFLAGS: ${SRCDIR}/libs/darwinlibz.a */ import "C" . ├── cgo.go ├── libs │ ├── darwinlibz.a │ ├── linuxlibz.a │ └── winlibz.a └── includes ├── zconf.h └── zlib.h
9. C types in Cgo char => C.char unsigned char => C.uchar int => C.int void* => unsafe.Pointer struct S => C.struct_S enum E => C.enum_E
10. Data conversion in Cgo func C.CString(string) *C.char func C.CBytes([]byte) unsafe.Pointer func C.free(unsafe.Pointer) Copy Go string / []byte to C Allocated with malloc() Must be freed with C.free()
11. Data conversion in Cgo func C.GoString(*C.char) string func C.GoStringN(*C.char, C.int) string func C.GoBytes(unsafe.Pointer, C.int) []byte Copy C string / bytes to Go
12. /* #include <stdlib.h> char *get_string() { return strdup("hello from C"); } void release_string(char *s) { free(s); } */ import "C" func main() { cs := C.CString("Hello from go\n") C.puts(cs) C.free(unsafe.Pointer(cs)) cs = C.get_string() gs := C.GoString(cs) fmt.Printf("%s\n", gs) C.release_string(cs) }
14. “ Cgo is not Go - Rob Pike
15. Why not to use Cgo Break Go’s awesome tooling Break your static binary Break cross-compiling Manage memory in C by hand Cgo calls are much slower than native Go calls
16. //#include <unistd.h> //void foo() { } import "C" //go:noinline func foo() {} func CallCgo(n int) { for i := 0; i < n; i++ { C.foo() } } func CallGo(n int) { for i := 0; i < n; i++ { foo() } } BenchmarkCGO-16 3772150 308.3 ns/op BenchmarkGo-16 931552690 1.231 ns/op
17. When to use Cgo No Go equivalent library Integrate with legacy C code Need to consume a proprietary library in C
18. Any other choice?
19. Cgo Tips and Pitfalls
20. Write your own C bridge func Get more control with your C interface More precise in C header Merge multiple Cgo calls into 1 Do loop-calls in C is more efficient foos := []C.struct_Foo{ { a: C.int(1), b: C.int(2) }, { a: C.int(3), b: C.int(4) }, } C.pass_array((*C.struct_Foo)(unsafe.Pointer(&foos[0])), C.int(len(foos)))
21. Isolate the Cgo part Isolate the Cgo wrapper into its own package even in a standalone process and communicate with RPC Can't call Cgo in Go test code
22. Use static link library Try hard to static link your libraries, otherwise your dependency might be complicated go build -ldflags='-extldflags "-static"'
23. Be careful for memory mgmt Never keep any pointer outside of the called function the memory might be GC or free
24. Be careful for memory mgmt Be aware of the C-memory ownership Use "defer C.free()" to release memory when func returns d := C.GoBytes([]byte{...}) C.foo(d) // <- if foo() takes the ownership // C.free(d)
25. Be careful for memory mgmt ZeroCopy: Go => C func Foo(payload []byte) { p := (*C.uchar)(unsafe.Pointer(&payload[0])) size := C.int(len(payload)) C.foo(p, size) } func Foo(str string) { strHdr := (*reflect.StringHeader)(unsafe.Pointer(&str)) p := (*C.char)(unsafe.Pointer(strHdr.Data)) size := C.int(len(payload)) C.foo(p, size) } ZeroCopy (Go => C): []bytes ZeroCopy (Go => C): string
26. Cgoroutines != Goroutines Cgo calls block your system thread func main() { var wg sync.WaitGroup wg.Add(1000) for i := 0; i < 1000; i++ { go func() { time.Sleep(100 * time.Second) wg.Done() }() } wg.Wait() } func main() { var wg sync.WaitGroup wg.Add(1000) for i := 0; i < 1000; i++ { go func() { time.Sleep(100 * time.Second) wg.Done() }() } wg.Wait() } // use 13 threads // use 1004 threads
27. Cgoroutines != Goroutines // Call from Go to C. func cgocall(fn, arg unsafe.Pointer) int32 { ... // Announce we are entering a system call // so that the scheduler knows to create another // M to run goroutines while we are in the // foreign code. // entersyscall() errno := asmcgocall(fn, arg) exitsyscall() ... }
28. C callbacks to Go Go function could be exported for use by C //export Add func Add(arg1, arg2 int) int {...} It will be available in the C code as extern int Add(int arg1, int arg2);
29. C callbacks to Go C can't callbacks to a receiver function because of Cgo pointer passing rule need to be creative func (c *MyCounter) Add(i int) { c.count += i }
30. var mu sync.Mutex var index int var fns = make(map[int]func(int)) func register(fn func(int)) int { mu.Lock() defer mu.Unlock() index++ for fns[index] != nil { index++ } fns[index] = fn return index } func lookup(i int) func(int) { ... } func unregister(i int) { ... } Use an extra map to keep the callbacks
31. /* extern void go_callback_int( int foo, int p1); static inline void CallMyFunction(int foo) { go_callback_int(foo, 5); } */ import "C" //export go_callback_int func go_callback_int(idx C.int, p1 C.int) { fn := lookup(int(idx)) fn(int(p1)) } func (c *MyCounter) Add(i int) { ... } func main() { c := &MyCounter{count: 0} i := register(c.Add) C.CallMyFunction(C.int(i)) unregister(i) }
32. /* extern void go_callback_int( uintptr_t h, int p1); static inline void CallMyFunction(uintptr_t h) { go_callback_int(h, 5); } */ import "C" //export go_callback_int func go_callback_int(h C.uintptr_t, p C.int){ v := cgo.Handle(h).Value() fn := v.(func(C.int)) fn(p) } func (c *MyCounter) Add(i int) { ... } func main() { c := &MyCounter{count: 0} i := register(c.Add) h := cgo.NewHandle(c.Add) C.CallMyFunction(C.uintptr_t(h)) h.Delete() } Go 1.17 cgo.Handle
33. Cgo preamble C <=> Go type conversion When to / not-to use Cgo Cgo tips and pitfalls Recap
34. We are hiring now! Feel free to chat with us in Room TR 312 ( 研揚⼤樓 312 室).