CGO journey: from using to understanding it

Transcript

1. CGO journey: from
   using to
   understanding it
   José Carlos Chávez
   @jcchavezs January Golang
   Meetup - GDG
   Berlin Golang
   

   View Slide

2. 2
   About me
   José Carlos Chávez
   - Software Engineer at
   Traceable.ai
   - Open source contributor in
   Observability projects
   - Maths student
   - Loving father
   @jcchavezs
   

   View Slide

3. What is CGO?
   Cgo lets Go packages call C code.
   Given a Go source file written with
   some special features, cgo outputs
   Go and C files that can be
   combined into a single Go package.
   Andrew Gerrand
   https://go.dev/blog/cgo
   3
   @jcchavezs Foto de Anne Nygård en Unsplash
   

   View Slide

4. There are many libraries written in
   C or C++. If we are willing to use
   them there are only a couple of
   options to go for:
   1. Rewrite them in Go
   2. Use them directly with CGO
   Why is CGO useful?
   4
   @jcchavezs
   

   View Slide

5. // source: main.go
   package main
   //int sum(int a, int b) { return a+b; }
   import "C"
   func main() {
   println(C.sum(1, 1))
   }
   5
   @jcchavezs
   How does it look like?
   

   View Slide

6. // source: main.go
   package main
   // Linker Options:
   #cgo darwin LDFLAGS: -framework Cocoa -framework OpenGL
   -framework IOKit -framework CoreVideo
   // Linux Build Tags
   #cgo linux,!wayland CFLAGS: -D_GLFW_X11 -D_GLFW_GLX
   #cgo linux,wayland CFLAGS: -D_GLFW_WAYLAND -D_GLFW_EGL
   ...
   6
   @jcchavezs
   How does it really look like?
   

   View Slide

7. 7
   Looking inside
   

   View Slide

8. 8
   @jcchavezs
   What happens inside?
   $ go tool cgo main.go
   $ ls _obj | awk '{print $NF}'
   >>_cgo_.o
   >>_cgo_export.c
   >>_cgo_export.h
   >>_cgo_flags
   >>_cgo_gotypes.go
   >>_cgo_main.c
   >>main.cgo1.go
   >>main.cgo2.c
   

   View Slide

9. //main.cgo1.go
   package main
   //int sum(int a, int b) { return a+b; }
   import _ "unsafe"
   func main() {
   println(( /*line :11:13*/_Cfunc_sum /*line :11:17*/)(1, 1))
   }
   9
   @jcchavezs
   What happens inside?
   

   View Slide

10. 10
    @jcchavezs
    Stack dance
    

    View Slide

11. Some people, when confronted with a
    problem, think “I know, I’ll use cgo.”
    Now they have two problems.
    Dave Cheney - cgo is not Go
    11
    @jcchavezs
    

    View Slide

12. - Programming gotchas
    - Build gotchas
    - Runtime gotchas
    - Tooling gotchas
    Gotchas
    12
    @jcchavezs
    

    View Slide

13. Programming gotchas
    13
    Manual memory management
    - Go is a garbage-collected runtime,
    but C is not.
    - Copying data aren’t always avoidable
    and eats up CPU.
    - Slices can be cumbersome due to its
    lifecycle.
    Data conversion
    - Most of the types are 1-1 mapping but
    some special cases use
    unsafe.pointer (e.g. string).
    - Special types lead to manual memory
    management.
    - C type used in one Go package is
    different from the same C type used
    in another.
    @jcchavezs
    

    View Slide

14. Programming gotchas
    14
    Passing a function pointer to C
    Callbacks’ overhead: Go code may pass a
    Go pointer to C provided the Go memory
    to which it points does not contain any
    Go pointers.
    See: https://golang.org/cmd/cgo/#hdr-Passing_pointers
    Cgoroutines != goroutines
    A blocking cgo call occupies a system
    thread. Go runtime can’t schedule them
    like it can a goroutine: Kb vs Mb in
    allocations.
    More: http://bit.ly/cgo-bench-test
    @jcchavezs
    

    View Slide

15. Build gotchas
    15
    Complicated builds
    - The build process is not anymore self
    contained: e.g. go get, go build and go
    test now require preconditions (e.g.
    ldflags, library location, etc)
    - Build caching mechanics will impose a
    structure for C/library code.
    - Loading libraries is cumbersome. The
    location of the library can be a pain
    depending on the nature of the
    source code:
    - Go applications
    - Go libraries
    Slower build times
    - The cgo tool is invoked to generate
    the C to Go and Go to C thunks and
    stubs.
    - Your system C compiler has to be
    invoked for every C file in the
    package.
    - The individual compilation units are
    combined together into a single .o file.
    - The resulting .o file take a trip through
    the system linker for fix-ups against
    shared objects they reference.
    @jcchavezs
    

    View Slide

16. Build gotchas
    16
    Static builds
    - The premise “single, static binary”
    isn’t easily achievable.
    - Unless static libraries you can’t
    guarantee 100% consistency between
    test - build - run.
    (Lack of) Cross compilation
    - By default CGO is disabled when
    cross compiling. You can still enable it
    passing a C cross-compiler for each
    GOOS+GOARCH
    - One extra degree of compatibility is
    added by the C library you are loading.
    @jcchavezs
    

    View Slide

17. Runtime gotchas
    17
    Dynamic linking
    - Usage of different libraries for
    different architectures (e.g. amd64,
    arm64). Not even talking about
    distros.
    - If a library isn’t installed in the host
    we need to link the library in runtime
    using LD_LIBRARY_PATH or a CWD
    relative location (predefined in the
    code).
    - It is possible to link the library after a
    lookup but that requires a manual
    loading (see example)
    Call overhead
    - CGO calls are expensive compared to
    a call within Go, although the
    overhead is negligible (see this
    benchmark)
    @jcchavezs
    

    View Slide

18. Tooling gotchas
    18
    Debugging
    - The portions residing in C aren’t as
    readily accessed through Go’s tooling.
    - Race detector, pprof for profiling
    code, coverage, fuzz testing, and
    source code analysis tools don’t work
    across the board
    @jcchavezs
    

    View Slide

19. 19
    @jcchavezs
    Summary
    - CGO is not a union, is an intersection
    of C and Go worlds.
    - Using libraries introduce another layer
    of complexity, they don’t come for
    free.
    - CGO was a foundational piece in Go
    evolution.
    

    View Slide

20. 20
    References
    ● https://blog.marlin.org/cgo-referencing-c-library-in-go
    ● https://pkg.go.dev/cmd/cgo#hdr-Passing_pointers
    ● https://www.programmerall.com/article/2072595424/
    ● https://medium.com/mysterium-network/golang-c-interopera
    bility-caf0ba9f7bf3
    ● https://www.cockroachlabs.com/blog/the-cost-and-complexi
    ty-of-cgo/
    ● https://honnef.co/posts/2015/06/statically_compiled_go_p
    rograms__always__even_with_cgo__using_musl/
    @jcchavezs
    

    View Slide