Moments before main()

GopherCon Poland 2021

Transcript

1. Moments before main()
   GopherCon Poland 2021
   Oleg Kovalov
   🌴
   olegk.dev
   

   View Slide

2. Me
   2
   

   View Slide

3. - Open source addicted gopher
   - Fan of linters (co-author go-critic)
   Me
   3
   

   View Slide

4. - Open source addicted gopher
   - Fan of linters (co-author go-critic)
   - Father of a labrador
   Me
   4
   

   View Slide

5. - Open source addicted gopher
   - Fan of linters (co-author go-critic)
   - Father of a labrador
   - Also Twitter/Telegram @go_perf
   - ...
   Me
   5
   

   View Slide

6. - Open source addicted gopher
   - Fan of linters (co-author go-critic)
   - Father of a labrador
   - Also Twitter/Telegram @go_perf
   - ...
   olegk.dev
   Me
   6
   

   View Slide

7. Agenda
   7
   

   View Slide

8. - Why???
   Agenda
   8
   

   View Slide

9. - Why???
   - We have *.exe
   Agenda
   9
   

   View Slide

10. - Why???
    - We have *.exe
    - Let’s get code from binary
    - Let’s scroll the code
    Agenda
    10
    

    View Slide

11. - Why???
    - We have *.exe
    - Let’s get code from binary
    - Let’s scroll the code
    - Let’s scroll the code
    - Let’s scroll the code
    - Let’s scroll the code
    - Let’s scroll the code
    - Let’s scroll the code
    - Let’s scroll the code
    Agenda
    11
    

    View Slide

12. - Why???
    - We have *.exe
    - Let’s get code from binary
    - Let’s scroll the code
    - Let’s scroll the code
    - Let’s scroll the code
    - Let’s scroll the code
    - Let’s scroll the code
    - Let’s scroll the code
    - Let’s scroll the code
    - Inside stdlib
    Go 1.17.1, OS Darwin (macOS Big Sur), YMMV
    Agenda
    12
    

    View Slide

13. Why???
    13
    

    View Slide

14. Why???
    14
    

    View Slide

15. Why???
    15
    

    View Slide

16. Why???
    16
    

    View Slide

17. Why???
    17
    

    View Slide

18. Life story
    18
    

    View Slide

19. Life story
    19
    Generated by https://github.com/knz/go-binsize-viz
    

    View Slide

20. Life story: runtime part
    20
    Generated by https://github.com/knz/go-binsize-viz
    pclntab
    rodata
    types
    

    View Slide

21. Good tool
    21
    Nice tool by Brad Fitzpatrick https://github.com/bradfitz/shotizam
    pclntab
    rodata
    types
    

    View Slide

22. What is this pclntab?
    22
    

    View Slide

23. Ok that’s pclntab, problem solved
    23
    

    View Slide

24. Hm, maybe not pclntab only
    24
    https://lobste.rs/s/gvtstv/my_go_executable_files_are_still_getting#c_dbfije
    

    View Slide

25. So what I’m trying to say
    25
    

    View Slide

26. - Looks like there is a problem
    So what I’m trying to say
    26
    

    View Slide

27. - Looks like there is a problem
    - But the problem is not so bad
    So what I’m trying to say
    27
    

    View Slide

28. - Looks like there is a problem
    - But the problem is not so bad
    - Docker image is more important
    - if your base image is Ubuntu/Debian
    So what I’m trying to say
    28
    

    View Slide

29. - Looks like there is a problem
    - But the problem is not so bad
    - Docker image is more important
    - if your base image is Ubuntu/Debian
    - also there is UPX
    So what I’m trying to say
    29
    

    View Slide

30. So what I’m trying to say
    30
    - Looks like there is a problem
    - But the problem is not so bad
    - Docker image is more important
    - if your base image is Ubuntu/Debian
    - also there is UPX
    - But what is inside?
    

    View Slide

31. So what I’m trying to say
    31
    - Looks like there is a problem
    - But the problem is not so bad
    - Docker image is more important
    - if your base image is Ubuntu/Debian
    - also there is UPX
    - But what is inside?
    - Let’s talk about that
    

    View Slide

32. Simplest program
    32
    

    View Slide

33. ❯ cat main.go
    package main
    func main() {
    // ʕ◔ϖ◔ʔ
    }
    Simplest program
    33
    

    View Slide

34. ❯ cat main.go
    package main
    func main() {
    // ʕ◔ϖ◔ʔ
    }
    ❯ go build -o main.exe main.go
    Simplest program
    34
    

    View Slide

35. ❯ cat main.go
    package main
    func main() {
    // ʕ◔ϖ◔ʔ
    }
    ❯ go build -o main.exe main.go
    ❯ du -sh main.exe
    1.1M main.exe
    Simplest program
    35
    

    View Slide

36. ❯ cat main.go
    package main
    func main() {
    // ʕ◔ϖ◔ʔ
    }
    ❯ go build -o main.exe main.go
    ❯ du -sh main.exe
    1.1M main.exe
    ❯ 👀
    Simplest program
    36
    

    View Slide

37. ❯ go build -o main.exe main.go
    ❯ cat main.exe
    What is inside *.exe? /1
    37
    

    View Slide

38. cat
    38
    

    View Slide

39. ❯ go build -o main.exe main.go
    ❯ go list -json . > main.json
    What is inside *.exe? /2
    39
    

    View Slide

40. go list -json
    40
    {
    "Dir": "/Users/olegkovalov/go/src/github.com/cristaloleg/funf",
    "ImportPath": "github.com/cristaloleg/funf",
    "Name": "main",
    "Module": {...},
    "GoFiles": [
    "main.go"
    ],
    "Deps": [
    "internal/bytealg",
    "internal/cpu",
    "runtime",
    "runtime/internal/atomic",
    "runtime/internal/math",
    "runtime/internal/sys",
    "unsafe"
    ]
    }
    

    View Slide

41. ❯ go build -o main.exe main.go
    ❯ go tool objdump -S main.exe > main.dump
    What is inside *.exe? /3
    41
    

    View Slide

42. - Disassembles executable files
    go tool objdump
    42
    

    View Slide

43. - Disassembles executable files
    var printCode = flag.Bool("S", false, "print Go code alongside assembly")
    go tool objdump
    43
    

    View Slide

44. - Disassembles executable files
    var printCode = flag.Bool("S", false, "print Go code alongside assembly")
    - Also symregexp flag
    flag.String("s", "", "only dump symbols matching this regexp")
    ❯ go tool objdump -S -s runtime.makech main.exe
    go tool objdump
    44
    

    View Slide

45. Example from main.dump
    45
    

    View Slide

46. ❯ go build -o main.exe main.go
    ❯ go tool objdump -S main.exe > main.dump
    ❯ du -sh main.dump
    4.5M main.dump
    So objdump
    46
    

    View Slide

47. ❯ go build -o main.exe main.go
    ❯ go tool objdump -S main.exe > main.dump
    ❯ du -sh main.dump
    4.5M main.dump
    ❯ rg "TEXT runtime." -c main.dump
    945
    So objdump
    47
    

    View Slide

48. ❯ go build -o main.exe main.go
    ❯ go tool objdump -S main.exe > main.dump
    ❯ du -sh main.dump
    4.5M main.dump
    ❯ rg "TEXT runtime." -c main.dump
    945
    ❯ 🔥🔥🔥
    So objdump
    48
    

    View Slide

49. Close look at main.dump
    49
    

    View Slide

50. - Just a text file
    Close look at main.dump
    50
    

    View Slide

51. - Just a text file
    - With all possible definitions
    Close look at main.dump
    51
    

    View Slide

52. Close look at main.dump
    52
    - Just a text file
    - With all possible definitions
    - Useful if you are doing low-level optimisations
    

    View Slide

53. Close look at main.dump
    53
    - Just a text file
    - With all possible definitions
    - Useful if you are doing low-level optimisations
    - or a conference talk :)
    

    View Slide

54. To sum up /0
    54
    - ...
    - main()
    

    View Slide

55. ...124k lines later
    0x1054bde cc INT $0x3
    0x1054bdf cc INT $0x3
    TEXT main.main(SB) /Users/olegkovalov/code/funf/main.go
    func main() {}
    0x1054be0 c3 RET
    0x1054be1 cc INT $0x3
    0x1054be2 cc INT $0x3
    0x1054be3 cc INT $0x3
    Lovely main()
    55
    

    View Slide

56. ...124k lines later
    0x1054bde cc INT $0x3
    0x1054bdf cc INT $0x3
    TEXT main.main(SB) /Users/olegkovalov/code/funf/main.go
    func main() {}
    0x1054be0 c3 RET
    0x1054be1 cc INT $0x3
    0x1054be2 cc INT $0x3
    0x1054be3 cc INT $0x3
    Hm...main.main ?
    56
    

    View Slide

57. Real main
    57
    ...60k lines
    0x102d7cd e8eebeffff CALL runtime.deferreturn(SB)
    0x102d7d2 488b6c2450 MOVQ 0x50(SP), BP
    0x102d7d7 4883c458 ADDQ $0x58, SP
    0x102d7db c3 RET
    func main() {
    0x102d7dc 0f1f4000 NOPL 0(AX)
    0x102d7e0 e8bb440200 CALL runtime.morestack_noctxt.abi0(SB)
    0x102d7e5 e9b6fcffff JMP runtime.main(SB)
    0x102d7ea cc INT $0x3
    ...60k lines
    

    View Slide

58. Real main...is closer
    ...60k lines
    0x102d7cd e8eebeffff CALL runtime.deferreturn(SB)
    0x102d7d2 488b6c2450 MOVQ 0x50(SP), BP
    0x102d7d7 4883c458 ADDQ $0x58, SP
    0x102d7db c3 RET
    func main() {
    0x102d7dc 0f1f4000 NOPL 0(AX)
    0x102d7e0 e8bb440200 CALL runtime.morestack_noctxt.abi0(SB)
    0x102d7e5 e9b6fcffff JMP runtime.main(SB)
    0x102d7ea cc INT $0x3
    ...60k lines
    58
    

    View Slide

59. runtime.main
    TEXT runtime.main(SB) /usr/local/Cellar/go/1.17/libexec/src/runtime/proc.go
    func main() {
    0x102d4a0 493b6610 CMPQ 0x10(R14), SP
    0x102d4a4 0f8632030000 JBE 0x102d7dc
    0x102d4aa 4883ec58 SUBQ $0x58, SP
    0x102d4ae 48896c2450 MOVQ BP, 0x50(SP)
    0x102d4b3 488d6c2450 LEAQ 0x50(SP), BP
    0x102d4b8 49c7c500000000 MOVQ $0x0, R13
    0x102d4bf 4c896c2448 MOVQ R13, 0x48(SP)
    0x102d4c4 c644242700 MOVB $0x0, 0x27(SP)
    g := getg()
    0x102d4c9 4c89742430 MOVQ R14, 0x30(SP)
    g.m.g0.racectx = 0
    0x102d4ce 498b4630 MOVQ 0x30(R14), AX
    0x102d4d2 488b00 MOVQ 0(AX), AX
    0x102d4d5 48c7804001000000000000 MOVQ $0x0, 0x140(AX)
    maxstacksize = 1000000000
    59
    

    View Slide

60. Inside runtime/proc.go
    60
    //go:linkname main_main main.main
    func main_main()
    // ...
    // The main goroutine.
    func main() {
    g := getg()
    // ...
    fn := main_main // make an indirect call,
    // as the linker doesn't know the address
    // of the main package when laying down the runtime
    fn()
    // ...
    

    View Slide

61. To sum up /1
    61
    - ...
    - runtime.main()
    - main_main ~ main.main // linker’s magic
    - main.main // that’s us!
    

    View Slide

62. But that's not all
    62
    

    View Slide

63. - There are strange runtime/asm_.s files
    But that's not all
    63
    

    View Slide

64. - There are strange runtime/asm_.s files
    - arch is some CPU architecture (amd64, arm64, mips, …)
    - *.s - Assembler
    But that's not all
    64
    

    View Slide

65. - There are strange runtime/asm_.s files
    - arch is some CPU architecture (amd64, arm64, mips, …)
    - *.s - Assembler
    - 👀
    But that's not all
    65
    

    View Slide

66. - There are strange runtime/asm_.s files
    - arch is some CPU architecture (amd64, arm64, mips, …)
    - *.s - Assembler
    - 👀
    - Let’s check asm_amd64.s
    - just 2k lines
    But that's not all
    66
    

    View Slide

67. - There are strange runtime/asm_.s files
    - arch is some CPU architecture (amd64, arm64, mips, …)
    - *.s - Assembler
    - 👀
    - Let’s check asm_amd64.s
    - just 2k lines
    - others are + - 1k
    - arm64.s is also heavy
    But that's not all
    67
    

    View Slide

68. - FP: Frame pointer: arguments and locals.
    - PC: Program counter: jumps and branches.
    - SB: Static base pointer: global symbols.
    - SP: Stack pointer: top of stack.
    Go assembler cheat sheet
    68
    

    View Slide

69. - FP: Frame pointer: arguments and locals.
    - PC: Program counter: jumps and branches.
    - SB: Static base pointer: global symbols.
    - SP: Stack pointer: top of stack.
    - JMP: jump
    - LEA: load effective address
    - MOV dst, src: move src to dst
    - CALL: invoke function
    - DATA(+GLOBL): data symbols (global)
    Go assembler cheat sheet
    69
    

    View Slide

70. // _rt0_amd64 is common startup code for most amd64 systems when using
    // internal linking. This is the entry point for the program from the
    // kernel for an ordinary -buildmode=exe program. The stack holds the
    // number of arguments and the C-style argv.
    TEXT _rt0_amd64(SB),NOSPLIT,$-8
    MOVQ 0(SP), DI // argc
    LEAQ 8(SP), SI // argv
    JMP runtime·rt0_go(SB)
    Ah, that the trick
    70
    

    View Slide

71. // Defined as ABIInternal since it does not use the stack-based Go ABI (and
    // in addition there are no calls to this entry point from Go code).
    TEXT runtime·rt0_go(SB),NOSPLIT|TOPFRAME,$0
    // copy arguments forward on an even stack
    MOVQ DI, AX // argc
    MOVQ SI, BX // argv
    // ...
    // create istack out of the given (operating system) stack.
    MOVQ $runtime·g0(SB), DI
    // ...
    // find out information about the processor we're on
    MOVL $0, AX
    CPUID
    // ...
    Scroll further
    71
    

    View Slide

72. Scroll further
    72
    TEXT runtime·rt0_go(SB),NOSPLIT|TOPFRAME,$0
    // ...
    ok:
    // set the per-goroutine and per-mach "registers"
    get_tls(BX)
    LEAQ runtime·g0(SB), CX
    MOVQ CX, g(BX)
    LEAQ runtime·m0(SB), AX
    // save m->g0 = g0
    MOVQ CX, m_g0(AX)
    // save m0 to g0->m
    MOVQ AX, g_m(CX)
    

    View Slide

73. Scroll further
    73
    TEXT runtime·rt0_go(SB),NOSPLIT|TOPFRAME,$0
    // ...
    CALL runtime·check(SB)
    MOVL 16(SP), AX // copy argc
    MOVL AX, 0(SP)
    MOVQ 24(SP), AX // copy argv
    MOVQ AX, 8(SP)
    CALL runtime·args(SB)
    CALL runtime·osinit(SB)
    CALL runtime·schedinit(SB)
    

    View Slide

74. - Platform dependent things
    runtime/os_.go
    74
    

    View Slide

75. - Platform dependent things
    - Signal handlers
    - Timers
    runtime/os_.go
    75
    

    View Slide

76. - Platform dependent things
    - Signal handlers
    - Timers
    - Random
    - Thread sizes and other consts
    - Envs and cmd params
    - I mean environment variables и argc & argv
    runtime/os_.go
    76
    

    View Slide

77. - Platform dependent things
    - Signal handlers
    - Timers
    - Random
    - Thread sizes and other consts
    - Envs and cmd params
    - I mean environment variables и argc & argv
    - Machine sanity checks
    - Oh, interesting!
    runtime/os_.go
    77
    

    View Slide

78. - In runtime1.go there is an interesting function
    - check()
    Machine sanity checks
    78
    

    View Slide

79. - In runtime1.go there is an interesting function
    - check()
    - checks primitive type sizes
    - d uint16
    - if unsafe.Sizeof(d) != 2 { throw("bad d") }
    Machine sanity checks
    79
    

    View Slide

80. - In runtime1.go there is an interesting function
    - check()
    - checks primitive type sizes
    - d uint16
    - if unsafe.Sizeof(d) != 2 { throw("bad d") }
    - check() calls testAtomic64()
    Machine sanity checks
    80
    

    View Slide

81. - In runtime1.go there is an interesting function
    - check()
    - checks primitive type sizes
    - d uint16
    - if unsafe.Sizeof(d) != 2 { throw("bad d") }
    - check() calls testAtomic64()
    - As you can guess testAtomic64 is about atomics
    - test_z64 = 42, test_x64 = 0
    - if atomic.Cas64(&test_z64, test_x64, 1) { throw("cas64 failed") }
    Machine sanity checks
    81
    

    View Slide

82. Scroll further
    82
    TEXT runtime·rt0_go(SB),NOSPLIT|TOPFRAME,$0
    // ...
    CALL runtime·check(SB)
    MOVL 16(SP), AX // copy argc
    MOVL AX, 0(SP)
    MOVQ 24(SP), AX // copy argv
    MOVQ AX, 8(SP)
    CALL runtime·args(SB)
    CALL runtime·osinit(SB)
    CALL runtime·schedinit(SB)
    // with some magic and linker this will be os.Args
    

    View Slide

83. /Users/olegkovalov/go/src/golang/go/src/runtime/runtime1.go
    func args(c int32, v **byte) {
    argc = c
    argv = v
    sysargs(c, v)
    }
    Scroll further
    83
    

    View Slide

84. Scroll further
    84
    /Users/olegkovalov/go/src/golang/go/src/runtime/runtime1.go
    func args(c int32, v **byte) {
    argc = c
    argv = v
    sysargs(c, v)
    }
    // os_darwin.go
    func sysargs(argc int32, argv **byte) {
    // skip over argv, envv and the first string will be the path
    n := argc + 1
    for argv_index(argv, n) != nil {
    n++
    }
    executablePath = gostringnocopy(argv_index(argv, n+1))
    // ...
    }
    

    View Slide

85. Scroll further
    85
    TEXT runtime·rt0_go(SB),NOSPLIT,$0
    // ...
    CALL runtime·check(SB)
    MOVL 16(SP), AX // copy argc
    MOVL AX, 0(SP)
    MOVQ 24(SP), AX // copy argv
    MOVQ AX, 8(SP)
    CALL runtime·args(SB)
    CALL runtime·osinit(SB)
    CALL runtime·schedinit(SB)
    

    View Slide

86. Scroll further
    86
    // os_darwin.go
    // BSD interface for threading.
    func osinit() {
    // pthread_create delayed until end of goenvs so that we
    // can look at the environment first.
    ncpu = getncpu()
    physPageSize = getPageSize()
    }
    

    View Slide

87. Scroll further
    87
    TEXT runtime·rt0_go(SB),NOSPLIT,$0
    // ...
    CALL runtime·check(SB)
    MOVL 16(SP), AX // copy argc
    MOVL AX, 0(SP)
    MOVQ 24(SP), AX // copy argv
    MOVQ AX, 8(SP)
    CALL runtime·args(SB)
    CALL runtime·osinit(SB)
    CALL runtime·schedinit(SB)
    

    View Slide

88. To sum up /2
    88
    - runtime·_rt0_amd64 // or which architecture you have
    - runtime·rt0_go
    -
    - check
    - args
    - osinit
    - schedinit
    - runtime.main()
    - main_main ~ main.main // linker’s magic
    - main.main // that’s us!
    

    View Slide

89. Scroll further
    89
    // The bootstrap sequence is:
    //
    // call osinit
    // call schedinit
    // make & queue new G
    // call runtime·mstart
    //
    // The new G calls runtime·main.
    func schedinit() {
    // ...
    

    View Slide

90. - G - goroutine
    - M - worker thread, or machine.
    - P - processor, a resource that is required to execute Go code.
    - getg() - returns the pointer to the current g
    - _g_ := getg()
    - p := getg().m.p.ptr()
    - tls - thread-local storage
    Everything else is if’s, atomics & unclear algorithms. Ez.
    Go runtime cheat sheet
    90
    

    View Slide

91. func schedinit() {
    // ...
    _g_ := getg()
    sched.maxmcount = 10000 // maximum number of m's allowed (or die)
    // The world starts stopped.
    worldStopped()
    moduledataverify() // pclntab is correct
    stackinit() //
    mallocinit()
    fastrandinit() // must run before mcommoninit
    mcommoninit(_g_.m, -1)
    schedinit /1
    91
    

    View Slide

92. func schedinit() {
    // ...
    _g_ := getg()
    sched.maxmcount = 10000 // maximum number of m's allowed (or die)
    // The world starts stopped.
    worldStopped()
    moduledataverify() // pclntab is correct
    stackinit() //
    mallocinit()
    fastrandinit() // must run before mcommoninit
    mcommoninit(_g_.m, -1)
    cpuinit() // must run before alginit
    alginit() // maps must not be used before this call
    modulesinit() // provides activeModules
    typelinksinit() // uses maps, activeModules
    itabsinit() // uses activeModules
    schedinit /1
    92
    

    View Slide

93. func schedinit() {
    // ...
    goargs()
    goenvs()
    parsedebugvars()
    gcinit()
    lock(&sched.lock)
    // ...
    if procresize(procs) != nil {
    throw("unknown runnable goroutine during bootstrap")
    }
    unlock(&sched.lock)
    // World is effectively started now, as P's can run.
    worldStarted()
    // ...
    }
    schedinit /2
    93
    

    View Slide

94. TEXT runtime·rt0_go(SB),NOSPLIT|TOPFRAME,$0
    // ...
    // create a new goroutine to start program
    MOVQ $runtime·mainPC(SB), AX // entry
    PUSHQ AX
    PUSHQ $0 // arg size
    CALL runtime·newproc(SB)
    POPQ AX
    POPQ AX
    ...
    // mainPC is a function value for runtime.main, to be passed to newproc.
    // The reference to runtime.main is made via ABIInternal, since the
    // actual function (not the ABI0 wrapper) is needed by newproc.
    DATA runtime·mainPC+0(SB)/8,$runtime·main(SB)
    GLOBL runtime·mainPC(SB),RODATA,$8
    A bit of assembler
    94
    

    View Slide

95. One more bit of assembler
    95
    TEXT runtime·rt0_go(SB),NOSPLIT|TOPFRAME,$0
    // ...
    // create a new goroutine to start program
    MOVQ $runtime·mainPC(SB), AX // entry
    PUSHQ AX
    PUSHQ $0 // arg size
    CALL runtime·newproc(SB)
    POPQ AX
    POPQ AX
    // start this M
    CALL runtime·mstart(SB)
    CALL runtime·abort(SB)// mstart should never return
    RET
    TEXT runtime·mstart(SB),NOSPLIT|TOPFRAME,$0
    CALL runtime·mstart0(SB)
    RET // not reached
    

    View Slide

96. newproc
    // Create a new g running fn.
    // Put it on the queue of g's waiting to run.
    // The compiler turns a go statement into a call to this.
    func newproc(fn *funcval) {
    gp := getg()
    pc := getcallerpc()
    systemstack(func() {
    newg := newproc1(fn, gp, pc)
    _p_ := getg().m.p.ptr()
    runqput(_p_, newg, true)
    if mainStarted {
    wakep()
    }
    })
    }
    96
    

    View Slide

97. // Create a new g running fn.
    // Put it on the queue of g's waiting to run.
    // The compiler turns a go statement into a call to this.
    func newproc(fn *funcval) {
    gp := getg()
    pc := getcallerpc()
    systemstack(func() {
    newg := newproc1(fn, gp, pc)
    _p_ := getg().m.p.ptr()
    runqput(_p_, newg, true)
    if mainStarted {
    wakep()
    }
    })
    }
    newproc
    97
    

    View Slide

98. newproc
    98
    // Create a new g running fn.
    // Put it on the queue of g's waiting to run.
    // The compiler turns a go statement into a call to this.
    func newproc(fn *funcval) {
    gp := getg()
    pc := getcallerpc()
    systemstack(func() {
    newg := newproc1(fn, gp, pc)
    _p_ := getg().m.p.ptr()
    runqput(_p_, newg, true)
    if mainStarted {
    wakep()
    }
    })
    }
    

    View Slide

99. systemstack
    99
    // systemstack runs fn on a system stack.
    // If systemstack is called from the per-OS-thread (g0) stack, or
    // if systemstack is called from the signal handling (gsignal) stack,
    // systemstack calls fn directly and returns.
    //
    // Otherwise, systemstack is being called from the limited stack
    // of an ordinary goroutine. In this case, systemstack switches
    // to the per-OS-thread stack, calls fn, and switches back.
    //
    //go:noescape
    func systemstack(fn func())
    

    View Slide

100. Finally scheduler
     // mstart is the entry-point for new Ms.
     func mstart()
     // mstart0 is the Go entry-point for new Ms.
     func mstart0() {
     _g_ := getg()
     // ...
     mstart1()
     }
     func mstart1() {
     _g_ := getg()
     // ...
     schedule()
     }
     100
     

     View Slide

101. schedule
     101
     // One round of scheduler: find a runnable goroutine and execute it.
     // Never returns.
     func schedule() {
     _g_ := getg()
     top:
     pp := _g_.m.p.ptr()
     pp.preempt = false
     var gp *g
     // get some gp
     execute(gp, inheritTime)
     }
     

     View Slide

102. To sum up /3
     102
     - runtime·_rt0_amd64 // or which architecture you have
     - runtime·rt0_go
     - check
     - args
     - osinit
     - schedinit
     - ...
     - newproc // will create goroutine for runtime.main
     - mstart // to start goroutine above
     - mstart0 -> mstart1
     - schedule
     - runtime.main()
     - main_main ~ main.main // linker’s magic
     - main.main // that’s us!
     

     View Slide

103. To sum up /3.1 (WebAssembly)
     103
     

     View Slide

104. - Everything discussed is ~250 lines
     Ok, but why 2k lines?
     104
     

     View Slide

105. - Everything discussed is ~250 lines
     - Everything else is runtime things (obvious)
     Ok, but why 2k lines?
     105
     

     View Slide

106. - Everything discussed is ~250 lines
     - Everything else is runtime things (obvious)
     - stack handling
     - defer’s
     - different panics
     Ok, but why 2k lines?
     106
     

     View Slide

107. - Everything discussed is ~250 lines
     - Everything else is runtime things (obvious)
     - stack handling
     - defer’s
     - different panics
     - debug helpers
     - lovely cgo
     - gc barriers
     Ok, but why 2k lines?
     107
     

     View Slide

108. - Everything discussed is ~250 lines
     - Everything else is runtime things (obvious)
     - stack handling
     - defer’s
     - different panics
     - debug helpers
     - lovely cgo
     - gc barriers
     - timers
     - signals
     - hashes
     Ok, but why 2k lines?
     108
     

     View Slide

109. Popular pattern
     // func memhash(p unsafe.Pointer, h, s uintptr) uintptr
     // hash function using AES hardware instructions
     TEXT runtime·memhash(SB),NOSPLIT,$0-32
     // AX = ptr to data
     // BX = seed
     // CX = size
     CMPB runtime·useAeshash(SB), $0
     JEQ noaes
     JMP aeshashbody<>(SB)
     noaes:
     JMP runtime·memhashFallback(SB)
     /Users/olegkovalov/go/src/golang/go/src/runtime/hash64.go
     func memhashFallback(p unsafe.Pointer, seed, s uintptr) uintptr {
     109
     

     View Slide

110. - add GOAMD64 architecture variants #25489
     - https://github.com/golang/go/issues/25489
     - Now you can easily find such lines
     - x86HasSSE41 = cpu.X86.HasSSE41
     - to be honest a lot of them
     - Wasting our CPU on nothing
     - every nanosecond counts :(
     GOAMD64 proposal
     110
     

     View Slide

111. What about init()’s ??
     111
     

     View Slide

112. - Runtime packages has a lot inits
     - Most of them are OS/machine checks
     - type size
     - available CPU flags
     - But there is one interesting
     (then we will scroll through other stdlib packages)
     What about init()’s ??
     112
     

     View Slide

113. /Users/olegkovalov/go/src/golang/go/src/runtime/proc.go
     // start forcegc helper goroutine
     func init() {
     go forcegchelper()
     }
     Popular claim regarding Go GC
     113
     

     View Slide

114. /Users/olegkovalov/go/src/golang/go/src/runtime/proc.go
     // start forcegc helper goroutine
     func init() {
     go forcegchelper()
     }
     // 5k lines later...
     Popular claim regarding Go GC
     114
     

     View Slide

115. Popular claim regarding Go GC
     115
     /Users/olegkovalov/go/src/golang/go/src/runtime/proc.go
     // start forcegc helper goroutine
     func init() {
     go forcegchelper()
     }
     // 5k lines later...
     // forcegcperiod is the maximum time in nanoseconds between garbage
     // collections. If we go this long without a garbage collection, one
     // is forced to run.
     //
     // This is a variable for testing purposes. It normally doesn't change.
     var forcegcperiod int64 = 2 * 60 * 1e9
     

     View Slide

116. func init() {
     http.HandleFunc("/debug/vars", expvarHandler)
     Publish("cmdline", Func(cmdline))
     Publish("memstats", Func(memstats))
     }
     Inside expvar
     116
     

     View Slide

117. func init() {
     crypto.RegisterHash(crypto.MD5, New)
     }
     Inside crypto/*
     117
     

     View Slide

118. func init() {
     crypto.RegisterHash(crypto.MD5, New)
     }
     // do THIS with crypto packages:
     import (
     _ "crypto/sha256" // to register a sha256
     _ "crypto/sha512" // to register a sha384/512
     )
     Inside crypto/*
     118
     

     View Slide

119. func init() {
     crypto.RegisterHash(crypto.MD5, New)
     }
     // do THIS with crypto packages:
     import (
     _ "crypto/sha256" // to register a sha256
     _ "crypto/sha512" // to register a sha384/512
     )
     // and NOT this:
     import (
     _ "crypto/sha256" //nolint
     "crypto/sha512"
     )
     var _ = sha512.
     Inside crypto/*
     119
     

     View Slide

120. func init() {
     // ...
     registerBasics()
     // ...
     }
     Inside encoding/gob
     120
     

     View Slide

121. func init() {
     // ...
     registerBasics()
     // ...
     }
     func registerBasics() {
     Register(int(0))
     Register(int8(0))
     Register(int16(0))
     Register(int32(0))
     Register(int64(0))
     // ...
     Inside encoding/gob
     121
     

     View Slide

122. Inside net
     122
     // net/conf_netcgo.go
     //go:build netcgo
     // +build netcgo
     package net
     // The build tag "netcgo" forces use of the cgo DNS resolver.
     // It is the opposite of "netgo".
     func init() { netCgo = true }
     // net/conf.go
     netGo bool // go DNS resolution forced
     netCgo bool // cgo DNS resolution forced
     

     View Slide

123. Inside net
     123
     // net/conf_netcgo.go
     //go:build netcgo
     // +build netcgo
     package net
     // The build tag "netcgo" forces use of the cgo DNS resolver.
     // It is the opposite of "netgo".
     func init() { netCgo = true }
     // net/conf.go
     netGo bool // go DNS resolution forced
     netCgo bool // cgo DNS resolution forced
     

     View Slide

124. // When debugging a particular http server-based test,
     // this flag lets you run
     // go test -run=BrokenTest -httptest.serve=127.0.0.1:8000
     // to start the broken server so you can interact with it manually.
     // ...
     // isn't really part of our API. Don't depend on this.
     var serveFlag string
     func init() {
     if has(os.Args, "-httptest.serve=") {
     flag.StringVar(&serveFlag, "httptest.serve", "", "...")
     }
     }
     Inside net/http/httptest
     124
     

     View Slide

125. func init() {
     http.HandleFunc("/debug/pprof/", Index)
     http.HandleFunc("/debug/pprof/cmdline", Cmdline)
     http.HandleFunc("/debug/pprof/profile", Profile)
     http.HandleFunc("/debug/pprof/symbol", Symbol)
     http.HandleFunc("/debug/pprof/trace", Trace)
     }
     Inside net/http/pprof
     125
     

     View Slide

126. // mime/type_freebsd.go
     func init() {
     typeFiles = append(typeFiles, "/usr/local/etc/mime.types")
     }
     // mime/type_openbsd.go
     func init() {
     typeFiles = append(typeFiles, "/usr/share/misc/mime.types")
     }
     ...
     Inside mime
     126
     MIME == Multipurpose Internet Mail Extensions
     

     View Slide

127. func init() {
     runtime_registerPoolCleanup(poolCleanup)
     }
     func poolCleanup() {
     for _, p := range oldPools {
     // ...
     }
     for _, p := range allPools {
     // ...
     }
     // The pools with non-empty primary caches now have non-empty
     // victim caches and no pools have primary caches.
     oldPools, allPools = allPools, nil
     }
     Inside sync
     127
     

     View Slide

128. Inside sync /2
     // sync/runtime.go
     // Ensure that sync and runtime agree on size of notifyList.
     func runtime_notifyListCheck(size uintptr)
     func init() {
     var n notifyList
     runtime_notifyListCheck(unsafe.Sizeof(n))
     }
     // runtime/sema.go
     //go:linkname notifyListCheck sync.runtime_notifyListCheck
     func notifyListCheck(sz uintptr) {
     if sz != unsafe.Sizeof(notifyList{}) {
     print("runtime: bad notifyList")
     throw("bad notifyList size")
     }
     } 128
     

     View Slide

129. Inside time/tzdata
     129
     

     View Slide

130. // Package tzdata provides an embedded copy of the timezone database.
     // If this package is imported anywhere in the program, then if
     // the time package cannot find tzdata files on the system,
     // it will use this embedded information.
     //
     // Importing this package will increase the size of a program by about
     // 450 KB.
     //
     Inside time/tzdata
     130
     

     View Slide

131. // Package tzdata provides an embedded copy of the timezone database.
     // If this package is imported anywhere in the program, then if
     // the time package cannot find tzdata files on the system,
     // it will use this embedded information.
     //
     // Importing this package will increase the size of a program by about
     // 450 KB.
     //
     Inside time/tzdata
     131
     

     View Slide

132. Inside time/tzdata
     132
     // Package tzdata provides an embedded copy of the timezone database.
     // If this package is imported anywhere in the program, then if
     // the time package cannot find tzdata files on the system,
     // it will use this embedded information.
     //
     // Importing this package will increase the size of a program by about
     // 450 KB.
     //
     // This package should normally be imported by a program's main package,
     // not by a library. Libraries normally shouldn't decide whether to
     // include the timezone database in a program.
     //
     // This package will be automatically imported if you build with
     // -tags timetzdata.
     package tzdata
     

     View Slide

133. Inside time/tzdata
     133
     // Package tzdata provides an embedded copy of the timezone database.
     // If this package is imported anywhere in the program, then if
     // the time package cannot find tzdata files on the system,
     // it will use this embedded information.
     //
     // Importing this package will increase the size of a program by about
     // 450 KB.
     //
     // This package should normally be imported by a program's main package,
     // not by a library. Libraries normally shouldn't decide whether to
     // include the timezone database in a program.
     //
     // This package will be automatically imported if you build with
     // -tags timetzdata.
     package tzdata
     

     View Slide

134. 1 slide notes
     134
     

     View Slide

135. - Function definition before usage
     - do you remember that first versions of Go compiler was in C ? :)
     1 slide notes
     135
     

     View Slide

136. - Function definition before usage
     - do you remember that first versions of Go compiler was in C ? :)
     - Many underscores in var and func names
     - again the C legacy
     1 slide notes
     136
     

     View Slide

137. - Function definition before usage
     - do you remember that first versions of Go compiler was in C ? :)
     - Many underscores in var and func names
     - again the C legacy
     - Windows, NetBSD, Wasm, JS & Plan9 are special
     - often requires special treatment
     1 slide notes
     137
     

     View Slide

138. - Function definition before usage
     - do you remember that first versions of Go compiler was in C ? :)
     - Many underscores in var and func names
     - again the C legacy
     - Windows, NetBSD, Wasm, JS & Plan9 are special
     - often requires special treatment
     - Cgo is not Go, as Rob Pike said
     - basically explains everything
     1 slide notes
     138
     

     View Slide

139. - Function definition before usage
     - do you remember that first versions of Go compiler was in C ? :)
     - Many underscores in var and func names
     - again the C legacy
     - Windows, NetBSD, Wasm, JS & Plan9 are special
     - often requires special treatment
     - Cgo is not Go, as Rob Pike said
     - basically explains everything
     - Runtime is all about corner cases
     - that’s my IMHO
     1 slide notes
     139
     

     View Slide

140. Conclusions
     140
     

     View Slide

141. - Meeting a machine
     Conclusions
     141
     

     View Slide

142. Conclusions
     142
     - Meeting a machine
     - Creating goroutine
     

     View Slide

143. Conclusions
     143
     - Meeting a machine
     - Creating goroutine
     - Creating thread to run it
     

     View Slide

144. Conclusions
     144
     - Meeting a machine
     - Creating goroutine
     - Creating thread to run it
     - Running and looking for another to run
     

     View Slide

145. Conclusions
     145
     - Meeting a machine
     - Creating goroutine
     - Creating thread to run it
     - Running and looking for another to run
     - That’s all
     

     View Slide

146. - A Quick Guide to Go's Assembler
     - https://golang.org/doc/asm
     - The Go Memory Model
     - https://golang.org/ref/mem
     - Go 1.2 Runtime Symbol Information (2013)
     - https://docs.google.com/document/d/1lyPIbmsYbXnpNj57a261hgOYVpNRcgydurVQIyZO
     z_o/pub
     - runtime/HACKING.md
     - https://github.com/golang/go/blob/master/src/runtime/HACKING.md
     References
     146
     

     View Slide

147. - Anton Repushko
     - Bogdan Storozhuk
     - Iskander Sharipov
     Thanks
     147
     

     View Slide

148. - Anton Repushko
     - Bogdan Storozhuk
     - Iskander Sharipov
     - and you
     Thanks
     148
     

     View Slide

149. - Anton Repushko
     - Bogdan Storozhuk
     - Iskander Sharipov
     - and you
     GopherCon Poland team 👌
     Thanks
     149
     

     View Slide

150. Thank you
     Questions?
     Telegram: @olegkovalov
     Twitter: @oleg_kovalov
     That’s all folks
     

     View Slide