Hacking Go Compiler Internals - 2nd season

Hacking Go Compiler Internals - 2nd season

Since the previous talk at Go Con 2014 Autumn, lots of things in the internals have changed. In this talk, I will try to give an overview of Go compiler internals and update the information as much as possible, along with my new hacks.

0bafbcfcfb548a4ac20406692858f68b?s=128

Moriyoshi Koizumi

May 18, 2019
Tweet

Transcript

  1. Hacking Go Compiler Internals Hacking Go Compiler Internals 2nd season

    2nd season May 18, 2019 May 18, 2019 Moriyoshi Koizumi Moriyoshi Koizumi Open Collector, Inc. Open Collector, Inc.
  2. Agenda Agenda Compiler basics recap Compiler basics recap A brief

    history of Go language infrastructure A brief history of Go language infrastructure Hacking the internals Hacking the internals Who Who am am I I: : @moriyoshi at github.com / @moriyoshit at twitter.com. @moriyoshi at github.com / @moriyoshit at twitter.com. An early Go contributor. An early Go contributor. Reviewed the Japanese translation of "Concurrency in Go". Reviewed the Japanese translation of "Concurrency in Go". commit a8fbf5dc2cd5b58167402df47bb06217c5e8fd22 commit a8fbf5dc2cd5b58167402df47bb06217c5e8fd22 Author: Moriyoshi Koizumi <mozo@mozo.jp> Author: Moriyoshi Koizumi <mozo@mozo.jp> Date: Tue Dec 15 21:24:17 2009 -0800 Date: Tue Dec 15 21:24:17 2009 -0800 This patch enables cgo utility to correctly convert enums in the C source This patch enables cgo utility to correctly convert enums in the C source into consts in the resulting Go source. Previously known as issue 161047, into consts in the resulting Go source. Previously known as issue 161047, which I deleted accidentally. Fixes issue 207. which I deleted accidentally. Fixes issue 207. R=rsc R=rsc https://golang.org/cl/166059 https://golang.org/cl/166059
  3. Previous Talk on Internals Previous Talk on Internals speakerdeck.com/moriyoshi/hacking-go-compiler-internals speakerdeck.com/moriyoshi/hacking-go-compiler-internals

    (https://speakerdeck.com/moriyoshi/hacking-go-compiler-internals) (https://speakerdeck.com/moriyoshi/hacking-go-compiler-internals)
  4. Compiler Basics Compiler Basics

  5. What is compiler? What does "compiling source" code mean? What

    is compiler? What does "compiling source" code mean? Translating source code into a binary form which a CPU can comprehend: Translating source code into a binary form which a CPU can comprehend: func hello() { func hello() { print("hello\n") print("hello\n") } } Into Into 000000000044fa40 <main.hello>: 000000000044fa40 <main.hello>: 44fa40: 48 83 ec 18 sub $0x18,%rsp 44fa40: 48 83 ec 18 sub $0x18,%rsp 44fa44: 48 89 6c 24 10 mov %rbp,0x10(%rsp) 44fa44: 48 89 6c 24 10 mov %rbp,0x10(%rsp) 44fa49: 48 8d 6c 24 10 lea 0x10(%rsp),%rbp 44fa49: 48 8d 6c 24 10 lea 0x10(%rsp),%rbp 44fa4e: 48 8d 05 93 d1 02 00 lea 0x2d193(%rip),%rax # 47cbe8 <hello> 44fa4e: 48 8d 05 93 d1 02 00 lea 0x2d193(%rip),%rax # 47cbe8 <hello> 44fa55: 48 89 04 24 mov %rax,(%rsp) 44fa55: 48 89 04 24 mov %rax,(%rsp) 44fa59: 48 c7 44 24 08 06 00 movq $0x6,0x8(%rsp) 44fa59: 48 c7 44 24 08 06 00 movq $0x6,0x8(%rsp) 44fa60: 00 00 44fa60: 00 00 44fa62: e8 89 4d fd ff callq 4247f0 <runtime.printstring> 44fa62: e8 89 4d fd ff callq 4247f0 <runtime.printstring> 44fa67: 48 8b 6c 24 10 mov 0x10(%rsp),%rbp 44fa67: 48 8b 6c 24 10 mov 0x10(%rsp),%rbp 44fa6c: 48 83 c4 18 add $0x18,%rsp 44fa6c: 48 83 c4 18 add $0x18,%rsp 44fa70: c3 retq 44fa70: c3 retq
  6. Compiling Phases Compiling Phases 1. Lexer 1. Lexer 2. Parser

    2. Parser 3. Annotated AST construction 3. Annotated AST construction 4. Type checking 4. Type checking 5. Variable capturing 5. Variable capturing 6. Inlining 6. Inlining 7. Escape analysis 7. Escape analysis 8. Closure rewriting 8. Closure rewriting 9. Walk 9. Walk 10. SSA generation 10. SSA generation 11. Machine code generation 11. Machine code generation
  7. Lexer Lexer The lexer scans over the source code and

    cut it into a series of meaningful chunks. The lexer scans over the source code and cut it into a series of meaningful chunks. src/cmd/compile/internal/syntax/tokens.go src/cmd/compile/internal/syntax/tokens.go src/cmd/compile/internal/syntax/scanner.go src/cmd/compile/internal/syntax/scanner.go a := b + c(12) _Name _Define _IncOp _Name _Lparen _Rparen _Name _Literal
  8. Parser Parser The parser reads the generated tokens and build

    an AST (abstract syntax tree). The parser reads the generated tokens and build an AST (abstract syntax tree). src/cmd/compile/internal/syntax/nodes.go src/cmd/compile/internal/syntax/nodes.go src/cmd/compile/internal/syntax/parser.go src/cmd/compile/internal/syntax/parser.go CallExpr Operation Name _Name _Define _IncOp _Name _Lparen _Rparen _Name _Literal BasicLit Name AssignStmt Name Tokens AST + a b c 12 :=
  9. Parser (cont'd) Parser (cont'd) A Go source file corresponds to

    a *syntax.File node, which consist of declaration nodes. A Go source file corresponds to a *syntax.File node, which consist of declaration nodes. package main import "fmt" const world = "world" type myString string var hello myString = "hello, " + world func main() { fmt.Println(hello) } FuncDecl File ImportDecl ConstDecl TypeDecl VarDecl ImportDecl ImportDecl ConstDecl ConstDecl TypeDecl TypeDecl VarDecl VarDecl FuncDecl FuncDecl CallExpr SelectorExpr Name Name Name fmt Println hello
  10. Annotated AST construction Annotated AST construction "Noder" translates the AST

    to the annotated AST (node tree). "Noder" translates the AST to the annotated AST (node tree). src/cmd/compile/internal/gc/noder.go src/cmd/compile/internal/gc/noder.go Translation is done on a one-to-one basis. Translation is done on a one-to-one basis. CallExpr Operation Name BasicLit Name AssignStmt Name AST OCALL OADD ONAME OLITERAL ONAME OAS ONAME Node Tree + a b c 12 :=
  11. Typechecking Typechecking "Typecheck" walks through the node tree and tries

    to... "Typecheck" walks through the node tree and tries to... Determine the type of each node. Determine the type of each node. Annotate the nodes with the information used in later stages. Annotate the nodes with the information used in later stages. Replace the nodes for special function calls (len, cap, append and so on) to dedicated Replace the nodes for special function calls (len, cap, append and so on) to dedicated nodes. nodes. Translate the references to methods to closures. Translate the references to methods to closures. etc. etc. Typechecking happens occationally in the following stages to deal with the "synthesized" Typechecking happens occationally in the following stages to deal with the "synthesized" nodes. nodes. src/cmd/compile/internal/gc/typecheck.go src/cmd/compile/internal/gc/typecheck.go
  12. Typechecking (cont'd) Typechecking (cont'd) OCALL OADD ONAME OLITERAL ONAME OAS

    ONAME OCALLFUNC OADD ONAME OLITERAL ONAME OAS ONAME + a b c 12 := + a b c 12 := ONAME OCALL ONAME len d ONAME OLEN d type=string type=int type=int type=int type=int type=func() type=int type=int
  13. Variable Capturing Variable Capturing On each declared function, checks if

    its body contains any closure function and finds out On each declared function, checks if its body contains any closure function and finds out how the outer variables are referenced in the closure. how the outer variables are referenced in the closure. a := 1 b := 2 func () { fmt.Println(a, b) }() a = 1 OCALLFUNC OCLOSURE OLITERAL OAS ONAME type=int assigned type=int ODCLFUNC = a 1 ODCL ONAME a type=func() ONAME type=int assigned type=int b ODCL Chosen strategies: a → pass by reference b → pass by value
  14. Inlining Inlining Checks inlining feasibility for each declared function. Checks

    inlining feasibility for each declared function. Weaves synthesized nodes into the caller. Weaves synthesized nodes into the caller. src/cmd/compile/internal/gc/inl.go src/cmd/compile/internal/gc/inl.go func div(dividend, divisor int) int { if divisor != 0 { return dividend / divisor } else { panic("division by zero!") } } func fact(n int) int { if n == 0 { return 1 } else if n > 0 { return n * fact(n-1) } else { panic("domain error") } } func div: Budget: 80 Inlining cost: 1 (if) + 3 ("divisor != 0") + 4 (return ...) + 2 (panic call) + 1 (panic penalty) = 11 → Inlineable func fact: Budget: 80 Inlining cost: ∞ (recursive function) → Uninlineable
  15. Inlining (cont'd) Inlining (cont'd) func c(n int) int { return

    n + 1 } a := b + c(1) OINLCALL OADD ONAME ODCL OAS ONAME ODCL OAS OAS OINLMARK OAS OGOTO ONAME OADD ONAME ONAME OCALL OADD ONAME OLITERAL ONAME OAS ONAME A generated temporary variable { n := 1 ~r2 := n + 1 goto end end: a = b + ~r2 }
  16. Escape Analysis Escape Analysis Checks on every variable if its

    address leaks off the stack. Checks on every variable if its address leaks off the stack. src/cmd/compile/internal/gc/escape.go src/cmd/compile/internal/gc/escape.go var z *int func main() { ... a := 1 z = &a ... } OLITERAL OAS ONAME type=int OADDR OAS ONAME type=*int type=*int ONAME type=int addrtaken ODCLFUNC = a 1 := z & a ODCL ONAME a type=int addrtaken type=int addrtaken typecheck escAnalyze OADDR ONAME type=*int type=*int ONAME type=int addrtaken class(PAUTOHEAP) z & a Located off-stack →mark the right operand as PAUTOHEAP
  17. Closure Rewriting Closure Rewriting Transform the immediate call to the

    closure to a simpler form. Transform the immediate call to the closure to a simpler form. src/cmd/compile/internal/gc/closure.go src/cmd/compile/internal/gc/closure.go func do() { a := 1 func() { fmt.Println(a) a = 2 }() } func func1(a *int) { fmt.Println(*a) *a = 2 } func do() { a := 1 func1(&a) } This can be done after escape analysis because there is no chance the outer variables will This can be done after escape analysis because there is no chance the outer variables will leak. leak.
  18. Walk Walk "Walk" phase happen right before the code generation

    takes place. "Walk" phase happen right before the code generation takes place. Transforms the nodes to simpler forms Transforms the nodes to simpler forms Transforms the nodes into some function calls Transforms the nodes into some function calls Promote PAUTOHEAP variables (see Escape Analysis) into pointers and initialize them Promote PAUTOHEAP variables (see Escape Analysis) into pointers and initialize them with ONEWOBJ. with ONEWOBJ. etc. etc. src/cmd/compile/internal/gc/walk.go src/cmd/compile/internal/gc/walk.go
  19. Walk (cont'd) Walk (cont'd) v, ok := m["foo"] autotmp_1, ok

    := runtime.mapaccess2_fast64(typeOf(m), m, "foo") v := *autotmp_1
  20. SSA Generation SSA Generation SSA (Static Single Assignment) form is

    an intermediate representation that is often used SSA (Static Single Assignment) form is an intermediate representation that is often used to mediate an AST form of source code with corresponding machine code. to mediate an AST form of source code with corresponding machine code. As SSA form, every variable gets assigned only once during its lifecycle. As SSA form, every variable gets assigned only once during its lifecycle. This ensures each basic block has exactly a single path, and thus the same This ensures each basic block has exactly a single path, and thus the same optimization strategy can be applied. optimization strategy can be applied. A basic block: a node of a control flow graph, which contains no branch operation by A basic block: a node of a control flow graph, which contains no branch operation by definition (branches are represented as edges in CFG.) definition (branches are represented as edges in CFG.) GOSSAFUNC environment variable GOSSAFUNC environment variable ~$ GOSSAFUNC=foo go tool compile foo.go ~$ GOSSAFUNC=foo go tool compile foo.go dumped SSA to /home/moriyoshi/ssa.html dumped SSA to /home/moriyoshi/ssa.html
  21. SSA Generation (cont'd) SSA Generation (cont'd) func pos(n int) (r

    bool) { if n > 0 { r = true } else { r = false } return } OIF OGT ONAME OLITERAL OAS ONAME OLITERAL OAS ONAME OLITERAL n > 0 r true r false else v1 = InitMem <mem> v2 = SP <uintptr> v3 = SB <uintptr> v4 = LocalAddr <*int> {n} v2 v1 v5 = LocalAddr <*bool> {r} v2 v1 v6 = Arg <int> {n} (n[int]) v7 = ConstBool <bool> [false] v8 = Const64 <int> [0] v9 = Greater64 <bool> v6 v8 v10 = ConstBool <bool> [true] v11 = (uninitialized) v12 = (uninitialized) v13 = (uninitialized) v14 = (uninitialized) v9 → b3 b4 (4) v11 = Phi <bool> v10 v7 v12 = Copy <mem> v1 v13 = VarDef <mem> {r} v12 v14 = Store <mem> {bool} v5 v11 v1 Ret v14 (+9) (empty) (empty) b1 b3 b4 b2
  22. SSA Generation (cont'd) SSA Generation (cont'd) v1 = InitMem <mem>

    v2 = SP <uintptr> v5 = LocalAddr <*bool> {r} v2 v1 v6 = Arg <int> {n} (n[int]) v7 = ConstBool <bool> [false] v8 = Const64 <int> [0] v9 = Greater64 <bool> v6 v8 v10 = ConstBool <bool> [true] v13 = VarDef <mem> {r} v12 v14 = Store <mem> {bool} v5 v11 v1 Ret v14 (+9) b2
  23. Machine Code Generation (lowering / register allocation) Machine Code Generation

    (lowering / register allocation) The process of transforming IRs into the concrete machine code. The process of transforming IRs into the concrete machine code. In Go, this gradually happens in the course of the SSA pipeline. In Go, this gradually happens in the course of the SSA pipeline. v1 = InitMem <mem> v2 = SP <uintptr> v5 = LEAQ <*bool> {r} v2 v6 = Arg <int> {n} (n[int]) v8 = MOVQconst <int> [0] v13 = VarDef <mem> {r} v1 v10 = TESTQ <flags> v6 v6 v9 = SETG <bool> v10 v14 = SETGstore <mem> {r} v2 v10 v13 v6 = Arg <int> {n} : n[int] (n[int]) v1 = InitMem <mem> v13 = VarDef <mem> {r} v1 v2 = SP <uintptr> : SP v9 = LoadReg <int> v6 : AX v10 = TESTQ <flags> v9 v9 v14 = SETGstore <mem> {r} v2 v10 v13 v9 -> MOVQ "".n(SP), AX v10 -> TESTQ AX, AX v14 -> SETGT "".r+8(SP)
  24. History History

  25. Prehistoric Age Prehistoric Age The earliest commits of Go imply

    the "symbolic" lineage. The earliest commits of Go imply the "symbolic" lineage. (ken=Thompson / dmr=Ritchie / bwk=Kernighan) (ken=Thompson / dmr=Ritchie / bwk=Kernighan) There should've ever existed no git thing at that time..., did it? There should've ever existed no git thing at that time..., did it? commit 7d7c6a97f815e9279d08cfaea7d5efb5e90695a8 commit 7d7c6a97f815e9279d08cfaea7d5efb5e90695a8 Author: Brian Kernighan <bwk> Author: Brian Kernighan <bwk> Date: Tue Jul 18 19:05:45 1972 -0500 Date: Tue Jul 18 19:05:45 1972 -0500 hello, world hello, world R=ken R=ken DELTA=7 (7 added, 0 deleted, 0 changed) DELTA=7 (7 added, 0 deleted, 0 changed) commit 0bb0b61d6a85b2a1a33dcbc418089656f2754d32 commit 0bb0b61d6a85b2a1a33dcbc418089656f2754d32 Author: Brian Kernighan <bwk> Author: Brian Kernighan <bwk> Date: Sun Jan 20 01:02:03 1974 -0400 Date: Sun Jan 20 01:02:03 1974 -0400 convert to C convert to C R=dmr R=dmr DELTA=6 (0 added, 3 deleted, 3 changed) DELTA=6 (0 added, 3 deleted, 3 changed) commit 0744ac969119db8a0ad3253951d375eb77cfce9e commit 0744ac969119db8a0ad3253951d375eb77cfce9e Author: Brian Kernighan <research!bwk> Author: Brian Kernighan <research!bwk> Date: Fri Apr 1 02:02:04 1988 -0500 Date: Fri Apr 1 02:02:04 1988 -0500
  26. convert to Draft-Proposed ANSI C convert to Draft-Proposed ANSI C

    R=dmr R=dmr DELTA=5 (2 added, 0 deleted, 3 changed) DELTA=5 (2 added, 0 deleted, 3 changed) commit d82b11e4a46307f1f1415024f33263e819c222b8 commit d82b11e4a46307f1f1415024f33263e819c222b8 Author: Brian Kernighan <bwk@research.att.com> Author: Brian Kernighan <bwk@research.att.com> Date: Fri Apr 1 02:03:04 1988 -0500 Date: Fri Apr 1 02:03:04 1988 -0500 last-minute fix: convert to ANSI C last-minute fix: convert to ANSI C R=dmr R=dmr DELTA=3 (2 added, 0 deleted, 1 changed) DELTA=3 (2 added, 0 deleted, 1 changed)
  27. Pre-1.0 to 1.4 Pre-1.0 to 1.4 Most of the bootstrapping

    tools derived from Plan 9 / Inferno Most of the bootstrapping tools derived from Plan 9 / Inferno 6a / 6c / 6g / lib9... 6a / 6c / 6g / lib9... The toolchain was written in The toolchain was written in Plan-9 Plan-9 flavored flavored C C (http://doc.cat-v.org/plan_9/4th_edition/papers/comp) (http://doc.cat-v.org/plan_9/4th_edition/papers/comp) Naive code generation facility (~1.6) Naive code generation facility (~1.6) Semi-concrete IRs were generated directly from annotated ASTs. Semi-concrete IRs were generated directly from annotated ASTs.
  28. 1.3 1.3 Near-precise GC of pointers in stack. Near-precise GC

    of pointers in stack. Based on the construction of control flow graphs (CFG). Based on the construction of control flow graphs (CFG). Contiguous stack model. Contiguous stack model. 1.5 1.5 Achieved complete self-hosting. Achieved complete self-hosting. 1.7 1.7 SSA-based codegen introduced. SSA-based codegen introduced.
  29. Hacking Internals Hacking Internals

  30. Example 1: Let Go accept emojis for various identifiers. Example

    1: Let Go accept emojis for various identifiers. Modify the portion of the lexer so it will accept "" (U+1F363) for identifiers: Modify the portion of the lexer so it will accept "" (U+1F363) for identifiers: func (s *scanner) isIdentRune(c rune, first bool) bool { func (s *scanner) isIdentRune(c rune, first bool) bool { switch { switch { case unicode.IsLetter(c) || c == '_' || c == 0x1f363: // Modified case unicode.IsLetter(c) || c == '_' || c == 0x1f363: // Modified // ok // ok case unicode.IsDigit(c): case unicode.IsDigit(c): if first { if first { s.errorf("identifier cannot begin with digit %#U", c) s.errorf("identifier cannot begin with digit %#U", c) } } case c >= utf8.RuneSelf: case c >= utf8.RuneSelf: s.errorf("invalid identifier character %#U", c) s.errorf("invalid identifier character %#U", c) default: default: return false return false } } return true return true } }
  31. Example 2: Add a New Operator Example 2: Add a

    New Operator Modify the lexer so it will generate a token value for the added operator. Modify the lexer so it will generate a token value for the added operator. Modify the parser so it will understand the token and give a right AST Node for it. Modify the parser so it will understand the token and give a right AST Node for it. Modify either the typecheck or walk facility so an annotated node that corresponds to Modify either the typecheck or walk facility so an annotated node that corresponds to the AST node will be transformed into a non-fancy branch of nodes. the AST node will be transformed into a non-fancy branch of nodes. type Foo struct { type Foo struct { } } func (*Foo) ->B(i int) { func (*Foo) ->B(i int) { fmt.Println(i) fmt.Println(i) } } func (*Foo) B->() int { func (*Foo) B->() int { return 1 return 1 } } type X interface { type X interface { ->B(int) ->B(int) B->() int B->() int } } func do(x X) { func do(x X) { x->B = 0 x->B = 0 x->B, c := 2, 3 x->B, c := 2, 3 fmt.Println(c) fmt.Println(c) } }
  32. None
  33. Questions? Questions?

  34. Thank you Thank you May 18, 2019 May 18, 2019

    Tags: golang, compiler, internals Tags: golang, compiler, internals (#ZgotmplZ) (#ZgotmplZ) Moriyoshi Koizumi Moriyoshi Koizumi Open Collector, Inc. Open Collector, Inc. mozo@mozo.jp mozo@mozo.jp (mailto:mozo@mozo.jp) (mailto:mozo@mozo.jp) http://www.mozo.jp/ http://www.mozo.jp/ (http://www.mozo.jp/) (http://www.mozo.jp/) @moriyoshit @moriyoshit (http://twitter.com/moriyoshit) (http://twitter.com/moriyoshit)
  35. None