Better x86 Assembly Generation with Go (Gophercon 2019)

Slide 1

Slide 1 text

Better x86 Assembly Generation with Go Michael McLoughlin Gophercon 2019 Uber Advanced Technologies Group

Slide 2

Slide 2 text

Assembly Language Go provides the ability to write functions in assembly language. Assembly language is a general term for low-level languages that allow programming at the architecture instruction level.

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

Should I write Go functions in Assembly?

Slide 5

Slide 5 text

Slide 6

Slide 6 text

No content

Slide 7

Slide 7 text

Go Proverbs which Might Have Been Cgo Assembly is not Go. My Inner Rob Pike With the unsafe package assembly there are no guarantees. Made-up Go Proverb

Slide 8

Slide 8 text

Go Proverbs which Might Have Been Cgo Assembly is not Go. My Inner Rob Pike With the unsafe package assembly there are no guarantees. Made-up Go Proverb

Slide 9

Slide 9 text

No content

Slide 10

Slide 10 text

We should forget about small efﬁciencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%. Donald Knuth, 1974

Slide 11

Slide 11 text

Slide 12

Slide 12 text

The Critical 3%? To take advantage of: • Missed optimizations by the compiler • Special hardware instructions Common use cases: • Math compute kernels • System Calls • Low-level Runtime Details • Cryptography

Slide 13

Slide 13 text

No content

Slide 14

Slide 14 text

Outline Go Assembly Primer Problem Statement Code Generation The avo Library Examples Dot Product SHA-1 Future

Slide 15

Slide 15 text

Go Assembly Primer

Slide 16

Slide 16 text

Hello, World! package add // Add x and y. func Add(x, y uint64) uint64 { return x + y }

Slide 17

Slide 17 text

Go Disassembler The Go disassembler may be used to inspect generated machine code. go build -o add.a go tool objdump add.a

Slide 18

Slide 18 text

TEXT %22%22.Add(SB) gofile../Users/michaelmcloughlin/Dev... add.go:5 0x2e7 488b442410 MOVQ 0x10(SP), AX add.go:5 0x2ec 488b4c2408 MOVQ 0x8(SP), CX add.go:5 0x2f1 4801c8 ADDQ CX, AX add.go:5 0x2f4 4889442418 MOVQ AX, 0x18(SP) add.go:5 0x2f9 c3 RET

Slide 19

Slide 19 text

Slide 20

Slide 20 text

Function Stubs package add // Add x and y. func Add(x, y uint64) uint64 Missing function body will be implemented in assembly.

Slide 21

Slide 21 text

Implementation provided in add_amd64.s. #include "textflag.h" // func Add(x, y uint64) uint64 TEXT ·Add(SB), NOSPLIT, $0-24 MOVQ x+0(FP), AX MOVQ y+8(FP), CX ADDQ CX, AX MOVQ AX, ret+16(FP) RET

Slide 22

Slide 22 text

Implementation provided in add_amd64.s. #include "textflag.h" // func Add(x, y uint64) uint64 TEXT ·Add(SB), NOSPLIT, $0-24 ‹ Declaration MOVQ x+0(FP), AX MOVQ y+8(FP), CX ADDQ CX, AX MOVQ AX, ret+16(FP) RET

Slide 23

Slide 23 text

Implementation provided in add_amd64.s. #include "textflag.h" // func Add(x, y uint64) uint64 TEXT ·Add(SB), NOSPLIT, $0-24 MOVQ x+0(FP), AX ‹ Read x from stack frame MOVQ y+8(FP), CX ‹ Read y ADDQ CX, AX MOVQ AX, ret+16(FP) RET

Slide 24

Slide 24 text

Implementation provided in add_amd64.s. #include "textflag.h" // func Add(x, y uint64) uint64 TEXT ·Add(SB), NOSPLIT, $0-24 MOVQ x+0(FP), AX MOVQ y+8(FP), CX ADDQ CX, AX MOVQ AX, ret+16(FP) RET

Slide 25

Slide 25 text

Implementation provided in add_amd64.s. #include "textflag.h" // func Add(x, y uint64) uint64 TEXT ·Add(SB), NOSPLIT, $0-24 MOVQ x+0(FP), AX MOVQ y+8(FP), CX ADDQ CX, AX MOVQ AX, ret+16(FP) ‹ Write return value RET

Slide 26

Slide 26 text

Problem Statement

Slide 27

Slide 27 text

24,962

Slide 28

Slide 28 text

Table 1: Assembly Lines by Top-Level Packages Lines Package 8140 crypto 8069 runtime 5686 internal 1173 math 1005 syscall 574 cmd 279 hash 36 reflect

Slide 29

Slide 29 text

Table 1: Assembly Lines by Top-Level Packages Lines Package 8140 crypto 8069 runtime 5686 internal 1173 math 1005 syscall 574 cmd 279 hash 36 reflect

Slide 30

Slide 30 text

Table 2: Top 10 Assembly Files by Lines Lines File 2695 internal/x/crypto/.../chacha20poly1305_amd64.s 2348 crypto/elliptic/p256_asm_amd64.s 1632 runtime/asm_amd64.s 1500 crypto/sha1/sha1block_amd64.s 1468 crypto/sha512/sha512block_amd64.s 1377 internal/x/crypto/curve25519/ladderstep_amd64.s 1286 crypto/aes/gcm_amd64.s 1031 crypto/sha256/sha256block_amd64.s 743 runtime/sys_darwin_amd64.s 727 runtime/sys_linux_amd64.s

Slide 31

Slide 31 text

Slide 32

Slide 32 text

openAVX2InternalLoop: // Lets just say this spaghetti loop interleaves 2 quarter rounds with 3 poly multiplications // Effectively per 512 bytes of stream we hash 480 bytes of ciphertext polyAdd(0*8(inp)(itr1*1)) VPADDD BB0, AA0, AA0; VPADDD BB1, AA1, AA1; VPADDD BB2, AA2, AA2; VPADDD BB3, AA3, AA3 polyMulStage1_AVX2 VPXOR AA0, DD0, DD0; VPXOR AA1, DD1, DD1; VPXOR AA2, DD2, DD2; VPXOR AA3, DD3, DD3 VPSHUFB ·rol16<>(SB), DD0, DD0; VPSHUFB ·rol16<>(SB), DD1, DD1; VPSHUFB ·rol16<>(SB), DD2, DD2; VPSHUFB ·rol16<>(S polyMulStage2_AVX2 VPADDD DD0, CC0, CC0; VPADDD DD1, CC1, CC1; VPADDD DD2, CC2, CC2; VPADDD DD3, CC3, CC3 VPXOR CC0, BB0, BB0; VPXOR CC1, BB1, BB1; VPXOR CC2, BB2, BB2; VPXOR CC3, BB3, BB3 polyMulStage3_AVX2 VMOVDQA CC3, tmpStoreAVX2 VPSLLD $12, BB0, CC3; VPSRLD $20, BB0, BB0; VPXOR CC3, BB0, BB0 VPSLLD $12, BB1, CC3; VPSRLD $20, BB1, BB1; VPXOR CC3, BB1, BB1 VPSLLD $12, BB2, CC3; VPSRLD $20, BB2, BB2; VPXOR CC3, BB2, BB2 VPSLLD $12, BB3, CC3; VPSRLD $20, BB3, BB3; VPXOR CC3, BB3, BB3 VMOVDQA tmpStoreAVX2, CC3 polyMulReduceStage VPADDD BB0, AA0, AA0; VPADDD BB1, AA1, AA1; VPADDD BB2, AA2, AA2; VPADDD BB3, AA3, AA3 VPXOR AA0, DD0, DD0; VPXOR AA1, DD1, DD1; VPXOR AA2, DD2, DD2; VPXOR AA3, DD3, DD3 VPSHUFB ·rol8<>(SB), DD0, DD0; VPSHUFB ·rol8<>(SB), DD1, DD1; VPSHUFB ·rol8<>(SB), DD2, DD2; VPSHUFB ·rol8<>(SB), polyAdd(2*8(inp)(itr1*1)) VPADDD DD0, CC0, CC0; VPADDD DD1, CC1, CC1; VPADDD DD2, CC2, CC2; VPADDD DD3, CC3, CC3 internal/x/.../chacha20poly1305_amd64.s lines 856-879 (go1.12)

Slide 33

Slide 33 text

// Special optimization for buffers smaller than 321 bytes openAVX2320: // For up to 320 bytes of ciphertext and 64 bytes for the poly key, we process six blocks VMOVDQA AA0, AA1; VMOVDQA BB0, BB1; VMOVDQA CC0, CC1; VPADDD ·avx2IncMask<>(SB), DD0, DD1 VMOVDQA AA0, AA2; VMOVDQA BB0, BB2; VMOVDQA CC0, CC2; VPADDD ·avx2IncMask<>(SB), DD1, DD2 VMOVDQA BB0, TT1; VMOVDQA CC0, TT2; VMOVDQA DD0, TT3 MOVQ $10, itr2 openAVX2320InnerCipherLoop: chachaQR_AVX2(AA0, BB0, CC0, DD0, TT0); chachaQR_AVX2(AA1, BB1, CC1, DD1, TT0); chachaQR_AVX2(AA2, BB2, CC2, DD2, T VPALIGNR $4, BB0, BB0, BB0; VPALIGNR $4, BB1, BB1, BB1; VPALIGNR $4, BB2, BB2, BB2 VPALIGNR $8, CC0, CC0, CC0; VPALIGNR $8, CC1, CC1, CC1; VPALIGNR $8, CC2, CC2, CC2 VPALIGNR $12, DD0, DD0, DD0; VPALIGNR $12, DD1, DD1, DD1; VPALIGNR $12, DD2, DD2, DD2 chachaQR_AVX2(AA0, BB0, CC0, DD0, TT0); chachaQR_AVX2(AA1, BB1, CC1, DD1, TT0); chachaQR_AVX2(AA2, BB2, CC2, DD2, T VPALIGNR $12, BB0, BB0, BB0; VPALIGNR $12, BB1, BB1, BB1; VPALIGNR $12, BB2, BB2, BB2 VPALIGNR $8, CC0, CC0, CC0; VPALIGNR $8, CC1, CC1, CC1; VPALIGNR $8, CC2, CC2, CC2 VPALIGNR $4, DD0, DD0, DD0; VPALIGNR $4, DD1, DD1, DD1; VPALIGNR $4, DD2, DD2, DD2 DECQ itr2 JNE openAVX2320InnerCipherLoop VMOVDQA ·chacha20Constants<>(SB), TT0 VPADDD TT0, AA0, AA0; VPADDD TT0, AA1, AA1; VPADDD TT0, AA2, AA2 VPADDD TT1, BB0, BB0; VPADDD TT1, BB1, BB1; VPADDD TT1, BB2, BB2 VPADDD TT2, CC0, CC0; VPADDD TT2, CC1, CC1; VPADDD TT2, CC2, CC2 internal/x/.../chacha20poly1305_amd64.s lines 1072-1095 (go1.12)

Slide 34

Slide 34 text

openAVX2Tail512LoopA: VPADDD BB0, AA0, AA0; VPADDD BB1, AA1, AA1; VPADDD BB2, AA2, AA2; VPADDD BB3, AA3, AA3 VPXOR AA0, DD0, DD0; VPXOR AA1, DD1, DD1; VPXOR AA2, DD2, DD2; VPXOR AA3, DD3, DD3 VPSHUFB ·rol16<>(SB), DD0, DD0; VPSHUFB ·rol16<>(SB), DD1, DD1; VPSHUFB ·rol16<>(SB), DD2, DD2; VPSHUFB ·rol16<>(S VPADDD DD0, CC0, CC0; VPADDD DD1, CC1, CC1; VPADDD DD2, CC2, CC2; VPADDD DD3, CC3, CC3 VPXOR CC0, BB0, BB0; VPXOR CC1, BB1, BB1; VPXOR CC2, BB2, BB2; VPXOR CC3, BB3, BB3 VMOVDQA CC3, tmpStoreAVX2 VPSLLD $12, BB0, CC3; VPSRLD $20, BB0, BB0; VPXOR CC3, BB0, BB0 VPSLLD $12, BB1, CC3; VPSRLD $20, BB1, BB1; VPXOR CC3, BB1, BB1 VPSLLD $12, BB2, CC3; VPSRLD $20, BB2, BB2; VPXOR CC3, BB2, BB2 VPSLLD $12, BB3, CC3; VPSRLD $20, BB3, BB3; VPXOR CC3, BB3, BB3 VMOVDQA tmpStoreAVX2, CC3 polyAdd(0*8(itr2)) polyMulAVX2 VPADDD BB0, AA0, AA0; VPADDD BB1, AA1, AA1; VPADDD BB2, AA2, AA2; VPADDD BB3, AA3, AA3 VPXOR AA0, DD0, DD0; VPXOR AA1, DD1, DD1; VPXOR AA2, DD2, DD2; VPXOR AA3, DD3, DD3 VPSHUFB ·rol8<>(SB), DD0, DD0; VPSHUFB ·rol8<>(SB), DD1, DD1; VPSHUFB ·rol8<>(SB), DD2, DD2; VPSHUFB ·rol8<>(SB), VPADDD DD0, CC0, CC0; VPADDD DD1, CC1, CC1; VPADDD DD2, CC2, CC2; VPADDD DD3, CC3, CC3 VPXOR CC0, BB0, BB0; VPXOR CC1, BB1, BB1; VPXOR CC2, BB2, BB2; VPXOR CC3, BB3, BB3 VMOVDQA CC3, tmpStoreAVX2 VPSLLD $7, BB0, CC3; VPSRLD $25, BB0, BB0; VPXOR CC3, BB0, BB0 VPSLLD $7, BB1, CC3; VPSRLD $25, BB1, BB1; VPXOR CC3, BB1, BB1 VPSLLD $7, BB2, CC3; VPSRLD $25, BB2, BB2; VPXOR CC3, BB2, BB2 VPSLLD $7, BB3, CC3; VPSRLD $25, BB3, BB3; VPXOR CC3, BB3, BB3 internal/x/.../chacha20poly1305_amd64.s lines 1374-1397 (go1.12)

Slide 35

Slide 35 text

sealAVX2Tail512LoopB: VPADDD BB0, AA0, AA0; VPADDD BB1, AA1, AA1; VPADDD BB2, AA2, AA2; VPADDD BB3, AA3, AA3 VPXOR AA0, DD0, DD0; VPXOR AA1, DD1, DD1; VPXOR AA2, DD2, DD2; VPXOR AA3, DD3, DD3 VPSHUFB ·rol16<>(SB), DD0, DD0; VPSHUFB ·rol16<>(SB), DD1, DD1; VPSHUFB ·rol16<>(SB), DD2, DD2; VPSHUFB ·rol16<>(S VPADDD DD0, CC0, CC0; VPADDD DD1, CC1, CC1; VPADDD DD2, CC2, CC2; VPADDD DD3, CC3, CC3 VPXOR CC0, BB0, BB0; VPXOR CC1, BB1, BB1; VPXOR CC2, BB2, BB2; VPXOR CC3, BB3, BB3 VMOVDQA CC3, tmpStoreAVX2 VPSLLD $12, BB0, CC3; VPSRLD $20, BB0, BB0; VPXOR CC3, BB0, BB0 VPSLLD $12, BB1, CC3; VPSRLD $20, BB1, BB1; VPXOR CC3, BB1, BB1 VPSLLD $12, BB2, CC3; VPSRLD $20, BB2, BB2; VPXOR CC3, BB2, BB2 VPSLLD $12, BB3, CC3; VPSRLD $20, BB3, BB3; VPXOR CC3, BB3, BB3 VMOVDQA tmpStoreAVX2, CC3 polyAdd(0*8(oup)) polyMulAVX2 VPADDD BB0, AA0, AA0; VPADDD BB1, AA1, AA1; VPADDD BB2, AA2, AA2; VPADDD BB3, AA3, AA3 VPXOR AA0, DD0, DD0; VPXOR AA1, DD1, DD1; VPXOR AA2, DD2, DD2; VPXOR AA3, DD3, DD3 VPSHUFB ·rol8<>(SB), DD0, DD0; VPSHUFB ·rol8<>(SB), DD1, DD1; VPSHUFB ·rol8<>(SB), DD2, DD2; VPSHUFB ·rol8<>(SB), VPADDD DD0, CC0, CC0; VPADDD DD1, CC1, CC1; VPADDD DD2, CC2, CC2; VPADDD DD3, CC3, CC3 VPXOR CC0, BB0, BB0; VPXOR CC1, BB1, BB1; VPXOR CC2, BB2, BB2; VPXOR CC3, BB3, BB3 VMOVDQA CC3, tmpStoreAVX2 VPSLLD $7, BB0, CC3; VPSRLD $25, BB0, BB0; VPXOR CC3, BB0, BB0 VPSLLD $7, BB1, CC3; VPSRLD $25, BB1, BB1; VPXOR CC3, BB1, BB1 VPSLLD $7, BB2, CC3; VPSRLD $25, BB2, BB2; VPXOR CC3, BB2, BB2 VPSLLD $7, BB3, CC3; VPSRLD $25, BB3, BB3; VPXOR CC3, BB3, BB3 internal/x/.../chacha20poly1305_amd64.s lines 2593-2616 (go1.12)

Slide 36

Slide 36 text

Is this ﬁne?

Slide 37

Slide 37 text

TEXT p256SubInternal(SB),NOSPLIT,$0 XORQ mul0, mul0 SUBQ t0, acc4 SBBQ t1, acc5 SBBQ t2, acc6 SBBQ t3, acc7 SBBQ $0, mul0 MOVQ acc4, acc0 MOVQ acc5, acc1 MOVQ acc6, acc2 MOVQ acc7, acc3 ADDQ $-1, acc4 ADCQ p256const0<>(SB), acc5 ADCQ $0, acc6 ADCQ p256const1<>(SB), acc7 ADCQ $0, mul0 CMOVQNE acc0, acc4 CMOVQNE acc1, acc5 CMOVQNE acc2, acc6 CMOVQNE acc3, acc7 RET crypto/elliptic/p256_asm_amd64.s lines 1300-1324 (94e44a9c8e)

Slide 38

Slide 38 text

No content

Slide 39

Slide 39 text

No content

Slide 40

Slide 40 text

No content

Slide 41

Slide 41 text

No content

Slide 42

Slide 42 text

Go Assembly Policy 1. Prefer Go, not assembly 2. Minimize use of assembly 3. Explain root causes 4. Test it well 5. Make assembly easy to review

Slide 43

Slide 43 text

Make your assembly easy to review; ideally, auto-generate it using a simpler Go program. Comment it well. Go Assembly Policy, Rule IV

Slide 44

Slide 44 text

Code Generation

Slide 45

Slide 45 text

There’s a reason people use compilers.

Slide 46

Slide 46 text

Intrinsics __m256d latq = _mm256_loadu_pd(lat); latq = _mm256_mul_pd(latq, _mm256_set1_pd(1 / 180.0)); latq = _mm256_add_pd(latq, _mm256_set1_pd(1.5)); __m256i lati = _mm256_srli_epi64(_mm256_castpd_si256(latq), __m256d lngq = _mm256_loadu_pd(lng); lngq = _mm256_mul_pd(lngq, _mm256_set1_pd(1 / 360.0)); lngq = _mm256_add_pd(lngq, _mm256_set1_pd(1.5)); __m256i lngi = _mm256_srli_epi64(_mm256_castpd_si256(lngq),

Slide 47

Slide 47 text

High-level Assembler Assembly language plus high-level language features. Macro assemblers: Microsoft Macro Assembler (MASM), Netwide Assembler (NASM), ...

Slide 48

Slide 48 text

High-level Assembler Assembly language plus high-level language features. Macro assemblers: Microsoft Macro Assembler (MASM), Netwide Assembler (NASM), ...

Slide 49

Slide 49 text

No content

Slide 50

Slide 50 text

No content

Slide 51

Slide 51 text

PeachPy Python-based High-Level Assembler

Slide 52

Slide 52 text

What about Go?

Slide 53

Slide 53 text

The avo Library

Slide 54

Slide 54 text

https://github.com/mmcloughlin/avo

Slide 55

Slide 55 text

Go framework that presents an assembly-like DSL.

Slide 56

Slide 56 text

Not a compiler. Not an assembler.

Slide 57

Slide 57 text

Programmer retains complete control, but without tedium.

Slide 58

Slide 58 text

Use Go control structures for assembly generation; avo programs are Go programs

Slide 59

Slide 59 text

Slide 60

Slide 60 text

Automatically load arguments and store return values: ensure memory offsets are correct for complex structures

Slide 61

Slide 61 text

Generation of stub ﬁles to interface with your Go package

Slide 62

Slide 62 text

import . "github.com/mmcloughlin/avo/build" func main() { TEXT("Add", NOSPLIT, "func(x, y uint64) uint64") Doc("Add adds x and y.") x := Load(Param("x"), GP64()) y := Load(Param("y"), GP64()) ADDQ(x, y) Store(y, ReturnIndex(0)) RET() Generate() }

Slide 63

Slide 63 text

Slide 64

Slide 64 text

Slide 65

Slide 65 text

import . "github.com/mmcloughlin/avo/build" func main() { TEXT("Add", NOSPLIT, "func(x, y uint64) uint64") Doc("Add adds x and y.") x := Load(Param("x"), GP64()) ‹ Param references y := Load(Param("y"), GP64()) ‹ Allocates register ADDQ(x, y) Store(y, ReturnIndex(0)) RET() Generate() }

Slide 66

Slide 66 text

Slide 67

Slide 67 text

Slide 68

Slide 68 text

Slide 69

Slide 69 text

Build go run asm.go -out add.s -stubs stubs.go

Slide 70

Slide 70 text

Generated Assembly // Code generated by command: go run asm.go -out add.s -stubs stu #include "textflag.h" // func Add(x uint64, y uint64) uint64 TEXT ·Add(SB), NOSPLIT, $0-24 MOVQ x+0(FP), AX MOVQ y+8(FP), CX ADDQ AX, CX MOVQ CX, ret+16(FP) RET

Slide 71

Slide 71 text

Generated Assembly // Code generated by command: go run asm.go -out add.s -stubs stu #include "textflag.h" // func Add(x uint64, y uint64) uint64 TEXT ·Add(SB), NOSPLIT, $0-24 ‹ Computed stack sizes MOVQ x+0(FP), AX MOVQ y+8(FP), CX ADDQ AX, CX MOVQ CX, ret+16(FP) RET

Slide 72

Slide 72 text

Generated Assembly // Code generated by command: go run asm.go -out add.s -stubs stu #include "textflag.h" // func Add(x uint64, y uint64) uint64 TEXT ·Add(SB), NOSPLIT, $0-24 MOVQ x+0(FP), AX ‹ Computed offsets MOVQ y+8(FP), CX ADDQ AX, CX MOVQ CX, ret+16(FP) RET

Slide 73

Slide 73 text

Generated Assembly // Code generated by command: go run asm.go -out add.s -stubs stu #include "textflag.h" // func Add(x uint64, y uint64) uint64 TEXT ·Add(SB), NOSPLIT, $0-24 MOVQ x+0(FP), AX ‹ Registers allocated MOVQ y+8(FP), CX ADDQ AX, CX MOVQ CX, ret+16(FP) RET

Slide 74

Slide 74 text

Auto-generated Stubs // Code generated by command: go run asm.go -out add.s -stubs stu package addavo // Add adds x and y. func Add(x uint64, y uint64) uint64

Slide 75

Slide 75 text

Go Control Structures TEXT("Mul5", NOSPLIT, "func(x uint64) uint64") Doc("Mul5 adds x to itself five times.") x := Load(Param("x"), GP64()) p := GP64() MOVQ(x, p) for i := 0; i < 4; i++ { ADDQ(x, p) } Store(p, ReturnIndex(0)) RET()

Slide 76

Slide 76 text

Slide 77

Slide 77 text

Generated Assembly // func Mul5(x uint64) uint64 TEXT ·Mul5(SB), NOSPLIT, $0-16 MOVQ x+0(FP), AX MOVQ AX, CX ADDQ AX, CX ADDQ AX, CX ADDQ AX, CX ADDQ AX, CX MOVQ CX, ret+8(FP) RET

Slide 78

Slide 78 text

Generated Assembly // func Mul5(x uint64) uint64 TEXT ·Mul5(SB), NOSPLIT, $0-16 MOVQ x+0(FP), AX MOVQ AX, CX ADDQ AX, CX ‹ Look, there's four of them! ADDQ AX, CX ADDQ AX, CX ADDQ AX, CX MOVQ CX, ret+8(FP) RET

Slide 79

Slide 79 text

Complex Parameter Loading type Struct struct { A byte B uint32 Sub [7]complex64 C uint16 }

Slide 80

Slide 80 text

Complex Parameter Loading Package("github.com/mmcloughlin/params") TEXT("Sub5Imag", NOSPLIT, "func(s Struct) float32") Doc("Returns the imaginary part of s.Sub[5]") x := Load(Param("s").Field("Sub").Index(5).Imag(), XMM()) Store(x, ReturnIndex(0)) RET()

Slide 81

Slide 81 text

Complex Parameter Loading Package("github.com/mmcloughlin/params") ‹ Types TEXT("Sub5Imag", NOSPLIT, "func(s Struct) float32") Doc("Returns the imaginary part of s.Sub[5]") x := Load(Param("s").Field("Sub").Index(5).Imag(), XMM()) Store(x, ReturnIndex(0)) RET()

Slide 82

Slide 82 text

Slide 83

Slide 83 text

Generated Assembly // func Sub5Imag(s Struct) float32 TEXT ·Sub5Imag(SB), NOSPLIT, $0-76 MOVSS s_Sub_5_imag+52(FP), X0 MOVSS X0, ret+72(FP) RET

Slide 84

Slide 84 text

Generated Assembly // func Sub5Imag(s Struct) float32 TEXT ·Sub5Imag(SB), NOSPLIT, $0-76 MOVSS s_Sub_5_imag+52(FP), X0 ‹ Of course it was 52 bytes MOVSS X0, ret+72(FP) RET

Slide 85

Slide 85 text

Examples

Slide 86

Slide 86 text

Vector Dot Product Maps two equal-length vectors x = (xi), y = (yi) to a single number. x · y = ∑ i xi × yi

Slide 87

Slide 87 text

x 0.8 2.0 2.0 3.1 1.9 4.1 1.5 4.7 4.5 4.5 3.6 3.0 2.4 2.0 1.4 2.4 y 3.0 3.4 3.8 2.0 5.0 2.0 2.4 2.0 2.0 1.6 2.0 3.1 2.0 2.0 4.0 2.0

Slide 88

Slide 88 text

x 0.8 2.0 2.0 3.1 1.9 4.1 1.5 4.7 4.5 4.5 3.6 3.0 2.4 2.0 1.4 2.4 y 3.0 3.4 3.8 2.0 5.0 2.0 2.4 2.0 2.0 1.6 2.0 3.1 2.0 2.0 4.0 2.0 × × × × × × × × × × × × × × × × = (xi × yi) 2.4 6.8 7.6 6.2 9.5 8.2 3.6 9.4 9.0 7.2 7.2 9.3 4.8 4.0 5.6 4.8

Slide 89

Slide 89 text

Slide 90

Slide 90 text

Pure Go Implementation package dot // DotGeneric implements vector dot product in pure Go. func DotGeneric(x, y []float32) float32 { var d float32 for i := range x { d += x[i] * y[i] } return d }

Slide 91

Slide 91 text

Pure Go Implementation package dot // DotGeneric implements vector dot product in pure Go. func DotGeneric(x, y []float32) float32 { var d float32 for i := range x { d += x[i] * y[i] ‹ Multiply and accumulate } return d }

Slide 92

Slide 92 text

Pure Go 970 M/s 1.0x (Vector size 4096, Intel Core i7-7567U at 3.5GHz)

Slide 93

Slide 93 text

Preamble TEXT("DotAsm", NOSPLIT, "func(x, y []float32) float32") Doc("DotAsm computes the dot product of x and y.") x := Mem{Base: Load(Param("x").Base(), GP64())} y := Mem{Base: Load(Param("y").Base(), GP64())} n := Load(Param("x").Len(), GP64())

Slide 94

Slide 94 text

Slide 95

Slide 95 text

Slide 96

Slide 96 text

Initialization Comment("Initialize dot product and index to zero.") d := XMM() XORPS(d, d) idx := GP64() XORQ(idx, idx)

Slide 97

Slide 97 text

Initialization Comment("Initialize dot product and index to zero.") d := XMM() ‹ Dot product XORPS(d, d) idx := GP64() XORQ(idx, idx)

Slide 98

Slide 98 text

Initialization Comment("Initialize dot product and index to zero.") d := XMM() XORPS(d, d) idx := GP64() ‹ Index register XORQ(idx, idx)

Slide 99

Slide 99 text

Main Loop Label("loop") CMPQ(idx, n) JGE(LabelRef("done")) xy := XMM() MOVSS(x.Idx(idx, 4), xy) MULSS(y.Idx(idx, 4), xy) ADDSS(xy, d) INCQ(idx) JMP(LabelRef("loop"))

Slide 100

Slide 100 text

Main Loop Label("loop") CMPQ(idx, n) JGE(LabelRef("done")) xy := XMM() MOVSS(x.Idx(idx, 4), xy) MULSS(y.Idx(idx, 4), xy) ADDSS(xy, d) INCQ(idx) JMP(LabelRef("loop"))

Slide 101

Slide 101 text

Main Loop Label("loop") CMPQ(idx, n) ‹ if idx < n JGE(LabelRef("done")) ‹ goto done xy := XMM() MOVSS(x.Idx(idx, 4), xy) MULSS(y.Idx(idx, 4), xy) ADDSS(xy, d) INCQ(idx) JMP(LabelRef("loop"))

Slide 102

Slide 102 text

Main Loop Label("loop") CMPQ(idx, n) JGE(LabelRef("done")) xy := XMM() ‹ Temporary register for product MOVSS(x.Idx(idx, 4), xy) ‹ Load x MULSS(y.Idx(idx, 4), xy) ‹ Multiply by y ADDSS(xy, d) ‹ Add into result INCQ(idx) JMP(LabelRef("loop"))

Slide 103

Slide 103 text

Main Loop Label("loop") CMPQ(idx, n) JGE(LabelRef("done")) xy := XMM() MOVSS(x.Idx(idx, 4), xy) MULSS(y.Idx(idx, 4), xy) ADDSS(xy, d) INCQ(idx) ‹ idx++ JMP(LabelRef("loop"))

Slide 104

Slide 104 text

Return Label("done") Comment("Store dot product to return value.") Store(d, ReturnIndex(0)) RET() Generate()

Slide 105

Slide 105 text

Assembly 976 M/s 1.0x (Vector size 4096, Intel Core i7-7567U at 3.5GHz)

Slide 106

Slide 106 text

No content

Slide 107

Slide 107 text

Special Fused-Multiply-Add (FMA) instructions combine the multiply and accumulate.

Slide 108

Slide 108 text

Vectorized VFMADD231PS instruction does 8 single-precision FMAs.

Slide 109

Slide 109 text

No content

Slide 110

Slide 110 text

0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 × × × × × × × × × × × × × × × × × × × × × × × × 0.8 2.0 2.0 3.1 1.9 4.1 1.5 4.7 4.5 4.5 3.6 3.0 2.4 2.0 1.4 2.4 3.2 2.0 1.9 2.9 2.0 2.0 0.6 3.4 · · · 3.0 3.4 3.8 2.0 5.0 2.0 2.4 2.0 2.0 1.6 2.0 3.1 2.0 2.0 4.0 2.0 3.0 2.6 4.0 2.0 3.1 4.0 3.0 2.5 · · ·

Slide 111

Slide 111 text

× × × × × × × × += 2.4 6.8 7.6 6.2 9.5 8.2 3.6 9.4 × × × × × × × × × × × × × × × × 0.8 2.0 2.0 3.1 1.9 4.1 1.5 4.7 4.5 4.5 3.6 3.0 2.4 2.0 1.4 2.4 3.2 2.0 1.9 2.9 2.0 2.0 0.6 3.4 · · · 3.0 3.4 3.8 2.0 5.0 2.0 2.4 2.0 2.0 1.6 2.0 3.1 2.0 2.0 4.0 2.0 3.0 2.6 4.0 2.0 3.1 4.0 3.0 2.5 · · ·

Slide 112

Slide 112 text

× × × × × × × × × × × × × × × × += 11.4 14.0 14.8 15.5 14.3 12.2 9.2 14.2 × × × × × × × × 0.8 2.0 2.0 3.1 1.9 4.1 1.5 4.7 4.5 4.5 3.6 3.0 2.4 2.0 1.4 2.4 3.2 2.0 1.9 2.9 2.0 2.0 0.6 3.4 · · · 3.0 3.4 3.8 2.0 5.0 2.0 2.4 2.0 2.0 1.6 2.0 3.1 2.0 2.0 4.0 2.0 3.0 2.6 4.0 2.0 3.1 4.0 3.0 2.5 · · ·

Slide 113

Slide 113 text

× × × × × × × × × × × × × × × × × × × × × × × × += 21.0 19.2 22.4 21.3 20.5 20.2 11.0 22.7 0.8 2.0 2.0 3.1 1.9 4.1 1.5 4.7 4.5 4.5 3.6 3.0 2.4 2.0 1.4 2.4 3.2 2.0 1.9 2.9 2.0 2.0 0.6 3.4 · · · 3.0 3.4 3.8 2.0 5.0 2.0 2.4 2.0 2.0 1.6 2.0 3.1 2.0 2.0 4.0 2.0 3.0 2.6 4.0 2.0 3.1 4.0 3.0 2.5 · · ·

Slide 114

Slide 114 text

Loop unrolling and pipelining.

Slide 115

Slide 115 text

0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 × × × × × × × × × × × × × × × × × × × × × × × × × × × × × × × × × × × × × × × × × × × × × × × × 0.8 2.0 2.0 3.1 1.9 4.1 1.5 4.7 4.5 4.5 3.6 3.0 2.4 2.0 1.4 2.4 3.2 2.0 1.9 2.9 2.0 2.0 0.6 3.4 9.5 3.4 2.0 1.4 5.0 1.5 2.1 4.0 5.0 7.0 1.5 5.0 4.3 2.5 2.8 5.0 2.6 5.8 2.0 6.5 1.1 1.4 0.8 3.0 · · · 3.0 3.4 3.8 2.0 5.0 2.0 2.4 2.0 2.0 1.6 2.0 3.1 2.0 2.0 4.0 2.0 3.0 2.6 4.0 2.0 3.1 4.0 3.0 2.5 0.6 2.5 4.0 7.0 1.3 3.0 4.0 2.4 1.4 1.2 1.2 1.7 2.0 3.0 3.5 1.7 0.5 0.5 1.1 1.4 5.0 2.0 0.5 3.3 · · ·

Slide 116

Slide 116 text

× × × × × × × × += 2.4 6.8 7.6 6.2 9.5 8.2 3.6 9.4 × × × × × × × × += 9.0 7.2 7.2 9.3 4.8 4.0 5.6 4.8 × × × × × × × × × × × × × × × × × × × × × × × × × × × × × × × × 0.8 2.0 2.0 3.1 1.9 4.1 1.5 4.7 4.5 4.5 3.6 3.0 2.4 2.0 1.4 2.4 3.2 2.0 1.9 2.9 2.0 2.0 0.6 3.4 9.5 3.4 2.0 1.4 5.0 1.5 2.1 4.0 5.0 7.0 1.5 5.0 4.3 2.5 2.8 5.0 2.6 5.8 2.0 6.5 1.1 1.4 0.8 3.0 · · · 3.0 3.4 3.8 2.0 5.0 2.0 2.4 2.0 2.0 1.6 2.0 3.1 2.0 2.0 4.0 2.0 3.0 2.6 4.0 2.0 3.1 4.0 3.0 2.5 0.6 2.5 4.0 7.0 1.3 3.0 4.0 2.4 1.4 1.2 1.2 1.7 2.0 3.0 3.5 1.7 0.5 0.5 1.1 1.4 5.0 2.0 0.5 3.3 · · ·

Slide 117

Slide 117 text

× × × × × × × × × × × × × × × × × × × × × × × × += 12.0 12.0 15.2 12.0 15.7 16.2 5.4 17.9 × × × × × × × × += 14.7 15.7 15.2 19.1 11.3 8.5 14.0 14.4 × × × × × × × × × × × × × × × × 0.8 2.0 2.0 3.1 1.9 4.1 1.5 4.7 4.5 4.5 3.6 3.0 2.4 2.0 1.4 2.4 3.2 2.0 1.9 2.9 2.0 2.0 0.6 3.4 9.5 3.4 2.0 1.4 5.0 1.5 2.1 4.0 5.0 7.0 1.5 5.0 4.3 2.5 2.8 5.0 2.6 5.8 2.0 6.5 1.1 1.4 0.8 3.0 · · · 3.0 3.4 3.8 2.0 5.0 2.0 2.4 2.0 2.0 1.6 2.0 3.1 2.0 2.0 4.0 2.0 3.0 2.6 4.0 2.0 3.1 4.0 3.0 2.5 0.6 2.5 4.0 7.0 1.3 3.0 4.0 2.4 1.4 1.2 1.2 1.7 2.0 3.0 3.5 1.7 0.5 0.5 1.1 1.4 5.0 2.0 0.5 3.3 · · ·

Slide 118

Slide 118 text

× × × × × × × × × × × × × × × × × × × × × × × × × × × × × × × × × × × × × × × × += 19.0 20.4 17.0 20.5 24.3 23.7 15.2 26.4 × × × × × × × × += 16.0 18.6 17.4 28.2 16.8 11.3 14.4 24.3 + + 0.8 2.0 2.0 3.1 1.9 4.1 1.5 4.7 4.5 4.5 3.6 3.0 2.4 2.0 1.4 2.4 3.2 2.0 1.9 2.9 2.0 2.0 0.6 3.4 9.5 3.4 2.0 1.4 5.0 1.5 2.1 4.0 5.0 7.0 1.5 5.0 4.3 2.5 2.8 5.0 2.6 5.8 2.0 6.5 1.1 1.4 0.8 3.0 · · · 3.0 3.4 3.8 2.0 5.0 2.0 2.4 2.0 2.0 1.6 2.0 3.1 2.0 2.0 4.0 2.0 3.0 2.6 4.0 2.0 3.1 4.0 3.0 2.5 0.6 2.5 4.0 7.0 1.3 3.0 4.0 2.4 1.4 1.2 1.2 1.7 2.0 3.0 3.5 1.7 0.5 0.5 1.1 1.4 5.0 2.0 0.5 3.3 · · ·

Slide 119

Slide 119 text

Preamble func dot(unroll int) { name := fmt.Sprintf("DotVecUnroll%d", unroll) TEXT(name, NOSPLIT, "func(x, y []float32) float32") x := Mem{Base: Load(Param("x").Base(), GP64())} y := Mem{Base: Load(Param("y").Base(), GP64())} n := Load(Param("x").Len(), GP64())

Slide 120

Slide 120 text

Preamble func dot(unroll int) { ‹ Parameterized code generation name := fmt.Sprintf("DotVecUnroll%d", unroll) TEXT(name, NOSPLIT, "func(x, y []float32) float32") x := Mem{Base: Load(Param("x").Base(), GP64())} y := Mem{Base: Load(Param("y").Base(), GP64())} n := Load(Param("x").Len(), GP64())

Slide 121

Slide 121 text

Slide 122

Slide 122 text

Slide 123

Slide 123 text

Initialization // Allocate and zero accumulation registers. acc := make([]VecVirtual, unroll) for i := 0; i < unroll; i++ { acc[i] = YMM() VXORPS(acc[i], acc[i], acc[i]) }

Slide 124

Slide 124 text

Initialization // Allocate and zero accumulation registers. acc := make([]VecVirtual, unroll) for i := 0; i < unroll; i++ { acc[i] = YMM() ‹ 256-bit registers VXORPS(acc[i], acc[i], acc[i]) ‹ XOR to zero }

Slide 125

Slide 125 text

Loop Check blockitems := 8 * unroll blocksize := 4 * blockitems Label("blockloop") CMPQ(n, U32(blockitems)) JL(LabelRef("tail"))

Slide 126

Slide 126 text

Loop Check blockitems := 8 * unroll blocksize := 4 * blockitems Label("blockloop") ‹ start loop over blocks CMPQ(n, U32(blockitems)) ‹ if have full block JL(LabelRef("tail"))

Slide 127

Slide 127 text

Loop Body // Load x. xs := make([]VecVirtual, unroll) for i := 0; i < unroll; i++ { xs[i] = YMM() VMOVUPS(x.Offset(32*i), xs[i]) } // The actual FMA. for i := 0; i < unroll; i++ { VFMADD231PS(y.Offset(32*i), xs[i], acc[i]) }

Slide 128

Slide 128 text

Loop Body // Load x. xs := make([]VecVirtual, unroll) for i := 0; i < unroll; i++ { xs[i] = YMM() VMOVUPS(x.Offset(32*i), xs[i]) ‹ Move x to registers } // The actual FMA. for i := 0; i < unroll; i++ { VFMADD231PS(y.Offset(32*i), xs[i], acc[i]) }

Slide 129

Slide 129 text

Slide 130

Slide 130 text

Tail Loop Process last non-full block. Label("tail") tail := XMM() VXORPS(tail, tail, tail) Label("tailloop") CMPQ(n, U32(0)) JE(LabelRef("reduce")) xt := XMM() VMOVSS(x, xt) VFMADD231SS(y, xt, tail) ADDQ(U32(4), x.Base) ADDQ(U32(4), y.Base) DECQ(n) JMP(LabelRef("tailloop"))

Slide 131

Slide 131 text

Final Reduce Label("reduce") for i := 1; i < unroll; i++ { VADDPS(acc[0], acc[i], acc[0]) } result := acc[0].AsX() top := XMM() VEXTRACTF128(U8(1), acc[0], top) VADDPS(result, top, result) VADDPS(result, tail, result) VHADDPS(result, result, result) VHADDPS(result, result, result)

Slide 132

Slide 132 text

main() var unrolls = flag.String("unroll", "4", "unroll factors") func main() { flag.Parse() for _, s := range strings.Split(*unrolls, ",") { unroll, _ := strconv.Atoi(s) dot(unroll) } Generate() }

Slide 133

Slide 133 text

Unrolled ×2 15,193 M/s 15.7x (Vector size 4096, Intel Core i7-7567U at 3.5GHz)

Slide 134

Slide 134 text

Unrolled ×4 25,786 M/s 26.6x (Vector size 4096, Intel Core i7-7567U at 3.5GHz)

Slide 135

Slide 135 text

Unrolled ×6 24,456 M/s 25.2x (Vector size 4096, Intel Core i7-7567U at 3.5GHz)

Slide 136

Slide 136 text

SHA-1 Cryptographic hash function. • 80 rounds • Constants and bitwise functions vary • Message update rule • State update rule avo can be used to create a completely unrolled implementation.

Slide 137

Slide 137 text

SHA-1 Subroutines func majority(b, c, d Register) Register { t, r := GP32(), GP32() MOVL(b, t) ORL(c, t) ANDL(d, t) MOVL(b, r) ANDL(c, r) ORL(t, r) return r }

Slide 138

Slide 138 text

SHA-1 Subroutines func xor(b, c, d Register) Register { r := GP32() MOVL(b, r) XORL(c, r) XORL(d, r) return r }

Slide 139

Slide 139 text

SHA-1 Loops Comment("Load initial hash.") hash := [5]Register{GP32(), GP32(), GP32(), GP32(), GP32()} for i, r := range hash { MOVL(h.Offset(4*i), r) } Comment("Initialize registers.") a, b, c, d, e := GP32(), GP32(), GP32(), GP32(), GP32() for i, r := range []Register{a, b, c, d, e} { MOVL(hash[i], r) }

Slide 140

Slide 140 text

for r := 0; r < 80; r++ { Commentf("Round %d.", r) ... q := quarter[r/20] t := GP32() MOVL(a, t) ROLL(U8(5), t) ADDL(q.F(b, c, d), t) ADDL(e, t) ADDL(U32(q.K), t) ADDL(u, t) ROLL(Imm(30), b) a, b, c, d, e = t, a, b, c, d }

Slide 141

Slide 141 text

for r := 0; r < 80; r++ { ‹ Loop over rounds Commentf("Round %d.", r) ... q := quarter[r/20] t := GP32() MOVL(a, t) ROLL(U8(5), t) ADDL(q.F(b, c, d), t) ADDL(e, t) ADDL(U32(q.K), t) ADDL(u, t) ROLL(Imm(30), b) a, b, c, d, e = t, a, b, c, d }

Slide 142

Slide 142 text

for r := 0; r < 80; r++ { Commentf("Round %d.", r) ... q := quarter[r/20] t := GP32() ‹ State update MOVL(a, t) ROLL(U8(5), t) ADDL(q.F(b, c, d), t) ADDL(e, t) ADDL(U32(q.K), t) ADDL(u, t) ROLL(Imm(30), b) a, b, c, d, e = t, a, b, c, d }

Slide 143

Slide 143 text

SHA-1 Conditionals u := GP32() if r < 16 { MOVL(m.Offset(4*r), u) BSWAPL(u) } else { MOVL(W(r-3), u) XORL(W(r-8), u) XORL(W(r-14), u) XORL(W(r-16), u) ROLL(U8(1), u) }

Slide 144

Slide 144 text

SHA-1 Conditionals u := GP32() if r < 16 { ‹ Early rounds MOVL(m.Offset(4*r), u) ‹ Read from memory BSWAPL(u) } else { MOVL(W(r-3), u) XORL(W(r-8), u) XORL(W(r-14), u) XORL(W(r-16), u) ROLL(U8(1), u) }

Slide 145

Slide 145 text

SHA-1 Conditionals u := GP32() if r < 16 { MOVL(m.Offset(4*r), u) BSWAPL(u) } else { MOVL(W(r-3), u) ‹ Formula in later rounds XORL(W(r-8), u) XORL(W(r-14), u) XORL(W(r-16), u) ROLL(U8(1), u) }

Slide 146

Slide 146 text

116 avo lines to 1507 assembly lines

Slide 147

Slide 147 text

Future

Slide 148

Slide 148 text

Use avo!

Slide 149

Slide 149 text

Real avo Examples • Farmhash32/64 • BLS12-381 Curve • Bitmap Indexes • Bloom Index • Marvin32 • Sip13 • SPECK • Chaskey MAC • SHA-1 • FNV-1a • Vector Dot Product With thanks to Damian Gryski, Marko Kevac and Julian Meyer (Phore Project).