善用 Go 語言效能測試工具來提升執行效率

Slide 1

Slide 1 text

善⽤ Go 語⾔效能測試⼯具來提升執⾏效率 Bo-Yi Wu 2020/10/24

Slide 2

Slide 2 text

About me • Software Engineer in Mediatek • Member of Drone CI/CD Platform • Member of Gitea Platform • Member of Gin Golang Framework • Maintain Some GitHub Actions Plugins.

Slide 3

Slide 3 text

使⽤ AI 來偵測異常減少晶片設計發⽣的問題

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

Data 資料量 • 1 ~ 2 萬個檔案 • 每個檔案 1000 萬⾏以上 • 總共 8 億 ~ 1x 億個資料點 (單⼀製程)

Slide 6

Slide 6 text

效能結果 (原先使⽤ 400 台機器) 1BSTFS ,# .# . (# .)8 PME VOJUTFDPOE ఏঋYYഒ DPSF( লԼ)8㑌࣍୆(3BN7.

Slide 7

Slide 7 text

Python to Golang ఏঋ଎౓۰౸ළᰍ

Slide 8

Slide 8 text

效能結果 (原先使⽤ 400 台機器) 1BSTFS ,# .# . (# .)8 PME VOJUTFDPOE ఏঋYYഒ DPSF( লԼ)8㑌࣍୆(3BN7.

Slide 9

Slide 9 text

Write Parser in Golang ᅷยိݯIUUQT[IVBOMBO[IJIVDPNQ

Slide 10

Slide 10 text

%BUB㚎༰֨ࣜ

Slide 11

Slide 11 text

⾃⾏研發 Parser 原因 • 提⾼部⾨產能 (太多單位有分析需求) • 減少 Computing Farm 使⽤ (省錢) • 整合 AI Model 花費時間過久 (inference time 不到⼀秒啊) • ⽬前在市⾯上找不到任何⼀套開源 Parser?

Slide 12

Slide 12 text

效能結果 (原先使⽤ 400 台機器) 1BSTFS ,# .# . (# .)8 PME VOJUTFDPOE ఏঋYYഒ DPSF( লԼ)8㑌࣍୆(3BN7.

Slide 13

Slide 13 text

第⼀個版本 var group = regexp.MustCompile(`^(\s*)(\S+)\s*$(.*?)$\s*{\s*$`) var groupDone = regexp.MustCompile(`^\s*}\s*$`) /// Simple attribute compile. /// key : value; var simpleAttribute = regexp.MustCompile(`^(\s*)(\S+)\s*:\s*(.+)\s*;.*$`) var specialSimpleAttribute = regexp.MustCompile(`^(\s*)(\S+)\s*:\s*(.+)\s*$`) /// Complex attribute compile. /// key (valueList); var complexAttribute = regexp.MustCompile(`^(\s*)(\S+)\s*($.+$)\s*;.*$`) var specialComplexAttribute = regexp.MustCompile(`^(\s*)(\S+)\s*($.+$)\s*$`) Ꮘೳළᰍ

Slide 14

Slide 14 text

效能評估⽅式

Slide 15

Slide 15 text

記錄程式執⾏時間 $ time your_program SFBMNT VTFSNT TZTNT

Slide 16

Slide 16 text

埋入程式碼 func main() { start := time.Now() fmt.Println(foobar(1000, 10)) elapsed := time.Since(start) log.Printf("foobar took %s", elapsed) }

Slide 17

Slide 17 text

Go 內建性能⼯具

Slide 18

Slide 18 text

Go benchmark $ go test -v -bench=foobar -run=none

Slide 19

Slide 19 text

var fixtures = []struct { path string }{ {"cells.lib"}, {"cells_timing.lib"}, {"sample2_Late.lib"}, {"small.lib"}, } func withFixtures(b *testing.B, fn func(b *testing.B, input []byte)) { for _, fix ::= range fixtures { data, err ::= ioutil.ReadFile(filepath.Join(".../testdata", fix.path)) if err !!= nil { b.Fatal(err) } b.Run(fix.path, func(b *testing.B) { b.ReportAllocs() b.SetBytes(int64(len(data))) b.ResetTimer() fn(b, data) }) } } 3VO1BSBMMFMฏߦ႔ཧ 3FQPSU"MMPDTهԱᱪ࢖༻ྔ 4FU#ZUFTᄸ࣍॥؀࢖༻ଟগهԱᱪ 3FTFU5JNFSॏஔ5JNFSܭࢉ

Slide 20

Slide 20 text

func BenchmarkWhitespaceArrayInlined(b *testing.B) { withFixtures(b, benchmarkWhitespaceArrayInlined) } func BenchmarkWhitespaceArrayLoop(b *testing.B) { withFixtures(b, benchmarkWhitespaceArray) } func BenchmarkWhitespaceSwitch(b *testing.B) { withFixtures(b, benchmarkWhitespaceSwitch) } func BenchmarkWhitespaceIf(b *testing.B) { withFixtures(b, benchmarkWhitespaceIf) } func BenchmarkWhitespaceIfInlined(b *testing.B) { withFixtures(b, benchmarkWhitespaceIfInlined) } ଟ䈕Ҋෆಉํ๏Ꮘೳଌࢼ

Slide 21

Slide 21 text

No content

Slide 22

Slide 22 text

RunParallel (平⾏處理) 根據系統 CPU 數量加速處理 HPUFTUDQV SVO?CFODICFODINBSL@UFTUHP

Slide 23

Slide 23 text

n ::= uint64(0) numProcs ::= b.parallelism * runtime.GOMAXPROCS(0) var wg sync.WaitGroup wg.Add(numProcs) for p ::= 0; p < numProcs; p+++ { go func() { defer wg.Done() pb ::= &PB{ globalN: &n, grain: grain, bN: uint64(b.N), } body(pb) }() } wg.Wait() CQBSBMMFMJTN༬ઃҝ ༬ઃҝ$16ݸᏐ IUUQTQLHHPEFWUFTUJOH#3VO1BSBMMFM

Slide 24

Slide 24 text

No content

Slide 25

Slide 25 text

ུաෆඞཁతܭࢉ C4UPQ5JNFSC4UBSU5JNFS init() b.ResetTimer() for i::=0; i

Slide 26

Slide 26 text

測試多久+測試次數 CFODIUJNFIT CFODIUJNFY

Slide 27

Slide 27 text

產⽣多組樣本數據 DPVOU

Slide 28

Slide 28 text

ᒬຊᏐᎦ Ҡআ༗ྀٙతࢿ㘤

Slide 29

Slide 29 text

Benchmark 比較 golang.org/x/perf/cmd/benchstat

Slide 30

Slide 30 text

比較兩種不同版本差異 HPUFTUWDPVOUCFODIYYYYYSVOOPOFMFYFScUFFPMEUYU HPUFTUWDPVOUCFODIYYYYYSVOOPOFMFYFScUFFOFXUYU

Slide 31

Slide 31 text

ᄸҰଌࢼ܈ᱪ benchstat old.txt

Slide 32

Slide 32 text

func (m *Metrics) computeStats() { /// Discard outliers. values ::= stats.Sample{Xs: m.Values} q1, q3 ::= values.Percentile(0.25), values.Percentile(0.75) lo, hi ::= q1-1.5*(q3-q1), q3+1.5*(q3-q1) for _, value ::= range m.Values { if lo <<= value &&& value <<= hi { m.RValues = append(m.RValues, value) } } /// Compute statistics of remaining data. m.Min, m.Max = stats.Bounds(m.RValues) m.Mean = stats.Mean(m.RValues) } ഒ࢛෼Ґڑํࣜ ࣗಈҠআ༗ྀٙతᏐᎦ golang.org/x/perf/cmd/benchstat

Slide 33

Slide 33 text

Ꮘೳલޙൺֱ benchstat old.txt new.txt

Slide 34

Slide 34 text

如何找出效能瓶頸 ࢖༻QQSPG෼ੳ$16҃هԱᱪ࢖༻ྔ DQVQSPpMFDQVPVUNFNQSPpMFNFNPVU

Slide 35

Slide 35 text

HPUFTUCFODIYYYYYDPVOUSVO?DQVQSPpMFDQVPVUMFYFS HPUPPMQQSPGDQVPVU

Slide 36

Slide 36 text

؃ᄸҰവࣔଌࢼᏈೳ݁Ռ MJTUGVOD

Slide 37

Slide 37 text

ݺڣ࣍Ꮠ౷ܭ QFFLGVOD

Slide 38

Slide 38 text

QQSPG㗞ੜ֤छෆಉใࠂ

Slide 39

Slide 39 text

⽤ Web 直接看比較快 HPUPPMQQSPGIUUQDQVPVU

Slide 40

Slide 40 text

No content

Slide 41

Slide 41 text

第⼀版 Parser 效能 3FHFY1BSTFSWT-FYFS1BSTFS

Slide 42

Slide 42 text

ఏঋେ໿ഒᏈೳ

Slide 43

Slide 43 text

ఏঋେ໿ഒᏈೳ

Slide 44

Slide 44 text

效能結果 1BSTFS ,# .# . (# .)8 PME VOJUTFDPOE ఏঋ઀ۙഒ DPSF(

Slide 45

Slide 45 text

/FYU5PLFOՖඅ࣌ؒ࠷ٱ

Slide 46

Slide 46 text

No content

Slide 47

Slide 47 text

OFX5PLFOՖඅ኷ٱ

Slide 48

Slide 48 text

func (l *Lexer) NextToken() { LOOP: switch l.char { case '{': l.token.Type = token.LeftBrace l.token.Literal = []byte{l.char} l.token.Line = l.line l.token.Start = l.position l.token.End = l.position + 1 case ' ', '\t', '\r', '\n': if l.char === '\n' { l.line+++ } l.readChar() goto LOOP

Slide 49

Slide 49 text

4LJQ8IJUFTQBDF func (l *Lexer) NextToken() { l.skipWhitespace() switch l.char { case '{': l.token.Type = token.LeftBrace l.token.Literal = []byte{l.char} l.token.Line = l.line l.token.Start = l.position l.token.End = l.position + 1

Slide 50

Slide 50 text

4LJQ8IJUFTQBDFमਖ਼݁Ռ

Slide 51

Slide 51 text

values ( \ "0, 0.00014625, 0.0003375, 0.00043875, 0.000545625", \ "0, 0.065, 0.15, 0.195, 0.2425, 0.295, 0.35, 0.42" \ ); values ( "0.1, 0.2, 0.3", \ "0.11, 0.21, 0.31", \ "0.12, 0.22, 0.32" ); index_2 ("0.02239") index_2 (TVDD, 0.633) statetable ( " D CP SE SI " , "Q" ) ႔ཧଟछࣈ۲ ૬Ճ

Slide 52

Slide 52 text

func (p *Parser) parseMultipleValue() ast.Value { val ::= ast.Literal{Type: ast.LiteralType} values ::= []string{} for !p.currentTokenTypeIs(token.EOF) { if p.currentTokenTypeIs(token.Escape) { p.nextToken() } else if p.currentTokenTypeIs(token.String) { values = append(values, p.parseString()) p.nextToken() } else if p.currentTokenTypeIs(token.Comma) { values = append(values, p.parseString()) p.nextToken() } else if p.currentTokenTypeIs(token.RightParentheses) { break } } val.Value = strings.Join(values, "") return val }

Slide 53

Slide 53 text

func (p *Parser) parseMultipleValue() ast.Value { val ::= ast.Literal{Type: ast.LiteralType} buf.Reset() for !p.currentTokenTypeIs(token.EOF) { if p.currentTokenTypeIs(token.Escape) { p.nextToken() } else if p.currentTokenTypeIs(token.String) { buf.WriteString(p.parseString()) p.nextToken() } else if p.currentTokenTypeIs(token.Comma) { buf.WriteString(p.parseString()) p.nextToken() } else if p.currentTokenTypeIs(token.RightParentheses) { break } } val.Value = buf.String() return val } ༻CZUFT#V⒎FSऔ୅

Slide 54

Slide 54 text

ࣈ۲႔ཧमਖ਼݁Ռ

Slide 55

Slide 55 text

Slide 56

Slide 56 text

case ast.PropertyParentheses: var val ast.Value if !p.peekTokenTypeIs(token.RightParentheses) { val = p.parseMultipleValue() prop.Value = val } else { /// single value /// index_2 ("0.02239") val = p.parseValue() prop.Value = val } ࣈ۲႔ཧᬓाௐ੔

Slide 57

Slide 57 text

ࣈ۲႔ཧᬓाௐ੔

Slide 58

Slide 58 text

case ',': l.token.Type = token.Comma l.token.Literal = []byte{l.char} l.token.Line = l.line l.token.Start = l.position l.token.End = l.position + 1 case '"': l.token.Type = token.String l.token.Literal = l.readString() l.token.Line = l.line l.token.Start = l.position l.token.End = l.position + 1 ݮগهԱᱪ࢖༻ྔ

Slide 59

Slide 59 text

case ',': l.token.Type = token.Comma l.token.Start = l.position l.token.End = l.position + 1 l.token.Literal = l.Data[l.token.Start:l.token.End] l.token.Line = l.line case '"': l.token.Type = token.String l.token.Literal = l.readString() l.token.Line = l.line l.token.Start = l.position l.token.End = l.position + 1 ݮগهԱᱪ࢖༻ྔ

Slide 60

Slide 60 text

ݮগهԱᱪ࢖༻ྔ

Slide 61

Slide 61 text

No content

Slide 62

Slide 62 text

ᩇऔძҾᥒؒతࣈ۲

Slide 63

Slide 63 text

func (l *Lexer) readString() []byte { l.position+++ data ::= l.Data[l.position:] _, length ::= findStringLen(data) l.position += length l.readPosition = l.position + 1 return data[:length] } ༏ԽSFBETUJOH

Slide 64

Slide 64 text

func findStringLen(data []byte) (isValid bool, length int) { for { idx ::= bytes.IndexByte(data, '"') if idx === -1 { return false, len(data) } if idx === 0 ||| (idx > 0 &&& data[idx-1] !!= '\\') { return true, length + idx } length += idx + 1 data = data[idx+1:] } } ፙ౸ୈҰݸ݁ଋූᥒ

Slide 65

Slide 65 text

༏Խ݁Ռ

Slide 66

Slide 66 text

༏ԽSFBE*EFOUJpFS

Slide 67

Slide 67 text

/// check previous token /// example: /// voltage_map ( TVDD, 0.633); /// voltage_map (TVDD, 0.633); if len(l.token.Literal) > 0 &&& l.token.Literal[0] === '(' { l.token.Type = token.String l.token.Literal = l.readMapString() l.token.Line = l.line l.token.Start = l.position l.token.End = l.position + 1 return } ࢿྉྔ኷େ ࢖༻CZUFT*OEFY#ZUF

Slide 68

Slide 68 text

༏Խ݁Ռ

Slide 69

Slide 69 text

मਖ਼4LJQ8IJUF4QBDF'VOD if whitespace[l.Data[l.position]] { if l.Data[l.position] === '\n' { l.line+++ } l.readChar() goto LOOP } var whitespace = [256]bool{ ' ': true, '\t': true, '\n': true, '\r': true, }

Slide 70

Slide 70 text

Ꮘೳଌࢼใࠂ

Slide 71

Slide 71 text

效能結果 (原先使⽤ 400 台機器) 1BSTFS ,# .# . (# .)8 PME VOJUTFDPOE ఏঋYYഒ DPSF( লԼ)8㑌࣍୆(3BN7.