Benchmarking - First Steps

An Introduction to Benchmarking in Go

Go Ireland is a Meetup open to anyone from all experience and technology backgrounds looking to discuss and dive deeper into the Go language.

The main objective of this Meetup is to share knowledge, demonstrate Go use cases, and connect with others in similar areas of interest.

Transcript

1. Benchmarking: First Steps Benchmarking: First Steps Go Ireland Meetup Go Ireland Meetup 6 February 2025 6 February 2025 Jakub Jarosz Jakub Jarosz
2. Agenda Agenda Testing vs Benchmarking Testing vs Benchmarking Writing benchmarks Writing benchmarks Running benchmarks Running benchmarks Interpreting results Interpreting results Gaming Time! Gaming Time! 2 2
3. Design Principles Design Principles Make it correct Make it correct Make it readable Make it readable Make it easy to understand Make it easy to understand ... ... Make it Make it fast fast (enough) (enough) 3 3
4. Go Optimization Principle Go Optimization Principle Optimizing Golang code for performance is almost certainly a waste of your time, for several Optimizing Golang code for performance is almost certainly a waste of your time, for several reasons: reasons: Performance doesn't matter Performance doesn't matter Go is fast Go is fast Readability beats speed Readability beats speed J. Arundel J. Arundel (https://bitfieldconsulting.com/posts/slower) (https://bitfieldconsulting.com/posts/slower) 4 4
5. Benchmarking Benchmarking Speed ? Speed ? Petrol ? Petrol ? Loops ? Loops ? Why ? Why ? 5 5
6. Testing vs Benchmarking Testing vs Benchmarking Test Test func TestMyFunc(t *testing.T) { func TestMyFunc(t *testing.T) { // test logic // test logic } } Benchmark Benchmark func BenchmarkMyFunc(b *testing.B) { func BenchmarkMyFunc(b *testing.B) { for i := 0; i < b.N; i++ { for i := 0; i < b.N; i++ { MyFunc() MyFunc() } } } } 6 6
7. Benchmarks Benchmarks var tt = []string{"uno", "due", "tre"} var tt = []string{"uno", "due", "tre"} func BenchmarkBar(b *testing.B) { func BenchmarkBar(b *testing.B) { for i := 0; i < b.N; i++ { for i := 0; i < b.N; i++ { for _, tc := range tt { for _, tc := range tt { Bar(tc) Bar(tc) } } } } } } 7 7
8. Running Benchmarks Running Benchmarks bench bench go test -run none -bench . go test -run none -bench . BenchmarkFoo-8 BenchmarkFoo-8 19146404 19146404 62.46 ns/op 62.46 ns/op 8 8
9. Running Benchmarks Running Benchmarks benchmem benchmem go test -run none -bench . -benchmem go test -run none -bench . -benchmem BenchmarkFoo-8 BenchmarkFoo-8 18934608 18934608 63.82 ns/op 63.82 ns/op 80 B/op 80 B/op 3 allocs/op 3 allocs/op 9 9
10. Running Benchmarks Running Benchmarks cpu cpu go test -run none -bench . -cpu 2,4,6,8 go test -run none -bench . -cpu 2,4,6,8 BenchmarkIsIsogram-2 BenchmarkIsIsogram-2 412042 412042 2895 ns/op 2895 ns/op BenchmarkIsIsogram-4 BenchmarkIsIsogram-4 406560 406560 2905 ns/op 2905 ns/op BenchmarkIsIsogram-6 BenchmarkIsIsogram-6 402758 402758 2903 ns/op 2903 ns/op BenchmarkIsIsogram-8 BenchmarkIsIsogram-8 402938 402938 2904 ns/op 2904 ns/op 10 10
11. Running Benchmarks Running Benchmarks benchtime benchtime go test -run none -bench . -benchtime 2s -benchmem -cpu 2,4,6,8 go test -run none -bench . -benchtime 2s -benchmem -cpu 2,4,6,8 BenchmarkIsIsogram-2 BenchmarkIsIsogram-2 833965 833965 2899 ns/op 2899 ns/op 1426 B/op 1426 B/op 16 allocs/op 16 allocs/op BenchmarkIsIsogram-4 BenchmarkIsIsogram-4 813745 813745 2894 ns/op 2894 ns/op 1426 B/op 1426 B/op 16 allocs/op 16 allocs/op BenchmarkIsIsogram-6 BenchmarkIsIsogram-6 806353 806353 2925 ns/op 2925 ns/op 1426 B/op 1426 B/op 16 allocs/op 16 allocs/op BenchmarkIsIsogram-8 BenchmarkIsIsogram-8 804255 804255 2928 ns/op 2928 ns/op 1426 B/op 1426 B/op 16 allocs/op 16 allocs/op 11 11
12. ShareWith... ShareWith... 12 12
13. ShareWith v1 ShareWith v1 func ShareWith(name string) string { func ShareWith(name string) string { if name == "" { if name == "" { name = "you" name = "you" } } return "One for " + name + ", one for me." return "One for " + name + ", one for me." } } 13 13
14. ShareWith v2 ShareWith v2 func ShareWith(name string) string { func ShareWith(name string) string { if name == "" { if name == "" { name = "you" name = "you" } } return fmt.Sprintf("One for %s, one for me.", name) return fmt.Sprintf("One for %s, one for me.", name) } } 14 14
15. ShareWith Benchmark ShareWith Benchmark func BenchmarkShareWith(b *testing.B) { func BenchmarkShareWith(b *testing.B) { for i := 0; i < b.N; i++ { for i := 0; i < b.N; i++ { for _, tc := range tt { for _, tc := range tt { hello.ShareWith(tc.input) hello.ShareWith(tc.input) } } } } } } 15 15
16. ShareWith Benchmark ShareWith Benchmark run benchmark run benchmark go test -run none -bench . -benchmem go test -run none -bench . -benchmem report report BenchmarkShareWith-8 BenchmarkShareWith-8 18975690 18975690 62.86 ns/op 62.86 ns/op 80 B/op 80 B/op 3 allocs/op 3 allocs/op Demo Demo 16 16
17. Let's play: Isogram Let's play: Isogram An isogram (also known as a "non-pattern word") is a word or phrase without a repeating An isogram (also known as a "non-pattern word") is a word or phrase without a repeating letter, however spaces and hyphens are allowed to appear multiple times. letter, however spaces and hyphens are allowed to appear multiple times. 17 17
18. Isogram v1 Isogram v1 func IsIsogram(word string) bool { func IsIsogram(word string) bool { letters := make(map[rune]bool) letters := make(map[rune]bool) for _, l := range word { for _, l := range word { letter := unicode.ToLower(l) letter := unicode.ToLower(l) if letter == '-' || unicode.IsSpace(letter) { if letter == '-' || unicode.IsSpace(letter) { continue continue } } if letters[letter] { if letters[letter] { return false return false } } letters[letter] = true letters[letter] = true } } return true return true } } 18 18
19. Isogram v2 Isogram v2 func IsIsogram(word string) bool { func IsIsogram(word string) bool { seen := make(map[rune]bool) seen := make(map[rune]bool) word = strings.ToLower(word) word = strings.ToLower(word) for _, l := range word { for _, l := range word { if !unicode.IsLetter(l) { if !unicode.IsLetter(l) { continue continue } } _, ok := seen[l] _, ok := seen[l] if ok { if ok { return false return false } } seen[l] = true seen[l] = true } } return true return true } } 19 19
20. Isogram v3 Isogram v3 func IsIsogram(word string) bool { func IsIsogram(word string) bool { word = strings.ToUpper(word) word = strings.ToUpper(word) for i, c := range word { for i, c := range word { if c == ' ' || c == '-' { if c == ' ' || c == '-' { continue continue } } for j := i + 1; j < len(word); j++ { for j := i + 1; j < len(word); j++ { if word[i] == word[j] { if word[i] == word[j] { return false return false } } } } } } return true return true } } 20 20
21. Isogram v4 Isogram v4 func IsIsogram(word string) bool { func IsIsogram(word string) bool { word = strings.ToLower(word) word = strings.ToLower(word) for i := 0; i < len(word); i++ { for i := 0; i < len(word); i++ { for j := i + 1; j < len(word); j++ { for j := i + 1; j < len(word); j++ { if word[i] == ' ' || word[j] == '-' { if word[i] == ' ' || word[j] == '-' { continue continue } } if word[i] == word[j] { if word[i] == word[j] { return false return false } } } } } } return true return true } } DEMO DEMO 21 21
22. Let's play: Scrabble Let's play: Scrabble 22 22
23. Scrabble v1 Scrabble v1 var points = [26]int{ var points = [26]int{ 1, 3, 3, 2, 1, 4, 2, 4, 1, 8, 5, 1, 3, 1, 3, 3, 2, 1, 4, 2, 4, 1, 8, 5, 1, 3, 1, 1, 3, 10, 1, 1, 1, 1, 4, 4, 8, 4, 10, 1, 1, 3, 10, 1, 1, 1, 1, 4, 4, 8, 4, 10, } } func Score(word string) int { func Score(word string) int { sum := 0 sum := 0 workString := strings.ToUpper(word) workString := strings.ToUpper(word) for _, set := range workString { for _, set := range workString { sum += points[set-65] sum += points[set-65] } } return sum return sum } } 23 23
24. Scrabble v2 Scrabble v2 func Score(word string) int { func Score(word string) int { letterScores := map[rune]int{ letterScores := map[rune]int{ 'A': 1, 'E': 1, 'I': 1, 'O': 1, 'U': 1, 'L': 1, 'N': 1, 'R': 1, 'S': 1, 'T': 1, 'A': 1, 'E': 1, 'I': 1, 'O': 1, 'U': 1, 'L': 1, 'N': 1, 'R': 1, 'S': 1, 'T': 1, 'D': 2, 'G': 2, 'D': 2, 'G': 2, 'B': 3, 'C': 3, 'M': 3, 'P': 3, 'B': 3, 'C': 3, 'M': 3, 'P': 3, 'F': 4, 'H': 4, 'V': 4, 'W': 4, 'Y': 4, 'F': 4, 'H': 4, 'V': 4, 'W': 4, 'Y': 4, 'K': 5, 'K': 5, 'J': 8, 'X': 8, 'J': 8, 'X': 8, 'Q': 10, 'Z': 10, 'Q': 10, 'Z': 10, } } score := 0 score := 0 for _, letter := range word { for _, letter := range word { score += letterScores[rune(unicode.ToUpper(letter))] score += letterScores[rune(unicode.ToUpper(letter))] } } return score return score } } 24 24
25. Scrabble v3 Scrabble v3 var letterScores = map[rune]int{ var letterScores = map[rune]int{ 'A': 1, 'E': 1, 'I': 1, 'O': 1, 'U': 1, 'L': 1, 'N': 1, 'R': 1, 'S': 1, 'T': 1, 'A': 1, 'E': 1, 'I': 1, 'O': 1, 'U': 1, 'L': 1, 'N': 1, 'R': 1, 'S': 1, 'T': 1, 'D': 2, 'G': 2, 'D': 2, 'G': 2, 'B': 3, 'C': 3, 'M': 3, 'P': 3, 'B': 3, 'C': 3, 'M': 3, 'P': 3, 'F': 4, 'H': 4, 'V': 4, 'W': 4, 'Y': 4, 'F': 4, 'H': 4, 'V': 4, 'W': 4, 'Y': 4, 'K': 5, 'K': 5, 'J': 8, 'X': 8, 'J': 8, 'X': 8, 'Q': 10, 'Z': 10, 'Q': 10, 'Z': 10, } } func Score(word string) int { func Score(word string) int { score := 0 score := 0 for _, letter := range word { for _, letter := range word { score += letterScores[rune(unicode.ToUpper(letter))] score += letterScores[rune(unicode.ToUpper(letter))] } } return score return score } } 25 25
26. Comparing Benchmarks - Benchstat Comparing Benchmarks - Benchstat 26 26
27. Comparing Benchmarks - Benchstat Comparing Benchmarks - Benchstat Running benchmarks Running benchmarks go test -run none -bench . -benchmem -count=10 > ../v1.txt go test -run none -bench . -benchmem -count=10 > ../v1.txt go test -run none -bench . -benchmem -count=10 > ../v2.txt go test -run none -bench . -benchmem -count=10 > ../v2.txt Running Running benchstat benchstat benchstat v1.txt v2.txt benchstat v1.txt v2.txt pkg: github.com/qba73/scrabble pkg: github.com/qba73/scrabble │ v1.txt │ v2.txt │ │ v1.txt │ v2.txt │ │ sec/op │ sec/op vs base │ │ sec/op │ sec/op vs base │ Score-8 3029.5n ± 1% 298.4n ± 1% -90.15% (p=0.000 n=10) Score-8 3029.5n ± 1% 298.4n ± 1% -90.15% (p=0.000 n=10) │ v1.txt │ v2.txt │ │ v1.txt │ v2.txt │ │ allocs/op │ allocs/op vs base │ │ allocs/op │ allocs/op vs base │ Score-8 63.000 ± 0% 9.000 ± 0% -85.71% (p=0.000 n=10) Score-8 63.000 ± 0% 9.000 ± 0% -85.71% (p=0.000 n=10) Benchstat Benchstat (https://pkg.go.dev/golang.org/x/perf/cmd/benchstat) (https://pkg.go.dev/golang.org/x/perf/cmd/benchstat) 27 27
28. Preventing Compiler Optimizations Preventing Compiler Optimizations var Res bool var Res bool func BenchmarkIsIsogram(b *testing.B) { func BenchmarkIsIsogram(b *testing.B) { var got bool var got bool for i := 0; i < b.N; i++ { for i := 0; i < b.N; i++ { for _, tc := range tt { for _, tc := range tt { got = isogram.IsIsogram(tc.input) got = isogram.IsIsogram(tc.input) } } } } Res = got Res = got } } 28 28
29. There is much more... There is much more... Quiet environment Quiet environment Managing benchmark loop Managing benchmark loop Managing benchmark timer Managing benchmark timer Improving benchmark accuracy Improving benchmark accuracy 29 29
30. Upcoming book Upcoming book Get the book when it's out Get the book when it's out (https://jarosz.dev/article/performance-tuning-and-benchmarking/) (https://jarosz.dev/article/performance-tuning-and-benchmarking/) . . 30 30
31. Thank you Thank you Jakub Jarosz Jakub Jarosz jakub@jarosz.dev jakub@jarosz.dev (mailto:jakub@jarosz.dev) (mailto:jakub@jarosz.dev) https://jarosz.dev https://jarosz.dev (https://jarosz.dev) (https://jarosz.dev) @qba73 @qba73 (http://twitter.com/qba73) (http://twitter.com/qba73)