Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
OSS Performance Tuning Tips
Search
orisano
October 28, 2019
Programming
8
5.4k
OSS Performance Tuning Tips
orisano
October 28, 2019
Tweet
Share
More Decks by orisano
See All by orisano
Docker-Compose & BuildKit
orisano
4
890
Container Build Talk
orisano
3
2.4k
dockerignore talk
orisano
2
6.8k
Better docker image+
orisano
6
5.7k
Socket.IO Introduction
orisano
0
2.9k
Profiling Go Application
orisano
11
7.5k
Multi-stage Builds Patterns & Practice
orisano
6
4.9k
better docker image
orisano
22
29k
the world of characters
orisano
8
1.3k
Other Decks in Programming
See All in Programming
Javaの現状2024夏 / Java current status 2024 summer
kishida
4
1.4k
君たちはどうコードをレビューする (される) か / 大吉祥寺.pm
utgwkk
15
8.5k
【Go言語】golangci-lintの使い方
tomo1227
0
270
Namespace on read
tagomoris
2
370
SDCon2024: Enabling DevOps and Team Topologies thru architecture: architecting for fast flow
cer
PRO
0
780
「2024年版 Kotlin サーバーサイドプログラミング実践開発」の補講 〜O/Rマッパー編〜
n_takehata
2
260
OpenAI/Gemini APIを使って EPUBを翻訳するCLIツールをつくってみた
tomiyan
0
790
Terraformテスト入門
msato
0
520
MIERUNE BBQにおけるユーザー中心設計()
mierune
PRO
1
110
ぼっちを避けて楽しむためのアノテコノテ / Various Tips and Tricks to Avoid Loneliness and Have Fun
nrslib
3
1.7k
【Go言語】ジェネリクス
tomo1227
0
170
初心者がおさえておきたいAWS CDKのベストプラクティス 2024
konokenj
15
7.3k
Featured
See All Featured
A Modern Web Designer's Workflow
chriscoyier
689
190k
jQuery: Nuts, Bolts and Bling
dougneiner
61
7.4k
Product Roadmaps are Hard
iamctodd
PRO
48
10k
A designer walks into a library…
pauljervisheath
201
24k
Bootstrapping a Software Product
garrettdimon
PRO
304
110k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
90
47k
The Invisible Customer
myddelton
117
13k
Practical Orchestrator
shlominoach
185
10k
Optimising Largest Contentful Paint
csswizardry
18
2.6k
Building Better People: How to give real-time feedback that sticks.
wjessup
357
18k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
228
16k
Dealing with People You Can't Stand - Big Design 2015
cassininazir
360
22k
Transcript
OSS Performance Tuning Tips #gocon #gocon_hall GoCon 2019 Autumn @orisano
త ̎ͭͷ͜ͱΛڞ༗͍ͨ͠
νϡʔχϯάΛͲ͏ਐΊΔͷ͔ Ҿ͖ग़͠Λ૿͢
νϡʔχϯάΛͲ͏ਐΊΔͷ͔ Ҿ͖ग़͠Λ૿͢
Ͳ͏ਐΊ͍ͯΔͷ͔
0. ෆຬͷϋʔυϧΛԼ͛Δ
ෆຬ͕ վળͷ͖͔͚ͬʹͳΓ·͢
ීஈ͍ͬͯΔ ϥΠϒϥϦɺιϑτΣΞ͕ ͍ͷͩͱࢥͬͯΈΔ
͖͔͚ͬͱ ϞνϕʔγϣϯΛ ࣗΒ࡞Γग़͠·͠ΐ͏
1. ܭଌͷϋʔυϧΛԼ͛Δ
͍ݪҼΛ ؾܰʹ ௐΒΕΔೳྗΛʹ͚ͭΔ
ख͕͔͔ؒΔߦҝΛ ਓؒ͋·Γ͠ͳ͍
ͨ͘͞Μܦݧ͢Δ
ؾܰʹ ௐࠪͰ͖ΔΑ͏ͳΈΛ ೖΕΔ
͜ΕOSSΛެ։͢Δଆͷ ڥมͰࢦఆͨ͠Γ σϑΥϧτͰprofile͕औΕΔ ΈΛఏڙ͢Δͱ վળ͍͢͠/͞Ε͍͢
ʹૺ۰ͨ͠ਓ͕ ݪҼௐ͍ࠪ͢͠ͱخ͍͠
2. ՕॴΛಛఆ͢Δ
͍ͱײͯ͡ Օॴ͕ؔϨϕϧͰ Θ͔Βͳ͍ͷ cliΛ࣮ߦ͍ͯ͠Δͱ͖
mainʹ github.com/pkg/profile ΛՃͯ͠ΈΔ
None
͜Ε͚ͩ
cliͷ࣮ߦ͕1ʹ 1࣌ؒҎ্͔͔Δ߹ͳͲ net/http/pprofͷ΄͏͕ ྑ͍߹
profileΛݟͯ ؔϨϕϧͰಛఆ͢Δ
pprofͷweb൛ʹ׳ΕΔ ͪΌΜͱҙຯ͕Θ͔Δ
͍ΖΜͳݟํͰݟͯΈΔ top, graph, flame-graph source, disasm
3. BenchmarkΛॻ͘
͢Ͱʹॻ͍ͯ͋Δ߹ ϥοΩʔ ͘ͳ͍ͬͯͳ͍߹ ͘ͳΔέʔεΛՃ͢Δ
BenchmarkΛॻ࣌͘ ࠷దԽͰফ͍͑ͯͳ͍͔֬ೝ
BenchmarkΛॻ͘ͱ ؆୯ʹprofile͕औΕΔ
go test -cpuprofile go test -memprofile Λ֮͑Α͏ (go help testflagΛಡ͏)
4. ղܾࡦΛߟ͑Δ
Ͳ͏ղܾ͢Δ͔ʹ͍ͭͯ “Ҿ͖ग़͠Λ૿͢”Ͱ
OSSͷ ύϑΥʔϚϯενϡʔχϯά Ϛʔδ͞Εͯྃ
Ͳ͏ΕϚʔδ͞ΕΔ ύονʹͳΔ͔
Ϛʔδ͢Δͱ͍͏͜ͱ ૬ख͕ϝϯςφϯε͢Δ ͱ͍͏͜ͱ
มߋ͕গͳ͍ ૬खͷίετΛ૿͞ͳ͍
ഁյతมߋ͕ͳ͍ ͘ΘΕ͍ͯΔͷ΄Ͳ ॏཁͳͱ͜Ζ
ҟৗͳίετ͕ͳ͍ มߋ, อक͕͘͠ͳΔ࣮ (ΞηϯϒϥΛͬͨΓ)
ґଘؔΛՃ͢Δ߹ ৻ॏʹબ͢Δ ΞΫςΟϒʹϝϯςφϯε͞ Ε͍ͯΔͷ͔ ຊʹඞཁ͔?
͘͢͝ύϑΥʔϚϯε͕ վળ͢Δ ܶతͳվળड͚ೖΕΒΕΔ
ςετ͕ॻ͍ͯ͋Δ ͢Ͱʹॻ͍ͯ͋Δ߹डཧ ͞Ε͍͢
5. ࢼߦࡨޡ͢Δ
·ͣॳظঢ়ଶͰ ेͳճ timeoutʹͳΒͳ͍Α͏ʹ Benchmark݁ՌΛऔ͓ͬͯ͘
go test -bench . -count=10 -timeout=30000s | tee old.txt
ύονΛૹΔͱ͖ʹ benchstatͷ݁ՌΛૹΔͷͰ ϕʔεϥΠϯेͳճͰ
timeout σϑΥϧτͩͱ10mͰ ҙ֎ͱΦʔόʔ͢Δ
ղܾࡦΛࢥ͍͍ͭͨΒ ·ͣ1Benchmark
ے͕ྑͦ͞͏ͳΒ ेͳճBenchmark
ଟ͘ࢼߦࡨޡ͍ͯ͠Δͱ profileͱsourceͷ͕ؔ Θ͔Βͳ͘ͳΔ
pprofʹ sourceΛݟΔػೳ͕͋Δ͕ อ͍࣋ͯ͠ΔΘ͚Ͱͳ͍ pathͷΈ
git add . git commit -m “$2” REV=$(git rev-parse HEAD)
go test -bench $1 -benchmem -cpuprofile cpu.${REV}.pb.gz -memprofile mem.$ {REV}.pb.gz | tee ${REV}.txt rm ./${REV}.* && git reset HEAD^
sourcegitʹཧͯ͠Β͏ ͪΌΜͱඥ͚ΒΕΔ
ଟ͘ͷprofile͕͋Δͱ ୯ମͷޮՌ͕Θ͔Βͳ͍ ͜ͱ͕͋Δ
go tool pprof -diff_base Ͱprofileؒͷ͕ࠩݟΕΔ
6. ύονΛૹΔ
࠷ऴͷBenchmarkΛ ेͳճߦ͏
ઐ༻ͷΠϯελϯεͰܭଌ ͍͠߹ ۃྗ֎෦ϓϩηεΛఀࢭ͢Δ
νϡʔχϯάΛͲ͏ਐΊΔͷ͔ Ҿ͖ग़͠Λ૿͢
࣮ྫϕʔεͰ ͲͷΑ͏ʹղܾ͔ͨ͠ ࣗͳΒͲ͏ղܾ͢Δ͔ ߟ͑ͯΒ͍͍ͨ
1. ࣮ࡍʹ͛ͨύον 2. ͍ϥΠϒϥϦ͕ॻ͖͍ͨ
1. ࣮ࡍʹ͛ͨύον 2. ͍ϥΠϒϥϦ͕ॻ͖͍ͨ
src-d/go-git
pure-goͰ gitͷૢ࡞Λ͢ΔͨΊͷ ϥΠϒϥϦ
ύονΛૹͬͨഎܠ
aquasecurity/trivy Ͱॳ͍͕ͬͯͨ େ͖ͳrepositoryͷcloneʹ ҟৗͳ࣌ؒ(10min~)͕͔͔Δ ͕͋ͬͨ
ݪҼ gitͷIndexͷߏஙΛߦ͏Օॴ ͩͬͨ
Indexͷ෦දݱ(public)͕ sliceʹͳ͓ͬͯΓ nameͰҰҙʹ͢ΔͨΊʹ deleteͱappend͕ ϑΝΠϧճ࣮ߦ͞Ε͍ͯͨ
deletenameΛࢦఆͯ͠ আ͢ΔͷͰ sliceͷཁૉΛͯࠪ͢͢Δ
ॳ ෦දݱΛmapʹมߋ͢Ε deleteͷܭࢉྔΛݮΒͤΔ ͷͰͦ͏͠Α͏ͱࢥͬͨ
͔͠͠ publicͳϑΟʔϧυ internalͰͳ͍ύοέʔδ 4500 star͑ͷ໊
౸ఈड͚ೖΕΒΕͳ͍ มߋͩͱࢥ͍ఘΊͨ
ޙ
ιʔείʔυΛ ݟ͍ͯ͠Δͱ
໌ه͞Ε͍ͯΔ ϑΟʔϧυͷ༷ (sliceͷॱংอূ͠ͳ͍) େྔͷIndexΛߏங͢Δ λΠϛϯά(clone࣌)
ݟͨ݁͠Ռ ഁյతมߋແ͠Ͱ मਖ਼͢Δํ๏Λࢥ͍͍ͭͨ
ଟ͘ݺͼग़͞ΕΔՕॴ ͷΈʹ͓͍ͯ mapͰอ࣋͠return͢Δࡍʹ sliceʹม͢Δ
mapΛprivateͳؔͰ Ҿ͖ճ͢Α͏ʹ͢Δ͜ͱͰ ഁյతมߋΛճආͰ͖ͨ
None
ͦ͜Λղܾ͢Δͱ mallocgc͕ॏ͘ͳͬͨ
ϝϞϦ༻ྔͷݪҼ io.Copyʹ͋ΔΑ͏ͩͬͨ
ϑΝΠϧ͝ͱʹio.Copy͕ ݺΕ͍ͯͨ
େྔʹݺΕΔՄೳੑ͕͋Δ ՕॴͰio.CopyͰͳ͘ io.CopyBufferΛ͏ͱ ϝϞϦ༻ྔΛ੍ޚͰ͖Δ
͏bufferΛͲ͏ࢦఆ͢Δ͔ ֎෦͔Β༩͑ΒΕͳ͍ͱ ݁ہϝϞϦ༻ྔมΘΒͳ͍
privateͳϝιουͱ͍͑ ҾʹՃ͢Δͱ มߋൣғ͕େ͖͘ͳΔ
ղܾࡦͱͯ͠ globalʹsync.PoolΛ ஔ͘͜ͱʹͨ͠
None
https://github.com/src-d/ go-git/pull/1179
None
605ඵ -> 249ඵ
͔͠͠ ·ͩ·ͩϝϞϦ༻ྔ͕ଟ͍ 56 GB/op
ϑΝΠϧʹࠩΛద༻͢Δ ॲཧ͕ϝϞϦΛେྔʹ༻
ؔ෦Ͱ publicͷؔͰ buffer͕֎෦͔Βͤͳ͍ ͷ͕ΘΕ͍ͯͨ
গͳ͘ͱ෦͔Βͷ༻ ͰbufferΛࢦఆ͍ͨ͠
None
෦͚ʹbufferࢦఆͰ͖Δ Α͏ʹͯ͠ղܾ
https://github.com/src-d/ go-git/pull/1180
56.1 GB -> 29.8 GB
None
None
image/png
ύονΛૹͬͨഎܠ
ࣾISUCONͰ QRίʔυΛߴʹੜ͢Δ ඞཁ͕͋ͬͨͨΊ
ࣾISUCONͷৼΓฦΓͰ ύϑΥʔϚϯενϡʔχϯά ΛҰਓͰָ͠ΜͰ͍ͨ
ऄ
ߴʹେྔͷpngΛ ग़ྗ͢ΔͨΊʹ
1.9͔Βೖͬͨ png.EncoderBufferPool
CompressionLevelͷઃఆ
ಠࣗimage.Image࣮Λ Θͳ͍
നࠇը૾Ͱ͋Ε image.GrayΛ͏ (Opaque͕bypassͰ͖Δ)
ऄऴྃ
image/png͕ bottleneckʹͳ͖ͬͯͨ
͢ͰʹBenchmark͕ ॻ͔Ε͍ͯͨͷͰ࣮ߦ
ࠓճ͍ͬͯΔ ՕॴͰͳ͍͕࿐ࠎʹ͍ ෦͕͋ͬͨ
https://go- review.googlesource.com/ c/go/+/187417
None
࣮ࡍॏ͔ͬͨՕॴ compress/deflate
pprof͕ॏ͍ͱࣔͨ͠ͷ for͕ॻ͔Ε͍ͯΔߦͩͬͨ
None
֘forจͷasmΛݟͯΈΔͱ ແବͳϝϞϦΞΫηε͕
compilerʹregisterΛ ͬͯΒ͏ͨΊʹ ϩʔΧϧมΛఆٛ
None
https://go- review.googlesource.com/ c/go/+/187837
None
None
GoogleContainerTools/ kaniko
ύονΛૹͬͨഎܠ
ΞΠσΟΞ͕εΩͩͬͨ
kanikoΛCIͰͬͯ Կʹ͕͔͔͍࣌ؒͬͯΔͷ͔ ؾʹͳͬͨ
kanikoϝϞϦ্ʹ filesystemͷsnapshotΛ࣋ͭ
ίϚϯυΛ࣮ߦ͢Δͨͼʹ ͕ࠩͳ͍͔ൺֱ͢Δ
md5Ͱൺֱ͢Δ
ͦͷmd5͕ॏ͔ͬͨ
ϑΝΠϧ͕ ಉҰ͔Ͳ͏͔͚ͩͰྑ͍ͷͰ md5Ͱ͋Δඞཁͳ͍
minio/HighwayHashʹมߋ (ຊΑ͘ͳ͍)
None
None
mount͞Ε͍ͯΔ σΟϨΫτϦ snapshotର֎
ϑΝΠϧ͕whitelistʹ ؚ·ΕΔఆ͢ΔՕॴͰ strings.SplitΛ༻
ϑΝΠϧ͕ଟ͔ͬͨΓ directory͕ਂ͍ͱ ແବʹϝϞϦΛ༻͢Δ
ಛੑΛߟ͑ͯ strings.SplitNΛ༻
None
None
https://github.com/ GoogleContainerTools/ kaniko/pull/694
129.54s -> 88.29s
None
1. ࣮ࡍʹ͛ͨύον 2. ͍ϥΠϒϥϦ͕ॻ͖͍ͨ
orisano/wyhash
kanikoͷύονΛ ॻ͍͍ͯΔͱ͖ʹ Կ͕ྑ͍hashͳͷͩΖ͏
Q. ͍hash?
Q. ͍hash? A. ܭଌ͠·͠ΐ͏
dgryski/trifles/hashbench खݩͰΒͤͯΈ·͠ΐ͏
(ݟ͔ͭΒͳ͍package͕ ͋ΔͷͰಈ͖·ͤΜ)
wyhash͕ GitHubͷTrendingͰ ্͕͖ͬͯͨ
ͯ͘ϙʔλϒϧͰڧ͍ Β͍͠
ඇৗʹ୯७ͳͷͰ GoʹҠ২ͯ͠ΈΑ͏ͱࢥͬͨ
2ҐͰҠ২͕ऴྃ
hashbenchʹՃ ֬ೝ͢ΔͱϘϩෛ͚͍ͯ͠Δ
͜Μͳܭࢉ͔͠ͳ͍ॲཧΛ Ͳ͏ͬͯߴԽ͢Δͷ͔
ྨࣅϥΠϒϥϦͷௐࠪ
҉߸ܥhashܥ جຊతʹasm͕ΘΕ͍ͯΔ
asmΛ͏ͱ͍?
ॻ͍ͯΈΑ͏
Go asmಠಛͳײ͡ ॻ͍͍ͯΔຊਓ͕ ΄ͱΜͲ͍ͳ͍? ͋·Γࢿྉ͕ͳ͍
ؤுͬͯ AVXΛͬͯॏ͍ॲཧΛॻ͘
lldbΛͬͯ bug(SEGV)Λमਖ਼͢Δ
Benchmark݁Ռ ͘ͳ͍ͬͯΔ
None
Կނ͔
asmͰॻ͍ͨؔ inlineԽ͞Εͳ͍
math/bits encoding/binary ίϯύΠϥ͕ݡ͘࠷దԽ͢Δ https://dave.cheney.net/ 2019/08/20/go-compiler- intrinsics
inlineԽ͞ΕΔΑ͏ͳ খ͞ͳؔ asmͷޮՌ͕ಘΒΕͳ͍
ࠓճͷΑ͏ͳ߹ͩͱ loop·ͰؚΊͯasmԽ͖͢
ෆ׳ΕͳasmͰ ଟ͘ͷίʔυΛॻ͖ͨ͘ͳ͍
mmcloughlin/avo Λ͓͏
None
GoͰasmΛੜ͢Δ ϓϩάϥϜΛॻ͘Ξϓϩʔν
Կ͕ྑ͍͔?
Go asmͷ͓࡞๏Λ avo͕ͬͯ͘ΕΔ
Go IDEͰͷิ͕ޮ͘
avoΛͬͯؤுͬͨ 5 GB/s -> 11 GB/s
ߴͳasmΛॻ͘ͷ͕͍͠
asmϨϕϧͰͳ͍ͥͷ͔ pprofͰΘ͔Βͳ͍
ύΠϓϥΠχϯάΛҙࣝ͢Δ 11 GB/s -> 14 GB/s
·ͱΊ
νϡʔχϯάΛͲ͏ਐΊΔͷ͔ • 0. ෆຬͷϋʔυϧΛԼ͛Δ • 1. ܭଌͷϋʔυϧΛԼ͛Δ • 2. ՕॴΛಛఆ͢Δ
• 3. BenchmarkΛॻ͘
νϡʔχϯάΛͲ͏ਐΊΔͷ͔ • 4. ղܾࡦΛߟ͑Δ • Ϛʔδ͞Ε͍͢ղܾࡦΛࢦ͢ • ϝϯςφϯείετΛ্͛ͳ͍ͷ • 5.
ࢼߦࡨޡ͢Δ • 6. ύονΛૹΔ
Ҿ͖ग़͠Λ૿͢ • ϝϞϦ༻ྔʹͳΓ͕ͪ • ֎෦͔ΒBuffer͕ड͚औΕΔAPIΛߟྀ͢Δ • มߋ͕༰қͰͳ͍߹sync.PoolΛߟྀ͢Δ
Ҿ͖ग़͠Λ૿͢ • దͳhashΛબ͢Δ • Ξηϯϒϥϝϯςφϯείετ͕ߴ͍ͷͰ ۃྗආ͚Δ • ॏ͍ॲཧͷ෦͚ͩσʔλͷ࣋ͪํΛม͑ͯ ΈΔ