Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
OSS Performance Tuning Tips
Search
orisano
October 28, 2019
Programming
8
5.2k
OSS Performance Tuning Tips
orisano
October 28, 2019
Tweet
Share
More Decks by orisano
See All by orisano
Docker-Compose & BuildKit
orisano
4
800
Container Build Talk
orisano
3
2.4k
dockerignore talk
orisano
2
6.6k
Better docker image+
orisano
6
5.5k
Socket.IO Introduction
orisano
0
2.7k
Profiling Go Application
orisano
11
7.3k
Multi-stage Builds Patterns & Practice
orisano
6
4.7k
better docker image
orisano
22
29k
the world of characters
orisano
8
1.2k
Other Decks in Programming
See All in Programming
スタートアップのフロントエンド事情 GENBA #2 〜Front-End Opsの現場〜
ebijun1007
1
780
syndicationd LT
ymgyt
0
110
OpCode目線で眺める PHPコードのカバレッジ
o0h
PRO
2
470
生成 AI の中身を覗いてみよう〜基礎から医療現場での応用まで〜
soh9834
2
700
とにかくHTTP3をライトニングに話す / Anyway, I'll talk to Lightning about HTTP3.
seike460
PRO
0
110
マイ隙間家具OSSたちのご紹介
karupanerura
2
120
オレオレkaggle開発環境に Formatter/Linter入れてみた
stgkrt
0
340
20240301_cocone_EMゆるミートアップvol6_LT資料
cocone
0
250
【KMC春合宿2024】実装視点で見るNeural Radiance Fields
runningoutrate
0
130
孤独のCTOグルメという やや奇抜な企画をやった目的と効果
shoheimitani
1
110
PHP で読む楽しいコアダンプ
sji
0
210
document.write再考
brn
5
1.8k
Featured
See All Featured
The Brand Is Dead. Long Live the Brand.
mthomps
48
19k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
18
1.7k
[RailsConf 2023] Rails as a piece of cake
palkan
21
3.8k
Infographics Made Easy
chrislema
237
17k
Web Components: a chance to create the future
zenorocha
304
41k
Navigating Team Friction
lara
177
13k
Art, The Web, and Tiny UX
lynnandtonic
288
19k
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
19
1.6k
A designer walks into a library…
pauljervisheath
199
23k
5 minutes of I Can Smell Your CMS
philhawksworth
199
19k
The Straight Up "How To Draw Better" Workshop
denniskardys
227
130k
Happy Clients
brianwarren
91
6.3k
Transcript
OSS Performance Tuning Tips #gocon #gocon_hall GoCon 2019 Autumn @orisano
త ̎ͭͷ͜ͱΛڞ༗͍ͨ͠
νϡʔχϯάΛͲ͏ਐΊΔͷ͔ Ҿ͖ग़͠Λ૿͢
νϡʔχϯάΛͲ͏ਐΊΔͷ͔ Ҿ͖ग़͠Λ૿͢
Ͳ͏ਐΊ͍ͯΔͷ͔
0. ෆຬͷϋʔυϧΛԼ͛Δ
ෆຬ͕ վળͷ͖͔͚ͬʹͳΓ·͢
ීஈ͍ͬͯΔ ϥΠϒϥϦɺιϑτΣΞ͕ ͍ͷͩͱࢥͬͯΈΔ
͖͔͚ͬͱ ϞνϕʔγϣϯΛ ࣗΒ࡞Γग़͠·͠ΐ͏
1. ܭଌͷϋʔυϧΛԼ͛Δ
͍ݪҼΛ ؾܰʹ ௐΒΕΔೳྗΛʹ͚ͭΔ
ख͕͔͔ؒΔߦҝΛ ਓؒ͋·Γ͠ͳ͍
ͨ͘͞Μܦݧ͢Δ
ؾܰʹ ௐࠪͰ͖ΔΑ͏ͳΈΛ ೖΕΔ
͜ΕOSSΛެ։͢Δଆͷ ڥมͰࢦఆͨ͠Γ σϑΥϧτͰprofile͕औΕΔ ΈΛఏڙ͢Δͱ վળ͍͢͠/͞Ε͍͢
ʹૺ۰ͨ͠ਓ͕ ݪҼௐ͍ࠪ͢͠ͱخ͍͠
2. ՕॴΛಛఆ͢Δ
͍ͱײͯ͡ Օॴ͕ؔϨϕϧͰ Θ͔Βͳ͍ͷ cliΛ࣮ߦ͍ͯ͠Δͱ͖
mainʹ github.com/pkg/profile ΛՃͯ͠ΈΔ
None
͜Ε͚ͩ
cliͷ࣮ߦ͕1ʹ 1࣌ؒҎ্͔͔Δ߹ͳͲ net/http/pprofͷ΄͏͕ ྑ͍߹
profileΛݟͯ ؔϨϕϧͰಛఆ͢Δ
pprofͷweb൛ʹ׳ΕΔ ͪΌΜͱҙຯ͕Θ͔Δ
͍ΖΜͳݟํͰݟͯΈΔ top, graph, flame-graph source, disasm
3. BenchmarkΛॻ͘
͢Ͱʹॻ͍ͯ͋Δ߹ ϥοΩʔ ͘ͳ͍ͬͯͳ͍߹ ͘ͳΔέʔεΛՃ͢Δ
BenchmarkΛॻ࣌͘ ࠷దԽͰফ͍͑ͯͳ͍͔֬ೝ
BenchmarkΛॻ͘ͱ ؆୯ʹprofile͕औΕΔ
go test -cpuprofile go test -memprofile Λ֮͑Α͏ (go help testflagΛಡ͏)
4. ղܾࡦΛߟ͑Δ
Ͳ͏ղܾ͢Δ͔ʹ͍ͭͯ “Ҿ͖ग़͠Λ૿͢”Ͱ
OSSͷ ύϑΥʔϚϯενϡʔχϯά Ϛʔδ͞Εͯྃ
Ͳ͏ΕϚʔδ͞ΕΔ ύονʹͳΔ͔
Ϛʔδ͢Δͱ͍͏͜ͱ ૬ख͕ϝϯςφϯε͢Δ ͱ͍͏͜ͱ
มߋ͕গͳ͍ ૬खͷίετΛ૿͞ͳ͍
ഁյతมߋ͕ͳ͍ ͘ΘΕ͍ͯΔͷ΄Ͳ ॏཁͳͱ͜Ζ
ҟৗͳίετ͕ͳ͍ มߋ, อक͕͘͠ͳΔ࣮ (ΞηϯϒϥΛͬͨΓ)
ґଘؔΛՃ͢Δ߹ ৻ॏʹબ͢Δ ΞΫςΟϒʹϝϯςφϯε͞ Ε͍ͯΔͷ͔ ຊʹඞཁ͔?
͘͢͝ύϑΥʔϚϯε͕ վળ͢Δ ܶతͳվળड͚ೖΕΒΕΔ
ςετ͕ॻ͍ͯ͋Δ ͢Ͱʹॻ͍ͯ͋Δ߹डཧ ͞Ε͍͢
5. ࢼߦࡨޡ͢Δ
·ͣॳظঢ়ଶͰ ेͳճ timeoutʹͳΒͳ͍Α͏ʹ Benchmark݁ՌΛऔ͓ͬͯ͘
go test -bench . -count=10 -timeout=30000s | tee old.txt
ύονΛૹΔͱ͖ʹ benchstatͷ݁ՌΛૹΔͷͰ ϕʔεϥΠϯेͳճͰ
timeout σϑΥϧτͩͱ10mͰ ҙ֎ͱΦʔόʔ͢Δ
ղܾࡦΛࢥ͍͍ͭͨΒ ·ͣ1Benchmark
ے͕ྑͦ͞͏ͳΒ ेͳճBenchmark
ଟ͘ࢼߦࡨޡ͍ͯ͠Δͱ profileͱsourceͷ͕ؔ Θ͔Βͳ͘ͳΔ
pprofʹ sourceΛݟΔػೳ͕͋Δ͕ อ͍࣋ͯ͠ΔΘ͚Ͱͳ͍ pathͷΈ
git add . git commit -m “$2” REV=$(git rev-parse HEAD)
go test -bench $1 -benchmem -cpuprofile cpu.${REV}.pb.gz -memprofile mem.$ {REV}.pb.gz | tee ${REV}.txt rm ./${REV}.* && git reset HEAD^
sourcegitʹཧͯ͠Β͏ ͪΌΜͱඥ͚ΒΕΔ
ଟ͘ͷprofile͕͋Δͱ ୯ମͷޮՌ͕Θ͔Βͳ͍ ͜ͱ͕͋Δ
go tool pprof -diff_base Ͱprofileؒͷ͕ࠩݟΕΔ
6. ύονΛૹΔ
࠷ऴͷBenchmarkΛ ेͳճߦ͏
ઐ༻ͷΠϯελϯεͰܭଌ ͍͠߹ ۃྗ֎෦ϓϩηεΛఀࢭ͢Δ
νϡʔχϯάΛͲ͏ਐΊΔͷ͔ Ҿ͖ग़͠Λ૿͢
࣮ྫϕʔεͰ ͲͷΑ͏ʹղܾ͔ͨ͠ ࣗͳΒͲ͏ղܾ͢Δ͔ ߟ͑ͯΒ͍͍ͨ
1. ࣮ࡍʹ͛ͨύον 2. ͍ϥΠϒϥϦ͕ॻ͖͍ͨ
1. ࣮ࡍʹ͛ͨύον 2. ͍ϥΠϒϥϦ͕ॻ͖͍ͨ
src-d/go-git
pure-goͰ gitͷૢ࡞Λ͢ΔͨΊͷ ϥΠϒϥϦ
ύονΛૹͬͨഎܠ
aquasecurity/trivy Ͱॳ͍͕ͬͯͨ େ͖ͳrepositoryͷcloneʹ ҟৗͳ࣌ؒ(10min~)͕͔͔Δ ͕͋ͬͨ
ݪҼ gitͷIndexͷߏஙΛߦ͏Օॴ ͩͬͨ
Indexͷ෦දݱ(public)͕ sliceʹͳ͓ͬͯΓ nameͰҰҙʹ͢ΔͨΊʹ deleteͱappend͕ ϑΝΠϧճ࣮ߦ͞Ε͍ͯͨ
deletenameΛࢦఆͯ͠ আ͢ΔͷͰ sliceͷཁૉΛͯࠪ͢͢Δ
ॳ ෦දݱΛmapʹมߋ͢Ε deleteͷܭࢉྔΛݮΒͤΔ ͷͰͦ͏͠Α͏ͱࢥͬͨ
͔͠͠ publicͳϑΟʔϧυ internalͰͳ͍ύοέʔδ 4500 star͑ͷ໊
౸ఈड͚ೖΕΒΕͳ͍ มߋͩͱࢥ͍ఘΊͨ
ޙ
ιʔείʔυΛ ݟ͍ͯ͠Δͱ
໌ه͞Ε͍ͯΔ ϑΟʔϧυͷ༷ (sliceͷॱংอূ͠ͳ͍) େྔͷIndexΛߏங͢Δ λΠϛϯά(clone࣌)
ݟͨ݁͠Ռ ഁյతมߋແ͠Ͱ मਖ਼͢Δํ๏Λࢥ͍͍ͭͨ
ଟ͘ݺͼग़͞ΕΔՕॴ ͷΈʹ͓͍ͯ mapͰอ࣋͠return͢Δࡍʹ sliceʹม͢Δ
mapΛprivateͳؔͰ Ҿ͖ճ͢Α͏ʹ͢Δ͜ͱͰ ഁյతมߋΛճආͰ͖ͨ
None
ͦ͜Λղܾ͢Δͱ mallocgc͕ॏ͘ͳͬͨ
ϝϞϦ༻ྔͷݪҼ io.Copyʹ͋ΔΑ͏ͩͬͨ
ϑΝΠϧ͝ͱʹio.Copy͕ ݺΕ͍ͯͨ
େྔʹݺΕΔՄೳੑ͕͋Δ ՕॴͰio.CopyͰͳ͘ io.CopyBufferΛ͏ͱ ϝϞϦ༻ྔΛ੍ޚͰ͖Δ
͏bufferΛͲ͏ࢦఆ͢Δ͔ ֎෦͔Β༩͑ΒΕͳ͍ͱ ݁ہϝϞϦ༻ྔมΘΒͳ͍
privateͳϝιουͱ͍͑ ҾʹՃ͢Δͱ มߋൣғ͕େ͖͘ͳΔ
ղܾࡦͱͯ͠ globalʹsync.PoolΛ ஔ͘͜ͱʹͨ͠
None
https://github.com/src-d/ go-git/pull/1179
None
605ඵ -> 249ඵ
͔͠͠ ·ͩ·ͩϝϞϦ༻ྔ͕ଟ͍ 56 GB/op
ϑΝΠϧʹࠩΛద༻͢Δ ॲཧ͕ϝϞϦΛେྔʹ༻
ؔ෦Ͱ publicͷؔͰ buffer͕֎෦͔Βͤͳ͍ ͷ͕ΘΕ͍ͯͨ
গͳ͘ͱ෦͔Βͷ༻ ͰbufferΛࢦఆ͍ͨ͠
None
෦͚ʹbufferࢦఆͰ͖Δ Α͏ʹͯ͠ղܾ
https://github.com/src-d/ go-git/pull/1180
56.1 GB -> 29.8 GB
None
None
image/png
ύονΛૹͬͨഎܠ
ࣾISUCONͰ QRίʔυΛߴʹੜ͢Δ ඞཁ͕͋ͬͨͨΊ
ࣾISUCONͷৼΓฦΓͰ ύϑΥʔϚϯενϡʔχϯά ΛҰਓͰָ͠ΜͰ͍ͨ
ऄ
ߴʹେྔͷpngΛ ग़ྗ͢ΔͨΊʹ
1.9͔Βೖͬͨ png.EncoderBufferPool
CompressionLevelͷઃఆ
ಠࣗimage.Image࣮Λ Θͳ͍
നࠇը૾Ͱ͋Ε image.GrayΛ͏ (Opaque͕bypassͰ͖Δ)
ऄऴྃ
image/png͕ bottleneckʹͳ͖ͬͯͨ
͢ͰʹBenchmark͕ ॻ͔Ε͍ͯͨͷͰ࣮ߦ
ࠓճ͍ͬͯΔ ՕॴͰͳ͍͕࿐ࠎʹ͍ ෦͕͋ͬͨ
https://go- review.googlesource.com/ c/go/+/187417
None
࣮ࡍॏ͔ͬͨՕॴ compress/deflate
pprof͕ॏ͍ͱࣔͨ͠ͷ for͕ॻ͔Ε͍ͯΔߦͩͬͨ
None
֘forจͷasmΛݟͯΈΔͱ ແବͳϝϞϦΞΫηε͕
compilerʹregisterΛ ͬͯΒ͏ͨΊʹ ϩʔΧϧมΛఆٛ
None
https://go- review.googlesource.com/ c/go/+/187837
None
None
GoogleContainerTools/ kaniko
ύονΛૹͬͨഎܠ
ΞΠσΟΞ͕εΩͩͬͨ
kanikoΛCIͰͬͯ Կʹ͕͔͔͍࣌ؒͬͯΔͷ͔ ؾʹͳͬͨ
kanikoϝϞϦ্ʹ filesystemͷsnapshotΛ࣋ͭ
ίϚϯυΛ࣮ߦ͢Δͨͼʹ ͕ࠩͳ͍͔ൺֱ͢Δ
md5Ͱൺֱ͢Δ
ͦͷmd5͕ॏ͔ͬͨ
ϑΝΠϧ͕ ಉҰ͔Ͳ͏͔͚ͩͰྑ͍ͷͰ md5Ͱ͋Δඞཁͳ͍
minio/HighwayHashʹมߋ (ຊΑ͘ͳ͍)
None
None
mount͞Ε͍ͯΔ σΟϨΫτϦ snapshotର֎
ϑΝΠϧ͕whitelistʹ ؚ·ΕΔఆ͢ΔՕॴͰ strings.SplitΛ༻
ϑΝΠϧ͕ଟ͔ͬͨΓ directory͕ਂ͍ͱ ແବʹϝϞϦΛ༻͢Δ
ಛੑΛߟ͑ͯ strings.SplitNΛ༻
None
None
https://github.com/ GoogleContainerTools/ kaniko/pull/694
129.54s -> 88.29s
None
1. ࣮ࡍʹ͛ͨύον 2. ͍ϥΠϒϥϦ͕ॻ͖͍ͨ
orisano/wyhash
kanikoͷύονΛ ॻ͍͍ͯΔͱ͖ʹ Կ͕ྑ͍hashͳͷͩΖ͏
Q. ͍hash?
Q. ͍hash? A. ܭଌ͠·͠ΐ͏
dgryski/trifles/hashbench खݩͰΒͤͯΈ·͠ΐ͏
(ݟ͔ͭΒͳ͍package͕ ͋ΔͷͰಈ͖·ͤΜ)
wyhash͕ GitHubͷTrendingͰ ্͕͖ͬͯͨ
ͯ͘ϙʔλϒϧͰڧ͍ Β͍͠
ඇৗʹ୯७ͳͷͰ GoʹҠ২ͯ͠ΈΑ͏ͱࢥͬͨ
2ҐͰҠ২͕ऴྃ
hashbenchʹՃ ֬ೝ͢ΔͱϘϩෛ͚͍ͯ͠Δ
͜Μͳܭࢉ͔͠ͳ͍ॲཧΛ Ͳ͏ͬͯߴԽ͢Δͷ͔
ྨࣅϥΠϒϥϦͷௐࠪ
҉߸ܥhashܥ جຊతʹasm͕ΘΕ͍ͯΔ
asmΛ͏ͱ͍?
ॻ͍ͯΈΑ͏
Go asmಠಛͳײ͡ ॻ͍͍ͯΔຊਓ͕ ΄ͱΜͲ͍ͳ͍? ͋·Γࢿྉ͕ͳ͍
ؤுͬͯ AVXΛͬͯॏ͍ॲཧΛॻ͘
lldbΛͬͯ bug(SEGV)Λमਖ਼͢Δ
Benchmark݁Ռ ͘ͳ͍ͬͯΔ
None
Կނ͔
asmͰॻ͍ͨؔ inlineԽ͞Εͳ͍
math/bits encoding/binary ίϯύΠϥ͕ݡ͘࠷దԽ͢Δ https://dave.cheney.net/ 2019/08/20/go-compiler- intrinsics
inlineԽ͞ΕΔΑ͏ͳ খ͞ͳؔ asmͷޮՌ͕ಘΒΕͳ͍
ࠓճͷΑ͏ͳ߹ͩͱ loop·ͰؚΊͯasmԽ͖͢
ෆ׳ΕͳasmͰ ଟ͘ͷίʔυΛॻ͖ͨ͘ͳ͍
mmcloughlin/avo Λ͓͏
None
GoͰasmΛੜ͢Δ ϓϩάϥϜΛॻ͘Ξϓϩʔν
Կ͕ྑ͍͔?
Go asmͷ͓࡞๏Λ avo͕ͬͯ͘ΕΔ
Go IDEͰͷิ͕ޮ͘
avoΛͬͯؤுͬͨ 5 GB/s -> 11 GB/s
ߴͳasmΛॻ͘ͷ͕͍͠
asmϨϕϧͰͳ͍ͥͷ͔ pprofͰΘ͔Βͳ͍
ύΠϓϥΠχϯάΛҙࣝ͢Δ 11 GB/s -> 14 GB/s
·ͱΊ
νϡʔχϯάΛͲ͏ਐΊΔͷ͔ • 0. ෆຬͷϋʔυϧΛԼ͛Δ • 1. ܭଌͷϋʔυϧΛԼ͛Δ • 2. ՕॴΛಛఆ͢Δ
• 3. BenchmarkΛॻ͘
νϡʔχϯάΛͲ͏ਐΊΔͷ͔ • 4. ղܾࡦΛߟ͑Δ • Ϛʔδ͞Ε͍͢ղܾࡦΛࢦ͢ • ϝϯςφϯείετΛ্͛ͳ͍ͷ • 5.
ࢼߦࡨޡ͢Δ • 6. ύονΛૹΔ
Ҿ͖ग़͠Λ૿͢ • ϝϞϦ༻ྔʹͳΓ͕ͪ • ֎෦͔ΒBuffer͕ड͚औΕΔAPIΛߟྀ͢Δ • มߋ͕༰қͰͳ͍߹sync.PoolΛߟྀ͢Δ
Ҿ͖ग़͠Λ૿͢ • దͳhashΛબ͢Δ • Ξηϯϒϥϝϯςφϯείετ͕ߴ͍ͷͰ ۃྗආ͚Δ • ॏ͍ॲཧͷ෦͚ͩσʔλͷ࣋ͪํΛม͑ͯ ΈΔ