Upgrade to PRO for Only $50/Year—Limited-Time Offer! 🔥
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
OSS Performance Tuning Tips
Search
orisano
October 28, 2019
Programming
8
5.6k
OSS Performance Tuning Tips
orisano
October 28, 2019
Tweet
Share
More Decks by orisano
See All by orisano
Docker-Compose & BuildKit
orisano
4
960
Container Build Talk
orisano
3
2.4k
dockerignore talk
orisano
2
7k
Better docker image+
orisano
6
5.9k
Socket.IO Introduction
orisano
0
3k
Profiling Go Application
orisano
11
7.6k
Multi-stage Builds Patterns & Practice
orisano
6
5k
better docker image
orisano
22
30k
the world of characters
orisano
8
1.3k
Other Decks in Programming
See All in Programming
これでLambdaが不要に?!Step FunctionsのJSONata対応について
iwatatomoya
0
230
【re:Growth 2024】 Aurora DSQL をちゃんと話します!
maroon1st
0
600
Webエンジニア主体のモバイルチームの 生産性を高く保つためにやったこと
igreenwood
0
160
Modular Monolith Monorepo ~シンプルさを保ちながらmonorepoのメリットを最大化する~
yuisakamoto
11
4k
RWC 2024 DICOM & ISO/IEC 2022
m_seki
0
140
PipeCDの歩き方
kuro_kurorrr
4
140
Djangoの開発環境で工夫したこと - pre-commit / DevContainer
hiroki_yod
1
670
eBPF Deep Dive: Architecture and Safety Mechanisms
takehaya
12
1.3k
「Chatwork」Android版アプリを 支える単体テストの現在
okuzawats
0
120
Haze - Real time background blurring
chrisbanes
1
450
新規学習のハードルを下げる方法とは?/ How to Make Learning Something New Easier?
nobuoooo
1
140
flutterkaigi_2024.pdf
kyoheig3
0
470
Featured
See All Featured
Bash Introduction
62gerente
608
210k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
330
21k
Embracing the Ebb and Flow
colly
84
4.5k
Fantastic passwords and where to find them - at NoRuKo
philnash
50
2.9k
Product Roadmaps are Hard
iamctodd
PRO
49
11k
Practical Orchestrator
shlominoach
186
10k
Automating Front-end Workflow
addyosmani
1366
200k
How To Stay Up To Date on Web Technology
chriscoyier
789
250k
Facilitating Awesome Meetings
lara
50
6.1k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
32
2.7k
Music & Morning Musume
bryan
46
6.2k
Bootstrapping a Software Product
garrettdimon
PRO
305
110k
Transcript
OSS Performance Tuning Tips #gocon #gocon_hall GoCon 2019 Autumn @orisano
త ̎ͭͷ͜ͱΛڞ༗͍ͨ͠
νϡʔχϯάΛͲ͏ਐΊΔͷ͔ Ҿ͖ग़͠Λ૿͢
νϡʔχϯάΛͲ͏ਐΊΔͷ͔ Ҿ͖ग़͠Λ૿͢
Ͳ͏ਐΊ͍ͯΔͷ͔
0. ෆຬͷϋʔυϧΛԼ͛Δ
ෆຬ͕ վળͷ͖͔͚ͬʹͳΓ·͢
ීஈ͍ͬͯΔ ϥΠϒϥϦɺιϑτΣΞ͕ ͍ͷͩͱࢥͬͯΈΔ
͖͔͚ͬͱ ϞνϕʔγϣϯΛ ࣗΒ࡞Γग़͠·͠ΐ͏
1. ܭଌͷϋʔυϧΛԼ͛Δ
͍ݪҼΛ ؾܰʹ ௐΒΕΔೳྗΛʹ͚ͭΔ
ख͕͔͔ؒΔߦҝΛ ਓؒ͋·Γ͠ͳ͍
ͨ͘͞Μܦݧ͢Δ
ؾܰʹ ௐࠪͰ͖ΔΑ͏ͳΈΛ ೖΕΔ
͜ΕOSSΛެ։͢Δଆͷ ڥมͰࢦఆͨ͠Γ σϑΥϧτͰprofile͕औΕΔ ΈΛఏڙ͢Δͱ վળ͍͢͠/͞Ε͍͢
ʹૺ۰ͨ͠ਓ͕ ݪҼௐ͍ࠪ͢͠ͱخ͍͠
2. ՕॴΛಛఆ͢Δ
͍ͱײͯ͡ Օॴ͕ؔϨϕϧͰ Θ͔Βͳ͍ͷ cliΛ࣮ߦ͍ͯ͠Δͱ͖
mainʹ github.com/pkg/profile ΛՃͯ͠ΈΔ
None
͜Ε͚ͩ
cliͷ࣮ߦ͕1ʹ 1࣌ؒҎ্͔͔Δ߹ͳͲ net/http/pprofͷ΄͏͕ ྑ͍߹
profileΛݟͯ ؔϨϕϧͰಛఆ͢Δ
pprofͷweb൛ʹ׳ΕΔ ͪΌΜͱҙຯ͕Θ͔Δ
͍ΖΜͳݟํͰݟͯΈΔ top, graph, flame-graph source, disasm
3. BenchmarkΛॻ͘
͢Ͱʹॻ͍ͯ͋Δ߹ ϥοΩʔ ͘ͳ͍ͬͯͳ͍߹ ͘ͳΔέʔεΛՃ͢Δ
BenchmarkΛॻ࣌͘ ࠷దԽͰফ͍͑ͯͳ͍͔֬ೝ
BenchmarkΛॻ͘ͱ ؆୯ʹprofile͕औΕΔ
go test -cpuprofile go test -memprofile Λ֮͑Α͏ (go help testflagΛಡ͏)
4. ղܾࡦΛߟ͑Δ
Ͳ͏ղܾ͢Δ͔ʹ͍ͭͯ “Ҿ͖ग़͠Λ૿͢”Ͱ
OSSͷ ύϑΥʔϚϯενϡʔχϯά Ϛʔδ͞Εͯྃ
Ͳ͏ΕϚʔδ͞ΕΔ ύονʹͳΔ͔
Ϛʔδ͢Δͱ͍͏͜ͱ ૬ख͕ϝϯςφϯε͢Δ ͱ͍͏͜ͱ
มߋ͕গͳ͍ ૬खͷίετΛ૿͞ͳ͍
ഁյతมߋ͕ͳ͍ ͘ΘΕ͍ͯΔͷ΄Ͳ ॏཁͳͱ͜Ζ
ҟৗͳίετ͕ͳ͍ มߋ, อक͕͘͠ͳΔ࣮ (ΞηϯϒϥΛͬͨΓ)
ґଘؔΛՃ͢Δ߹ ৻ॏʹબ͢Δ ΞΫςΟϒʹϝϯςφϯε͞ Ε͍ͯΔͷ͔ ຊʹඞཁ͔?
͘͢͝ύϑΥʔϚϯε͕ վળ͢Δ ܶతͳվળड͚ೖΕΒΕΔ
ςετ͕ॻ͍ͯ͋Δ ͢Ͱʹॻ͍ͯ͋Δ߹डཧ ͞Ε͍͢
5. ࢼߦࡨޡ͢Δ
·ͣॳظঢ়ଶͰ ेͳճ timeoutʹͳΒͳ͍Α͏ʹ Benchmark݁ՌΛऔ͓ͬͯ͘
go test -bench . -count=10 -timeout=30000s | tee old.txt
ύονΛૹΔͱ͖ʹ benchstatͷ݁ՌΛૹΔͷͰ ϕʔεϥΠϯेͳճͰ
timeout σϑΥϧτͩͱ10mͰ ҙ֎ͱΦʔόʔ͢Δ
ղܾࡦΛࢥ͍͍ͭͨΒ ·ͣ1Benchmark
ے͕ྑͦ͞͏ͳΒ ेͳճBenchmark
ଟ͘ࢼߦࡨޡ͍ͯ͠Δͱ profileͱsourceͷ͕ؔ Θ͔Βͳ͘ͳΔ
pprofʹ sourceΛݟΔػೳ͕͋Δ͕ อ͍࣋ͯ͠ΔΘ͚Ͱͳ͍ pathͷΈ
git add . git commit -m “$2” REV=$(git rev-parse HEAD)
go test -bench $1 -benchmem -cpuprofile cpu.${REV}.pb.gz -memprofile mem.$ {REV}.pb.gz | tee ${REV}.txt rm ./${REV}.* && git reset HEAD^
sourcegitʹཧͯ͠Β͏ ͪΌΜͱඥ͚ΒΕΔ
ଟ͘ͷprofile͕͋Δͱ ୯ମͷޮՌ͕Θ͔Βͳ͍ ͜ͱ͕͋Δ
go tool pprof -diff_base Ͱprofileؒͷ͕ࠩݟΕΔ
6. ύονΛૹΔ
࠷ऴͷBenchmarkΛ ेͳճߦ͏
ઐ༻ͷΠϯελϯεͰܭଌ ͍͠߹ ۃྗ֎෦ϓϩηεΛఀࢭ͢Δ
νϡʔχϯάΛͲ͏ਐΊΔͷ͔ Ҿ͖ग़͠Λ૿͢
࣮ྫϕʔεͰ ͲͷΑ͏ʹղܾ͔ͨ͠ ࣗͳΒͲ͏ղܾ͢Δ͔ ߟ͑ͯΒ͍͍ͨ
1. ࣮ࡍʹ͛ͨύον 2. ͍ϥΠϒϥϦ͕ॻ͖͍ͨ
1. ࣮ࡍʹ͛ͨύον 2. ͍ϥΠϒϥϦ͕ॻ͖͍ͨ
src-d/go-git
pure-goͰ gitͷૢ࡞Λ͢ΔͨΊͷ ϥΠϒϥϦ
ύονΛૹͬͨഎܠ
aquasecurity/trivy Ͱॳ͍͕ͬͯͨ େ͖ͳrepositoryͷcloneʹ ҟৗͳ࣌ؒ(10min~)͕͔͔Δ ͕͋ͬͨ
ݪҼ gitͷIndexͷߏஙΛߦ͏Օॴ ͩͬͨ
Indexͷ෦දݱ(public)͕ sliceʹͳ͓ͬͯΓ nameͰҰҙʹ͢ΔͨΊʹ deleteͱappend͕ ϑΝΠϧճ࣮ߦ͞Ε͍ͯͨ
deletenameΛࢦఆͯ͠ আ͢ΔͷͰ sliceͷཁૉΛͯࠪ͢͢Δ
ॳ ෦දݱΛmapʹมߋ͢Ε deleteͷܭࢉྔΛݮΒͤΔ ͷͰͦ͏͠Α͏ͱࢥͬͨ
͔͠͠ publicͳϑΟʔϧυ internalͰͳ͍ύοέʔδ 4500 star͑ͷ໊
౸ఈड͚ೖΕΒΕͳ͍ มߋͩͱࢥ͍ఘΊͨ
ޙ
ιʔείʔυΛ ݟ͍ͯ͠Δͱ
໌ه͞Ε͍ͯΔ ϑΟʔϧυͷ༷ (sliceͷॱংอূ͠ͳ͍) େྔͷIndexΛߏங͢Δ λΠϛϯά(clone࣌)
ݟͨ݁͠Ռ ഁյతมߋແ͠Ͱ मਖ਼͢Δํ๏Λࢥ͍͍ͭͨ
ଟ͘ݺͼग़͞ΕΔՕॴ ͷΈʹ͓͍ͯ mapͰอ࣋͠return͢Δࡍʹ sliceʹม͢Δ
mapΛprivateͳؔͰ Ҿ͖ճ͢Α͏ʹ͢Δ͜ͱͰ ഁյతมߋΛճආͰ͖ͨ
None
ͦ͜Λղܾ͢Δͱ mallocgc͕ॏ͘ͳͬͨ
ϝϞϦ༻ྔͷݪҼ io.Copyʹ͋ΔΑ͏ͩͬͨ
ϑΝΠϧ͝ͱʹio.Copy͕ ݺΕ͍ͯͨ
େྔʹݺΕΔՄೳੑ͕͋Δ ՕॴͰio.CopyͰͳ͘ io.CopyBufferΛ͏ͱ ϝϞϦ༻ྔΛ੍ޚͰ͖Δ
͏bufferΛͲ͏ࢦఆ͢Δ͔ ֎෦͔Β༩͑ΒΕͳ͍ͱ ݁ہϝϞϦ༻ྔมΘΒͳ͍
privateͳϝιουͱ͍͑ ҾʹՃ͢Δͱ มߋൣғ͕େ͖͘ͳΔ
ղܾࡦͱͯ͠ globalʹsync.PoolΛ ஔ͘͜ͱʹͨ͠
None
https://github.com/src-d/ go-git/pull/1179
None
605ඵ -> 249ඵ
͔͠͠ ·ͩ·ͩϝϞϦ༻ྔ͕ଟ͍ 56 GB/op
ϑΝΠϧʹࠩΛద༻͢Δ ॲཧ͕ϝϞϦΛେྔʹ༻
ؔ෦Ͱ publicͷؔͰ buffer͕֎෦͔Βͤͳ͍ ͷ͕ΘΕ͍ͯͨ
গͳ͘ͱ෦͔Βͷ༻ ͰbufferΛࢦఆ͍ͨ͠
None
෦͚ʹbufferࢦఆͰ͖Δ Α͏ʹͯ͠ղܾ
https://github.com/src-d/ go-git/pull/1180
56.1 GB -> 29.8 GB
None
None
image/png
ύονΛૹͬͨഎܠ
ࣾISUCONͰ QRίʔυΛߴʹੜ͢Δ ඞཁ͕͋ͬͨͨΊ
ࣾISUCONͷৼΓฦΓͰ ύϑΥʔϚϯενϡʔχϯά ΛҰਓͰָ͠ΜͰ͍ͨ
ऄ
ߴʹେྔͷpngΛ ग़ྗ͢ΔͨΊʹ
1.9͔Βೖͬͨ png.EncoderBufferPool
CompressionLevelͷઃఆ
ಠࣗimage.Image࣮Λ Θͳ͍
നࠇը૾Ͱ͋Ε image.GrayΛ͏ (Opaque͕bypassͰ͖Δ)
ऄऴྃ
image/png͕ bottleneckʹͳ͖ͬͯͨ
͢ͰʹBenchmark͕ ॻ͔Ε͍ͯͨͷͰ࣮ߦ
ࠓճ͍ͬͯΔ ՕॴͰͳ͍͕࿐ࠎʹ͍ ෦͕͋ͬͨ
https://go- review.googlesource.com/ c/go/+/187417
None
࣮ࡍॏ͔ͬͨՕॴ compress/deflate
pprof͕ॏ͍ͱࣔͨ͠ͷ for͕ॻ͔Ε͍ͯΔߦͩͬͨ
None
֘forจͷasmΛݟͯΈΔͱ ແବͳϝϞϦΞΫηε͕
compilerʹregisterΛ ͬͯΒ͏ͨΊʹ ϩʔΧϧมΛఆٛ
None
https://go- review.googlesource.com/ c/go/+/187837
None
None
GoogleContainerTools/ kaniko
ύονΛૹͬͨഎܠ
ΞΠσΟΞ͕εΩͩͬͨ
kanikoΛCIͰͬͯ Կʹ͕͔͔͍࣌ؒͬͯΔͷ͔ ؾʹͳͬͨ
kanikoϝϞϦ্ʹ filesystemͷsnapshotΛ࣋ͭ
ίϚϯυΛ࣮ߦ͢Δͨͼʹ ͕ࠩͳ͍͔ൺֱ͢Δ
md5Ͱൺֱ͢Δ
ͦͷmd5͕ॏ͔ͬͨ
ϑΝΠϧ͕ ಉҰ͔Ͳ͏͔͚ͩͰྑ͍ͷͰ md5Ͱ͋Δඞཁͳ͍
minio/HighwayHashʹมߋ (ຊΑ͘ͳ͍)
None
None
mount͞Ε͍ͯΔ σΟϨΫτϦ snapshotର֎
ϑΝΠϧ͕whitelistʹ ؚ·ΕΔఆ͢ΔՕॴͰ strings.SplitΛ༻
ϑΝΠϧ͕ଟ͔ͬͨΓ directory͕ਂ͍ͱ ແବʹϝϞϦΛ༻͢Δ
ಛੑΛߟ͑ͯ strings.SplitNΛ༻
None
None
https://github.com/ GoogleContainerTools/ kaniko/pull/694
129.54s -> 88.29s
None
1. ࣮ࡍʹ͛ͨύον 2. ͍ϥΠϒϥϦ͕ॻ͖͍ͨ
orisano/wyhash
kanikoͷύονΛ ॻ͍͍ͯΔͱ͖ʹ Կ͕ྑ͍hashͳͷͩΖ͏
Q. ͍hash?
Q. ͍hash? A. ܭଌ͠·͠ΐ͏
dgryski/trifles/hashbench खݩͰΒͤͯΈ·͠ΐ͏
(ݟ͔ͭΒͳ͍package͕ ͋ΔͷͰಈ͖·ͤΜ)
wyhash͕ GitHubͷTrendingͰ ্͕͖ͬͯͨ
ͯ͘ϙʔλϒϧͰڧ͍ Β͍͠
ඇৗʹ୯७ͳͷͰ GoʹҠ২ͯ͠ΈΑ͏ͱࢥͬͨ
2ҐͰҠ২͕ऴྃ
hashbenchʹՃ ֬ೝ͢ΔͱϘϩෛ͚͍ͯ͠Δ
͜Μͳܭࢉ͔͠ͳ͍ॲཧΛ Ͳ͏ͬͯߴԽ͢Δͷ͔
ྨࣅϥΠϒϥϦͷௐࠪ
҉߸ܥhashܥ جຊతʹasm͕ΘΕ͍ͯΔ
asmΛ͏ͱ͍?
ॻ͍ͯΈΑ͏
Go asmಠಛͳײ͡ ॻ͍͍ͯΔຊਓ͕ ΄ͱΜͲ͍ͳ͍? ͋·Γࢿྉ͕ͳ͍
ؤுͬͯ AVXΛͬͯॏ͍ॲཧΛॻ͘
lldbΛͬͯ bug(SEGV)Λमਖ਼͢Δ
Benchmark݁Ռ ͘ͳ͍ͬͯΔ
None
Կނ͔
asmͰॻ͍ͨؔ inlineԽ͞Εͳ͍
math/bits encoding/binary ίϯύΠϥ͕ݡ͘࠷దԽ͢Δ https://dave.cheney.net/ 2019/08/20/go-compiler- intrinsics
inlineԽ͞ΕΔΑ͏ͳ খ͞ͳؔ asmͷޮՌ͕ಘΒΕͳ͍
ࠓճͷΑ͏ͳ߹ͩͱ loop·ͰؚΊͯasmԽ͖͢
ෆ׳ΕͳasmͰ ଟ͘ͷίʔυΛॻ͖ͨ͘ͳ͍
mmcloughlin/avo Λ͓͏
None
GoͰasmΛੜ͢Δ ϓϩάϥϜΛॻ͘Ξϓϩʔν
Կ͕ྑ͍͔?
Go asmͷ͓࡞๏Λ avo͕ͬͯ͘ΕΔ
Go IDEͰͷิ͕ޮ͘
avoΛͬͯؤுͬͨ 5 GB/s -> 11 GB/s
ߴͳasmΛॻ͘ͷ͕͍͠
asmϨϕϧͰͳ͍ͥͷ͔ pprofͰΘ͔Βͳ͍
ύΠϓϥΠχϯάΛҙࣝ͢Δ 11 GB/s -> 14 GB/s
·ͱΊ
νϡʔχϯάΛͲ͏ਐΊΔͷ͔ • 0. ෆຬͷϋʔυϧΛԼ͛Δ • 1. ܭଌͷϋʔυϧΛԼ͛Δ • 2. ՕॴΛಛఆ͢Δ
• 3. BenchmarkΛॻ͘
νϡʔχϯάΛͲ͏ਐΊΔͷ͔ • 4. ղܾࡦΛߟ͑Δ • Ϛʔδ͞Ε͍͢ղܾࡦΛࢦ͢ • ϝϯςφϯείετΛ্͛ͳ͍ͷ • 5.
ࢼߦࡨޡ͢Δ • 6. ύονΛૹΔ
Ҿ͖ग़͠Λ૿͢ • ϝϞϦ༻ྔʹͳΓ͕ͪ • ֎෦͔ΒBuffer͕ड͚औΕΔAPIΛߟྀ͢Δ • มߋ͕༰қͰͳ͍߹sync.PoolΛߟྀ͢Δ
Ҿ͖ग़͠Λ૿͢ • దͳhashΛબ͢Δ • Ξηϯϒϥϝϯςφϯείετ͕ߴ͍ͷͰ ۃྗආ͚Δ • ॏ͍ॲཧͷ෦͚ͩσʔλͷ࣋ͪํΛม͑ͯ ΈΔ