OSS Performance Tuning Tips

OSS
Performance Tuning
Tips
#gocon #gocon_hall GoCon 2019 Autumn
@orisano

View Slide

໨త
̎ͭͷ͜ͱΛڞ༗͍ͨ͠

View Slide

νϡʔχϯάΛͲ͏ਐΊΔͷ͔
Ҿ͖ग़͠Λ૿΍͢

View Slide

Ͳ͏ਐΊ͍ͯΔͷ͔

View Slide

0. ෆຬͷϋʔυϧΛԼ͛Δ

View Slide

ෆຬ͕
վળͷ͖͔͚ͬʹͳΓ·͢

View Slide

ීஈ࢖͍ͬͯΔ
ϥΠϒϥϦɺιϑτ΢ΣΞ͕
஗͍΋ͷͩͱࢥͬͯΈΔ

View Slide

͖͔͚ͬͱ
ϞνϕʔγϣϯΛ
ࣗΒ࡞Γग़͠·͠ΐ͏

View Slide

1. ܭଌͷϋʔυϧΛԼ͛Δ

View Slide

஗͍ݪҼΛ
ؾܰʹ
ௐ΂ΒΕΔೳྗΛ਎ʹ͚ͭΔ

View Slide

ख͕͔͔ؒΔߦҝΛ
ਓؒ͸͋·Γ͠ͳ͍

View Slide

ͨ͘͞Μܦݧ͢Δ

View Slide

ؾܰʹ
ௐࠪͰ͖ΔΑ͏ͳ࢓૊ΈΛ
ೖΕΔ

View Slide

͜Ε͸OSSΛެ։͢Δଆͷ࿩
؀ڥม਺Ͱࢦఆͨ͠Γ
σϑΥϧτͰproﬁle͕औΕΔ
࢓૊ΈΛఏڙ͢Δͱ
վળ͠΍͍͢/͞Ε΍͍͢

View Slide

໰୊ʹૺ۰ͨ͠ਓ͕
ݪҼௐࠪ͠΍͍͢ͱخ͍͠

View Slide

2. ໰୊ՕॴΛಛఆ͢Δ

View Slide

஗͍ͱײͯ͡
໰୊Օॴ͕ؔ਺ϨϕϧͰ
Θ͔Βͳ͍ͷ͸
cliΛ࣮ߦ͍ͯ͠Δͱ͖

View Slide

mainʹ
github.com/pkg/proﬁle
Λ௥Ճͯ͠ΈΔ

View Slide

View Slide

͜Ε͚ͩ

View Slide

cliͷ࣮ߦ͕1౓ʹ
1࣌ؒҎ্͔͔Δ৔߹ͳͲ͸
net/http/pprofͷ΄͏͕
ྑ͍৔߹΋

View Slide

proﬁleΛݟͯ
ؔ਺ϨϕϧͰಛఆ͢Δ

View Slide

pprofͷweb൛ʹ׳ΕΔ
ͪΌΜͱҙຯ͕Θ͔Δ

View Slide

͍ΖΜͳݟํͰݟͯΈΔ
top, graph, ﬂame-graph
source, disasm

View Slide

3. BenchmarkΛॻ͘

View Slide

͢Ͱʹॻ͍ͯ͋Δ৔߹͸
ϥοΩʔ
஗͘ͳ͍ͬͯͳ͍৔߹͸
஗͘ͳΔέʔεΛ௥Ճ͢Δ

View Slide

BenchmarkΛॻ࣌͘͸
࠷దԽͰফ͍͑ͯͳ͍͔֬ೝ

View Slide

BenchmarkΛॻ͘ͱ
؆୯ʹproﬁle͕औΕΔ

View Slide

go test -cpuprofile
go test -memprofile
Λ֮͑Α͏
(go help testflagΛಡ΋͏)

View Slide

4. ղܾࡦΛߟ͑Δ

View Slide

Ͳ͏ղܾ͢Δ͔ʹ͍ͭͯ͸
“Ҿ͖ग़͠Λ૿΍͢”Ͱ

View Slide

OSSͷ
ύϑΥʔϚϯενϡʔχϯά
͸Ϛʔδ͞Εͯ׬ྃ

View Slide

Ͳ͏΍Ε͹Ϛʔδ͞ΕΔ
ύονʹͳΔ͔

View Slide

Ϛʔδ͢Δͱ͍͏͜ͱ͸
૬ख͕ϝϯςφϯε͢Δ
ͱ͍͏͜ͱ

View Slide

มߋ͕গͳ͍
૬खͷίετΛ૿΍͞ͳ͍

View Slide

ഁյతมߋ͕ͳ͍
޿͘࢖ΘΕ͍ͯΔ΋ͷ΄Ͳ
ॏཁͳͱ͜Ζ

View Slide

ҟৗͳίετ͕ͳ͍
มߋ, อक͕೉͘͠ͳΔ࣮૷
(ΞηϯϒϥΛ࢖ͬͨΓ)

View Slide

ґଘؔ܎Λ௥Ճ͢Δ৔߹͸
৻ॏʹબ୒͢Δ
ΞΫςΟϒʹϝϯςφϯε͞
Ε͍ͯΔ΋ͷ͔
ຊ౰ʹඞཁ͔?

View Slide

͘͢͝ύϑΥʔϚϯε͕
վળ͢Δ
ܶతͳվળ͸ड͚ೖΕΒΕΔ

View Slide

ςετ͕ॻ͍ͯ͋Δ
͢Ͱʹॻ͍ͯ͋Δ৔߹͸डཧ
͞Ε΍͍͢

View Slide

5. ࢼߦࡨޡ͢Δ

View Slide

·ͣॳظঢ়ଶͰ
े෼ͳճ਺
timeoutʹͳΒͳ͍Α͏ʹ
Benchmark݁ՌΛऔ͓ͬͯ͘

View Slide

go test -bench .
-count=10
-timeout=30000s
| tee old.txt

View Slide

ύονΛૹΔͱ͖ʹ
benchstatͷ݁ՌΛૹΔͷͰ
ϕʔεϥΠϯ͸े෼ͳճ਺Ͱ

View Slide

timeout͸
σϑΥϧτͩͱ10mͰ
ҙ֎ͱΦʔόʔ͢Δ

View Slide

ղܾࡦΛࢥ͍͍ͭͨΒ
·ͣ͸1౓Benchmark

View Slide

ے͕ྑͦ͞͏ͳΒ
े෼ͳճ਺Benchmark

View Slide

ଟ͘ࢼߦࡨޡ͍ͯ͠Δͱ
proﬁleͱsourceͷؔ܎͕
Θ͔Βͳ͘ͳΔ

View Slide

pprofʹ͸
sourceΛݟΔػೳ͕͋Δ͕
อ͍࣋ͯ͠ΔΘ͚Ͱ͸ͳ͍
pathͷΈ

View Slide

git add .
git commit -m “$2”
REV=$(git rev-parse HEAD)
go test -bench $1 -benchmem -cpuproﬁle
cpu.${REV}.pb.gz -memproﬁle mem.$
{REV}.pb.gz | tee ${REV}.txt
rm ./${REV}.* && git reset HEAD^

View Slide

source͸gitʹ؅ཧͯ͠΋Β͏
ͪΌΜͱඥ෇͚ΒΕΔ

View Slide

ଟ͘ͷproﬁle͕͋Δͱ
୯ମͷޮՌ͕Θ͔Βͳ͍
͜ͱ͕͋Δ

View Slide

go tool pprof -diff_base
Ͱproﬁleؒͷࠩ෼͕ݟΕΔ

View Slide

6. ύονΛૹΔ

View Slide

࠷ऴͷBenchmarkΛ
े෼ͳճ਺ߦ͏

View Slide

ઐ༻ͷΠϯελϯεͰܭଌ
೉͍͠৔߹͸
ۃྗ֎෦ϓϩηεΛఀࢭ͢Δ

View Slide

Ҿ͖ग़͠Λ૿΍͢

View Slide

࣮ྫϕʔεͰ
ͲͷΑ͏ʹղܾ͔ͨ͠
ࣗ෼ͳΒͲ͏ղܾ͢Δ͔
ߟ͑ͯ΋Β͍͍ͨ

View Slide

1. ࣮ࡍʹ౤͛ͨύον
2. ଎͍ϥΠϒϥϦ͕ॻ͖͍ͨ

View Slide

src-d/go-git

View Slide

pure-goͰ
gitͷૢ࡞Λ͢ΔͨΊͷ
ϥΠϒϥϦ

View Slide

ύονΛૹͬͨഎܠ

View Slide

aquasecurity/trivy
Ͱ౰ॳ࢖͍͕ͬͯͨ
େ͖ͳrepositoryͷcloneʹ
ҟৗͳ࣌ؒ(10min~)͕͔͔Δ
໰୊͕͋ͬͨ

View Slide

ݪҼ͸
gitͷIndexͷߏஙΛߦ͏Օॴ
ͩͬͨ

View Slide

Indexͷ಺෦දݱ(public)͕
sliceʹͳ͓ͬͯΓ
nameͰҰҙʹ͢ΔͨΊʹ
deleteͱappend͕
ϑΝΠϧ਺ճ࣮ߦ͞Ε͍ͯͨ

View Slide

delete͸nameΛࢦఆͯ͠
࡟আ͢ΔͷͰ
sliceͷཁૉΛ͢΂ͯ૸ࠪ͢Δ

View Slide

౰ॳ͸
಺෦දݱΛmapʹมߋ͢Ε͹
deleteͷܭࢉྔΛݮΒͤΔ
ͷͰͦ͏͠Α͏ͱࢥͬͨ

View Slide

͔͠͠
publicͳϑΟʔϧυ
internalͰͳ͍ύοέʔδ
4500 star௒͑ͷ஌໊౓

View Slide

౸ఈड͚ೖΕΒΕͳ͍
มߋͩͱࢥ͍ఘΊͨ

View Slide

ޙ೔

View Slide

ιʔείʔυΛ
ݟ௚͍ͯ͠Δͱ

View Slide

໌ه͞Ε͍ͯΔ
ϑΟʔϧυͷ࢓༷
(sliceͷॱং͸อূ͠ͳ͍)
େྔͷIndexΛߏங͢Δ
λΠϛϯά(clone࣌)

View Slide

ݟ௚ͨ݁͠Ռ
ഁյతมߋແ͠Ͱ
मਖ਼͢Δํ๏Λࢥ͍͍ͭͨ

View Slide

ଟ͘ݺͼग़͞ΕΔՕॴ
ͷΈʹ͓͍ͯ
mapͰอ࣋͠return͢Δࡍʹ
sliceʹม׵͢Δ

View Slide

mapΛprivateͳؔ਺Ͱ
Ҿ͖ճ͢Α͏ʹ͢Δ͜ͱͰ
ഁյతมߋΛճආͰ͖ͨ

View Slide

View Slide

ͦ͜Λղܾ͢Δͱ
mallocgc͕ॏ͘ͳͬͨ

View Slide

ϝϞϦ࢖༻ྔͷݪҼ͸
io.Copyʹ͋ΔΑ͏ͩͬͨ

View Slide

ϑΝΠϧ͝ͱʹio.Copy͕
ݺ͹Ε͍ͯͨ

View Slide

େྔʹݺ͹ΕΔՄೳੑ͕͋Δ
ՕॴͰ͸io.CopyͰ͸ͳ͘
io.CopyBufferΛ࢖͏ͱ
ϝϞϦ࢖༻ྔΛ੍ޚͰ͖Δ

View Slide

࢖͏bufferΛͲ͏ࢦఆ͢Δ͔
֎෦͔Β༩͑ΒΕͳ͍ͱ
݁ہϝϞϦ࢖༻ྔมΘΒͳ͍

View Slide

privateͳϝιουͱ͸͍͑
Ҿ਺ʹ௥Ճ͢Δͱ
มߋൣғ͕େ͖͘ͳΔ

View Slide

ղܾࡦͱͯ͠
globalʹsync.PoolΛ
ஔ͘͜ͱʹͨ͠

View Slide

View Slide

https://github.com/src-d/
go-git/pull/1179

View Slide

View Slide

605ඵ -> 249ඵ

View Slide

͔͠͠
·ͩ·ͩϝϞϦ࢖༻ྔ͕ଟ͍
56 GB/op

View Slide

ϑΝΠϧʹࠩ෼Λద༻͢Δ
ॲཧ͕ϝϞϦΛେྔʹ࢖༻

View Slide

ؔ਺಺෦Ͱ͸
publicͷؔ਺Ͱ
buffer͕֎෦͔Β౉ͤͳ͍
΋ͷ͕࢖ΘΕ͍ͯͨ

View Slide

গͳ͘ͱ΋಺෦͔Βͷ࢖༻
Ͱ͸bufferΛࢦఆ͍ͨ͠

View Slide

View Slide

಺෦޲͚ʹbufferࢦఆͰ͖Δ
Α͏ʹͯ͠ղܾ

View Slide

https://github.com/src-d/
go-git/pull/1180

View Slide

56.1 GB -> 29.8 GB

View Slide

View Slide

image/png

View Slide


View Slide

ࣾ಺ISUCONͰ
QRίʔυΛߴ଎ʹੜ੒͢Δ
ඞཁ͕͋ͬͨͨΊ

View Slide

ࣾ಺ISUCONͷৼΓฦΓͰ
ύϑΥʔϚϯενϡʔχϯά
ΛҰਓͰָ͠ΜͰ͍ͨ

View Slide

ऄ଍

View Slide

ߴ଎ʹେྔͷpngΛ
ग़ྗ͢ΔͨΊʹ͸

View Slide

1.9͔Βೖͬͨ
png.EncoderBufferPool

View Slide

CompressionLevelͷઃఆ

View Slide

ಠࣗimage.Image࣮૷Λ
࢖Θͳ͍

View Slide

നࠇը૾Ͱ͋Ε͹
image.GrayΛ࢖͏
(Opaque͕bypassͰ͖Δ)

View Slide

ऄ଍ऴྃ

View Slide

image/png͕
bottleneckʹͳ͖ͬͯͨ

View Slide

͢ͰʹBenchmark͕
ॻ͔Ε͍ͯͨͷͰ࣮ߦ

View Slide

ࠓճ࢖͍ͬͯΔ
ՕॴͰ͸ͳ͍͕࿐ࠎʹ஗͍
෦෼͕͋ͬͨ

View Slide

https://go-
review.googlesource.com/
c/go/+/187417

View Slide

View Slide

࣮ࡍॏ͔ͬͨՕॴ͸
compress/deﬂate

View Slide

pprof͕ॏ͍ͱࣔͨ͠ͷ͸
for͕ॻ͔Ε͍ͯΔߦͩͬͨ

View Slide

View Slide

౰֘forจͷasmΛݟͯΈΔͱ
ແବͳϝϞϦΞΫηε͕

View Slide

compilerʹregisterΛ
࢖ͬͯ΋Β͏ͨΊʹ
ϩʔΧϧม਺Λఆٛ

View Slide

View Slide

https://go-
review.googlesource.com/
c/go/+/187837

View Slide

View Slide

GoogleContainerTools/
kaniko

View Slide


View Slide

ΞΠσΟΞ͕εΩͩͬͨ

View Slide

kanikoΛCIͰ࢖ͬͯ
Կʹ͕͔͔͍࣌ؒͬͯΔͷ͔
ؾʹͳͬͨ

View Slide

kaniko͸ϝϞϦ্ʹ
ﬁlesystemͷsnapshotΛ࣋ͭ

View Slide

ίϚϯυΛ࣮ߦ͢Δͨͼʹ
ࠩ෼͕ͳ͍͔ൺֱ͢Δ

View Slide

md5Ͱൺֱ͢Δ

View Slide

ͦͷmd5͕ॏ͔ͬͨ

View Slide

ϑΝΠϧ͕
ಉҰ͔Ͳ͏͔͚ͩͰྑ͍ͷͰ
md5Ͱ͋Δඞཁ͸ͳ͍

View Slide

minio/HighwayHashʹมߋ
(ຊ౰͸Α͘ͳ͍)

View Slide

View Slide

mount͞Ε͍ͯΔ
σΟϨΫτϦ͸
snapshotର৅֎

View Slide

ϑΝΠϧ͕whitelistʹ
ؚ·ΕΔ൑ఆ͢ΔՕॴͰ
strings.SplitΛ࢖༻

View Slide

ϑΝΠϧ਺͕ଟ͔ͬͨΓ
directory͕ਂ͍ͱ
ແବʹϝϞϦΛ࢖༻͢Δ

View Slide

ಛੑΛߟ͑ͯ
strings.SplitNΛ࢖༻

View Slide

View Slide

https://github.com/
GoogleContainerTools/
kaniko/pull/694

View Slide

129.54s -> 88.29s

View Slide

View Slide

1. ࣮ࡍʹ౤͛ͨύον
2. ଎͍ϥΠϒϥϦ͕ॻ͖͍ͨ

View Slide

orisano/wyhash

View Slide

kanikoͷύονΛ
ॻ͍͍ͯΔͱ͖ʹ
Կ͕ྑ͍hashͳͷͩΖ͏

View Slide

Q. ଎͍hash͸?

View Slide

Q. ଎͍hash͸?
A. ܭଌ͠·͠ΐ͏

View Slide

dgryski/triﬂes/hashbench
खݩͰ૸ΒͤͯΈ·͠ΐ͏

View Slide

(ݟ͔ͭΒͳ͍package͕
͋ΔͷͰಈ͖·ͤΜ)

View Slide

wyhash͕
GitHubͷTrendingͰ
্͕͖ͬͯͨ

View Slide

଎ͯ͘ϙʔλϒϧͰڧ͍
Β͍͠

View Slide

ඇৗʹ୯७ͳͷͰ
GoʹҠ২ͯ͠ΈΑ͏ͱࢥͬͨ

View Slide

2೔ҐͰҠ২͕ऴྃ

View Slide

hashbenchʹ௥Ճ
֬ೝ͢ΔͱϘϩෛ͚͍ͯ͠Δ

View Slide

͜Μͳܭࢉ͔͠ͳ͍ॲཧΛ
Ͳ͏΍ͬͯߴ଎Խ͢Δͷ͔

View Slide

ྨࣅϥΠϒϥϦͷௐࠪ

View Slide

҉߸ܥ΍hashܥ͸
جຊతʹasm͕࢖ΘΕ͍ͯΔ

View Slide

asmΛ࢖͏ͱ଎͍?

View Slide

ॻ͍ͯΈΑ͏

View Slide

Go asm͸ಠಛͳײ͡
ॻ͍͍ͯΔ೔ຊਓ͕
΄ͱΜͲ͍ͳ͍?
͋·Γࢿྉ͕ͳ͍

View Slide

ؤுͬͯ
AVXΛ࢖ͬͯॏ͍ॲཧΛॻ͘

View Slide

lldbΛ࢖ͬͯ
bug(SEGV)Λमਖ਼͢Δ

View Slide

Benchmark݁Ռ
஗͘ͳ͍ͬͯΔ

View Slide

View Slide

Կނ͔

View Slide

asmͰॻ͍ͨؔ਺͸
inlineԽ͞Εͳ͍

View Slide

math/bits
encoding/binary
ίϯύΠϥ͕ݡ͘࠷దԽ͢Δ
https://dave.cheney.net/
2019/08/20/go-compiler-
intrinsics

View Slide

inlineԽ͞ΕΔΑ͏ͳ
খ͞ͳؔ਺͸
asmͷޮՌ͕ಘΒΕͳ͍

View Slide

ࠓճͷΑ͏ͳ৔߹ͩͱ
loop·ͰؚΊͯasmԽ͢΂͖

View Slide

ෆ׳ΕͳasmͰ
ଟ͘ͷίʔυΛॻ͖ͨ͘ͳ͍

View Slide

mmcloughlin/avo
Λ࢖͓͏

View Slide

View Slide

GoͰasmΛੜ੒͢Δ
ϓϩάϥϜΛॻ͘Ξϓϩʔν

View Slide

Կ͕ྑ͍͔?

View Slide

Go asmͷ͓࡞๏Λ
avo͕΍ͬͯ͘ΕΔ

View Slide

Go IDEͰͷิ׬͕ޮ͘

View Slide

avoΛ࢖ͬͯؤுͬͨ
5 GB/s -> 11 GB/s

View Slide

ߴ଎ͳasmΛॻ͘ͷ͕೉͍͠

View Slide

asmϨϕϧͰͳͥ஗͍ͷ͔
pprofͰ͸Θ͔Βͳ͍

View Slide

ύΠϓϥΠχϯάΛҙࣝ͢Δ
11 GB/s -> 14 GB/s

View Slide

·ͱΊ

View Slide

• 0. ෆຬͷϋʔυϧΛԼ͛Δ
• 1. ܭଌͷϋʔυϧΛԼ͛Δ
• 2. ໰୊ՕॴΛಛఆ͢Δ
• 3. BenchmarkΛॻ͘

View Slide

• 4. ղܾࡦΛߟ͑Δ
• Ϛʔδ͞Ε΍͍͢ղܾࡦΛ໨ࢦ͢
• ϝϯςφϯείετΛ্͛ͳ͍΋ͷ
• 5. ࢼߦࡨޡ͢Δ
• 6. ύονΛૹΔ

View Slide

Ҿ͖ग़͠Λ૿΍͢
• ϝϞϦ࢖༻ྔ͸໰୊ʹͳΓ͕ͪ
• ֎෦͔ΒBuffer͕ड͚औΕΔAPIΛߟྀ͢Δ
• มߋ͕༰қͰ͸ͳ͍৔߹sync.PoolΛߟྀ͢Δ

View Slide

Ҿ͖ग़͠Λ૿΍͢
• ద੾ͳhashΛબ୒͢Δ
• Ξηϯϒϥ͸ϝϯςφϯείετ͕ߴ͍ͷͰ
ۃྗආ͚Δ
• ॏ͍ॲཧͷ෦෼͚ͩσʔλͷ࣋ͪํΛม͑ͯ
ΈΔ

View Slide

OSS Performance Tuning Tips

OSS Performance Tuning Tips

orisano

More Decks by orisano

Other Decks in Programming

Featured

Transcript