Slide 1
Slide 1 text
OSS
Performance Tuning
Tips
#gocon #gocon_hall GoCon 2019 Autumn
@orisano
Slide 2
Slide 2 text
త
̎ͭͷ͜ͱΛڞ༗͍ͨ͠
Slide 3
Slide 3 text
νϡʔχϯάΛͲ͏ਐΊΔͷ͔
Ҿ͖ग़͠Λ૿͢
Slide 4
Slide 4 text
νϡʔχϯάΛͲ͏ਐΊΔͷ͔
Ҿ͖ग़͠Λ૿͢
Slide 5
Slide 5 text
Ͳ͏ਐΊ͍ͯΔͷ͔
Slide 6
Slide 6 text
0. ෆຬͷϋʔυϧΛԼ͛Δ
Slide 7
Slide 7 text
ෆຬ͕
վળͷ͖͔͚ͬʹͳΓ·͢
Slide 8
Slide 8 text
ීஈ͍ͬͯΔ
ϥΠϒϥϦɺιϑτΣΞ͕
͍ͷͩͱࢥͬͯΈΔ
Slide 9
Slide 9 text
͖͔͚ͬͱ
ϞνϕʔγϣϯΛ
ࣗΒ࡞Γग़͠·͠ΐ͏
Slide 10
Slide 10 text
1. ܭଌͷϋʔυϧΛԼ͛Δ
Slide 11
Slide 11 text
͍ݪҼΛ
ؾܰʹ
ௐΒΕΔೳྗΛʹ͚ͭΔ
Slide 12
Slide 12 text
ख͕͔͔ؒΔߦҝΛ
ਓؒ͋·Γ͠ͳ͍
Slide 13
Slide 13 text
ͨ͘͞Μܦݧ͢Δ
Slide 14
Slide 14 text
ؾܰʹ
ௐࠪͰ͖ΔΑ͏ͳΈΛ
ೖΕΔ
Slide 15
Slide 15 text
͜ΕOSSΛެ։͢Δଆͷ
ڥมͰࢦఆͨ͠Γ
σϑΥϧτͰprofile͕औΕΔ
ΈΛఏڙ͢Δͱ
վળ͍͢͠/͞Ε͍͢
Slide 16
Slide 16 text
ʹૺ۰ͨ͠ਓ͕
ݪҼௐ͍ࠪ͢͠ͱخ͍͠
Slide 17
Slide 17 text
2. ՕॴΛಛఆ͢Δ
Slide 18
Slide 18 text
͍ͱײͯ͡
Օॴ͕ؔϨϕϧͰ
Θ͔Βͳ͍ͷ
cliΛ࣮ߦ͍ͯ͠Δͱ͖
Slide 19
Slide 19 text
mainʹ
github.com/pkg/profile
ΛՃͯ͠ΈΔ
Slide 20
Slide 20 text
No content
Slide 21
Slide 21 text
͜Ε͚ͩ
Slide 22
Slide 22 text
cliͷ࣮ߦ͕1ʹ
1࣌ؒҎ্͔͔Δ߹ͳͲ
net/http/pprofͷ΄͏͕
ྑ͍߹
Slide 23
Slide 23 text
profileΛݟͯ
ؔϨϕϧͰಛఆ͢Δ
Slide 24
Slide 24 text
pprofͷweb൛ʹ׳ΕΔ
ͪΌΜͱҙຯ͕Θ͔Δ
Slide 25
Slide 25 text
͍ΖΜͳݟํͰݟͯΈΔ
top, graph, flame-graph
source, disasm
Slide 26
Slide 26 text
3. BenchmarkΛॻ͘
Slide 27
Slide 27 text
͢Ͱʹॻ͍ͯ͋Δ߹
ϥοΩʔ
͘ͳ͍ͬͯͳ͍߹
͘ͳΔέʔεΛՃ͢Δ
Slide 28
Slide 28 text
BenchmarkΛॻ࣌͘
࠷దԽͰফ͍͑ͯͳ͍͔֬ೝ
Slide 29
Slide 29 text
BenchmarkΛॻ͘ͱ
؆୯ʹprofile͕औΕΔ
Slide 30
Slide 30 text
go test -cpuprofile
go test -memprofile
Λ֮͑Α͏
(go help testflagΛಡ͏)
Slide 31
Slide 31 text
4. ղܾࡦΛߟ͑Δ
Slide 32
Slide 32 text
Ͳ͏ղܾ͢Δ͔ʹ͍ͭͯ
“Ҿ͖ग़͠Λ૿͢”Ͱ
Slide 33
Slide 33 text
OSSͷ
ύϑΥʔϚϯενϡʔχϯά
Ϛʔδ͞Εͯྃ
Slide 34
Slide 34 text
Ͳ͏ΕϚʔδ͞ΕΔ
ύονʹͳΔ͔
Slide 35
Slide 35 text
Ϛʔδ͢Δͱ͍͏͜ͱ
૬ख͕ϝϯςφϯε͢Δ
ͱ͍͏͜ͱ
Slide 36
Slide 36 text
มߋ͕গͳ͍
૬खͷίετΛ૿͞ͳ͍
Slide 37
Slide 37 text
ഁյతมߋ͕ͳ͍
͘ΘΕ͍ͯΔͷ΄Ͳ
ॏཁͳͱ͜Ζ
Slide 38
Slide 38 text
ҟৗͳίετ͕ͳ͍
มߋ, อक͕͘͠ͳΔ࣮
(ΞηϯϒϥΛͬͨΓ)
Slide 39
Slide 39 text
ґଘؔΛՃ͢Δ߹
৻ॏʹબ͢Δ
ΞΫςΟϒʹϝϯςφϯε͞
Ε͍ͯΔͷ͔
ຊʹඞཁ͔?
Slide 40
Slide 40 text
͘͢͝ύϑΥʔϚϯε͕
վળ͢Δ
ܶతͳվળड͚ೖΕΒΕΔ
Slide 41
Slide 41 text
ςετ͕ॻ͍ͯ͋Δ
͢Ͱʹॻ͍ͯ͋Δ߹डཧ
͞Ε͍͢
Slide 42
Slide 42 text
5. ࢼߦࡨޡ͢Δ
Slide 43
Slide 43 text
·ͣॳظঢ়ଶͰ
ेͳճ
timeoutʹͳΒͳ͍Α͏ʹ
Benchmark݁ՌΛऔ͓ͬͯ͘
Slide 44
Slide 44 text
go test -bench .
-count=10
-timeout=30000s
| tee old.txt
Slide 45
Slide 45 text
ύονΛૹΔͱ͖ʹ
benchstatͷ݁ՌΛૹΔͷͰ
ϕʔεϥΠϯेͳճͰ
Slide 46
Slide 46 text
timeout
σϑΥϧτͩͱ10mͰ
ҙ֎ͱΦʔόʔ͢Δ
Slide 47
Slide 47 text
ղܾࡦΛࢥ͍͍ͭͨΒ
·ͣ1Benchmark
Slide 48
Slide 48 text
ے͕ྑͦ͞͏ͳΒ
ेͳճBenchmark
Slide 49
Slide 49 text
ଟ͘ࢼߦࡨޡ͍ͯ͠Δͱ
profileͱsourceͷ͕ؔ
Θ͔Βͳ͘ͳΔ
Slide 50
Slide 50 text
pprofʹ
sourceΛݟΔػೳ͕͋Δ͕
อ͍࣋ͯ͠ΔΘ͚Ͱͳ͍
pathͷΈ
Slide 51
Slide 51 text
git add .
git commit -m “$2”
REV=$(git rev-parse HEAD)
go test -bench $1 -benchmem -cpuprofile
cpu.${REV}.pb.gz -memprofile mem.$
{REV}.pb.gz | tee ${REV}.txt
rm ./${REV}.* && git reset HEAD^
Slide 52
Slide 52 text
sourcegitʹཧͯ͠Β͏
ͪΌΜͱඥ͚ΒΕΔ
Slide 53
Slide 53 text
ଟ͘ͷprofile͕͋Δͱ
୯ମͷޮՌ͕Θ͔Βͳ͍
͜ͱ͕͋Δ
Slide 54
Slide 54 text
go tool pprof -diff_base
Ͱprofileؒͷ͕ࠩݟΕΔ
Slide 55
Slide 55 text
6. ύονΛૹΔ
Slide 56
Slide 56 text
࠷ऴͷBenchmarkΛ
ेͳճߦ͏
Slide 57
Slide 57 text
ઐ༻ͷΠϯελϯεͰܭଌ
͍͠߹
ۃྗ֎෦ϓϩηεΛఀࢭ͢Δ
Slide 58
Slide 58 text
νϡʔχϯάΛͲ͏ਐΊΔͷ͔
Ҿ͖ग़͠Λ૿͢
Slide 59
Slide 59 text
࣮ྫϕʔεͰ
ͲͷΑ͏ʹղܾ͔ͨ͠
ࣗͳΒͲ͏ղܾ͢Δ͔
ߟ͑ͯΒ͍͍ͨ
Slide 60
Slide 60 text
1. ࣮ࡍʹ͛ͨύον
2. ͍ϥΠϒϥϦ͕ॻ͖͍ͨ
Slide 61
Slide 61 text
1. ࣮ࡍʹ͛ͨύον
2. ͍ϥΠϒϥϦ͕ॻ͖͍ͨ
Slide 62
Slide 62 text
src-d/go-git
Slide 63
Slide 63 text
pure-goͰ
gitͷૢ࡞Λ͢ΔͨΊͷ
ϥΠϒϥϦ
Slide 64
Slide 64 text
ύονΛૹͬͨഎܠ
Slide 65
Slide 65 text
aquasecurity/trivy
Ͱॳ͍͕ͬͯͨ
େ͖ͳrepositoryͷcloneʹ
ҟৗͳ࣌ؒ(10min~)͕͔͔Δ
͕͋ͬͨ
Slide 66
Slide 66 text
ݪҼ
gitͷIndexͷߏஙΛߦ͏Օॴ
ͩͬͨ
Slide 67
Slide 67 text
Indexͷ෦දݱ(public)͕
sliceʹͳ͓ͬͯΓ
nameͰҰҙʹ͢ΔͨΊʹ
deleteͱappend͕
ϑΝΠϧճ࣮ߦ͞Ε͍ͯͨ
Slide 68
Slide 68 text
deletenameΛࢦఆͯ͠
আ͢ΔͷͰ
sliceͷཁૉΛͯࠪ͢͢Δ
Slide 69
Slide 69 text
ॳ
෦දݱΛmapʹมߋ͢Ε
deleteͷܭࢉྔΛݮΒͤΔ
ͷͰͦ͏͠Α͏ͱࢥͬͨ
Slide 70
Slide 70 text
͔͠͠
publicͳϑΟʔϧυ
internalͰͳ͍ύοέʔδ
4500 star͑ͷ໊
Slide 71
Slide 71 text
౸ఈड͚ೖΕΒΕͳ͍
มߋͩͱࢥ͍ఘΊͨ
Slide 72
Slide 72 text
ޙ
Slide 73
Slide 73 text
ιʔείʔυΛ
ݟ͍ͯ͠Δͱ
Slide 74
Slide 74 text
໌ه͞Ε͍ͯΔ
ϑΟʔϧυͷ༷
(sliceͷॱংอূ͠ͳ͍)
େྔͷIndexΛߏங͢Δ
λΠϛϯά(clone࣌)
Slide 75
Slide 75 text
ݟͨ݁͠Ռ
ഁյతมߋແ͠Ͱ
मਖ਼͢Δํ๏Λࢥ͍͍ͭͨ
Slide 76
Slide 76 text
ଟ͘ݺͼग़͞ΕΔՕॴ
ͷΈʹ͓͍ͯ
mapͰอ࣋͠return͢Δࡍʹ
sliceʹม͢Δ
Slide 77
Slide 77 text
mapΛprivateͳؔͰ
Ҿ͖ճ͢Α͏ʹ͢Δ͜ͱͰ
ഁյతมߋΛճආͰ͖ͨ
Slide 78
Slide 78 text
No content
Slide 79
Slide 79 text
ͦ͜Λղܾ͢Δͱ
mallocgc͕ॏ͘ͳͬͨ
Slide 80
Slide 80 text
ϝϞϦ༻ྔͷݪҼ
io.Copyʹ͋ΔΑ͏ͩͬͨ
Slide 81
Slide 81 text
ϑΝΠϧ͝ͱʹio.Copy͕
ݺΕ͍ͯͨ
Slide 82
Slide 82 text
େྔʹݺΕΔՄೳੑ͕͋Δ
ՕॴͰio.CopyͰͳ͘
io.CopyBufferΛ͏ͱ
ϝϞϦ༻ྔΛ੍ޚͰ͖Δ
Slide 83
Slide 83 text
͏bufferΛͲ͏ࢦఆ͢Δ͔
֎෦͔Β༩͑ΒΕͳ͍ͱ
݁ہϝϞϦ༻ྔมΘΒͳ͍
Slide 84
Slide 84 text
privateͳϝιουͱ͍͑
ҾʹՃ͢Δͱ
มߋൣғ͕େ͖͘ͳΔ
Slide 85
Slide 85 text
ղܾࡦͱͯ͠
globalʹsync.PoolΛ
ஔ͘͜ͱʹͨ͠
Slide 86
Slide 86 text
No content
Slide 87
Slide 87 text
https://github.com/src-d/
go-git/pull/1179
Slide 88
Slide 88 text
No content
Slide 89
Slide 89 text
605ඵ -> 249ඵ
Slide 90
Slide 90 text
͔͠͠
·ͩ·ͩϝϞϦ༻ྔ͕ଟ͍
56 GB/op
Slide 91
Slide 91 text
ϑΝΠϧʹࠩΛద༻͢Δ
ॲཧ͕ϝϞϦΛେྔʹ༻
Slide 92
Slide 92 text
ؔ෦Ͱ
publicͷؔͰ
buffer͕֎෦͔Βͤͳ͍
ͷ͕ΘΕ͍ͯͨ
Slide 93
Slide 93 text
গͳ͘ͱ෦͔Βͷ༻
ͰbufferΛࢦఆ͍ͨ͠
Slide 94
Slide 94 text
No content
Slide 95
Slide 95 text
෦͚ʹbufferࢦఆͰ͖Δ
Α͏ʹͯ͠ղܾ
Slide 96
Slide 96 text
https://github.com/src-d/
go-git/pull/1180
Slide 97
Slide 97 text
56.1 GB -> 29.8 GB
Slide 98
Slide 98 text
No content
Slide 99
Slide 99 text
No content
Slide 100
Slide 100 text
image/png
Slide 101
Slide 101 text
ύονΛૹͬͨഎܠ
Slide 102
Slide 102 text
ࣾISUCONͰ
QRίʔυΛߴʹੜ͢Δ
ඞཁ͕͋ͬͨͨΊ
Slide 103
Slide 103 text
ࣾISUCONͷৼΓฦΓͰ
ύϑΥʔϚϯενϡʔχϯά
ΛҰਓͰָ͠ΜͰ͍ͨ
Slide 104
Slide 104 text
ऄ
Slide 105
Slide 105 text
ߴʹେྔͷpngΛ
ग़ྗ͢ΔͨΊʹ
Slide 106
Slide 106 text
1.9͔Βೖͬͨ
png.EncoderBufferPool
Slide 107
Slide 107 text
CompressionLevelͷઃఆ
Slide 108
Slide 108 text
ಠࣗimage.Image࣮Λ
Θͳ͍
Slide 109
Slide 109 text
നࠇը૾Ͱ͋Ε
image.GrayΛ͏
(Opaque͕bypassͰ͖Δ)
Slide 110
Slide 110 text
ऄऴྃ
Slide 111
Slide 111 text
image/png͕
bottleneckʹͳ͖ͬͯͨ
Slide 112
Slide 112 text
͢ͰʹBenchmark͕
ॻ͔Ε͍ͯͨͷͰ࣮ߦ
Slide 113
Slide 113 text
ࠓճ͍ͬͯΔ
ՕॴͰͳ͍͕࿐ࠎʹ͍
෦͕͋ͬͨ
Slide 114
Slide 114 text
https://go-
review.googlesource.com/
c/go/+/187417
Slide 115
Slide 115 text
No content
Slide 116
Slide 116 text
࣮ࡍॏ͔ͬͨՕॴ
compress/deflate
Slide 117
Slide 117 text
pprof͕ॏ͍ͱࣔͨ͠ͷ
for͕ॻ͔Ε͍ͯΔߦͩͬͨ
Slide 118
Slide 118 text
No content
Slide 119
Slide 119 text
֘forจͷasmΛݟͯΈΔͱ
ແବͳϝϞϦΞΫηε͕
Slide 120
Slide 120 text
compilerʹregisterΛ
ͬͯΒ͏ͨΊʹ
ϩʔΧϧมΛఆٛ
Slide 121
Slide 121 text
No content
Slide 122
Slide 122 text
https://go-
review.googlesource.com/
c/go/+/187837
Slide 123
Slide 123 text
No content
Slide 124
Slide 124 text
No content
Slide 125
Slide 125 text
GoogleContainerTools/
kaniko
Slide 126
Slide 126 text
ύονΛૹͬͨഎܠ
Slide 127
Slide 127 text
ΞΠσΟΞ͕εΩͩͬͨ
Slide 128
Slide 128 text
kanikoΛCIͰͬͯ
Կʹ͕͔͔͍࣌ؒͬͯΔͷ͔
ؾʹͳͬͨ
Slide 129
Slide 129 text
kanikoϝϞϦ্ʹ
filesystemͷsnapshotΛ࣋ͭ
Slide 130
Slide 130 text
ίϚϯυΛ࣮ߦ͢Δͨͼʹ
͕ࠩͳ͍͔ൺֱ͢Δ
Slide 131
Slide 131 text
md5Ͱൺֱ͢Δ
Slide 132
Slide 132 text
ͦͷmd5͕ॏ͔ͬͨ
Slide 133
Slide 133 text
ϑΝΠϧ͕
ಉҰ͔Ͳ͏͔͚ͩͰྑ͍ͷͰ
md5Ͱ͋Δඞཁͳ͍
Slide 134
Slide 134 text
minio/HighwayHashʹมߋ
(ຊΑ͘ͳ͍)
Slide 135
Slide 135 text
No content
Slide 136
Slide 136 text
No content
Slide 137
Slide 137 text
mount͞Ε͍ͯΔ
σΟϨΫτϦ
snapshotର֎
Slide 138
Slide 138 text
ϑΝΠϧ͕whitelistʹ
ؚ·ΕΔఆ͢ΔՕॴͰ
strings.SplitΛ༻
Slide 139
Slide 139 text
ϑΝΠϧ͕ଟ͔ͬͨΓ
directory͕ਂ͍ͱ
ແବʹϝϞϦΛ༻͢Δ
Slide 140
Slide 140 text
ಛੑΛߟ͑ͯ
strings.SplitNΛ༻
Slide 141
Slide 141 text
No content
Slide 142
Slide 142 text
No content
Slide 143
Slide 143 text
https://github.com/
GoogleContainerTools/
kaniko/pull/694
Slide 144
Slide 144 text
129.54s -> 88.29s
Slide 145
Slide 145 text
No content
Slide 146
Slide 146 text
1. ࣮ࡍʹ͛ͨύον
2. ͍ϥΠϒϥϦ͕ॻ͖͍ͨ
Slide 147
Slide 147 text
orisano/wyhash
Slide 148
Slide 148 text
kanikoͷύονΛ
ॻ͍͍ͯΔͱ͖ʹ
Կ͕ྑ͍hashͳͷͩΖ͏
Slide 149
Slide 149 text
Q. ͍hash?
Slide 150
Slide 150 text
Q. ͍hash?
A. ܭଌ͠·͠ΐ͏
Slide 151
Slide 151 text
dgryski/trifles/hashbench
खݩͰΒͤͯΈ·͠ΐ͏
Slide 152
Slide 152 text
(ݟ͔ͭΒͳ͍package͕
͋ΔͷͰಈ͖·ͤΜ)
Slide 153
Slide 153 text
wyhash͕
GitHubͷTrendingͰ
্͕͖ͬͯͨ
Slide 154
Slide 154 text
ͯ͘ϙʔλϒϧͰڧ͍
Β͍͠
Slide 155
Slide 155 text
ඇৗʹ୯७ͳͷͰ
GoʹҠ২ͯ͠ΈΑ͏ͱࢥͬͨ
Slide 156
Slide 156 text
2ҐͰҠ২͕ऴྃ
Slide 157
Slide 157 text
hashbenchʹՃ
֬ೝ͢ΔͱϘϩෛ͚͍ͯ͠Δ
Slide 158
Slide 158 text
͜Μͳܭࢉ͔͠ͳ͍ॲཧΛ
Ͳ͏ͬͯߴԽ͢Δͷ͔
Slide 159
Slide 159 text
ྨࣅϥΠϒϥϦͷௐࠪ
Slide 160
Slide 160 text
҉߸ܥhashܥ
جຊతʹasm͕ΘΕ͍ͯΔ
Slide 161
Slide 161 text
asmΛ͏ͱ͍?
Slide 162
Slide 162 text
ॻ͍ͯΈΑ͏
Slide 163
Slide 163 text
Go asmಠಛͳײ͡
ॻ͍͍ͯΔຊਓ͕
΄ͱΜͲ͍ͳ͍?
͋·Γࢿྉ͕ͳ͍
Slide 164
Slide 164 text
ؤுͬͯ
AVXΛͬͯॏ͍ॲཧΛॻ͘
Slide 165
Slide 165 text
lldbΛͬͯ
bug(SEGV)Λमਖ਼͢Δ
Slide 166
Slide 166 text
Benchmark݁Ռ
͘ͳ͍ͬͯΔ
Slide 167
Slide 167 text
No content
Slide 168
Slide 168 text
Կނ͔
Slide 169
Slide 169 text
asmͰॻ͍ͨؔ
inlineԽ͞Εͳ͍
Slide 170
Slide 170 text
math/bits
encoding/binary
ίϯύΠϥ͕ݡ͘࠷దԽ͢Δ
https://dave.cheney.net/
2019/08/20/go-compiler-
intrinsics
Slide 171
Slide 171 text
inlineԽ͞ΕΔΑ͏ͳ
খ͞ͳؔ
asmͷޮՌ͕ಘΒΕͳ͍
Slide 172
Slide 172 text
ࠓճͷΑ͏ͳ߹ͩͱ
loop·ͰؚΊͯasmԽ͖͢
Slide 173
Slide 173 text
ෆ׳ΕͳasmͰ
ଟ͘ͷίʔυΛॻ͖ͨ͘ͳ͍
Slide 174
Slide 174 text
mmcloughlin/avo
Λ͓͏
Slide 175
Slide 175 text
No content
Slide 176
Slide 176 text
GoͰasmΛੜ͢Δ
ϓϩάϥϜΛॻ͘Ξϓϩʔν
Slide 177
Slide 177 text
Կ͕ྑ͍͔?
Slide 178
Slide 178 text
Go asmͷ͓࡞๏Λ
avo͕ͬͯ͘ΕΔ
Slide 179
Slide 179 text
Go IDEͰͷิ͕ޮ͘
Slide 180
Slide 180 text
avoΛͬͯؤுͬͨ
5 GB/s -> 11 GB/s
Slide 181
Slide 181 text
ߴͳasmΛॻ͘ͷ͕͍͠
Slide 182
Slide 182 text
asmϨϕϧͰͳ͍ͥͷ͔
pprofͰΘ͔Βͳ͍
Slide 183
Slide 183 text
ύΠϓϥΠχϯάΛҙࣝ͢Δ
11 GB/s -> 14 GB/s
Slide 184
Slide 184 text
·ͱΊ
Slide 185
Slide 185 text
νϡʔχϯάΛͲ͏ਐΊΔͷ͔
• 0. ෆຬͷϋʔυϧΛԼ͛Δ
• 1. ܭଌͷϋʔυϧΛԼ͛Δ
• 2. ՕॴΛಛఆ͢Δ
• 3. BenchmarkΛॻ͘
Slide 186
Slide 186 text
νϡʔχϯάΛͲ͏ਐΊΔͷ͔
• 4. ղܾࡦΛߟ͑Δ
• Ϛʔδ͞Ε͍͢ղܾࡦΛࢦ͢
• ϝϯςφϯείετΛ্͛ͳ͍ͷ
• 5. ࢼߦࡨޡ͢Δ
• 6. ύονΛૹΔ
Slide 187
Slide 187 text
Ҿ͖ग़͠Λ૿͢
• ϝϞϦ༻ྔʹͳΓ͕ͪ
• ֎෦͔ΒBuffer͕ड͚औΕΔAPIΛߟྀ͢Δ
• มߋ͕༰қͰͳ͍߹sync.PoolΛߟྀ͢Δ
Slide 188
Slide 188 text
Ҿ͖ग़͠Λ૿͢
• దͳhashΛબ͢Δ
• Ξηϯϒϥϝϯςφϯείετ͕ߴ͍ͷͰ
ۃྗආ͚Δ
• ॏ͍ॲཧͷ෦͚ͩσʔλͷ࣋ͪํΛม͑ͯ
ΈΔ