Optimization for Number of goroutines Using Feedback Control

Yusuke MIYAKE / Pepabo R&D Institute, GMO Pepabo, Inc.
2019.07.25 GopherCon 2019
Optimization for number of goroutines
using feedback control

View Slide

1SJODJQBMFOHJOFFS
:VTVLF.*:",&!NPOPDISPNFHBOF
1FQBCP3%*OTUJUVUF (.01FQBCP *OD
IUUQTCMPHNPOPDISPNFHBOFDPN

View Slide

Go and I
3
• A Japanese gopher who loves writing OSS
using Go.
• monochromegane/the_platinum_searcher
• An organizer of the local Go community in
Fukuoka, Japan.
• Fukuoka.go (https://fukuokago.dev)
• Held Go Conference’19 summer in Fukuoka
on 7/13.
• https://fukuoka.gocon.jp 5IF(PHPQIFSXBTEFTJHOFECZ3FOFF'SFODI
5IFHPQIFSJMMVTUSBUJPOXBTESBXOCZ,FJUB,BXBNPUP
-JDFOTFEVOEFSUIF$SFBUJWF$PNNPOT"UUSJCVUJPOTMJDFOTF

View Slide

1. Introduction
2. Background
3. Proposal
4. Evaluation
5. Conclusion
4
Agenda

View Slide

1.
Introduction

View Slide

How many is
the optimal
number of goroutines?

View Slide

Performance tuning in case of pt
7
'JOE
(PSPVUJOFT
$IBO $IBO
4FBSDI .BUDI
QU
1SJOU
ʜ
HPSPVUJOFT /VN$16

-JOVY $16 (0."9130$4

View Slide

8
'JOE
(PSPVUJOFT
$IBO $IBO
4FBSDI .BUDI
QU
1SJOU
ʜ
3FBMUJNFWJSVTTDBOQSPDFTT
6OTUBCMF
HPSPVUJOFT /VN$16

NBD04 $16 (0."9130$4

View Slide

9
'JOE
(PSPVUJOFT
$IBO $IBO
4FBSDI .BUDI
QU
1SJOU
ʜ
3FBMUJNFWJSVTTDBOQSPDFTT
NBD04 $16 (0."9130$4
❌
HPSPVUJOFT /VN$16

View Slide

10
• I chose stability rather than speed…

View Slide

How many is
the optimal
number of goroutines
in each case?

View Slide

2.
Background

View Slide

1. Concurrency and complexity
2. Concurrency and Go
3. Concurrency and application

View Slide

• Concurrency brings our application good performance and complexity.
• We design a task to have concurrency and request runtime to run them in
parallel.
14
Concurrency and complexity
%FTJHO *NQMFNFOUBUJPO 3VOUJNF
$PODVSSFOU 1BSBMMFM
w 5ISFBENBOBHFNFOU
w 3BDFDPOEJUJPOT
w .FNPSZBDDFTTTZOD
w %FBEMPDLT
w 4UBSWBUJPO
w (SBOVMBSJUZ
w /VNCFSPG
DPODVSSFODZ

View Slide


View Slide

Concurrency and Go
16
• Go hides the complexity by smart runtime scheduler and features.
3VOUJNFTDIFEVMFS
$IBOOFM
$PODVSSFOU 1BSBMMFM
w 5ISFBENBOBHFNFOU
w (SBOVMBSJUZ
w /VNCFSPG
DPODVSSFODZ
w 3BDFDPOEJUJPOT
w .FNPSZBDDFTTTZOD
w %FBEMPDLT
w 4UBSWBUJPO

View Slide

Runtime scheduler of Go
17
(
.
1 ( (
-PDBM3VO2VFVF
(MPCBM3VO2VFVF
( (
SVOUJNFTZTNPO
.
TDIFEVMF
FYFDVUF
(GO
HPFYJU
• G: Goroutine
• M: OS thread
• P: Processor (Scheduling context)
• It provides M:N Scheduler (Some
goroutines:Some threads).
• It has local run queue.
• We can think of Goroutines as
application-level threads. 
OS thread behaves like a worker for
goroutine using a run queue.
3FGIUUQTTQFBLFSEFDLDPNSFUFSWJTJPOHPSVOUJNFTDIFEVMFS

View Slide

New “g”
18
YD
YD
YD
TUBDLHVBSE
)JHI
-PX
YDE
TIFETQ
YDF
TULUPQTQ
Stack (2048bytes)
TULUPQ
4UBDL
4UBDLHVBSE

Sched (type gobuf)
type gobuf struct {
sp uintptr // 824633919448 -> 0xc0000307d8
pc uintptr // 4359616 -> 0x4285c0
g guintptr // newg
ctxt unsafe.Pointer // 0x4c96c0
ret sys.Uintreg. // 0
lr uintptr // 0
bp uintptr // 0
}
• Runtime allocates 2kB stack to “g”.
• “g” has gobuf named “sched”.
• “sched” has goroutines state. (e.g.: stack pointer,
program counter…)

View Slide


View Slide

Concurrency and application
20
• The design for the number of concurrency is important to achieve both
speed and stability.
• Running many tasks tend to cause starvation.
$PODVSSFOU 1BSBMMFM
w 5ISFBENBOBHFNFOU
w (SBOVMBSJUZ
w /VNCFSPG
DPODVSSFODZ
w 3BDFDPOEJUJPOT
w .FNPSZBDDFTTTZOD
w %FBEMPDLT
w 4UBSWBUJPO

View Slide

Concurrency and application
21
• The design for the number of concurrency is difﬁcult.
• The optimal number depends on the app, environments and load
condition.
• The environment in which the program is tuned and the environment in
which it will be executed is different.
• It's desirable for the number of concurrency to be determined dynamically
and be controlled rapidly by detection the bottleneck on running program.

View Slide

3.
Proposal

View Slide

Goal
23
• The optimal number of concurrency to be determined dynamically and be
controlled rapidly.

View Slide

Basic idea
24
1. Increase the number of goroutines to performance target
1FSGPSNBODF

/VNCFSPGHPSPVUJOFT

4UFQ

(PSPVUJOFT 1FSGPSNBODF 1FSGPSNBODFUBSHFU

View Slide

Basic idea
25
2. Stop increasing the number of goroutines if it meets the performance target
1FSGPSNBODF

/VNCFSPGHPSPVUJOFT

4UFQ


View Slide

Basic idea
26
2. Stop increasing the number of goroutines if it meets the performance target
3. Attempt to decrease the number of goroutines
1FSGPSNBODF

/VNCFSPGHPSPVUJOFT

4UFQ


View Slide

Issues to solve for the realization
27
1. Selection of performance metrics
2. Finding how to control rapidly and continuously

View Slide

Performance metrics

View Slide

Performance metrics
29
• Independent resource type which task use.
• e.g.: CPU usage, Throughput.
• Go's scheduler turns blocking tasks into as CPU bound as possible by
switching tasks continuously.

View Slide

Performance metrics
30
• The metrics upper limit is calculated on running.
1. Set the target value high
1FSGPSNBODF

/VNCFSPGHPSPVUJOFT

4UFQ


View Slide

Performance metrics
31
• The metrics upper limit is calculated on running.
1. Set the target value high
2. gradually adjust it to the upper limit value.
1FSGPSNBODF

/VNCFSPGHPSPVUJOFT

4UFQ


View Slide

Determining

View Slide

Determine the number of goroutines
33
• Determine the number of goroutines based on the metrics
• Continuously
• Rapidly
• Accurately
• Feedback control meets these conditions.

View Slide

• Tracking a given set-point using errors from the set-point.
• Applying an automatic correction continuously.
• Identifying suitable set-point and input/output is important.
• Dealing with the system as a black box.
Feedback control
$POUSPMMFS 4ZTUFN
*OQVU 0VUQVU
&SSPS
'FFECBDLQBUI
4FUQPJOU

View Slide

• Applying a correction rapidly and accurately.
• The output is a combination of its {Proportional, Integral, Derivative}
control.
35
PID Controller (1/2)
$POUSPMMFS
&SSPS
4FUQPJOU 1 Kpe(t)
AAACmHichVG7SgNBFL1ZXzE+ErURbYIhEi3CrAiKVdBCxSYP85AYwu46iUv2xe4kEIM/YGclmErBQiz8Axsbf8AinyApI9hYePcBosF4l9m5c+aeM/dwRUORLUZIx8cNDY+MjvnHAxOTU9PB0MxsztLrpkSzkq7oZkEULKrIGs0ymSm0YJhUUEWF5sXajn2fb1DTknXtkDUNWlKFqiZXZElgCOUPygaNsZVyKELixIlwf8J7SQS8SOqhJziGE9BBgjqoQEEDhrkCAlj4FYEHAgZiJWghZmImO/cUziGA3DpWUawQEK3hv4qnoodqeLY1LYct4SsKLhOZYYiSV3JPeuSFPJA38vmnVsvRsHtp4i66XGqUgxfzmY9/WSruDE6/WQMYIla7nXXII/bVPVot5gd6ZFCBTcebjF4NB7FdS65O4+yql9lKR1vL5JZ00e8NKj+jstZ4l+5SNN3GcfG/h9Of5NbiPInzqfVIYtsbnB8WYQliOJ0NSMAeJCHr9HYJ19DmFrgEt8vtu6Wcz+PMwY/g0l/eDZev
* Ki
Z t
0
e(r)dr
AAACpnichVFNS+NQFD3Gz6lf1W4G3JQpSp1FuRFBcSW6EQakrVOrVM0k8amPpklIXgu1+Af8Ay5mMYwgKC78B7OZzfyBWfgTxKWCm1nMbRoQFfWGvHfeefecdy/X8h0ZKqLrLq27p7evf+BDYnBoeGQ0OTa+Hnr1wBYl23O8YMMyQ+FIV5SUVI7Y8ANh1ixHlK3qcvu+3BBBKD33q2r6Yrtm7rtyT9qmYspIpr4Ycku6yqAdlRbZYDq9GxjJDOUoivRLoMcggzjyXvIXtrALDzbqqEHAhWLswETIXwU6CD5z22gxFzCS0b3AERKsrXOW4AyT2Sqv+3yqxKzL57ZnGKltfsXhP2BlGpP0ly7ojv7QJd3Qv1e9WpFHu5Ym71ZHK3xj9Pjj2sO7qhrvCgePqjcUFmd3KrumK67rdvNzpfxmjwp7mI96k9yrHzHtru2OT+Pw5G5toTjZmqJTuuV+f7Lzb3Z2G/f2WUEUv/O49OfDeQnWZ3I65fTCbGZxKR7cACbwCVmezhwWsYI8SvxqEz9wjgstq61qJa3cSdW6Yk0KT0L79h9J15xt
% Kd
de(t)
dt
AAACp3ichVHLSsNAFD2Nr/quj4XgplgUdVEmIiiuim4EF9rWPqRKSdKphqZJSKYFDf0Bf8CFGxVcVBf+gRs3/oALP0FcKrhx4W0aEBX1hsycOXPPmXu5qm3ormDsMSR1dHZ194R7+/oHBoeGIyOjWdeqORrPaJZhOXlVcbmhmzwjdGHwvO1wpaoaPKdW1lr3uTp3XN0yt8Whzfeqyr6pl3VNEUQVI+MbxVJ0t+womlfis2Ku4ZVEoxiJsTjzI/oTyAGIIYgtK3KLXZRgQUMNVXCYEIQNKHDpK0AGg03cHjziHEK6f8/RQB9pa5TFKUMhtkLrPp0KAWvSueXp+mqNXjHod0gZxTR7YE32wu7ZNXti7796eb5Hq5ZD2tW2ltvF4eOJ9Nu/qirtAgefqj8UKmW3K3tkN1TX8858IfdnjwJlLPu96dSr7TOtrrW2T/3o5CW9kpr2ZtgFe6Z+z8n5jpzN+qt2meSpUxqX/H04P0F2IS6zuJxcjCVWg8GFMYkpzNJ0lpDAOraQoVePcIYmrqQ5aVPKSvl2qhQKNGP4EpLyAfEdnZo=

r
AAACknichVG7SgNBFD1ZXzE+Eh+FYCMGRSzCrAiKVdTGwsIk5iFRZHcddchmd9mdBDT4A9oqFlYKFmLhH9jY+AMWfoKkjGBj4c1mQVTUu+zMmTP3nLmXqzum8CRjzyGlrb2jsyvcHenp7euPxgYGc55dcQ2eNWzTdgu65nFTWDwrhTR5wXG5VtZNntdLy837fJW7nrCtdXng8K2ytmeJXWFokqiUux2LswTzY+wnUAMQRxBrduwem9iBDQMVlMFhQRI2ocGjrwgVDA5xW6gR5xIS/j3HESKkrVAWpwyN2BKte3QqBqxF56an56sNesWk3yXlGCbYE7thDfbIbtkLe//Vq+Z7NGs5oF1vabmzHT0eybz9qyrTLrH/qfpDoVN2q7Jndkd11Temi/k/e5TYxbzfm6BeHZ9pdm20fKqH543MQnqiNsmuWJ36vSTnB3K2qq/GdYqnL2hc6vfh/AS5mYTKEmpqNp5cCgYXxijGMUXTmUMSK1hDll7lOMEpzpRhZUFZVJZbqUoo0AzhSyirH/1ElaE=
e
AAACknichVG7SgNBFD1Z3/EVH4VgIwZFLMKsCEoqNY2FRR7GKFFkdx11cLO77E4CMfgD2ioWVgoWYuEf2Nj4AxZ+glgq2Fh49wGiot5lZ86cuefMvVzdMYUnGXuMKU3NLa1t7R3xzq7unt5EX/+KZ1ddgxcN27TdVV3zuCksXpRCmnzVcblW0U1e0vcy/n2pxl1P2NayrDt8o6LtWGJbGJokKsc3E0mWYkGM/ARqBJKIImsnbrGOLdgwUEUFHBYkYRMaPPrKUMHgELeBBnEuIRHccxwgTtoqZXHK0Ijdo3WHTuWItejse3qB2qBXTPpdUo5gjD2wK/bC7tk1e2Lvv3o1Ag+/ljrteqjlzmbv4VDh7V9VhXaJ3U/VHwqdssPKHtkN1fW8Nlku/dmjxDZmg94E9eoEjN+1EfrU9k9fCun8WGOcXbBn6vecnO/I2aq9Gpc5nj+jcanfh/MTrEylVJZSc9PJuYVocO0YxigmaDozmMMisijSqxxHOMaJMqiklXklE6YqsUgzgC+hLH0A4O6VlA==

4ZTUFN

'FFECBDLQBUI
*OQVU 0VUQVU

View Slide

• Proportional control(P) output is proportional to the error.
• Integral control(I) output is proportional to the integral of the error over
time.
• Derivative control(D) output is proportional to the derivative of the error.
36
PID Controller (2/2)
$POUSPMMFS
&SSPS
4FUQPJOU 1 Kpe(t)
* Ki
Z t
0
e(r)dr
% Kd
de(t)
dt

r
e

4ZTUFN

'FFECBDLQBUI
*OQVU 0VUQVU

View Slide

37
Dynamic Target Controller
&SSPS
4FUQPJOU
1
*
%

r
e

(PBQQ

'FFECBDLQBUI
4FNBQIPSF $166TBHF
$IBOHFSBUF
• Having two nested control loops to change set-point on running.
• The outer loop sets new set-points of the inner loop when CPU usage
reaches the upper limit.
$POUSPMMFS

View Slide

Bounding

View Slide

Bounding concurrency using
buffered channel as semaphore
39
To bound concurrency in Go
func main() {
var wg sync.WaitGroup
sem := make(chan struct{}, 3)
for i := 0; i < 10; i++ {
wg.Add(1)
sem <- struct{}{}
go func() {
defer wg.Done()
defer func() { <-sem }()
task()
}()
}
wg.Wait()
close(sem)
}

View Slide

40
Elastic semaphore
DIBO
&MBTUJDTFNBQIPSF
$PVOUFS
XBJU

TJHOBM

JODSFNFOU-JNJU

$PVOUFS
$PVOUFS
8BJUJOH
/PUXBJUJOH

• wait() & signal() as same as
semaphore
• incrementLimit() changes
upper of semaphore variable
• Ensure atomicity of above
operations

View Slide

Kaburaya

View Slide

Architecture of kaburaya
42
• The optimal number of concurrency to be determined dynamically and be
controlled rapidly
• Use CPU usage upper limit as performance metrics.
• Use feedback control to determine the number of goroutines continuosly.
• Use elastic semaphore to bound concurrency.
• Kaburaya is “ధ໼” which is a Japanese arrow with a whistle.
• https://github.com/monochromegane/kaburaya

View Slide

• Kaburaya optimizes the number of goroutines by feedback control.
43
monochromegane/kaburaya
func main() {
sem := kaburaya.NewSem(100 * time.Millisecond)
var wg sync.WaitGroup
for i := 0; i < 10; i++ {
wg.Add(1)
sem.Wait()
go func() {
defer sem.Signal()
defer wg.Done()
task()
}()
}
wg.Wait()
sem.Stop()
}
Create semaphore and
specify time span for
update the semaphore and
CPU target.
Wait for decrement operation.
Increment operation.
Stop update the semaphore and CPU target.

View Slide

4.
Evaluation

View Slide

• Apply kaburaya to some tasks to evaluate efﬁcacy.
• Measure speedup, CPU usage, and limit of semaphore.
• Find good and not so good cases for the kaburaya controller.
45
Evaluation

View Slide

Task: pt_mac-scan
46
• pt runs on macOS with realtime virus scan
• CPU 8, Memory 16GB
• GOMAXPROCS=8
• Better concurrent are from 16 to 64.
• More degradation of performance due to
increased goroutines.

View Slide

Task: pt_mac-scan (Good)
47
• Kaburaya found good set points and adjust numbers within ideal range.
Duration: 50ms, Gain: 0.1, Change rate: ±0.3
Median

View Slide

Task: pt_mac-no-scan
48
• Runs pt on Mac without realtime virus
scan
• GOMAXPROCS=8
• Better concurrent is 4
• Less degradation of performance due
to increased goroutines.

View Slide

Task: pt_mac-no-scan (Good)
49
Median

View Slide

Task: pt_mac-no-scan (Bad)
50
• Find set points and adjust numbers within ideal range.
• Increase the number due to failure to reset the set point at end of period.
Median

View Slide

Task: pt_linux
51
• pt runs on Linux
• GOMAXPROCS=4

View Slide

Task: pt_linux (Not so bad)
52
• Increase the number due to failure to reset the set point at end of period.
Median

View Slide

Task: mem_4096000_10000
53
• Runs task uses memory.
• 4MB/task x 10,000
• GOMAXPROCS=4
• Better concurrent is 512.
• More degradation of performance
(Swap out) due to increased
goroutines.

View Slide

Task: mem_4096000_10000 (Good)
54
• Kaburaya found set points and adjust numbers within ideal range.
• Avoid starvation of resource
Median

View Slide

Task: mem_40960_1000000
55
• 40kB/task x 1,000,000
• GOMAXPROCS=4

View Slide

Task: mem_40960_1000000 (Good)
56
• Kaburaya found set points and adjust numbers within ideal range.
Median

View Slide

Task: mem_409600_100000
57
• 400kB/task x 100,000
• GOMAXPROCS=4
• Better concurrent is over 1024?

View Slide

Task: mem_409600_100000 (Not so good)
58
• Kaburaya found set points and adjust numbers.
• Does not reach the limit of performance metrics…
Median

View Slide

• Kaburaya ﬁnds set point and adjusts numbers within ideal range by
feedback control and elastic semaphore.
• Towards to improve the controller
• prediction accuracy of set point
• Optimal parameters (K{p,i,d} and range of change rate)
• Solve them in the related researches about feedback control as the next
step.
59
Experimental results

View Slide

5.
Conclusion

View Slide

Conclusion
61
• I proposed kaburaya to control the number of goroutines without
depending on platform, runtime and current load.
• Experimental results show possible and some issues of kaburaya.
• In particular improvement detecting performance upper limit of task is
important.
• In the future, I will also consider the application to auto-scaling of cloud
computing.
• Go

View Slide

Appendix

View Slide

• Implementation of Kaburaya
• https://github.com/monochromegane/kaburaya
• Study of Kaburaya (In Japanese)
• https://blog.monochromegane.com/blog/2018/11/25/wsa_3_kaburaya/
• https://gist.github.com/monochromegane/5e9fc94371a9b2385eedd38d98341e6b
• https://blog.monochromegane.com/blog/2019/04/14/wsa_4_ebira/
• Learn more goroutine
• https://speakerdeck.com/monochromegane/road-to-your-goroutines
63
Reference

View Slide

64
Go gopher
• The Go gopher was designed by Renee French.
• The gopher illustrations were drawn by Keita Kawamoto.
• Licensed under the Creative Commons 3.0 Attributions license.

View Slide

Thank you!

View Slide

View Slide

Optimization for Number of goroutines Using Feedback Control

Optimization for Number of goroutines Using Feedback Control

monochromegane

More Decks by monochromegane

Other Decks in Programming

Featured

Transcript