Optimization for Number of goroutines Using Feedback Control

GopherCon 2019 - Marriott Marquis San Diego Marina, California
https://www.gophercon.com/

Optimization for Number of goroutines Using Feedback Control
https://www.gophercon.com/agenda/speakers/442434

The design for the number of concurrency is important to achieve both speed and stability. To give a good performance without depending on platform and load conditions, it’s desirable for the number to be dynamic and rapidly controlled. In this talk, I will propose an architecture to solve this by utilizing feedback control.

Transcript

1. Yusuke MIYAKE / Pepabo R&D Institute, GMO Pepabo, Inc. 2019.07.25 GopherCon 2019 Optimization for number of goroutines using feedback control
2. 1SJODJQBM
   FOHJOFFS :VTVLF
   .*:",&
   
   !NPOPDISPNFHBOF 1FQBCP
   3%
   *OTUJUVUF
   (.0
   1FQBCP
   *OD IUUQTCMPHNPOPDISPNFHBOFDPN
3. Go and I 3 • A Japanese gopher who loves writing OSS using Go. • monochromegane/the_platinum_searcher • An organizer of the local Go community in Fukuoka, Japan. • Fukuoka.go (https://fukuokago.dev) • Held Go Conference’19 summer in Fukuoka on 7/13. • https://fukuoka.gocon.jp 5IF
   (P
   HPQIFS
   XBT
   EFTJHOFE
   CZ
   3FOFF
   'SFODI
   5IF
   HPQIFS
   JMMVTUSBUJPO
   XBT
   ESBXO
   CZ
   ,FJUB
   ,BXBNPUP
   -JDFOTFE
   VOEFS
   UIF
   $SFBUJWF
   $PNNPOT
   
   "UUSJCVUJPOT
   MJDFOTF
4. 1. Introduction 2. Background 3. Proposal 4. Evaluation 5. Conclusion 4 Agenda
5. 1. Introduction
6. How many is the optimal number of goroutines?
7. Performance tuning in case of pt 7 'JOE (PSPVUJOFT $IBO $IBO 4FBSDI .BUDI QU 1SJOU ʜ 
   HPSPVUJOFT
   /VN$16  -JOVY
   $16
   (0."9130$4
8. Performance tuning in case of pt 8 'JOE (PSPVUJOFT $IBO $IBO 4FBSDI .BUDI QU 1SJOU ʜ 3FBMUJNF
   WJSVT
   TDBO
   QSPDFTT 6OTUBCMF 
   HPSPVUJOFT /VN$16  NBD04
   $16
   (0."9130$4
9. Performance tuning in case of pt 9 'JOE (PSPVUJOFT $IBO $IBO 4FBSDI .BUDI QU 1SJOU ʜ 3FBMUJNF
   WJSVT
   TDBO
   QSPDFTT NBD04
   $16
   (0."9130$4 ❌ 
   HPSPVUJOFT /VN$16
10. Performance tuning in case of pt 10 • I chose stability rather than speed…
11. How many is the optimal number of goroutines in each case?
12. 2. Background
13. 1. Concurrency and complexity 2. Concurrency and Go 3. Concurrency and application
14. • Concurrency brings our application good performance and complexity. • We design a task to have concurrency and request runtime to run them in parallel. 14 Concurrency and complexity %FTJHO *NQMFNFOUBUJPO 3VOUJNF $PODVSSFOU 1BSBMMFM w 5ISFBE
    NBOBHFNFOU w 3BDF
    DPOEJUJPOT
    w .FNPSZ
    BDDFTT
    TZOD
    w %FBEMPDLT
    w 4UBSWBUJPO w (SBOVMBSJUZ
    w /VNCFS
    PG
    DPODVSSFODZ
15. 1. Concurrency and complexity 2. Concurrency and Go 3. Concurrency and application
16. Concurrency and Go 16 • Go hides the complexity by smart runtime scheduler and features. 3VOUJNF
    TDIFEVMFS $IBOOFM %FTJHO *NQMFNFOUBUJPO 3VOUJNF $PODVSSFOU 1BSBMMFM w 5ISFBE
    NBOBHFNFOU w (SBOVMBSJUZ
    w /VNCFS
    PG
    DPODVSSFODZ w 3BDF
    DPOEJUJPOT
    w .FNPSZ
    BDDFTT
    TZOD
    w %FBEMPDLT
    w 4UBSWBUJPO
17. Runtime scheduler of Go 17 ( . 1 ( ( -PDBM
    3VO
    2VFVF (MPCBM
    3VO
    2VFVF ( ( SVOUJNFTZTNPO . TDIFEVMF FYFDVUF (GO HPFYJU • G: Goroutine • M: OS thread • P: Processor (Scheduling context) • It provides M:N Scheduler (Some goroutines:Some threads). • It has local run queue. • We can think of Goroutines as application-level threads.
 OS thread behaves like a worker for goroutine using a run queue. 3FG
    IUUQTTQFBLFSEFDLDPNSFUFSWJTJPOHPSVOUJNFTDIFEVMFS
18. New “g” 18 YD YD YD TUBDLHVBSE )JHI -PX YDE TIFETQ YDF TULUPQTQ Stack (2048bytes) TULUPQ 4UBDL 4UBDLHVBSE   Sched (type gobuf) type gobuf struct { sp uintptr // 824633919448 -> 0xc0000307d8 pc uintptr // 4359616 -> 0x4285c0 <runtime.main> g guintptr // newg ctxt unsafe.Pointer // 0x4c96c0 <runtime.mainPC> ret sys.Uintreg. // 0 lr uintptr // 0 bp uintptr // 0 } • Runtime allocates 2kB stack to “g”. • “g” has gobuf named “sched”. • “sched” has goroutines state. (e.g.: stack pointer, program counter…)
19. 1. Concurrency and complexity 2. Concurrency and Go 3. Concurrency and application
20. Concurrency and application 20 • The design for the number of concurrency is important to achieve both speed and stability. • Running many tasks tend to cause starvation. %FTJHO *NQMFNFOUBUJPO 3VOUJNF $PODVSSFOU 1BSBMMFM w 5ISFBE
    NBOBHFNFOU w (SBOVMBSJUZ
    w /VNCFS
    PG
    DPODVSSFODZ w 3BDF
    DPOEJUJPOT
    w .FNPSZ
    BDDFTT
    TZOD
    w %FBEMPDLT
    w 4UBSWBUJPO
21. Concurrency and application 21 • The design for the number of concurrency is difficult. • The optimal number depends on the app, environments and load condition. • The environment in which the program is tuned and the environment in which it will be executed is different. • It's desirable for the number of concurrency to be determined dynamically and be controlled rapidly by detection the bottleneck on running program.
22. 3. Proposal
23. Goal 23 • The optimal number of concurrency to be determined dynamically and be controlled rapidly.
24. Basic idea 24 1. Increase the number of goroutines to performance target 1FSGPSNBODF      /VNCFS
    PG
    HPSPVUJOFT      4UFQ           (PSPVUJOFT 1FSGPSNBODF 1FSGPSNBODF
    UBSHFU
25. Basic idea 25 1. Increase the number of goroutines to performance target 2. Stop increasing the number of goroutines if it meets the performance target 1FSGPSNBODF      /VNCFS
    PG
    HPSPVUJOFT      4UFQ           (PSPVUJOFT 1FSGPSNBODF 1FSGPSNBODF
    UBSHFU
26. Basic idea 26 1. Increase the number of goroutines to performance target 2. Stop increasing the number of goroutines if it meets the performance target 3. Attempt to decrease the number of goroutines 1FSGPSNBODF      /VNCFS
    PG
    HPSPVUJOFT      4UFQ           (PSPVUJOFT 1FSGPSNBODF 1FSGPSNBODF
    UBSHFU
27. Issues to solve for the realization 27 1. Selection of performance metrics 2. Finding how to control rapidly and continuously
28. Performance metrics
29. Performance metrics 29 • Independent resource type which task use. • e.g.: CPU usage, Throughput. • Go's scheduler turns blocking tasks into as CPU bound as possible by switching tasks continuously.
30. Performance metrics 30 • The metrics upper limit is calculated on running. 1. Set the target value high 1FSGPSNBODF       /VNCFS
    PG
    HPSPVUJOFT       4UFQ           (PSPVUJOFT 1FSGPSNBODF 1FSGPSNBODF
    UBSHFU
31. Performance metrics 31 • The metrics upper limit is calculated on running. 1. Set the target value high 2. gradually adjust it to the upper limit value. 1FSGPSNBODF       /VNCFS
    PG
    HPSPVUJOFT       4UFQ           (PSPVUJOFT 1FSGPSNBODF 1FSGPSNBODF
    UBSHFU
32. Determining
33. Determine the number of goroutines 33 • Determine the number of goroutines based on the metrics • Continuously • Rapidly • Accurately • Feedback control meets these conditions.
34. • Tracking a given set-point using errors from the set-point. • Applying an automatic correction continuously. • Identifying suitable set-point and input/output is important. • Dealing with the system as a black box. Feedback control $POUSPMMFS 4ZTUFN *OQVU 0VUQVU &SSPS 'FFECBDL
    QBUI 4FUQPJOU
35. • Applying a correction rapidly and accurately. • The output is a combination of its {Proportional, Integral, Derivative} control. 35 PID Controller (1/2) $POUSPMMFS &SSPS 4FUQPJOU 1 Kpe(t) <latexit sha1_base64="WSac9fWZ+QE538KWn4+gz2K9Jv4=">AAACmHichVG7SgNBFL1ZXzE+ErURbYIhEi3CrAiKVdBCxSYP85AYwu46iUv2xe4kEIM/YGclmErBQiz8Axsbf8AinyApI9hYePcBosF4l9m5c+aeM/dwRUORLUZIx8cNDY+MjvnHAxOTU9PB0MxsztLrpkSzkq7oZkEULKrIGs0ymSm0YJhUUEWF5sXajn2fb1DTknXtkDUNWlKFqiZXZElgCOUPygaNsZVyKELixIlwf8J7SQS8SOqhJziGE9BBgjqoQEEDhrkCAlj4FYEHAgZiJWghZmImO/cUziGA3DpWUawQEK3hv4qnoodqeLY1LYct4SsKLhOZYYiSV3JPeuSFPJA38vmnVsvRsHtp4i66XGqUgxfzmY9/WSruDE6/WQMYIla7nXXII/bVPVot5gd6ZFCBTcebjF4NB7FdS65O4+yql9lKR1vL5JZ00e8NKj+jstZ4l+5SNN3GcfG/h9Of5NbiPInzqfVIYtsbnB8WYQliOJ0NSMAeJCHr9HYJ19DmFrgEt8vtu6Wcz+PMwY/g0l/eDZev</latexit> <latexit sha1_base64="WSac9fWZ+QE538KWn4+gz2K9Jv4=">AAACmHichVG7SgNBFL1ZXzE+ErURbYIhEi3CrAiKVdBCxSYP85AYwu46iUv2xe4kEIM/YGclmErBQiz8Axsbf8AinyApI9hYePcBosF4l9m5c+aeM/dwRUORLUZIx8cNDY+MjvnHAxOTU9PB0MxsztLrpkSzkq7oZkEULKrIGs0ymSm0YJhUUEWF5sXajn2fb1DTknXtkDUNWlKFqiZXZElgCOUPygaNsZVyKELixIlwf8J7SQS8SOqhJziGE9BBgjqoQEEDhrkCAlj4FYEHAgZiJWghZmImO/cUziGA3DpWUawQEK3hv4qnoodqeLY1LYct4SsKLhOZYYiSV3JPeuSFPJA38vmnVsvRsHtp4i66XGqUgxfzmY9/WSruDE6/WQMYIla7nXXII/bVPVot5gd6ZFCBTcebjF4NB7FdS65O4+yql9lKR1vL5JZ00e8NKj+jstZ4l+5SNN3GcfG/h9Of5NbiPInzqfVIYtsbnB8WYQliOJ0NSMAeJCHr9HYJ19DmFrgEt8vtu6Wcz+PMwY/g0l/eDZev</latexit> <latexit sha1_base64="WSac9fWZ+QE538KWn4+gz2K9Jv4=">AAACmHichVG7SgNBFL1ZXzE+ErURbYIhEi3CrAiKVdBCxSYP85AYwu46iUv2xe4kEIM/YGclmErBQiz8Axsbf8AinyApI9hYePcBosF4l9m5c+aeM/dwRUORLUZIx8cNDY+MjvnHAxOTU9PB0MxsztLrpkSzkq7oZkEULKrIGs0ymSm0YJhUUEWF5sXajn2fb1DTknXtkDUNWlKFqiZXZElgCOUPygaNsZVyKELixIlwf8J7SQS8SOqhJziGE9BBgjqoQEEDhrkCAlj4FYEHAgZiJWghZmImO/cUziGA3DpWUawQEK3hv4qnoodqeLY1LYct4SsKLhOZYYiSV3JPeuSFPJA38vmnVsvRsHtp4i66XGqUgxfzmY9/WSruDE6/WQMYIla7nXXII/bVPVot5gd6ZFCBTcebjF4NB7FdS65O4+yql9lKR1vL5JZ00e8NKj+jstZ4l+5SNN3GcfG/h9Of5NbiPInzqfVIYtsbnB8WYQliOJ0NSMAeJCHr9HYJ19DmFrgEt8vtu6Wcz+PMwY/g0l/eDZev</latexit> <latexit sha1_base64="WSac9fWZ+QE538KWn4+gz2K9Jv4=">AAACmHichVG7SgNBFL1ZXzE+ErURbYIhEi3CrAiKVdBCxSYP85AYwu46iUv2xe4kEIM/YGclmErBQiz8Axsbf8AinyApI9hYePcBosF4l9m5c+aeM/dwRUORLUZIx8cNDY+MjvnHAxOTU9PB0MxsztLrpkSzkq7oZkEULKrIGs0ymSm0YJhUUEWF5sXajn2fb1DTknXtkDUNWlKFqiZXZElgCOUPygaNsZVyKELixIlwf8J7SQS8SOqhJziGE9BBgjqoQEEDhrkCAlj4FYEHAgZiJWghZmImO/cUziGA3DpWUawQEK3hv4qnoodqeLY1LYct4SsKLhOZYYiSV3JPeuSFPJA38vmnVsvRsHtp4i66XGqUgxfzmY9/WSruDE6/WQMYIla7nXXII/bVPVot5gd6ZFCBTcebjF4NB7FdS65O4+yql9lKR1vL5JZ00e8NKj+jstZ4l+5SNN3GcfG/h9Of5NbiPInzqfVIYtsbnB8WYQliOJ0NSMAeJCHr9HYJ19DmFrgEt8vtu6Wcz+PMwY/g0l/eDZev</latexit> * Ki Z t 0 e(r)dr <latexit sha1_base64="aDkxGx5q/sb9YJIQEipbOLXS4BA=">AAACpnichVFNS+NQFD3Gz6lf1W4G3JQpSp1FuRFBcSW6EQakrVOrVM0k8amPpklIXgu1+Af8Ay5mMYwgKC78B7OZzfyBWfgTxKWCm1nMbRoQFfWGvHfeefecdy/X8h0ZKqLrLq27p7evf+BDYnBoeGQ0OTa+Hnr1wBYl23O8YMMyQ+FIV5SUVI7Y8ANh1ixHlK3qcvu+3BBBKD33q2r6Yrtm7rtyT9qmYspIpr4Ycku6yqAdlRbZYDq9GxjJDOUoivRLoMcggzjyXvIXtrALDzbqqEHAhWLswETIXwU6CD5z22gxFzCS0b3AERKsrXOW4AyT2Sqv+3yqxKzL57ZnGKltfsXhP2BlGpP0ly7ojv7QJd3Qv1e9WpFHu5Ym71ZHK3xj9Pjj2sO7qhrvCgePqjcUFmd3KrumK67rdvNzpfxmjwp7mI96k9yrHzHtru2OT+Pw5G5toTjZmqJTuuV+f7Lzb3Z2G/f2WUEUv/O49OfDeQnWZ3I65fTCbGZxKR7cACbwCVmezhwWsYI8SvxqEz9wjgstq61qJa3cSdW6Yk0KT0L79h9J15xt</latexit> <latexit sha1_base64="aDkxGx5q/sb9YJIQEipbOLXS4BA=">AAACpnichVFNS+NQFD3Gz6lf1W4G3JQpSp1FuRFBcSW6EQakrVOrVM0k8amPpklIXgu1+Af8Ay5mMYwgKC78B7OZzfyBWfgTxKWCm1nMbRoQFfWGvHfeefecdy/X8h0ZKqLrLq27p7evf+BDYnBoeGQ0OTa+Hnr1wBYl23O8YMMyQ+FIV5SUVI7Y8ANh1ixHlK3qcvu+3BBBKD33q2r6Yrtm7rtyT9qmYspIpr4Ycku6yqAdlRbZYDq9GxjJDOUoivRLoMcggzjyXvIXtrALDzbqqEHAhWLswETIXwU6CD5z22gxFzCS0b3AERKsrXOW4AyT2Sqv+3yqxKzL57ZnGKltfsXhP2BlGpP0ly7ojv7QJd3Qv1e9WpFHu5Ym71ZHK3xj9Pjj2sO7qhrvCgePqjcUFmd3KrumK67rdvNzpfxmjwp7mI96k9yrHzHtru2OT+Pw5G5toTjZmqJTuuV+f7Lzb3Z2G/f2WUEUv/O49OfDeQnWZ3I65fTCbGZxKR7cACbwCVmezhwWsYI8SvxqEz9wjgstq61qJa3cSdW6Yk0KT0L79h9J15xt</latexit> <latexit sha1_base64="aDkxGx5q/sb9YJIQEipbOLXS4BA=">AAACpnichVFNS+NQFD3Gz6lf1W4G3JQpSp1FuRFBcSW6EQakrVOrVM0k8amPpklIXgu1+Af8Ay5mMYwgKC78B7OZzfyBWfgTxKWCm1nMbRoQFfWGvHfeefecdy/X8h0ZKqLrLq27p7evf+BDYnBoeGQ0OTa+Hnr1wBYl23O8YMMyQ+FIV5SUVI7Y8ANh1ixHlK3qcvu+3BBBKD33q2r6Yrtm7rtyT9qmYspIpr4Ycku6yqAdlRbZYDq9GxjJDOUoivRLoMcggzjyXvIXtrALDzbqqEHAhWLswETIXwU6CD5z22gxFzCS0b3AERKsrXOW4AyT2Sqv+3yqxKzL57ZnGKltfsXhP2BlGpP0ly7ojv7QJd3Qv1e9WpFHu5Ym71ZHK3xj9Pjj2sO7qhrvCgePqjcUFmd3KrumK67rdvNzpfxmjwp7mI96k9yrHzHtru2OT+Pw5G5toTjZmqJTuuV+f7Lzb3Z2G/f2WUEUv/O49OfDeQnWZ3I65fTCbGZxKR7cACbwCVmezhwWsYI8SvxqEz9wjgstq61qJa3cSdW6Yk0KT0L79h9J15xt</latexit> <latexit sha1_base64="aDkxGx5q/sb9YJIQEipbOLXS4BA=">AAACpnichVFNS+NQFD3Gz6lf1W4G3JQpSp1FuRFBcSW6EQakrVOrVM0k8amPpklIXgu1+Af8Ay5mMYwgKC78B7OZzfyBWfgTxKWCm1nMbRoQFfWGvHfeefecdy/X8h0ZKqLrLq27p7evf+BDYnBoeGQ0OTa+Hnr1wBYl23O8YMMyQ+FIV5SUVI7Y8ANh1ixHlK3qcvu+3BBBKD33q2r6Yrtm7rtyT9qmYspIpr4Ycku6yqAdlRbZYDq9GxjJDOUoivRLoMcggzjyXvIXtrALDzbqqEHAhWLswETIXwU6CD5z22gxFzCS0b3AERKsrXOW4AyT2Sqv+3yqxKzL57ZnGKltfsXhP2BlGpP0ly7ojv7QJd3Qv1e9WpFHu5Ym71ZHK3xj9Pjj2sO7qhrvCgePqjcUFmd3KrumK67rdvNzpfxmjwp7mI96k9yrHzHtru2OT+Pw5G5toTjZmqJTuuV+f7Lzb3Z2G/f2WUEUv/O49OfDeQnWZ3I65fTCbGZxKR7cACbwCVmezhwWsYI8SvxqEz9wjgstq61qJa3cSdW6Yk0KT0L79h9J15xt</latexit> % Kd de(t) dt <latexit sha1_base64="pK4ns8fNxecVndhwIulsD6E0PEo=">AAACp3ichVHLSsNAFD2Nr/quj4XgplgUdVEmIiiuim4EF9rWPqRKSdKphqZJSKYFDf0Bf8CFGxVcVBf+gRs3/oALP0FcKrhx4W0aEBX1hsycOXPPmXu5qm3ormDsMSR1dHZ194R7+/oHBoeGIyOjWdeqORrPaJZhOXlVcbmhmzwjdGHwvO1wpaoaPKdW1lr3uTp3XN0yt8Whzfeqyr6pl3VNEUQVI+MbxVJ0t+womlfis2Ku4ZVEoxiJsTjzI/oTyAGIIYgtK3KLXZRgQUMNVXCYEIQNKHDpK0AGg03cHjziHEK6f8/RQB9pa5TFKUMhtkLrPp0KAWvSueXp+mqNXjHod0gZxTR7YE32wu7ZNXti7796eb5Hq5ZD2tW2ltvF4eOJ9Nu/qirtAgefqj8UKmW3K3tkN1TX8858IfdnjwJlLPu96dSr7TOtrrW2T/3o5CW9kpr2ZtgFe6Z+z8n5jpzN+qt2meSpUxqX/H04P0F2IS6zuJxcjCVWg8GFMYkpzNJ0lpDAOraQoVePcIYmrqQ5aVPKSvl2qhQKNGP4EpLyAfEdnZo=</latexit> <latexit sha1_base64="pK4ns8fNxecVndhwIulsD6E0PEo=">AAACp3ichVHLSsNAFD2Nr/quj4XgplgUdVEmIiiuim4EF9rWPqRKSdKphqZJSKYFDf0Bf8CFGxVcVBf+gRs3/oALP0FcKrhx4W0aEBX1hsycOXPPmXu5qm3ormDsMSR1dHZ194R7+/oHBoeGIyOjWdeqORrPaJZhOXlVcbmhmzwjdGHwvO1wpaoaPKdW1lr3uTp3XN0yt8Whzfeqyr6pl3VNEUQVI+MbxVJ0t+womlfis2Ku4ZVEoxiJsTjzI/oTyAGIIYgtK3KLXZRgQUMNVXCYEIQNKHDpK0AGg03cHjziHEK6f8/RQB9pa5TFKUMhtkLrPp0KAWvSueXp+mqNXjHod0gZxTR7YE32wu7ZNXti7796eb5Hq5ZD2tW2ltvF4eOJ9Nu/qirtAgefqj8UKmW3K3tkN1TX8858IfdnjwJlLPu96dSr7TOtrrW2T/3o5CW9kpr2ZtgFe6Z+z8n5jpzN+qt2meSpUxqX/H04P0F2IS6zuJxcjCVWg8GFMYkpzNJ0lpDAOraQoVePcIYmrqQ5aVPKSvl2qhQKNGP4EpLyAfEdnZo=</latexit> <latexit sha1_base64="pK4ns8fNxecVndhwIulsD6E0PEo=">AAACp3ichVHLSsNAFD2Nr/quj4XgplgUdVEmIiiuim4EF9rWPqRKSdKphqZJSKYFDf0Bf8CFGxVcVBf+gRs3/oALP0FcKrhx4W0aEBX1hsycOXPPmXu5qm3ormDsMSR1dHZ194R7+/oHBoeGIyOjWdeqORrPaJZhOXlVcbmhmzwjdGHwvO1wpaoaPKdW1lr3uTp3XN0yt8Whzfeqyr6pl3VNEUQVI+MbxVJ0t+womlfis2Ku4ZVEoxiJsTjzI/oTyAGIIYgtK3KLXZRgQUMNVXCYEIQNKHDpK0AGg03cHjziHEK6f8/RQB9pa5TFKUMhtkLrPp0KAWvSueXp+mqNXjHod0gZxTR7YE32wu7ZNXti7796eb5Hq5ZD2tW2ltvF4eOJ9Nu/qirtAgefqj8UKmW3K3tkN1TX8858IfdnjwJlLPu96dSr7TOtrrW2T/3o5CW9kpr2ZtgFe6Z+z8n5jpzN+qt2meSpUxqX/H04P0F2IS6zuJxcjCVWg8GFMYkpzNJ0lpDAOraQoVePcIYmrqQ5aVPKSvl2qhQKNGP4EpLyAfEdnZo=</latexit> <latexit sha1_base64="pK4ns8fNxecVndhwIulsD6E0PEo=">AAACp3ichVHLSsNAFD2Nr/quj4XgplgUdVEmIiiuim4EF9rWPqRKSdKphqZJSKYFDf0Bf8CFGxVcVBf+gRs3/oALP0FcKrhx4W0aEBX1hsycOXPPmXu5qm3ormDsMSR1dHZ194R7+/oHBoeGIyOjWdeqORrPaJZhOXlVcbmhmzwjdGHwvO1wpaoaPKdW1lr3uTp3XN0yt8Whzfeqyr6pl3VNEUQVI+MbxVJ0t+womlfis2Ku4ZVEoxiJsTjzI/oTyAGIIYgtK3KLXZRgQUMNVXCYEIQNKHDpK0AGg03cHjziHEK6f8/RQB9pa5TFKUMhtkLrPp0KAWvSueXp+mqNXjHod0gZxTR7YE32wu7ZNXti7796eb5Hq5ZD2tW2ltvF4eOJ9Nu/qirtAgefqj8UKmW3K3tkN1TX8858IfdnjwJlLPu96dSr7TOtrrW2T/3o5CW9kpr2ZtgFe6Z+z8n5jpzN+qt2meSpUxqX/H04P0F2IS6zuJxcjCVWg8GFMYkpzNJ0lpDAOraQoVePcIYmrqQ5aVPKSvl2qhQKNGP4EpLyAfEdnZo=</latexit> r <latexit sha1_base64="l3XaGwPmE5Vm/Jhs1LG2xz0k9Bs=">AAACknichVG7SgNBFD1ZXzE+Eh+FYCMGRSzCrAiKVdTGwsIk5iFRZHcddchmd9mdBDT4A9oqFlYKFmLhH9jY+AMWfoKkjGBj4c1mQVTUu+zMmTP3nLmXqzum8CRjzyGlrb2jsyvcHenp7euPxgYGc55dcQ2eNWzTdgu65nFTWDwrhTR5wXG5VtZNntdLy837fJW7nrCtdXng8K2ytmeJXWFokqiUux2LswTzY+wnUAMQRxBrduwem9iBDQMVlMFhQRI2ocGjrwgVDA5xW6gR5xIS/j3HESKkrVAWpwyN2BKte3QqBqxF56an56sNesWk3yXlGCbYE7thDfbIbtkLe//Vq+Z7NGs5oF1vabmzHT0eybz9qyrTLrH/qfpDoVN2q7Jndkd11Temi/k/e5TYxbzfm6BeHZ9pdm20fKqH543MQnqiNsmuWJ36vSTnB3K2qq/GdYqnL2hc6vfh/AS5mYTKEmpqNp5cCgYXxijGMUXTmUMSK1hDll7lOMEpzpRhZUFZVJZbqUoo0AzhSyirH/1ElaE=</latexit> <latexit sha1_base64="l3XaGwPmE5Vm/Jhs1LG2xz0k9Bs=">AAACknichVG7SgNBFD1ZXzE+Eh+FYCMGRSzCrAiKVdTGwsIk5iFRZHcddchmd9mdBDT4A9oqFlYKFmLhH9jY+AMWfoKkjGBj4c1mQVTUu+zMmTP3nLmXqzum8CRjzyGlrb2jsyvcHenp7euPxgYGc55dcQ2eNWzTdgu65nFTWDwrhTR5wXG5VtZNntdLy837fJW7nrCtdXng8K2ytmeJXWFokqiUux2LswTzY+wnUAMQRxBrduwem9iBDQMVlMFhQRI2ocGjrwgVDA5xW6gR5xIS/j3HESKkrVAWpwyN2BKte3QqBqxF56an56sNesWk3yXlGCbYE7thDfbIbtkLe//Vq+Z7NGs5oF1vabmzHT0eybz9qyrTLrH/qfpDoVN2q7Jndkd11Temi/k/e5TYxbzfm6BeHZ9pdm20fKqH543MQnqiNsmuWJ36vSTnB3K2qq/GdYqnL2hc6vfh/AS5mYTKEmpqNp5cCgYXxijGMUXTmUMSK1hDll7lOMEpzpRhZUFZVJZbqUoo0AzhSyirH/1ElaE=</latexit> <latexit sha1_base64="l3XaGwPmE5Vm/Jhs1LG2xz0k9Bs=">AAACknichVG7SgNBFD1ZXzE+Eh+FYCMGRSzCrAiKVdTGwsIk5iFRZHcddchmd9mdBDT4A9oqFlYKFmLhH9jY+AMWfoKkjGBj4c1mQVTUu+zMmTP3nLmXqzum8CRjzyGlrb2jsyvcHenp7euPxgYGc55dcQ2eNWzTdgu65nFTWDwrhTR5wXG5VtZNntdLy837fJW7nrCtdXng8K2ytmeJXWFokqiUux2LswTzY+wnUAMQRxBrduwem9iBDQMVlMFhQRI2ocGjrwgVDA5xW6gR5xIS/j3HESKkrVAWpwyN2BKte3QqBqxF56an56sNesWk3yXlGCbYE7thDfbIbtkLe//Vq+Z7NGs5oF1vabmzHT0eybz9qyrTLrH/qfpDoVN2q7Jndkd11Temi/k/e5TYxbzfm6BeHZ9pdm20fKqH543MQnqiNsmuWJ36vSTnB3K2qq/GdYqnL2hc6vfh/AS5mYTKEmpqNp5cCgYXxijGMUXTmUMSK1hDll7lOMEpzpRhZUFZVJZbqUoo0AzhSyirH/1ElaE=</latexit> <latexit sha1_base64="l3XaGwPmE5Vm/Jhs1LG2xz0k9Bs=">AAACknichVG7SgNBFD1ZXzE+Eh+FYCMGRSzCrAiKVdTGwsIk5iFRZHcddchmd9mdBDT4A9oqFlYKFmLhH9jY+AMWfoKkjGBj4c1mQVTUu+zMmTP3nLmXqzum8CRjzyGlrb2jsyvcHenp7euPxgYGc55dcQ2eNWzTdgu65nFTWDwrhTR5wXG5VtZNntdLy837fJW7nrCtdXng8K2ytmeJXWFokqiUux2LswTzY+wnUAMQRxBrduwem9iBDQMVlMFhQRI2ocGjrwgVDA5xW6gR5xIS/j3HESKkrVAWpwyN2BKte3QqBqxF56an56sNesWk3yXlGCbYE7thDfbIbtkLe//Vq+Z7NGs5oF1vabmzHT0eybz9qyrTLrH/qfpDoVN2q7Jndkd11Temi/k/e5TYxbzfm6BeHZ9pdm20fKqH543MQnqiNsmuWJ36vSTnB3K2qq/GdYqnL2hc6vfh/AS5mYTKEmpqNp5cCgYXxijGMUXTmUMSK1hDll7lOMEpzpRhZUFZVJZbqUoo0AzhSyirH/1ElaE=</latexit> e <latexit sha1_base64="JucbY3V7iD4pyjirU4Kr9PWwJLM=">AAACknichVG7SgNBFD1Z3/EVH4VgIwZFLMKsCEoqNY2FRR7GKFFkdx11cLO77E4CMfgD2ioWVgoWYuEf2Nj4AxZ+glgq2Fh49wGiot5lZ86cuefMvVzdMYUnGXuMKU3NLa1t7R3xzq7unt5EX/+KZ1ddgxcN27TdVV3zuCksXpRCmnzVcblW0U1e0vcy/n2pxl1P2NayrDt8o6LtWGJbGJokKsc3E0mWYkGM/ARqBJKIImsnbrGOLdgwUEUFHBYkYRMaPPrKUMHgELeBBnEuIRHccxwgTtoqZXHK0Ijdo3WHTuWItejse3qB2qBXTPpdUo5gjD2wK/bC7tk1e2Lvv3o1Ag+/ljrteqjlzmbv4VDh7V9VhXaJ3U/VHwqdssPKHtkN1fW8Nlku/dmjxDZmg94E9eoEjN+1EfrU9k9fCun8WGOcXbBn6vecnO/I2aq9Gpc5nj+jcanfh/MTrEylVJZSc9PJuYVocO0YxigmaDozmMMisijSqxxHOMaJMqiklXklE6YqsUgzgC+hLH0A4O6VlA==</latexit> <latexit sha1_base64="JucbY3V7iD4pyjirU4Kr9PWwJLM=">AAACknichVG7SgNBFD1Z3/EVH4VgIwZFLMKsCEoqNY2FRR7GKFFkdx11cLO77E4CMfgD2ioWVgoWYuEf2Nj4AxZ+glgq2Fh49wGiot5lZ86cuefMvVzdMYUnGXuMKU3NLa1t7R3xzq7unt5EX/+KZ1ddgxcN27TdVV3zuCksXpRCmnzVcblW0U1e0vcy/n2pxl1P2NayrDt8o6LtWGJbGJokKsc3E0mWYkGM/ARqBJKIImsnbrGOLdgwUEUFHBYkYRMaPPrKUMHgELeBBnEuIRHccxwgTtoqZXHK0Ijdo3WHTuWItejse3qB2qBXTPpdUo5gjD2wK/bC7tk1e2Lvv3o1Ag+/ljrteqjlzmbv4VDh7V9VhXaJ3U/VHwqdssPKHtkN1fW8Nlku/dmjxDZmg94E9eoEjN+1EfrU9k9fCun8WGOcXbBn6vecnO/I2aq9Gpc5nj+jcanfh/MTrEylVJZSc9PJuYVocO0YxigmaDozmMMisijSqxxHOMaJMqiklXklE6YqsUgzgC+hLH0A4O6VlA==</latexit> <latexit sha1_base64="JucbY3V7iD4pyjirU4Kr9PWwJLM=">AAACknichVG7SgNBFD1Z3/EVH4VgIwZFLMKsCEoqNY2FRR7GKFFkdx11cLO77E4CMfgD2ioWVgoWYuEf2Nj4AxZ+glgq2Fh49wGiot5lZ86cuefMvVzdMYUnGXuMKU3NLa1t7R3xzq7unt5EX/+KZ1ddgxcN27TdVV3zuCksXpRCmnzVcblW0U1e0vcy/n2pxl1P2NayrDt8o6LtWGJbGJokKsc3E0mWYkGM/ARqBJKIImsnbrGOLdgwUEUFHBYkYRMaPPrKUMHgELeBBnEuIRHccxwgTtoqZXHK0Ijdo3WHTuWItejse3qB2qBXTPpdUo5gjD2wK/bC7tk1e2Lvv3o1Ag+/ljrteqjlzmbv4VDh7V9VhXaJ3U/VHwqdssPKHtkN1fW8Nlku/dmjxDZmg94E9eoEjN+1EfrU9k9fCun8WGOcXbBn6vecnO/I2aq9Gpc5nj+jcanfh/MTrEylVJZSc9PJuYVocO0YxigmaDozmMMisijSqxxHOMaJMqiklXklE6YqsUgzgC+hLH0A4O6VlA==</latexit> <latexit sha1_base64="JucbY3V7iD4pyjirU4Kr9PWwJLM=">AAACknichVG7SgNBFD1Z3/EVH4VgIwZFLMKsCEoqNY2FRR7GKFFkdx11cLO77E4CMfgD2ioWVgoWYuEf2Nj4AxZ+glgq2Fh49wGiot5lZ86cuefMvVzdMYUnGXuMKU3NLa1t7R3xzq7unt5EX/+KZ1ddgxcN27TdVV3zuCksXpRCmnzVcblW0U1e0vcy/n2pxl1P2NayrDt8o6LtWGJbGJokKsc3E0mWYkGM/ARqBJKIImsnbrGOLdgwUEUFHBYkYRMaPPrKUMHgELeBBnEuIRHccxwgTtoqZXHK0Ijdo3WHTuWItejse3qB2qBXTPpdUo5gjD2wK/bC7tk1e2Lvv3o1Ag+/ljrteqjlzmbv4VDh7V9VhXaJ3U/VHwqdssPKHtkN1fW8Nlku/dmjxDZmg94E9eoEjN+1EfrU9k9fCun8WGOcXbBn6vecnO/I2aq9Gpc5nj+jcanfh/MTrEylVJZSc9PJuYVocO0YxigmaDozmMMisijSqxxHOMaJMqiklXklE6YqsUgzgC+hLH0A4O6VlA==</latexit> 4ZTUFN  'FFECBDL
    QBUI *OQVU 0VUQVU
36. • Proportional control(P) output is proportional to the error. • Integral control(I) output is proportional to the integral of the error over time. • Derivative control(D) output is proportional to the derivative of the error. 36 PID Controller (2/2) $POUSPMMFS &SSPS 4FUQPJOU 1 Kpe(t) <latexit sha1_base64="WSac9fWZ+QE538KWn4+gz2K9Jv4=">AAACmHichVG7SgNBFL1ZXzE+ErURbYIhEi3CrAiKVdBCxSYP85AYwu46iUv2xe4kEIM/YGclmErBQiz8Axsbf8AinyApI9hYePcBosF4l9m5c+aeM/dwRUORLUZIx8cNDY+MjvnHAxOTU9PB0MxsztLrpkSzkq7oZkEULKrIGs0ymSm0YJhUUEWF5sXajn2fb1DTknXtkDUNWlKFqiZXZElgCOUPygaNsZVyKELixIlwf8J7SQS8SOqhJziGE9BBgjqoQEEDhrkCAlj4FYEHAgZiJWghZmImO/cUziGA3DpWUawQEK3hv4qnoodqeLY1LYct4SsKLhOZYYiSV3JPeuSFPJA38vmnVsvRsHtp4i66XGqUgxfzmY9/WSruDE6/WQMYIla7nXXII/bVPVot5gd6ZFCBTcebjF4NB7FdS65O4+yql9lKR1vL5JZ00e8NKj+jstZ4l+5SNN3GcfG/h9Of5NbiPInzqfVIYtsbnB8WYQliOJ0NSMAeJCHr9HYJ19DmFrgEt8vtu6Wcz+PMwY/g0l/eDZev</latexit> <latexit sha1_base64="WSac9fWZ+QE538KWn4+gz2K9Jv4=">AAACmHichVG7SgNBFL1ZXzE+ErURbYIhEi3CrAiKVdBCxSYP85AYwu46iUv2xe4kEIM/YGclmErBQiz8Axsbf8AinyApI9hYePcBosF4l9m5c+aeM/dwRUORLUZIx8cNDY+MjvnHAxOTU9PB0MxsztLrpkSzkq7oZkEULKrIGs0ymSm0YJhUUEWF5sXajn2fb1DTknXtkDUNWlKFqiZXZElgCOUPygaNsZVyKELixIlwf8J7SQS8SOqhJziGE9BBgjqoQEEDhrkCAlj4FYEHAgZiJWghZmImO/cUziGA3DpWUawQEK3hv4qnoodqeLY1LYct4SsKLhOZYYiSV3JPeuSFPJA38vmnVsvRsHtp4i66XGqUgxfzmY9/WSruDE6/WQMYIla7nXXII/bVPVot5gd6ZFCBTcebjF4NB7FdS65O4+yql9lKR1vL5JZ00e8NKj+jstZ4l+5SNN3GcfG/h9Of5NbiPInzqfVIYtsbnB8WYQliOJ0NSMAeJCHr9HYJ19DmFrgEt8vtu6Wcz+PMwY/g0l/eDZev</latexit> <latexit sha1_base64="WSac9fWZ+QE538KWn4+gz2K9Jv4=">AAACmHichVG7SgNBFL1ZXzE+ErURbYIhEi3CrAiKVdBCxSYP85AYwu46iUv2xe4kEIM/YGclmErBQiz8Axsbf8AinyApI9hYePcBosF4l9m5c+aeM/dwRUORLUZIx8cNDY+MjvnHAxOTU9PB0MxsztLrpkSzkq7oZkEULKrIGs0ymSm0YJhUUEWF5sXajn2fb1DTknXtkDUNWlKFqiZXZElgCOUPygaNsZVyKELixIlwf8J7SQS8SOqhJziGE9BBgjqoQEEDhrkCAlj4FYEHAgZiJWghZmImO/cUziGA3DpWUawQEK3hv4qnoodqeLY1LYct4SsKLhOZYYiSV3JPeuSFPJA38vmnVsvRsHtp4i66XGqUgxfzmY9/WSruDE6/WQMYIla7nXXII/bVPVot5gd6ZFCBTcebjF4NB7FdS65O4+yql9lKR1vL5JZ00e8NKj+jstZ4l+5SNN3GcfG/h9Of5NbiPInzqfVIYtsbnB8WYQliOJ0NSMAeJCHr9HYJ19DmFrgEt8vtu6Wcz+PMwY/g0l/eDZev</latexit> <latexit sha1_base64="WSac9fWZ+QE538KWn4+gz2K9Jv4=">AAACmHichVG7SgNBFL1ZXzE+ErURbYIhEi3CrAiKVdBCxSYP85AYwu46iUv2xe4kEIM/YGclmErBQiz8Axsbf8AinyApI9hYePcBosF4l9m5c+aeM/dwRUORLUZIx8cNDY+MjvnHAxOTU9PB0MxsztLrpkSzkq7oZkEULKrIGs0ymSm0YJhUUEWF5sXajn2fb1DTknXtkDUNWlKFqiZXZElgCOUPygaNsZVyKELixIlwf8J7SQS8SOqhJziGE9BBgjqoQEEDhrkCAlj4FYEHAgZiJWghZmImO/cUziGA3DpWUawQEK3hv4qnoodqeLY1LYct4SsKLhOZYYiSV3JPeuSFPJA38vmnVsvRsHtp4i66XGqUgxfzmY9/WSruDE6/WQMYIla7nXXII/bVPVot5gd6ZFCBTcebjF4NB7FdS65O4+yql9lKR1vL5JZ00e8NKj+jstZ4l+5SNN3GcfG/h9Of5NbiPInzqfVIYtsbnB8WYQliOJ0NSMAeJCHr9HYJ19DmFrgEt8vtu6Wcz+PMwY/g0l/eDZev</latexit> * Ki Z t 0 e(r)dr <latexit sha1_base64="aDkxGx5q/sb9YJIQEipbOLXS4BA=">AAACpnichVFNS+NQFD3Gz6lf1W4G3JQpSp1FuRFBcSW6EQakrVOrVM0k8amPpklIXgu1+Af8Ay5mMYwgKC78B7OZzfyBWfgTxKWCm1nMbRoQFfWGvHfeefecdy/X8h0ZKqLrLq27p7evf+BDYnBoeGQ0OTa+Hnr1wBYl23O8YMMyQ+FIV5SUVI7Y8ANh1ixHlK3qcvu+3BBBKD33q2r6Yrtm7rtyT9qmYspIpr4Ycku6yqAdlRbZYDq9GxjJDOUoivRLoMcggzjyXvIXtrALDzbqqEHAhWLswETIXwU6CD5z22gxFzCS0b3AERKsrXOW4AyT2Sqv+3yqxKzL57ZnGKltfsXhP2BlGpP0ly7ojv7QJd3Qv1e9WpFHu5Ym71ZHK3xj9Pjj2sO7qhrvCgePqjcUFmd3KrumK67rdvNzpfxmjwp7mI96k9yrHzHtru2OT+Pw5G5toTjZmqJTuuV+f7Lzb3Z2G/f2WUEUv/O49OfDeQnWZ3I65fTCbGZxKR7cACbwCVmezhwWsYI8SvxqEz9wjgstq61qJa3cSdW6Yk0KT0L79h9J15xt</latexit> <latexit sha1_base64="aDkxGx5q/sb9YJIQEipbOLXS4BA=">AAACpnichVFNS+NQFD3Gz6lf1W4G3JQpSp1FuRFBcSW6EQakrVOrVM0k8amPpklIXgu1+Af8Ay5mMYwgKC78B7OZzfyBWfgTxKWCm1nMbRoQFfWGvHfeefecdy/X8h0ZKqLrLq27p7evf+BDYnBoeGQ0OTa+Hnr1wBYl23O8YMMyQ+FIV5SUVI7Y8ANh1ixHlK3qcvu+3BBBKD33q2r6Yrtm7rtyT9qmYspIpr4Ycku6yqAdlRbZYDq9GxjJDOUoivRLoMcggzjyXvIXtrALDzbqqEHAhWLswETIXwU6CD5z22gxFzCS0b3AERKsrXOW4AyT2Sqv+3yqxKzL57ZnGKltfsXhP2BlGpP0ly7ojv7QJd3Qv1e9WpFHu5Ym71ZHK3xj9Pjj2sO7qhrvCgePqjcUFmd3KrumK67rdvNzpfxmjwp7mI96k9yrHzHtru2OT+Pw5G5toTjZmqJTuuV+f7Lzb3Z2G/f2WUEUv/O49OfDeQnWZ3I65fTCbGZxKR7cACbwCVmezhwWsYI8SvxqEz9wjgstq61qJa3cSdW6Yk0KT0L79h9J15xt</latexit> <latexit sha1_base64="aDkxGx5q/sb9YJIQEipbOLXS4BA=">AAACpnichVFNS+NQFD3Gz6lf1W4G3JQpSp1FuRFBcSW6EQakrVOrVM0k8amPpklIXgu1+Af8Ay5mMYwgKC78B7OZzfyBWfgTxKWCm1nMbRoQFfWGvHfeefecdy/X8h0ZKqLrLq27p7evf+BDYnBoeGQ0OTa+Hnr1wBYl23O8YMMyQ+FIV5SUVI7Y8ANh1ixHlK3qcvu+3BBBKD33q2r6Yrtm7rtyT9qmYspIpr4Ycku6yqAdlRbZYDq9GxjJDOUoivRLoMcggzjyXvIXtrALDzbqqEHAhWLswETIXwU6CD5z22gxFzCS0b3AERKsrXOW4AyT2Sqv+3yqxKzL57ZnGKltfsXhP2BlGpP0ly7ojv7QJd3Qv1e9WpFHu5Ym71ZHK3xj9Pjj2sO7qhrvCgePqjcUFmd3KrumK67rdvNzpfxmjwp7mI96k9yrHzHtru2OT+Pw5G5toTjZmqJTuuV+f7Lzb3Z2G/f2WUEUv/O49OfDeQnWZ3I65fTCbGZxKR7cACbwCVmezhwWsYI8SvxqEz9wjgstq61qJa3cSdW6Yk0KT0L79h9J15xt</latexit> <latexit sha1_base64="aDkxGx5q/sb9YJIQEipbOLXS4BA=">AAACpnichVFNS+NQFD3Gz6lf1W4G3JQpSp1FuRFBcSW6EQakrVOrVM0k8amPpklIXgu1+Af8Ay5mMYwgKC78B7OZzfyBWfgTxKWCm1nMbRoQFfWGvHfeefecdy/X8h0ZKqLrLq27p7evf+BDYnBoeGQ0OTa+Hnr1wBYl23O8YMMyQ+FIV5SUVI7Y8ANh1ixHlK3qcvu+3BBBKD33q2r6Yrtm7rtyT9qmYspIpr4Ycku6yqAdlRbZYDq9GxjJDOUoivRLoMcggzjyXvIXtrALDzbqqEHAhWLswETIXwU6CD5z22gxFzCS0b3AERKsrXOW4AyT2Sqv+3yqxKzL57ZnGKltfsXhP2BlGpP0ly7ojv7QJd3Qv1e9WpFHu5Ym71ZHK3xj9Pjj2sO7qhrvCgePqjcUFmd3KrumK67rdvNzpfxmjwp7mI96k9yrHzHtru2OT+Pw5G5toTjZmqJTuuV+f7Lzb3Z2G/f2WUEUv/O49OfDeQnWZ3I65fTCbGZxKR7cACbwCVmezhwWsYI8SvxqEz9wjgstq61qJa3cSdW6Yk0KT0L79h9J15xt</latexit> % Kd de(t) dt <latexit sha1_base64="pK4ns8fNxecVndhwIulsD6E0PEo=">AAACp3ichVHLSsNAFD2Nr/quj4XgplgUdVEmIiiuim4EF9rWPqRKSdKphqZJSKYFDf0Bf8CFGxVcVBf+gRs3/oALP0FcKrhx4W0aEBX1hsycOXPPmXu5qm3ormDsMSR1dHZ194R7+/oHBoeGIyOjWdeqORrPaJZhOXlVcbmhmzwjdGHwvO1wpaoaPKdW1lr3uTp3XN0yt8Whzfeqyr6pl3VNEUQVI+MbxVJ0t+womlfis2Ku4ZVEoxiJsTjzI/oTyAGIIYgtK3KLXZRgQUMNVXCYEIQNKHDpK0AGg03cHjziHEK6f8/RQB9pa5TFKUMhtkLrPp0KAWvSueXp+mqNXjHod0gZxTR7YE32wu7ZNXti7796eb5Hq5ZD2tW2ltvF4eOJ9Nu/qirtAgefqj8UKmW3K3tkN1TX8858IfdnjwJlLPu96dSr7TOtrrW2T/3o5CW9kpr2ZtgFe6Z+z8n5jpzN+qt2meSpUxqX/H04P0F2IS6zuJxcjCVWg8GFMYkpzNJ0lpDAOraQoVePcIYmrqQ5aVPKSvl2qhQKNGP4EpLyAfEdnZo=</latexit> <latexit sha1_base64="pK4ns8fNxecVndhwIulsD6E0PEo=">AAACp3ichVHLSsNAFD2Nr/quj4XgplgUdVEmIiiuim4EF9rWPqRKSdKphqZJSKYFDf0Bf8CFGxVcVBf+gRs3/oALP0FcKrhx4W0aEBX1hsycOXPPmXu5qm3ormDsMSR1dHZ194R7+/oHBoeGIyOjWdeqORrPaJZhOXlVcbmhmzwjdGHwvO1wpaoaPKdW1lr3uTp3XN0yt8Whzfeqyr6pl3VNEUQVI+MbxVJ0t+womlfis2Ku4ZVEoxiJsTjzI/oTyAGIIYgtK3KLXZRgQUMNVXCYEIQNKHDpK0AGg03cHjziHEK6f8/RQB9pa5TFKUMhtkLrPp0KAWvSueXp+mqNXjHod0gZxTR7YE32wu7ZNXti7796eb5Hq5ZD2tW2ltvF4eOJ9Nu/qirtAgefqj8UKmW3K3tkN1TX8858IfdnjwJlLPu96dSr7TOtrrW2T/3o5CW9kpr2ZtgFe6Z+z8n5jpzN+qt2meSpUxqX/H04P0F2IS6zuJxcjCVWg8GFMYkpzNJ0lpDAOraQoVePcIYmrqQ5aVPKSvl2qhQKNGP4EpLyAfEdnZo=</latexit> <latexit sha1_base64="pK4ns8fNxecVndhwIulsD6E0PEo=">AAACp3ichVHLSsNAFD2Nr/quj4XgplgUdVEmIiiuim4EF9rWPqRKSdKphqZJSKYFDf0Bf8CFGxVcVBf+gRs3/oALP0FcKrhx4W0aEBX1hsycOXPPmXu5qm3ormDsMSR1dHZ194R7+/oHBoeGIyOjWdeqORrPaJZhOXlVcbmhmzwjdGHwvO1wpaoaPKdW1lr3uTp3XN0yt8Whzfeqyr6pl3VNEUQVI+MbxVJ0t+womlfis2Ku4ZVEoxiJsTjzI/oTyAGIIYgtK3KLXZRgQUMNVXCYEIQNKHDpK0AGg03cHjziHEK6f8/RQB9pa5TFKUMhtkLrPp0KAWvSueXp+mqNXjHod0gZxTR7YE32wu7ZNXti7796eb5Hq5ZD2tW2ltvF4eOJ9Nu/qirtAgefqj8UKmW3K3tkN1TX8858IfdnjwJlLPu96dSr7TOtrrW2T/3o5CW9kpr2ZtgFe6Z+z8n5jpzN+qt2meSpUxqX/H04P0F2IS6zuJxcjCVWg8GFMYkpzNJ0lpDAOraQoVePcIYmrqQ5aVPKSvl2qhQKNGP4EpLyAfEdnZo=</latexit> <latexit sha1_base64="pK4ns8fNxecVndhwIulsD6E0PEo=">AAACp3ichVHLSsNAFD2Nr/quj4XgplgUdVEmIiiuim4EF9rWPqRKSdKphqZJSKYFDf0Bf8CFGxVcVBf+gRs3/oALP0FcKrhx4W0aEBX1hsycOXPPmXu5qm3ormDsMSR1dHZ194R7+/oHBoeGIyOjWdeqORrPaJZhOXlVcbmhmzwjdGHwvO1wpaoaPKdW1lr3uTp3XN0yt8Whzfeqyr6pl3VNEUQVI+MbxVJ0t+womlfis2Ku4ZVEoxiJsTjzI/oTyAGIIYgtK3KLXZRgQUMNVXCYEIQNKHDpK0AGg03cHjziHEK6f8/RQB9pa5TFKUMhtkLrPp0KAWvSueXp+mqNXjHod0gZxTR7YE32wu7ZNXti7796eb5Hq5ZD2tW2ltvF4eOJ9Nu/qirtAgefqj8UKmW3K3tkN1TX8858IfdnjwJlLPu96dSr7TOtrrW2T/3o5CW9kpr2ZtgFe6Z+z8n5jpzN+qt2meSpUxqX/H04P0F2IS6zuJxcjCVWg8GFMYkpzNJ0lpDAOraQoVePcIYmrqQ5aVPKSvl2qhQKNGP4EpLyAfEdnZo=</latexit> r <latexit sha1_base64="l3XaGwPmE5Vm/Jhs1LG2xz0k9Bs=">AAACknichVG7SgNBFD1ZXzE+Eh+FYCMGRSzCrAiKVdTGwsIk5iFRZHcddchmd9mdBDT4A9oqFlYKFmLhH9jY+AMWfoKkjGBj4c1mQVTUu+zMmTP3nLmXqzum8CRjzyGlrb2jsyvcHenp7euPxgYGc55dcQ2eNWzTdgu65nFTWDwrhTR5wXG5VtZNntdLy837fJW7nrCtdXng8K2ytmeJXWFokqiUux2LswTzY+wnUAMQRxBrduwem9iBDQMVlMFhQRI2ocGjrwgVDA5xW6gR5xIS/j3HESKkrVAWpwyN2BKte3QqBqxF56an56sNesWk3yXlGCbYE7thDfbIbtkLe//Vq+Z7NGs5oF1vabmzHT0eybz9qyrTLrH/qfpDoVN2q7Jndkd11Temi/k/e5TYxbzfm6BeHZ9pdm20fKqH543MQnqiNsmuWJ36vSTnB3K2qq/GdYqnL2hc6vfh/AS5mYTKEmpqNp5cCgYXxijGMUXTmUMSK1hDll7lOMEpzpRhZUFZVJZbqUoo0AzhSyirH/1ElaE=</latexit> <latexit sha1_base64="l3XaGwPmE5Vm/Jhs1LG2xz0k9Bs=">AAACknichVG7SgNBFD1ZXzE+Eh+FYCMGRSzCrAiKVdTGwsIk5iFRZHcddchmd9mdBDT4A9oqFlYKFmLhH9jY+AMWfoKkjGBj4c1mQVTUu+zMmTP3nLmXqzum8CRjzyGlrb2jsyvcHenp7euPxgYGc55dcQ2eNWzTdgu65nFTWDwrhTR5wXG5VtZNntdLy837fJW7nrCtdXng8K2ytmeJXWFokqiUux2LswTzY+wnUAMQRxBrduwem9iBDQMVlMFhQRI2ocGjrwgVDA5xW6gR5xIS/j3HESKkrVAWpwyN2BKte3QqBqxF56an56sNesWk3yXlGCbYE7thDfbIbtkLe//Vq+Z7NGs5oF1vabmzHT0eybz9qyrTLrH/qfpDoVN2q7Jndkd11Temi/k/e5TYxbzfm6BeHZ9pdm20fKqH543MQnqiNsmuWJ36vSTnB3K2qq/GdYqnL2hc6vfh/AS5mYTKEmpqNp5cCgYXxijGMUXTmUMSK1hDll7lOMEpzpRhZUFZVJZbqUoo0AzhSyirH/1ElaE=</latexit> <latexit sha1_base64="l3XaGwPmE5Vm/Jhs1LG2xz0k9Bs=">AAACknichVG7SgNBFD1ZXzE+Eh+FYCMGRSzCrAiKVdTGwsIk5iFRZHcddchmd9mdBDT4A9oqFlYKFmLhH9jY+AMWfoKkjGBj4c1mQVTUu+zMmTP3nLmXqzum8CRjzyGlrb2jsyvcHenp7euPxgYGc55dcQ2eNWzTdgu65nFTWDwrhTR5wXG5VtZNntdLy837fJW7nrCtdXng8K2ytmeJXWFokqiUux2LswTzY+wnUAMQRxBrduwem9iBDQMVlMFhQRI2ocGjrwgVDA5xW6gR5xIS/j3HESKkrVAWpwyN2BKte3QqBqxF56an56sNesWk3yXlGCbYE7thDfbIbtkLe//Vq+Z7NGs5oF1vabmzHT0eybz9qyrTLrH/qfpDoVN2q7Jndkd11Temi/k/e5TYxbzfm6BeHZ9pdm20fKqH543MQnqiNsmuWJ36vSTnB3K2qq/GdYqnL2hc6vfh/AS5mYTKEmpqNp5cCgYXxijGMUXTmUMSK1hDll7lOMEpzpRhZUFZVJZbqUoo0AzhSyirH/1ElaE=</latexit> <latexit sha1_base64="l3XaGwPmE5Vm/Jhs1LG2xz0k9Bs=">AAACknichVG7SgNBFD1ZXzE+Eh+FYCMGRSzCrAiKVdTGwsIk5iFRZHcddchmd9mdBDT4A9oqFlYKFmLhH9jY+AMWfoKkjGBj4c1mQVTUu+zMmTP3nLmXqzum8CRjzyGlrb2jsyvcHenp7euPxgYGc55dcQ2eNWzTdgu65nFTWDwrhTR5wXG5VtZNntdLy837fJW7nrCtdXng8K2ytmeJXWFokqiUux2LswTzY+wnUAMQRxBrduwem9iBDQMVlMFhQRI2ocGjrwgVDA5xW6gR5xIS/j3HESKkrVAWpwyN2BKte3QqBqxF56an56sNesWk3yXlGCbYE7thDfbIbtkLe//Vq+Z7NGs5oF1vabmzHT0eybz9qyrTLrH/qfpDoVN2q7Jndkd11Temi/k/e5TYxbzfm6BeHZ9pdm20fKqH543MQnqiNsmuWJ36vSTnB3K2qq/GdYqnL2hc6vfh/AS5mYTKEmpqNp5cCgYXxijGMUXTmUMSK1hDll7lOMEpzpRhZUFZVJZbqUoo0AzhSyirH/1ElaE=</latexit> e <latexit sha1_base64="JucbY3V7iD4pyjirU4Kr9PWwJLM=">AAACknichVG7SgNBFD1Z3/EVH4VgIwZFLMKsCEoqNY2FRR7GKFFkdx11cLO77E4CMfgD2ioWVgoWYuEf2Nj4AxZ+glgq2Fh49wGiot5lZ86cuefMvVzdMYUnGXuMKU3NLa1t7R3xzq7unt5EX/+KZ1ddgxcN27TdVV3zuCksXpRCmnzVcblW0U1e0vcy/n2pxl1P2NayrDt8o6LtWGJbGJokKsc3E0mWYkGM/ARqBJKIImsnbrGOLdgwUEUFHBYkYRMaPPrKUMHgELeBBnEuIRHccxwgTtoqZXHK0Ijdo3WHTuWItejse3qB2qBXTPpdUo5gjD2wK/bC7tk1e2Lvv3o1Ag+/ljrteqjlzmbv4VDh7V9VhXaJ3U/VHwqdssPKHtkN1fW8Nlku/dmjxDZmg94E9eoEjN+1EfrU9k9fCun8WGOcXbBn6vecnO/I2aq9Gpc5nj+jcanfh/MTrEylVJZSc9PJuYVocO0YxigmaDozmMMisijSqxxHOMaJMqiklXklE6YqsUgzgC+hLH0A4O6VlA==</latexit> <latexit sha1_base64="JucbY3V7iD4pyjirU4Kr9PWwJLM=">AAACknichVG7SgNBFD1Z3/EVH4VgIwZFLMKsCEoqNY2FRR7GKFFkdx11cLO77E4CMfgD2ioWVgoWYuEf2Nj4AxZ+glgq2Fh49wGiot5lZ86cuefMvVzdMYUnGXuMKU3NLa1t7R3xzq7unt5EX/+KZ1ddgxcN27TdVV3zuCksXpRCmnzVcblW0U1e0vcy/n2pxl1P2NayrDt8o6LtWGJbGJokKsc3E0mWYkGM/ARqBJKIImsnbrGOLdgwUEUFHBYkYRMaPPrKUMHgELeBBnEuIRHccxwgTtoqZXHK0Ijdo3WHTuWItejse3qB2qBXTPpdUo5gjD2wK/bC7tk1e2Lvv3o1Ag+/ljrteqjlzmbv4VDh7V9VhXaJ3U/VHwqdssPKHtkN1fW8Nlku/dmjxDZmg94E9eoEjN+1EfrU9k9fCun8WGOcXbBn6vecnO/I2aq9Gpc5nj+jcanfh/MTrEylVJZSc9PJuYVocO0YxigmaDozmMMisijSqxxHOMaJMqiklXklE6YqsUgzgC+hLH0A4O6VlA==</latexit> <latexit sha1_base64="JucbY3V7iD4pyjirU4Kr9PWwJLM=">AAACknichVG7SgNBFD1Z3/EVH4VgIwZFLMKsCEoqNY2FRR7GKFFkdx11cLO77E4CMfgD2ioWVgoWYuEf2Nj4AxZ+glgq2Fh49wGiot5lZ86cuefMvVzdMYUnGXuMKU3NLa1t7R3xzq7unt5EX/+KZ1ddgxcN27TdVV3zuCksXpRCmnzVcblW0U1e0vcy/n2pxl1P2NayrDt8o6LtWGJbGJokKsc3E0mWYkGM/ARqBJKIImsnbrGOLdgwUEUFHBYkYRMaPPrKUMHgELeBBnEuIRHccxwgTtoqZXHK0Ijdo3WHTuWItejse3qB2qBXTPpdUo5gjD2wK/bC7tk1e2Lvv3o1Ag+/ljrteqjlzmbv4VDh7V9VhXaJ3U/VHwqdssPKHtkN1fW8Nlku/dmjxDZmg94E9eoEjN+1EfrU9k9fCun8WGOcXbBn6vecnO/I2aq9Gpc5nj+jcanfh/MTrEylVJZSc9PJuYVocO0YxigmaDozmMMisijSqxxHOMaJMqiklXklE6YqsUgzgC+hLH0A4O6VlA==</latexit> <latexit sha1_base64="JucbY3V7iD4pyjirU4Kr9PWwJLM=">AAACknichVG7SgNBFD1Z3/EVH4VgIwZFLMKsCEoqNY2FRR7GKFFkdx11cLO77E4CMfgD2ioWVgoWYuEf2Nj4AxZ+glgq2Fh49wGiot5lZ86cuefMvVzdMYUnGXuMKU3NLa1t7R3xzq7unt5EX/+KZ1ddgxcN27TdVV3zuCksXpRCmnzVcblW0U1e0vcy/n2pxl1P2NayrDt8o6LtWGJbGJokKsc3E0mWYkGM/ARqBJKIImsnbrGOLdgwUEUFHBYkYRMaPPrKUMHgELeBBnEuIRHccxwgTtoqZXHK0Ijdo3WHTuWItejse3qB2qBXTPpdUo5gjD2wK/bC7tk1e2Lvv3o1Ag+/ljrteqjlzmbv4VDh7V9VhXaJ3U/VHwqdssPKHtkN1fW8Nlku/dmjxDZmg94E9eoEjN+1EfrU9k9fCun8WGOcXbBn6vecnO/I2aq9Gpc5nj+jcanfh/MTrEylVJZSc9PJuYVocO0YxigmaDozmMMisijSqxxHOMaJMqiklXklE6YqsUgzgC+hLH0A4O6VlA==</latexit> 4ZTUFN  'FFECBDL
    QBUI *OQVU 0VUQVU
37. 37 Dynamic Target Controller &SSPS 4FUQPJOU 1 * % r <latexit sha1_base64="l3XaGwPmE5Vm/Jhs1LG2xz0k9Bs=">AAACknichVG7SgNBFD1ZXzE+Eh+FYCMGRSzCrAiKVdTGwsIk5iFRZHcddchmd9mdBDT4A9oqFlYKFmLhH9jY+AMWfoKkjGBj4c1mQVTUu+zMmTP3nLmXqzum8CRjzyGlrb2jsyvcHenp7euPxgYGc55dcQ2eNWzTdgu65nFTWDwrhTR5wXG5VtZNntdLy837fJW7nrCtdXng8K2ytmeJXWFokqiUux2LswTzY+wnUAMQRxBrduwem9iBDQMVlMFhQRI2ocGjrwgVDA5xW6gR5xIS/j3HESKkrVAWpwyN2BKte3QqBqxF56an56sNesWk3yXlGCbYE7thDfbIbtkLe//Vq+Z7NGs5oF1vabmzHT0eybz9qyrTLrH/qfpDoVN2q7Jndkd11Temi/k/e5TYxbzfm6BeHZ9pdm20fKqH543MQnqiNsmuWJ36vSTnB3K2qq/GdYqnL2hc6vfh/AS5mYTKEmpqNp5cCgYXxijGMUXTmUMSK1hDll7lOMEpzpRhZUFZVJZbqUoo0AzhSyirH/1ElaE=</latexit> <latexit sha1_base64="l3XaGwPmE5Vm/Jhs1LG2xz0k9Bs=">AAACknichVG7SgNBFD1ZXzE+Eh+FYCMGRSzCrAiKVdTGwsIk5iFRZHcddchmd9mdBDT4A9oqFlYKFmLhH9jY+AMWfoKkjGBj4c1mQVTUu+zMmTP3nLmXqzum8CRjzyGlrb2jsyvcHenp7euPxgYGc55dcQ2eNWzTdgu65nFTWDwrhTR5wXG5VtZNntdLy837fJW7nrCtdXng8K2ytmeJXWFokqiUux2LswTzY+wnUAMQRxBrduwem9iBDQMVlMFhQRI2ocGjrwgVDA5xW6gR5xIS/j3HESKkrVAWpwyN2BKte3QqBqxF56an56sNesWk3yXlGCbYE7thDfbIbtkLe//Vq+Z7NGs5oF1vabmzHT0eybz9qyrTLrH/qfpDoVN2q7Jndkd11Temi/k/e5TYxbzfm6BeHZ9pdm20fKqH543MQnqiNsmuWJ36vSTnB3K2qq/GdYqnL2hc6vfh/AS5mYTKEmpqNp5cCgYXxijGMUXTmUMSK1hDll7lOMEpzpRhZUFZVJZbqUoo0AzhSyirH/1ElaE=</latexit> <latexit sha1_base64="l3XaGwPmE5Vm/Jhs1LG2xz0k9Bs=">AAACknichVG7SgNBFD1ZXzE+Eh+FYCMGRSzCrAiKVdTGwsIk5iFRZHcddchmd9mdBDT4A9oqFlYKFmLhH9jY+AMWfoKkjGBj4c1mQVTUu+zMmTP3nLmXqzum8CRjzyGlrb2jsyvcHenp7euPxgYGc55dcQ2eNWzTdgu65nFTWDwrhTR5wXG5VtZNntdLy837fJW7nrCtdXng8K2ytmeJXWFokqiUux2LswTzY+wnUAMQRxBrduwem9iBDQMVlMFhQRI2ocGjrwgVDA5xW6gR5xIS/j3HESKkrVAWpwyN2BKte3QqBqxF56an56sNesWk3yXlGCbYE7thDfbIbtkLe//Vq+Z7NGs5oF1vabmzHT0eybz9qyrTLrH/qfpDoVN2q7Jndkd11Temi/k/e5TYxbzfm6BeHZ9pdm20fKqH543MQnqiNsmuWJ36vSTnB3K2qq/GdYqnL2hc6vfh/AS5mYTKEmpqNp5cCgYXxijGMUXTmUMSK1hDll7lOMEpzpRhZUFZVJZbqUoo0AzhSyirH/1ElaE=</latexit> <latexit sha1_base64="l3XaGwPmE5Vm/Jhs1LG2xz0k9Bs=">AAACknichVG7SgNBFD1ZXzE+Eh+FYCMGRSzCrAiKVdTGwsIk5iFRZHcddchmd9mdBDT4A9oqFlYKFmLhH9jY+AMWfoKkjGBj4c1mQVTUu+zMmTP3nLmXqzum8CRjzyGlrb2jsyvcHenp7euPxgYGc55dcQ2eNWzTdgu65nFTWDwrhTR5wXG5VtZNntdLy837fJW7nrCtdXng8K2ytmeJXWFokqiUux2LswTzY+wnUAMQRxBrduwem9iBDQMVlMFhQRI2ocGjrwgVDA5xW6gR5xIS/j3HESKkrVAWpwyN2BKte3QqBqxF56an56sNesWk3yXlGCbYE7thDfbIbtkLe//Vq+Z7NGs5oF1vabmzHT0eybz9qyrTLrH/qfpDoVN2q7Jndkd11Temi/k/e5TYxbzfm6BeHZ9pdm20fKqH543MQnqiNsmuWJ36vSTnB3K2qq/GdYqnL2hc6vfh/AS5mYTKEmpqNp5cCgYXxijGMUXTmUMSK1hDll7lOMEpzpRhZUFZVJZbqUoo0AzhSyirH/1ElaE=</latexit> e <latexit sha1_base64="JucbY3V7iD4pyjirU4Kr9PWwJLM=">AAACknichVG7SgNBFD1Z3/EVH4VgIwZFLMKsCEoqNY2FRR7GKFFkdx11cLO77E4CMfgD2ioWVgoWYuEf2Nj4AxZ+glgq2Fh49wGiot5lZ86cuefMvVzdMYUnGXuMKU3NLa1t7R3xzq7unt5EX/+KZ1ddgxcN27TdVV3zuCksXpRCmnzVcblW0U1e0vcy/n2pxl1P2NayrDt8o6LtWGJbGJokKsc3E0mWYkGM/ARqBJKIImsnbrGOLdgwUEUFHBYkYRMaPPrKUMHgELeBBnEuIRHccxwgTtoqZXHK0Ijdo3WHTuWItejse3qB2qBXTPpdUo5gjD2wK/bC7tk1e2Lvv3o1Ag+/ljrteqjlzmbv4VDh7V9VhXaJ3U/VHwqdssPKHtkN1fW8Nlku/dmjxDZmg94E9eoEjN+1EfrU9k9fCun8WGOcXbBn6vecnO/I2aq9Gpc5nj+jcanfh/MTrEylVJZSc9PJuYVocO0YxigmaDozmMMisijSqxxHOMaJMqiklXklE6YqsUgzgC+hLH0A4O6VlA==</latexit> <latexit sha1_base64="JucbY3V7iD4pyjirU4Kr9PWwJLM=">AAACknichVG7SgNBFD1Z3/EVH4VgIwZFLMKsCEoqNY2FRR7GKFFkdx11cLO77E4CMfgD2ioWVgoWYuEf2Nj4AxZ+glgq2Fh49wGiot5lZ86cuefMvVzdMYUnGXuMKU3NLa1t7R3xzq7unt5EX/+KZ1ddgxcN27TdVV3zuCksXpRCmnzVcblW0U1e0vcy/n2pxl1P2NayrDt8o6LtWGJbGJokKsc3E0mWYkGM/ARqBJKIImsnbrGOLdgwUEUFHBYkYRMaPPrKUMHgELeBBnEuIRHccxwgTtoqZXHK0Ijdo3WHTuWItejse3qB2qBXTPpdUo5gjD2wK/bC7tk1e2Lvv3o1Ag+/ljrteqjlzmbv4VDh7V9VhXaJ3U/VHwqdssPKHtkN1fW8Nlku/dmjxDZmg94E9eoEjN+1EfrU9k9fCun8WGOcXbBn6vecnO/I2aq9Gpc5nj+jcanfh/MTrEylVJZSc9PJuYVocO0YxigmaDozmMMisijSqxxHOMaJMqiklXklE6YqsUgzgC+hLH0A4O6VlA==</latexit> <latexit sha1_base64="JucbY3V7iD4pyjirU4Kr9PWwJLM=">AAACknichVG7SgNBFD1Z3/EVH4VgIwZFLMKsCEoqNY2FRR7GKFFkdx11cLO77E4CMfgD2ioWVgoWYuEf2Nj4AxZ+glgq2Fh49wGiot5lZ86cuefMvVzdMYUnGXuMKU3NLa1t7R3xzq7unt5EX/+KZ1ddgxcN27TdVV3zuCksXpRCmnzVcblW0U1e0vcy/n2pxl1P2NayrDt8o6LtWGJbGJokKsc3E0mWYkGM/ARqBJKIImsnbrGOLdgwUEUFHBYkYRMaPPrKUMHgELeBBnEuIRHccxwgTtoqZXHK0Ijdo3WHTuWItejse3qB2qBXTPpdUo5gjD2wK/bC7tk1e2Lvv3o1Ag+/ljrteqjlzmbv4VDh7V9VhXaJ3U/VHwqdssPKHtkN1fW8Nlku/dmjxDZmg94E9eoEjN+1EfrU9k9fCun8WGOcXbBn6vecnO/I2aq9Gpc5nj+jcanfh/MTrEylVJZSc9PJuYVocO0YxigmaDozmMMisijSqxxHOMaJMqiklXklE6YqsUgzgC+hLH0A4O6VlA==</latexit> <latexit sha1_base64="JucbY3V7iD4pyjirU4Kr9PWwJLM=">AAACknichVG7SgNBFD1Z3/EVH4VgIwZFLMKsCEoqNY2FRR7GKFFkdx11cLO77E4CMfgD2ioWVgoWYuEf2Nj4AxZ+glgq2Fh49wGiot5lZ86cuefMvVzdMYUnGXuMKU3NLa1t7R3xzq7unt5EX/+KZ1ddgxcN27TdVV3zuCksXpRCmnzVcblW0U1e0vcy/n2pxl1P2NayrDt8o6LtWGJbGJokKsc3E0mWYkGM/ARqBJKIImsnbrGOLdgwUEUFHBYkYRMaPPrKUMHgELeBBnEuIRHccxwgTtoqZXHK0Ijdo3WHTuWItejse3qB2qBXTPpdUo5gjD2wK/bC7tk1e2Lvv3o1Ag+/ljrteqjlzmbv4VDh7V9VhXaJ3U/VHwqdssPKHtkN1fW8Nlku/dmjxDZmg94E9eoEjN+1EfrU9k9fCun8WGOcXbBn6vecnO/I2aq9Gpc5nj+jcanfh/MTrEylVJZSc9PJuYVocO0YxigmaDozmMMisijSqxxHOMaJMqiklXklE6YqsUgzgC+hLH0A4O6VlA==</latexit> (P
    BQQ  'FFECBDL
    QBUI 4FNBQIPSF $16
    6TBHF $IBOHF
    SBUF • Having two nested control loops to change set-point on running. • The outer loop sets new set-points of the inner loop when CPU usage reaches the upper limit. $POUSPMMFS
38. Bounding
39. Bounding concurrency using buffered channel as semaphore 39 To bound concurrency in Go func main() { var wg sync.WaitGroup sem := make(chan struct{}, 3) for i := 0; i < 10; i++ { wg.Add(1) sem <- struct{}{} go func() { defer wg.Done() defer func() { <-sem }() task() }() } wg.Wait() close(sem) }
40. 40 Elastic semaphore 
    DIBO
     &MBTUJD
    TFNBQIPSF $PVOUFS XBJU TJHOBM JODSFNFOU-JNJU   $PVOUFS
    
     $PVOUFS
    
     8BJUJOH /PU
    XBJUJOH  • wait() & signal() as same as semaphore • incrementLimit() changes upper of semaphore variable • Ensure atomicity of above operations
41. Kaburaya
42. Architecture of kaburaya 42 • The optimal number of concurrency to be determined dynamically and be controlled rapidly • Use CPU usage upper limit as performance metrics. • Use feedback control to determine the number of goroutines continuosly. • Use elastic semaphore to bound concurrency. • Kaburaya is “ధ໼” which is a Japanese arrow with a whistle. • https://github.com/monochromegane/kaburaya
43. • Kaburaya optimizes the number of goroutines by feedback control. 43 monochromegane/kaburaya func main() { sem := kaburaya.NewSem(100 * time.Millisecond) var wg sync.WaitGroup for i := 0; i < 10; i++ { wg.Add(1) sem.Wait() go func() { defer sem.Signal() defer wg.Done() task() }() } wg.Wait() sem.Stop() } Create semaphore and specify time span for update the semaphore and CPU target. Wait for decrement operation. Increment operation. Stop update the semaphore and CPU target.
44. 4. Evaluation
45. • Apply kaburaya to some tasks to evaluate efficacy. • Measure speedup, CPU usage, and limit of semaphore. • Find good and not so good cases for the kaburaya controller. 45 Evaluation
46. Task: pt_mac-scan 46 • pt runs on macOS with realtime virus scan • CPU 8, Memory 16GB • GOMAXPROCS=8 • Better concurrent are from 16 to 64. • More degradation of performance due to increased goroutines.
47. Task: pt_mac-scan (Good) 47 • Kaburaya found good set points and adjust numbers within ideal range. Duration: 50ms, Gain: 0.1, Change rate: ±0.3 Median
48. Task: pt_mac-no-scan 48 • Runs pt on Mac without realtime virus scan • CPU 8, Memory 16GB • GOMAXPROCS=8 • Better concurrent is 4 • Less degradation of performance due to increased goroutines.
49. Task: pt_mac-no-scan (Good) 49 • Kaburaya found good set points and adjust numbers within ideal range. Duration: 50ms, Gain: 0.1, Change rate: ±0.3 Median
50. Task: pt_mac-no-scan (Bad) 50 • Find set points and adjust numbers within ideal range. • Increase the number due to failure to reset the set point at end of period. Duration: 50ms, Gain: 0.1, Change rate: ±0.3 Median
51. Task: pt_linux 51 • pt runs on Linux • CPU 4, Memory 4GB • GOMAXPROCS=4 • Better concurrent are from 4 to 32. • Less degradation of performance due to increased goroutines.
52. Task: pt_linux (Not so bad) 52 • Kaburaya found good set points and adjust numbers within ideal range. • Increase the number due to failure to reset the set point at end of period. Duration: 50ms, Gain: 0.1, Change rate: ±0.3 Median
53. Task: mem_4096000_10000 53 • Runs task uses memory. • 4MB/task x 10,000 • CPU 4, Memory 4GB • GOMAXPROCS=4 • Better concurrent is 512. • More degradation of performance (Swap out) due to increased goroutines.
54. Task: mem_4096000_10000 (Good) 54 • Kaburaya found set points and adjust numbers within ideal range. • Avoid starvation of resource Duration: 500ms, Gain: 0.3, Change rate: ±0.3 Median
55. Task: mem_40960_1000000 55 • Runs task uses memory. • 40kB/task x 1,000,000 • CPU 4, Memory 4GB • GOMAXPROCS=4 • Better concurrent are from 4 to 64. • Less degradation of performance due to increased goroutines.
56. Task: mem_40960_1000000 (Good) 56 • Kaburaya found set points and adjust numbers within ideal range. Duration: 500ms, Gain: 0.3, Change rate: ±0.3 Median
57. Task: mem_409600_100000 57 • Runs task uses memory. • 400kB/task x 100,000 • CPU 4, Memory 4GB • GOMAXPROCS=4 • Better concurrent is over 1024? • Less degradation of performance due to increased goroutines.
58. Task: mem_409600_100000 (Not so good) 58 • Kaburaya found set points and adjust numbers. • Does not reach the limit of performance metrics… Duration: 500ms, Gain: 0.3, Change rate: ±0.3 Median
59. • Kaburaya finds set point and adjusts numbers within ideal range by feedback control and elastic semaphore. • Towards to improve the controller • prediction accuracy of set point • Optimal parameters (K{p,i,d} and range of change rate) • Solve them in the related researches about feedback control as the next step. 59 Experimental results
60. 5. Conclusion
61. Conclusion 61 • I proposed kaburaya to control the number of goroutines without depending on platform, runtime and current load. • Experimental results show possible and some issues of kaburaya. • In particular improvement detecting performance upper limit of task is important. • In the future, I will also consider the application to auto-scaling of cloud computing. • Go
62. Appendix
63. • Implementation of Kaburaya • https://github.com/monochromegane/kaburaya • Study of Kaburaya (In Japanese) • https://blog.monochromegane.com/blog/2018/11/25/wsa_3_kaburaya/ • https://gist.github.com/monochromegane/5e9fc94371a9b2385eedd38d98341e6b • https://blog.monochromegane.com/blog/2019/04/14/wsa_4_ebira/ • Learn more goroutine • https://speakerdeck.com/monochromegane/road-to-your-goroutines 63 Reference
64. 64 Go gopher • The Go gopher was designed by Renee French. • The gopher illustrations were drawn by Keita Kawamoto. • Licensed under the Creative Commons 3.0 Attributions license.
65. Thank you!