Lock in $30 Savings on PRO—Offer Ends Soon! ⏳

Optimization for Number of goroutines Using Fee...

Optimization for Number of goroutines Using Feedback Control

GopherCon 2019 - Marriott Marquis San Diego Marina, California
https://www.gophercon.com/

Optimization for Number of goroutines Using Feedback Control
https://www.gophercon.com/agenda/speakers/442434

The design for the number of concurrency is important to achieve both speed and stability. To give a good performance without depending on platform and load conditions, it’s desirable for the number to be dynamic and rapidly controlled. In this talk, I will propose an architecture to solve this by utilizing feedback control.

monochromegane

July 25, 2019
Tweet

More Decks by monochromegane

Other Decks in Programming

Transcript

  1. Yusuke MIYAKE / Pepabo R&D Institute, GMO Pepabo, Inc. 2019.07.25

    GopherCon 2019 Optimization for number of goroutines using feedback control
  2. Go and I 3 • A Japanese gopher who loves

    writing OSS using Go. • monochromegane/the_platinum_searcher • An organizer of the local Go community in Fukuoka, Japan. • Fukuoka.go (https://fukuokago.dev) • Held Go Conference’19 summer in Fukuoka on 7/13. • https://fukuoka.gocon.jp 5IF(PHPQIFSXBTEFTJHOFECZ3FOFF'SFODI 5IFHPQIFSJMMVTUSBUJPOXBTESBXOCZ,FJUB,BXBNPUP -JDFOTFEVOEFSUIF$SFBUJWF$PNNPOT"UUSJCVUJPOTMJDFOTF
  3. Performance tuning in case of pt 7 'JOE (PSPVUJOFT $IBO

    $IBO 4FBSDI .BUDI QU 1SJOU ʜ HPSPVUJOFT /VN$16  -JOVY $16 (0."9130$4
  4. Performance tuning in case of pt 8 'JOE (PSPVUJOFT $IBO

    $IBO 4FBSDI .BUDI QU 1SJOU ʜ 3FBMUJNFWJSVTTDBOQSPDFTT 6OTUBCMF HPSPVUJOFT /VN$16  NBD04 $16 (0."9130$4
  5. Performance tuning in case of pt 9 'JOE (PSPVUJOFT $IBO

    $IBO 4FBSDI .BUDI QU 1SJOU ʜ 3FBMUJNFWJSVTTDBOQSPDFTT NBD04 $16 (0."9130$4 ❌ HPSPVUJOFT /VN$16
  6. Performance tuning in case of pt 10 • I chose

    stability rather than speed…
  7. • Concurrency brings our application good performance and complexity. •

    We design a task to have concurrency and request runtime to run them in parallel. 14 Concurrency and complexity %FTJHO *NQMFNFOUBUJPO 3VOUJNF $PODVSSFOU 1BSBMMFM w 5ISFBENBOBHFNFOU w 3BDFDPOEJUJPOT w .FNPSZBDDFTTTZOD w %FBEMPDLT w 4UBSWBUJPO w (SBOVMBSJUZ w /VNCFSPG DPODVSSFODZ
  8. Concurrency and Go 16 • Go hides the complexity by

    smart runtime scheduler and features. 3VOUJNFTDIFEVMFS $IBOOFM %FTJHO *NQMFNFOUBUJPO 3VOUJNF $PODVSSFOU 1BSBMMFM w 5ISFBENBOBHFNFOU w (SBOVMBSJUZ w /VNCFSPG DPODVSSFODZ w 3BDFDPOEJUJPOT w .FNPSZBDDFTTTZOD w %FBEMPDLT w 4UBSWBUJPO
  9. Runtime scheduler of Go 17 ( . 1 ( (

    -PDBM3VO2VFVF (MPCBM3VO2VFVF ( ( SVOUJNFTZTNPO . TDIFEVMF FYFDVUF (GO HPFYJU • G: Goroutine • M: OS thread • P: Processor (Scheduling context) • It provides M:N Scheduler (Some goroutines:Some threads). • It has local run queue. • We can think of Goroutines as application-level threads.
 OS thread behaves like a worker for goroutine using a run queue. 3FGIUUQTTQFBLFSEFDLDPNSFUFSWJTJPOHPSVOUJNFTDIFEVMFS
  10. New “g” 18 YD YD YD TUBDLHVBSE )JHI -PX YDE

    TIFETQ YDF TULUPQTQ Stack (2048bytes) TULUPQ 4UBDL 4UBDLHVBSE   Sched (type gobuf) type gobuf struct { sp uintptr // 824633919448 -> 0xc0000307d8 pc uintptr // 4359616 -> 0x4285c0 <runtime.main> g guintptr // newg ctxt unsafe.Pointer // 0x4c96c0 <runtime.mainPC> ret sys.Uintreg. // 0 lr uintptr // 0 bp uintptr // 0 } • Runtime allocates 2kB stack to “g”. • “g” has gobuf named “sched”. • “sched” has goroutines state. (e.g.: stack pointer, program counter…)
  11. Concurrency and application 20 • The design for the number

    of concurrency is important to achieve both speed and stability. • Running many tasks tend to cause starvation. %FTJHO *NQMFNFOUBUJPO 3VOUJNF $PODVSSFOU 1BSBMMFM w 5ISFBENBOBHFNFOU w (SBOVMBSJUZ w /VNCFSPG DPODVSSFODZ w 3BDFDPOEJUJPOT w .FNPSZBDDFTTTZOD w %FBEMPDLT w 4UBSWBUJPO
  12. Concurrency and application 21 • The design for the number

    of concurrency is difficult. • The optimal number depends on the app, environments and load condition. • The environment in which the program is tuned and the environment in which it will be executed is different. • It's desirable for the number of concurrency to be determined dynamically and be controlled rapidly by detection the bottleneck on running program.
  13. Goal 23 • The optimal number of concurrency to be

    determined dynamically and be controlled rapidly.
  14. Basic idea 24 1. Increase the number of goroutines to

    performance target 1FSGPSNBODF      /VNCFSPGHPSPVUJOFT      4UFQ           (PSPVUJOFT 1FSGPSNBODF 1FSGPSNBODFUBSHFU
  15. Basic idea 25 1. Increase the number of goroutines to

    performance target 2. Stop increasing the number of goroutines if it meets the performance target 1FSGPSNBODF      /VNCFSPGHPSPVUJOFT      4UFQ           (PSPVUJOFT 1FSGPSNBODF 1FSGPSNBODFUBSHFU
  16. Basic idea 26 1. Increase the number of goroutines to

    performance target 2. Stop increasing the number of goroutines if it meets the performance target 3. Attempt to decrease the number of goroutines 1FSGPSNBODF      /VNCFSPGHPSPVUJOFT      4UFQ           (PSPVUJOFT 1FSGPSNBODF 1FSGPSNBODFUBSHFU
  17. Issues to solve for the realization 27 1. Selection of

    performance metrics 2. Finding how to control rapidly and continuously
  18. Performance metrics 29 • Independent resource type which task use.

    • e.g.: CPU usage, Throughput. • Go's scheduler turns blocking tasks into as CPU bound as possible by switching tasks continuously.
  19. Performance metrics 30 • The metrics upper limit is calculated

    on running. 1. Set the target value high 1FSGPSNBODF       /VNCFSPGHPSPVUJOFT       4UFQ           (PSPVUJOFT 1FSGPSNBODF 1FSGPSNBODFUBSHFU
  20. Performance metrics 31 • The metrics upper limit is calculated

    on running. 1. Set the target value high 2. gradually adjust it to the upper limit value. 1FSGPSNBODF       /VNCFSPGHPSPVUJOFT       4UFQ           (PSPVUJOFT 1FSGPSNBODF 1FSGPSNBODFUBSHFU
  21. Determine the number of goroutines 33 • Determine the number

    of goroutines based on the metrics • Continuously • Rapidly • Accurately • Feedback control meets these conditions.
  22. • Tracking a given set-point using errors from the set-point.

    • Applying an automatic correction continuously. • Identifying suitable set-point and input/output is important. • Dealing with the system as a black box. Feedback control $POUSPMMFS 4ZTUFN *OQVU 0VUQVU &SSPS 'FFECBDLQBUI 4FUQPJOU
  23. • Applying a correction rapidly and accurately. • The output

    is a combination of its {Proportional, Integral, Derivative} control. 35 PID Controller (1/2) $POUSPMMFS &SSPS 4FUQPJOU 1 Kpe(t) <latexit sha1_base64="WSac9fWZ+QE538KWn4+gz2K9Jv4=">AAACmHichVG7SgNBFL1ZXzE+ErURbYIhEi3CrAiKVdBCxSYP85AYwu46iUv2xe4kEIM/YGclmErBQiz8Axsbf8AinyApI9hYePcBosF4l9m5c+aeM/dwRUORLUZIx8cNDY+MjvnHAxOTU9PB0MxsztLrpkSzkq7oZkEULKrIGs0ymSm0YJhUUEWF5sXajn2fb1DTknXtkDUNWlKFqiZXZElgCOUPygaNsZVyKELixIlwf8J7SQS8SOqhJziGE9BBgjqoQEEDhrkCAlj4FYEHAgZiJWghZmImO/cUziGA3DpWUawQEK3hv4qnoodqeLY1LYct4SsKLhOZYYiSV3JPeuSFPJA38vmnVsvRsHtp4i66XGqUgxfzmY9/WSruDE6/WQMYIla7nXXII/bVPVot5gd6ZFCBTcebjF4NB7FdS65O4+yql9lKR1vL5JZ00e8NKj+jstZ4l+5SNN3GcfG/h9Of5NbiPInzqfVIYtsbnB8WYQliOJ0NSMAeJCHr9HYJ19DmFrgEt8vtu6Wcz+PMwY/g0l/eDZev</latexit> <latexit sha1_base64="WSac9fWZ+QE538KWn4+gz2K9Jv4=">AAACmHichVG7SgNBFL1ZXzE+ErURbYIhEi3CrAiKVdBCxSYP85AYwu46iUv2xe4kEIM/YGclmErBQiz8Axsbf8AinyApI9hYePcBosF4l9m5c+aeM/dwRUORLUZIx8cNDY+MjvnHAxOTU9PB0MxsztLrpkSzkq7oZkEULKrIGs0ymSm0YJhUUEWF5sXajn2fb1DTknXtkDUNWlKFqiZXZElgCOUPygaNsZVyKELixIlwf8J7SQS8SOqhJziGE9BBgjqoQEEDhrkCAlj4FYEHAgZiJWghZmImO/cUziGA3DpWUawQEK3hv4qnoodqeLY1LYct4SsKLhOZYYiSV3JPeuSFPJA38vmnVsvRsHtp4i66XGqUgxfzmY9/WSruDE6/WQMYIla7nXXII/bVPVot5gd6ZFCBTcebjF4NB7FdS65O4+yql9lKR1vL5JZ00e8NKj+jstZ4l+5SNN3GcfG/h9Of5NbiPInzqfVIYtsbnB8WYQliOJ0NSMAeJCHr9HYJ19DmFrgEt8vtu6Wcz+PMwY/g0l/eDZev</latexit> <latexit sha1_base64="WSac9fWZ+QE538KWn4+gz2K9Jv4=">AAACmHichVG7SgNBFL1ZXzE+ErURbYIhEi3CrAiKVdBCxSYP85AYwu46iUv2xe4kEIM/YGclmErBQiz8Axsbf8AinyApI9hYePcBosF4l9m5c+aeM/dwRUORLUZIx8cNDY+MjvnHAxOTU9PB0MxsztLrpkSzkq7oZkEULKrIGs0ymSm0YJhUUEWF5sXajn2fb1DTknXtkDUNWlKFqiZXZElgCOUPygaNsZVyKELixIlwf8J7SQS8SOqhJziGE9BBgjqoQEEDhrkCAlj4FYEHAgZiJWghZmImO/cUziGA3DpWUawQEK3hv4qnoodqeLY1LYct4SsKLhOZYYiSV3JPeuSFPJA38vmnVsvRsHtp4i66XGqUgxfzmY9/WSruDE6/WQMYIla7nXXII/bVPVot5gd6ZFCBTcebjF4NB7FdS65O4+yql9lKR1vL5JZ00e8NKj+jstZ4l+5SNN3GcfG/h9Of5NbiPInzqfVIYtsbnB8WYQliOJ0NSMAeJCHr9HYJ19DmFrgEt8vtu6Wcz+PMwY/g0l/eDZev</latexit> <latexit sha1_base64="WSac9fWZ+QE538KWn4+gz2K9Jv4=">AAACmHichVG7SgNBFL1ZXzE+ErURbYIhEi3CrAiKVdBCxSYP85AYwu46iUv2xe4kEIM/YGclmErBQiz8Axsbf8AinyApI9hYePcBosF4l9m5c+aeM/dwRUORLUZIx8cNDY+MjvnHAxOTU9PB0MxsztLrpkSzkq7oZkEULKrIGs0ymSm0YJhUUEWF5sXajn2fb1DTknXtkDUNWlKFqiZXZElgCOUPygaNsZVyKELixIlwf8J7SQS8SOqhJziGE9BBgjqoQEEDhrkCAlj4FYEHAgZiJWghZmImO/cUziGA3DpWUawQEK3hv4qnoodqeLY1LYct4SsKLhOZYYiSV3JPeuSFPJA38vmnVsvRsHtp4i66XGqUgxfzmY9/WSruDE6/WQMYIla7nXXII/bVPVot5gd6ZFCBTcebjF4NB7FdS65O4+yql9lKR1vL5JZ00e8NKj+jstZ4l+5SNN3GcfG/h9Of5NbiPInzqfVIYtsbnB8WYQliOJ0NSMAeJCHr9HYJ19DmFrgEt8vtu6Wcz+PMwY/g0l/eDZev</latexit> * Ki Z t 0 e(r)dr <latexit sha1_base64="aDkxGx5q/sb9YJIQEipbOLXS4BA=">AAACpnichVFNS+NQFD3Gz6lf1W4G3JQpSp1FuRFBcSW6EQakrVOrVM0k8amPpklIXgu1+Af8Ay5mMYwgKC78B7OZzfyBWfgTxKWCm1nMbRoQFfWGvHfeefecdy/X8h0ZKqLrLq27p7evf+BDYnBoeGQ0OTa+Hnr1wBYl23O8YMMyQ+FIV5SUVI7Y8ANh1ixHlK3qcvu+3BBBKD33q2r6Yrtm7rtyT9qmYspIpr4Ycku6yqAdlRbZYDq9GxjJDOUoivRLoMcggzjyXvIXtrALDzbqqEHAhWLswETIXwU6CD5z22gxFzCS0b3AERKsrXOW4AyT2Sqv+3yqxKzL57ZnGKltfsXhP2BlGpP0ly7ojv7QJd3Qv1e9WpFHu5Ym71ZHK3xj9Pjj2sO7qhrvCgePqjcUFmd3KrumK67rdvNzpfxmjwp7mI96k9yrHzHtru2OT+Pw5G5toTjZmqJTuuV+f7Lzb3Z2G/f2WUEUv/O49OfDeQnWZ3I65fTCbGZxKR7cACbwCVmezhwWsYI8SvxqEz9wjgstq61qJa3cSdW6Yk0KT0L79h9J15xt</latexit> <latexit sha1_base64="aDkxGx5q/sb9YJIQEipbOLXS4BA=">AAACpnichVFNS+NQFD3Gz6lf1W4G3JQpSp1FuRFBcSW6EQakrVOrVM0k8amPpklIXgu1+Af8Ay5mMYwgKC78B7OZzfyBWfgTxKWCm1nMbRoQFfWGvHfeefecdy/X8h0ZKqLrLq27p7evf+BDYnBoeGQ0OTa+Hnr1wBYl23O8YMMyQ+FIV5SUVI7Y8ANh1ixHlK3qcvu+3BBBKD33q2r6Yrtm7rtyT9qmYspIpr4Ycku6yqAdlRbZYDq9GxjJDOUoivRLoMcggzjyXvIXtrALDzbqqEHAhWLswETIXwU6CD5z22gxFzCS0b3AERKsrXOW4AyT2Sqv+3yqxKzL57ZnGKltfsXhP2BlGpP0ly7ojv7QJd3Qv1e9WpFHu5Ym71ZHK3xj9Pjj2sO7qhrvCgePqjcUFmd3KrumK67rdvNzpfxmjwp7mI96k9yrHzHtru2OT+Pw5G5toTjZmqJTuuV+f7Lzb3Z2G/f2WUEUv/O49OfDeQnWZ3I65fTCbGZxKR7cACbwCVmezhwWsYI8SvxqEz9wjgstq61qJa3cSdW6Yk0KT0L79h9J15xt</latexit> <latexit sha1_base64="aDkxGx5q/sb9YJIQEipbOLXS4BA=">AAACpnichVFNS+NQFD3Gz6lf1W4G3JQpSp1FuRFBcSW6EQakrVOrVM0k8amPpklIXgu1+Af8Ay5mMYwgKC78B7OZzfyBWfgTxKWCm1nMbRoQFfWGvHfeefecdy/X8h0ZKqLrLq27p7evf+BDYnBoeGQ0OTa+Hnr1wBYl23O8YMMyQ+FIV5SUVI7Y8ANh1ixHlK3qcvu+3BBBKD33q2r6Yrtm7rtyT9qmYspIpr4Ycku6yqAdlRbZYDq9GxjJDOUoivRLoMcggzjyXvIXtrALDzbqqEHAhWLswETIXwU6CD5z22gxFzCS0b3AERKsrXOW4AyT2Sqv+3yqxKzL57ZnGKltfsXhP2BlGpP0ly7ojv7QJd3Qv1e9WpFHu5Ym71ZHK3xj9Pjj2sO7qhrvCgePqjcUFmd3KrumK67rdvNzpfxmjwp7mI96k9yrHzHtru2OT+Pw5G5toTjZmqJTuuV+f7Lzb3Z2G/f2WUEUv/O49OfDeQnWZ3I65fTCbGZxKR7cACbwCVmezhwWsYI8SvxqEz9wjgstq61qJa3cSdW6Yk0KT0L79h9J15xt</latexit> <latexit sha1_base64="aDkxGx5q/sb9YJIQEipbOLXS4BA=">AAACpnichVFNS+NQFD3Gz6lf1W4G3JQpSp1FuRFBcSW6EQakrVOrVM0k8amPpklIXgu1+Af8Ay5mMYwgKC78B7OZzfyBWfgTxKWCm1nMbRoQFfWGvHfeefecdy/X8h0ZKqLrLq27p7evf+BDYnBoeGQ0OTa+Hnr1wBYl23O8YMMyQ+FIV5SUVI7Y8ANh1ixHlK3qcvu+3BBBKD33q2r6Yrtm7rtyT9qmYspIpr4Ycku6yqAdlRbZYDq9GxjJDOUoivRLoMcggzjyXvIXtrALDzbqqEHAhWLswETIXwU6CD5z22gxFzCS0b3AERKsrXOW4AyT2Sqv+3yqxKzL57ZnGKltfsXhP2BlGpP0ly7ojv7QJd3Qv1e9WpFHu5Ym71ZHK3xj9Pjj2sO7qhrvCgePqjcUFmd3KrumK67rdvNzpfxmjwp7mI96k9yrHzHtru2OT+Pw5G5toTjZmqJTuuV+f7Lzb3Z2G/f2WUEUv/O49OfDeQnWZ3I65fTCbGZxKR7cACbwCVmezhwWsYI8SvxqEz9wjgstq61qJa3cSdW6Yk0KT0L79h9J15xt</latexit> % Kd de(t) dt <latexit sha1_base64="pK4ns8fNxecVndhwIulsD6E0PEo=">AAACp3ichVHLSsNAFD2Nr/quj4XgplgUdVEmIiiuim4EF9rWPqRKSdKphqZJSKYFDf0Bf8CFGxVcVBf+gRs3/oALP0FcKrhx4W0aEBX1hsycOXPPmXu5qm3ormDsMSR1dHZ194R7+/oHBoeGIyOjWdeqORrPaJZhOXlVcbmhmzwjdGHwvO1wpaoaPKdW1lr3uTp3XN0yt8Whzfeqyr6pl3VNEUQVI+MbxVJ0t+womlfis2Ku4ZVEoxiJsTjzI/oTyAGIIYgtK3KLXZRgQUMNVXCYEIQNKHDpK0AGg03cHjziHEK6f8/RQB9pa5TFKUMhtkLrPp0KAWvSueXp+mqNXjHod0gZxTR7YE32wu7ZNXti7796eb5Hq5ZD2tW2ltvF4eOJ9Nu/qirtAgefqj8UKmW3K3tkN1TX8858IfdnjwJlLPu96dSr7TOtrrW2T/3o5CW9kpr2ZtgFe6Z+z8n5jpzN+qt2meSpUxqX/H04P0F2IS6zuJxcjCVWg8GFMYkpzNJ0lpDAOraQoVePcIYmrqQ5aVPKSvl2qhQKNGP4EpLyAfEdnZo=</latexit> <latexit sha1_base64="pK4ns8fNxecVndhwIulsD6E0PEo=">AAACp3ichVHLSsNAFD2Nr/quj4XgplgUdVEmIiiuim4EF9rWPqRKSdKphqZJSKYFDf0Bf8CFGxVcVBf+gRs3/oALP0FcKrhx4W0aEBX1hsycOXPPmXu5qm3ormDsMSR1dHZ194R7+/oHBoeGIyOjWdeqORrPaJZhOXlVcbmhmzwjdGHwvO1wpaoaPKdW1lr3uTp3XN0yt8Whzfeqyr6pl3VNEUQVI+MbxVJ0t+womlfis2Ku4ZVEoxiJsTjzI/oTyAGIIYgtK3KLXZRgQUMNVXCYEIQNKHDpK0AGg03cHjziHEK6f8/RQB9pa5TFKUMhtkLrPp0KAWvSueXp+mqNXjHod0gZxTR7YE32wu7ZNXti7796eb5Hq5ZD2tW2ltvF4eOJ9Nu/qirtAgefqj8UKmW3K3tkN1TX8858IfdnjwJlLPu96dSr7TOtrrW2T/3o5CW9kpr2ZtgFe6Z+z8n5jpzN+qt2meSpUxqX/H04P0F2IS6zuJxcjCVWg8GFMYkpzNJ0lpDAOraQoVePcIYmrqQ5aVPKSvl2qhQKNGP4EpLyAfEdnZo=</latexit> <latexit sha1_base64="pK4ns8fNxecVndhwIulsD6E0PEo=">AAACp3ichVHLSsNAFD2Nr/quj4XgplgUdVEmIiiuim4EF9rWPqRKSdKphqZJSKYFDf0Bf8CFGxVcVBf+gRs3/oALP0FcKrhx4W0aEBX1hsycOXPPmXu5qm3ormDsMSR1dHZ194R7+/oHBoeGIyOjWdeqORrPaJZhOXlVcbmhmzwjdGHwvO1wpaoaPKdW1lr3uTp3XN0yt8Whzfeqyr6pl3VNEUQVI+MbxVJ0t+womlfis2Ku4ZVEoxiJsTjzI/oTyAGIIYgtK3KLXZRgQUMNVXCYEIQNKHDpK0AGg03cHjziHEK6f8/RQB9pa5TFKUMhtkLrPp0KAWvSueXp+mqNXjHod0gZxTR7YE32wu7ZNXti7796eb5Hq5ZD2tW2ltvF4eOJ9Nu/qirtAgefqj8UKmW3K3tkN1TX8858IfdnjwJlLPu96dSr7TOtrrW2T/3o5CW9kpr2ZtgFe6Z+z8n5jpzN+qt2meSpUxqX/H04P0F2IS6zuJxcjCVWg8GFMYkpzNJ0lpDAOraQoVePcIYmrqQ5aVPKSvl2qhQKNGP4EpLyAfEdnZo=</latexit> <latexit sha1_base64="pK4ns8fNxecVndhwIulsD6E0PEo=">AAACp3ichVHLSsNAFD2Nr/quj4XgplgUdVEmIiiuim4EF9rWPqRKSdKphqZJSKYFDf0Bf8CFGxVcVBf+gRs3/oALP0FcKrhx4W0aEBX1hsycOXPPmXu5qm3ormDsMSR1dHZ194R7+/oHBoeGIyOjWdeqORrPaJZhOXlVcbmhmzwjdGHwvO1wpaoaPKdW1lr3uTp3XN0yt8Whzfeqyr6pl3VNEUQVI+MbxVJ0t+womlfis2Ku4ZVEoxiJsTjzI/oTyAGIIYgtK3KLXZRgQUMNVXCYEIQNKHDpK0AGg03cHjziHEK6f8/RQB9pa5TFKUMhtkLrPp0KAWvSueXp+mqNXjHod0gZxTR7YE32wu7ZNXti7796eb5Hq5ZD2tW2ltvF4eOJ9Nu/qirtAgefqj8UKmW3K3tkN1TX8858IfdnjwJlLPu96dSr7TOtrrW2T/3o5CW9kpr2ZtgFe6Z+z8n5jpzN+qt2meSpUxqX/H04P0F2IS6zuJxcjCVWg8GFMYkpzNJ0lpDAOraQoVePcIYmrqQ5aVPKSvl2qhQKNGP4EpLyAfEdnZo=</latexit> r <latexit sha1_base64="l3XaGwPmE5Vm/Jhs1LG2xz0k9Bs=">AAACknichVG7SgNBFD1ZXzE+Eh+FYCMGRSzCrAiKVdTGwsIk5iFRZHcddchmd9mdBDT4A9oqFlYKFmLhH9jY+AMWfoKkjGBj4c1mQVTUu+zMmTP3nLmXqzum8CRjzyGlrb2jsyvcHenp7euPxgYGc55dcQ2eNWzTdgu65nFTWDwrhTR5wXG5VtZNntdLy837fJW7nrCtdXng8K2ytmeJXWFokqiUux2LswTzY+wnUAMQRxBrduwem9iBDQMVlMFhQRI2ocGjrwgVDA5xW6gR5xIS/j3HESKkrVAWpwyN2BKte3QqBqxF56an56sNesWk3yXlGCbYE7thDfbIbtkLe//Vq+Z7NGs5oF1vabmzHT0eybz9qyrTLrH/qfpDoVN2q7Jndkd11Temi/k/e5TYxbzfm6BeHZ9pdm20fKqH543MQnqiNsmuWJ36vSTnB3K2qq/GdYqnL2hc6vfh/AS5mYTKEmpqNp5cCgYXxijGMUXTmUMSK1hDll7lOMEpzpRhZUFZVJZbqUoo0AzhSyirH/1ElaE=</latexit> <latexit sha1_base64="l3XaGwPmE5Vm/Jhs1LG2xz0k9Bs=">AAACknichVG7SgNBFD1ZXzE+Eh+FYCMGRSzCrAiKVdTGwsIk5iFRZHcddchmd9mdBDT4A9oqFlYKFmLhH9jY+AMWfoKkjGBj4c1mQVTUu+zMmTP3nLmXqzum8CRjzyGlrb2jsyvcHenp7euPxgYGc55dcQ2eNWzTdgu65nFTWDwrhTR5wXG5VtZNntdLy837fJW7nrCtdXng8K2ytmeJXWFokqiUux2LswTzY+wnUAMQRxBrduwem9iBDQMVlMFhQRI2ocGjrwgVDA5xW6gR5xIS/j3HESKkrVAWpwyN2BKte3QqBqxF56an56sNesWk3yXlGCbYE7thDfbIbtkLe//Vq+Z7NGs5oF1vabmzHT0eybz9qyrTLrH/qfpDoVN2q7Jndkd11Temi/k/e5TYxbzfm6BeHZ9pdm20fKqH543MQnqiNsmuWJ36vSTnB3K2qq/GdYqnL2hc6vfh/AS5mYTKEmpqNp5cCgYXxijGMUXTmUMSK1hDll7lOMEpzpRhZUFZVJZbqUoo0AzhSyirH/1ElaE=</latexit> <latexit sha1_base64="l3XaGwPmE5Vm/Jhs1LG2xz0k9Bs=">AAACknichVG7SgNBFD1ZXzE+Eh+FYCMGRSzCrAiKVdTGwsIk5iFRZHcddchmd9mdBDT4A9oqFlYKFmLhH9jY+AMWfoKkjGBj4c1mQVTUu+zMmTP3nLmXqzum8CRjzyGlrb2jsyvcHenp7euPxgYGc55dcQ2eNWzTdgu65nFTWDwrhTR5wXG5VtZNntdLy837fJW7nrCtdXng8K2ytmeJXWFokqiUux2LswTzY+wnUAMQRxBrduwem9iBDQMVlMFhQRI2ocGjrwgVDA5xW6gR5xIS/j3HESKkrVAWpwyN2BKte3QqBqxF56an56sNesWk3yXlGCbYE7thDfbIbtkLe//Vq+Z7NGs5oF1vabmzHT0eybz9qyrTLrH/qfpDoVN2q7Jndkd11Temi/k/e5TYxbzfm6BeHZ9pdm20fKqH543MQnqiNsmuWJ36vSTnB3K2qq/GdYqnL2hc6vfh/AS5mYTKEmpqNp5cCgYXxijGMUXTmUMSK1hDll7lOMEpzpRhZUFZVJZbqUoo0AzhSyirH/1ElaE=</latexit> <latexit sha1_base64="l3XaGwPmE5Vm/Jhs1LG2xz0k9Bs=">AAACknichVG7SgNBFD1ZXzE+Eh+FYCMGRSzCrAiKVdTGwsIk5iFRZHcddchmd9mdBDT4A9oqFlYKFmLhH9jY+AMWfoKkjGBj4c1mQVTUu+zMmTP3nLmXqzum8CRjzyGlrb2jsyvcHenp7euPxgYGc55dcQ2eNWzTdgu65nFTWDwrhTR5wXG5VtZNntdLy837fJW7nrCtdXng8K2ytmeJXWFokqiUux2LswTzY+wnUAMQRxBrduwem9iBDQMVlMFhQRI2ocGjrwgVDA5xW6gR5xIS/j3HESKkrVAWpwyN2BKte3QqBqxF56an56sNesWk3yXlGCbYE7thDfbIbtkLe//Vq+Z7NGs5oF1vabmzHT0eybz9qyrTLrH/qfpDoVN2q7Jndkd11Temi/k/e5TYxbzfm6BeHZ9pdm20fKqH543MQnqiNsmuWJ36vSTnB3K2qq/GdYqnL2hc6vfh/AS5mYTKEmpqNp5cCgYXxijGMUXTmUMSK1hDll7lOMEpzpRhZUFZVJZbqUoo0AzhSyirH/1ElaE=</latexit> e <latexit sha1_base64="JucbY3V7iD4pyjirU4Kr9PWwJLM=">AAACknichVG7SgNBFD1Z3/EVH4VgIwZFLMKsCEoqNY2FRR7GKFFkdx11cLO77E4CMfgD2ioWVgoWYuEf2Nj4AxZ+glgq2Fh49wGiot5lZ86cuefMvVzdMYUnGXuMKU3NLa1t7R3xzq7unt5EX/+KZ1ddgxcN27TdVV3zuCksXpRCmnzVcblW0U1e0vcy/n2pxl1P2NayrDt8o6LtWGJbGJokKsc3E0mWYkGM/ARqBJKIImsnbrGOLdgwUEUFHBYkYRMaPPrKUMHgELeBBnEuIRHccxwgTtoqZXHK0Ijdo3WHTuWItejse3qB2qBXTPpdUo5gjD2wK/bC7tk1e2Lvv3o1Ag+/ljrteqjlzmbv4VDh7V9VhXaJ3U/VHwqdssPKHtkN1fW8Nlku/dmjxDZmg94E9eoEjN+1EfrU9k9fCun8WGOcXbBn6vecnO/I2aq9Gpc5nj+jcanfh/MTrEylVJZSc9PJuYVocO0YxigmaDozmMMisijSqxxHOMaJMqiklXklE6YqsUgzgC+hLH0A4O6VlA==</latexit> <latexit sha1_base64="JucbY3V7iD4pyjirU4Kr9PWwJLM=">AAACknichVG7SgNBFD1Z3/EVH4VgIwZFLMKsCEoqNY2FRR7GKFFkdx11cLO77E4CMfgD2ioWVgoWYuEf2Nj4AxZ+glgq2Fh49wGiot5lZ86cuefMvVzdMYUnGXuMKU3NLa1t7R3xzq7unt5EX/+KZ1ddgxcN27TdVV3zuCksXpRCmnzVcblW0U1e0vcy/n2pxl1P2NayrDt8o6LtWGJbGJokKsc3E0mWYkGM/ARqBJKIImsnbrGOLdgwUEUFHBYkYRMaPPrKUMHgELeBBnEuIRHccxwgTtoqZXHK0Ijdo3WHTuWItejse3qB2qBXTPpdUo5gjD2wK/bC7tk1e2Lvv3o1Ag+/ljrteqjlzmbv4VDh7V9VhXaJ3U/VHwqdssPKHtkN1fW8Nlku/dmjxDZmg94E9eoEjN+1EfrU9k9fCun8WGOcXbBn6vecnO/I2aq9Gpc5nj+jcanfh/MTrEylVJZSc9PJuYVocO0YxigmaDozmMMisijSqxxHOMaJMqiklXklE6YqsUgzgC+hLH0A4O6VlA==</latexit> <latexit sha1_base64="JucbY3V7iD4pyjirU4Kr9PWwJLM=">AAACknichVG7SgNBFD1Z3/EVH4VgIwZFLMKsCEoqNY2FRR7GKFFkdx11cLO77E4CMfgD2ioWVgoWYuEf2Nj4AxZ+glgq2Fh49wGiot5lZ86cuefMvVzdMYUnGXuMKU3NLa1t7R3xzq7unt5EX/+KZ1ddgxcN27TdVV3zuCksXpRCmnzVcblW0U1e0vcy/n2pxl1P2NayrDt8o6LtWGJbGJokKsc3E0mWYkGM/ARqBJKIImsnbrGOLdgwUEUFHBYkYRMaPPrKUMHgELeBBnEuIRHccxwgTtoqZXHK0Ijdo3WHTuWItejse3qB2qBXTPpdUo5gjD2wK/bC7tk1e2Lvv3o1Ag+/ljrteqjlzmbv4VDh7V9VhXaJ3U/VHwqdssPKHtkN1fW8Nlku/dmjxDZmg94E9eoEjN+1EfrU9k9fCun8WGOcXbBn6vecnO/I2aq9Gpc5nj+jcanfh/MTrEylVJZSc9PJuYVocO0YxigmaDozmMMisijSqxxHOMaJMqiklXklE6YqsUgzgC+hLH0A4O6VlA==</latexit> <latexit sha1_base64="JucbY3V7iD4pyjirU4Kr9PWwJLM=">AAACknichVG7SgNBFD1Z3/EVH4VgIwZFLMKsCEoqNY2FRR7GKFFkdx11cLO77E4CMfgD2ioWVgoWYuEf2Nj4AxZ+glgq2Fh49wGiot5lZ86cuefMvVzdMYUnGXuMKU3NLa1t7R3xzq7unt5EX/+KZ1ddgxcN27TdVV3zuCksXpRCmnzVcblW0U1e0vcy/n2pxl1P2NayrDt8o6LtWGJbGJokKsc3E0mWYkGM/ARqBJKIImsnbrGOLdgwUEUFHBYkYRMaPPrKUMHgELeBBnEuIRHccxwgTtoqZXHK0Ijdo3WHTuWItejse3qB2qBXTPpdUo5gjD2wK/bC7tk1e2Lvv3o1Ag+/ljrteqjlzmbv4VDh7V9VhXaJ3U/VHwqdssPKHtkN1fW8Nlku/dmjxDZmg94E9eoEjN+1EfrU9k9fCun8WGOcXbBn6vecnO/I2aq9Gpc5nj+jcanfh/MTrEylVJZSc9PJuYVocO0YxigmaDozmMMisijSqxxHOMaJMqiklXklE6YqsUgzgC+hLH0A4O6VlA==</latexit> 4ZTUFN  'FFECBDLQBUI *OQVU 0VUQVU
  24. • Proportional control(P) output is proportional to the error. •

    Integral control(I) output is proportional to the integral of the error over time. • Derivative control(D) output is proportional to the derivative of the error. 36 PID Controller (2/2) $POUSPMMFS &SSPS 4FUQPJOU 1 Kpe(t) <latexit sha1_base64="WSac9fWZ+QE538KWn4+gz2K9Jv4=">AAACmHichVG7SgNBFL1ZXzE+ErURbYIhEi3CrAiKVdBCxSYP85AYwu46iUv2xe4kEIM/YGclmErBQiz8Axsbf8AinyApI9hYePcBosF4l9m5c+aeM/dwRUORLUZIx8cNDY+MjvnHAxOTU9PB0MxsztLrpkSzkq7oZkEULKrIGs0ymSm0YJhUUEWF5sXajn2fb1DTknXtkDUNWlKFqiZXZElgCOUPygaNsZVyKELixIlwf8J7SQS8SOqhJziGE9BBgjqoQEEDhrkCAlj4FYEHAgZiJWghZmImO/cUziGA3DpWUawQEK3hv4qnoodqeLY1LYct4SsKLhOZYYiSV3JPeuSFPJA38vmnVsvRsHtp4i66XGqUgxfzmY9/WSruDE6/WQMYIla7nXXII/bVPVot5gd6ZFCBTcebjF4NB7FdS65O4+yql9lKR1vL5JZ00e8NKj+jstZ4l+5SNN3GcfG/h9Of5NbiPInzqfVIYtsbnB8WYQliOJ0NSMAeJCHr9HYJ19DmFrgEt8vtu6Wcz+PMwY/g0l/eDZev</latexit> <latexit sha1_base64="WSac9fWZ+QE538KWn4+gz2K9Jv4=">AAACmHichVG7SgNBFL1ZXzE+ErURbYIhEi3CrAiKVdBCxSYP85AYwu46iUv2xe4kEIM/YGclmErBQiz8Axsbf8AinyApI9hYePcBosF4l9m5c+aeM/dwRUORLUZIx8cNDY+MjvnHAxOTU9PB0MxsztLrpkSzkq7oZkEULKrIGs0ymSm0YJhUUEWF5sXajn2fb1DTknXtkDUNWlKFqiZXZElgCOUPygaNsZVyKELixIlwf8J7SQS8SOqhJziGE9BBgjqoQEEDhrkCAlj4FYEHAgZiJWghZmImO/cUziGA3DpWUawQEK3hv4qnoodqeLY1LYct4SsKLhOZYYiSV3JPeuSFPJA38vmnVsvRsHtp4i66XGqUgxfzmY9/WSruDE6/WQMYIla7nXXII/bVPVot5gd6ZFCBTcebjF4NB7FdS65O4+yql9lKR1vL5JZ00e8NKj+jstZ4l+5SNN3GcfG/h9Of5NbiPInzqfVIYtsbnB8WYQliOJ0NSMAeJCHr9HYJ19DmFrgEt8vtu6Wcz+PMwY/g0l/eDZev</latexit> <latexit sha1_base64="WSac9fWZ+QE538KWn4+gz2K9Jv4=">AAACmHichVG7SgNBFL1ZXzE+ErURbYIhEi3CrAiKVdBCxSYP85AYwu46iUv2xe4kEIM/YGclmErBQiz8Axsbf8AinyApI9hYePcBosF4l9m5c+aeM/dwRUORLUZIx8cNDY+MjvnHAxOTU9PB0MxsztLrpkSzkq7oZkEULKrIGs0ymSm0YJhUUEWF5sXajn2fb1DTknXtkDUNWlKFqiZXZElgCOUPygaNsZVyKELixIlwf8J7SQS8SOqhJziGE9BBgjqoQEEDhrkCAlj4FYEHAgZiJWghZmImO/cUziGA3DpWUawQEK3hv4qnoodqeLY1LYct4SsKLhOZYYiSV3JPeuSFPJA38vmnVsvRsHtp4i66XGqUgxfzmY9/WSruDE6/WQMYIla7nXXII/bVPVot5gd6ZFCBTcebjF4NB7FdS65O4+yql9lKR1vL5JZ00e8NKj+jstZ4l+5SNN3GcfG/h9Of5NbiPInzqfVIYtsbnB8WYQliOJ0NSMAeJCHr9HYJ19DmFrgEt8vtu6Wcz+PMwY/g0l/eDZev</latexit> <latexit sha1_base64="WSac9fWZ+QE538KWn4+gz2K9Jv4=">AAACmHichVG7SgNBFL1ZXzE+ErURbYIhEi3CrAiKVdBCxSYP85AYwu46iUv2xe4kEIM/YGclmErBQiz8Axsbf8AinyApI9hYePcBosF4l9m5c+aeM/dwRUORLUZIx8cNDY+MjvnHAxOTU9PB0MxsztLrpkSzkq7oZkEULKrIGs0ymSm0YJhUUEWF5sXajn2fb1DTknXtkDUNWlKFqiZXZElgCOUPygaNsZVyKELixIlwf8J7SQS8SOqhJziGE9BBgjqoQEEDhrkCAlj4FYEHAgZiJWghZmImO/cUziGA3DpWUawQEK3hv4qnoodqeLY1LYct4SsKLhOZYYiSV3JPeuSFPJA38vmnVsvRsHtp4i66XGqUgxfzmY9/WSruDE6/WQMYIla7nXXII/bVPVot5gd6ZFCBTcebjF4NB7FdS65O4+yql9lKR1vL5JZ00e8NKj+jstZ4l+5SNN3GcfG/h9Of5NbiPInzqfVIYtsbnB8WYQliOJ0NSMAeJCHr9HYJ19DmFrgEt8vtu6Wcz+PMwY/g0l/eDZev</latexit> * Ki Z t 0 e(r)dr <latexit sha1_base64="aDkxGx5q/sb9YJIQEipbOLXS4BA=">AAACpnichVFNS+NQFD3Gz6lf1W4G3JQpSp1FuRFBcSW6EQakrVOrVM0k8amPpklIXgu1+Af8Ay5mMYwgKC78B7OZzfyBWfgTxKWCm1nMbRoQFfWGvHfeefecdy/X8h0ZKqLrLq27p7evf+BDYnBoeGQ0OTa+Hnr1wBYl23O8YMMyQ+FIV5SUVI7Y8ANh1ixHlK3qcvu+3BBBKD33q2r6Yrtm7rtyT9qmYspIpr4Ycku6yqAdlRbZYDq9GxjJDOUoivRLoMcggzjyXvIXtrALDzbqqEHAhWLswETIXwU6CD5z22gxFzCS0b3AERKsrXOW4AyT2Sqv+3yqxKzL57ZnGKltfsXhP2BlGpP0ly7ojv7QJd3Qv1e9WpFHu5Ym71ZHK3xj9Pjj2sO7qhrvCgePqjcUFmd3KrumK67rdvNzpfxmjwp7mI96k9yrHzHtru2OT+Pw5G5toTjZmqJTuuV+f7Lzb3Z2G/f2WUEUv/O49OfDeQnWZ3I65fTCbGZxKR7cACbwCVmezhwWsYI8SvxqEz9wjgstq61qJa3cSdW6Yk0KT0L79h9J15xt</latexit> <latexit sha1_base64="aDkxGx5q/sb9YJIQEipbOLXS4BA=">AAACpnichVFNS+NQFD3Gz6lf1W4G3JQpSp1FuRFBcSW6EQakrVOrVM0k8amPpklIXgu1+Af8Ay5mMYwgKC78B7OZzfyBWfgTxKWCm1nMbRoQFfWGvHfeefecdy/X8h0ZKqLrLq27p7evf+BDYnBoeGQ0OTa+Hnr1wBYl23O8YMMyQ+FIV5SUVI7Y8ANh1ixHlK3qcvu+3BBBKD33q2r6Yrtm7rtyT9qmYspIpr4Ycku6yqAdlRbZYDq9GxjJDOUoivRLoMcggzjyXvIXtrALDzbqqEHAhWLswETIXwU6CD5z22gxFzCS0b3AERKsrXOW4AyT2Sqv+3yqxKzL57ZnGKltfsXhP2BlGpP0ly7ojv7QJd3Qv1e9WpFHu5Ym71ZHK3xj9Pjj2sO7qhrvCgePqjcUFmd3KrumK67rdvNzpfxmjwp7mI96k9yrHzHtru2OT+Pw5G5toTjZmqJTuuV+f7Lzb3Z2G/f2WUEUv/O49OfDeQnWZ3I65fTCbGZxKR7cACbwCVmezhwWsYI8SvxqEz9wjgstq61qJa3cSdW6Yk0KT0L79h9J15xt</latexit> <latexit sha1_base64="aDkxGx5q/sb9YJIQEipbOLXS4BA=">AAACpnichVFNS+NQFD3Gz6lf1W4G3JQpSp1FuRFBcSW6EQakrVOrVM0k8amPpklIXgu1+Af8Ay5mMYwgKC78B7OZzfyBWfgTxKWCm1nMbRoQFfWGvHfeefecdy/X8h0ZKqLrLq27p7evf+BDYnBoeGQ0OTa+Hnr1wBYl23O8YMMyQ+FIV5SUVI7Y8ANh1ixHlK3qcvu+3BBBKD33q2r6Yrtm7rtyT9qmYspIpr4Ycku6yqAdlRbZYDq9GxjJDOUoivRLoMcggzjyXvIXtrALDzbqqEHAhWLswETIXwU6CD5z22gxFzCS0b3AERKsrXOW4AyT2Sqv+3yqxKzL57ZnGKltfsXhP2BlGpP0ly7ojv7QJd3Qv1e9WpFHu5Ym71ZHK3xj9Pjj2sO7qhrvCgePqjcUFmd3KrumK67rdvNzpfxmjwp7mI96k9yrHzHtru2OT+Pw5G5toTjZmqJTuuV+f7Lzb3Z2G/f2WUEUv/O49OfDeQnWZ3I65fTCbGZxKR7cACbwCVmezhwWsYI8SvxqEz9wjgstq61qJa3cSdW6Yk0KT0L79h9J15xt</latexit> <latexit sha1_base64="aDkxGx5q/sb9YJIQEipbOLXS4BA=">AAACpnichVFNS+NQFD3Gz6lf1W4G3JQpSp1FuRFBcSW6EQakrVOrVM0k8amPpklIXgu1+Af8Ay5mMYwgKC78B7OZzfyBWfgTxKWCm1nMbRoQFfWGvHfeefecdy/X8h0ZKqLrLq27p7evf+BDYnBoeGQ0OTa+Hnr1wBYl23O8YMMyQ+FIV5SUVI7Y8ANh1ixHlK3qcvu+3BBBKD33q2r6Yrtm7rtyT9qmYspIpr4Ycku6yqAdlRbZYDq9GxjJDOUoivRLoMcggzjyXvIXtrALDzbqqEHAhWLswETIXwU6CD5z22gxFzCS0b3AERKsrXOW4AyT2Sqv+3yqxKzL57ZnGKltfsXhP2BlGpP0ly7ojv7QJd3Qv1e9WpFHu5Ym71ZHK3xj9Pjj2sO7qhrvCgePqjcUFmd3KrumK67rdvNzpfxmjwp7mI96k9yrHzHtru2OT+Pw5G5toTjZmqJTuuV+f7Lzb3Z2G/f2WUEUv/O49OfDeQnWZ3I65fTCbGZxKR7cACbwCVmezhwWsYI8SvxqEz9wjgstq61qJa3cSdW6Yk0KT0L79h9J15xt</latexit> % Kd de(t) dt <latexit sha1_base64="pK4ns8fNxecVndhwIulsD6E0PEo=">AAACp3ichVHLSsNAFD2Nr/quj4XgplgUdVEmIiiuim4EF9rWPqRKSdKphqZJSKYFDf0Bf8CFGxVcVBf+gRs3/oALP0FcKrhx4W0aEBX1hsycOXPPmXu5qm3ormDsMSR1dHZ194R7+/oHBoeGIyOjWdeqORrPaJZhOXlVcbmhmzwjdGHwvO1wpaoaPKdW1lr3uTp3XN0yt8Whzfeqyr6pl3VNEUQVI+MbxVJ0t+womlfis2Ku4ZVEoxiJsTjzI/oTyAGIIYgtK3KLXZRgQUMNVXCYEIQNKHDpK0AGg03cHjziHEK6f8/RQB9pa5TFKUMhtkLrPp0KAWvSueXp+mqNXjHod0gZxTR7YE32wu7ZNXti7796eb5Hq5ZD2tW2ltvF4eOJ9Nu/qirtAgefqj8UKmW3K3tkN1TX8858IfdnjwJlLPu96dSr7TOtrrW2T/3o5CW9kpr2ZtgFe6Z+z8n5jpzN+qt2meSpUxqX/H04P0F2IS6zuJxcjCVWg8GFMYkpzNJ0lpDAOraQoVePcIYmrqQ5aVPKSvl2qhQKNGP4EpLyAfEdnZo=</latexit> <latexit sha1_base64="pK4ns8fNxecVndhwIulsD6E0PEo=">AAACp3ichVHLSsNAFD2Nr/quj4XgplgUdVEmIiiuim4EF9rWPqRKSdKphqZJSKYFDf0Bf8CFGxVcVBf+gRs3/oALP0FcKrhx4W0aEBX1hsycOXPPmXu5qm3ormDsMSR1dHZ194R7+/oHBoeGIyOjWdeqORrPaJZhOXlVcbmhmzwjdGHwvO1wpaoaPKdW1lr3uTp3XN0yt8Whzfeqyr6pl3VNEUQVI+MbxVJ0t+womlfis2Ku4ZVEoxiJsTjzI/oTyAGIIYgtK3KLXZRgQUMNVXCYEIQNKHDpK0AGg03cHjziHEK6f8/RQB9pa5TFKUMhtkLrPp0KAWvSueXp+mqNXjHod0gZxTR7YE32wu7ZNXti7796eb5Hq5ZD2tW2ltvF4eOJ9Nu/qirtAgefqj8UKmW3K3tkN1TX8858IfdnjwJlLPu96dSr7TOtrrW2T/3o5CW9kpr2ZtgFe6Z+z8n5jpzN+qt2meSpUxqX/H04P0F2IS6zuJxcjCVWg8GFMYkpzNJ0lpDAOraQoVePcIYmrqQ5aVPKSvl2qhQKNGP4EpLyAfEdnZo=</latexit> <latexit sha1_base64="pK4ns8fNxecVndhwIulsD6E0PEo=">AAACp3ichVHLSsNAFD2Nr/quj4XgplgUdVEmIiiuim4EF9rWPqRKSdKphqZJSKYFDf0Bf8CFGxVcVBf+gRs3/oALP0FcKrhx4W0aEBX1hsycOXPPmXu5qm3ormDsMSR1dHZ194R7+/oHBoeGIyOjWdeqORrPaJZhOXlVcbmhmzwjdGHwvO1wpaoaPKdW1lr3uTp3XN0yt8Whzfeqyr6pl3VNEUQVI+MbxVJ0t+womlfis2Ku4ZVEoxiJsTjzI/oTyAGIIYgtK3KLXZRgQUMNVXCYEIQNKHDpK0AGg03cHjziHEK6f8/RQB9pa5TFKUMhtkLrPp0KAWvSueXp+mqNXjHod0gZxTR7YE32wu7ZNXti7796eb5Hq5ZD2tW2ltvF4eOJ9Nu/qirtAgefqj8UKmW3K3tkN1TX8858IfdnjwJlLPu96dSr7TOtrrW2T/3o5CW9kpr2ZtgFe6Z+z8n5jpzN+qt2meSpUxqX/H04P0F2IS6zuJxcjCVWg8GFMYkpzNJ0lpDAOraQoVePcIYmrqQ5aVPKSvl2qhQKNGP4EpLyAfEdnZo=</latexit> <latexit sha1_base64="pK4ns8fNxecVndhwIulsD6E0PEo=">AAACp3ichVHLSsNAFD2Nr/quj4XgplgUdVEmIiiuim4EF9rWPqRKSdKphqZJSKYFDf0Bf8CFGxVcVBf+gRs3/oALP0FcKrhx4W0aEBX1hsycOXPPmXu5qm3ormDsMSR1dHZ194R7+/oHBoeGIyOjWdeqORrPaJZhOXlVcbmhmzwjdGHwvO1wpaoaPKdW1lr3uTp3XN0yt8Whzfeqyr6pl3VNEUQVI+MbxVJ0t+womlfis2Ku4ZVEoxiJsTjzI/oTyAGIIYgtK3KLXZRgQUMNVXCYEIQNKHDpK0AGg03cHjziHEK6f8/RQB9pa5TFKUMhtkLrPp0KAWvSueXp+mqNXjHod0gZxTR7YE32wu7ZNXti7796eb5Hq5ZD2tW2ltvF4eOJ9Nu/qirtAgefqj8UKmW3K3tkN1TX8858IfdnjwJlLPu96dSr7TOtrrW2T/3o5CW9kpr2ZtgFe6Z+z8n5jpzN+qt2meSpUxqX/H04P0F2IS6zuJxcjCVWg8GFMYkpzNJ0lpDAOraQoVePcIYmrqQ5aVPKSvl2qhQKNGP4EpLyAfEdnZo=</latexit> r <latexit sha1_base64="l3XaGwPmE5Vm/Jhs1LG2xz0k9Bs=">AAACknichVG7SgNBFD1ZXzE+Eh+FYCMGRSzCrAiKVdTGwsIk5iFRZHcddchmd9mdBDT4A9oqFlYKFmLhH9jY+AMWfoKkjGBj4c1mQVTUu+zMmTP3nLmXqzum8CRjzyGlrb2jsyvcHenp7euPxgYGc55dcQ2eNWzTdgu65nFTWDwrhTR5wXG5VtZNntdLy837fJW7nrCtdXng8K2ytmeJXWFokqiUux2LswTzY+wnUAMQRxBrduwem9iBDQMVlMFhQRI2ocGjrwgVDA5xW6gR5xIS/j3HESKkrVAWpwyN2BKte3QqBqxF56an56sNesWk3yXlGCbYE7thDfbIbtkLe//Vq+Z7NGs5oF1vabmzHT0eybz9qyrTLrH/qfpDoVN2q7Jndkd11Temi/k/e5TYxbzfm6BeHZ9pdm20fKqH543MQnqiNsmuWJ36vSTnB3K2qq/GdYqnL2hc6vfh/AS5mYTKEmpqNp5cCgYXxijGMUXTmUMSK1hDll7lOMEpzpRhZUFZVJZbqUoo0AzhSyirH/1ElaE=</latexit> <latexit sha1_base64="l3XaGwPmE5Vm/Jhs1LG2xz0k9Bs=">AAACknichVG7SgNBFD1ZXzE+Eh+FYCMGRSzCrAiKVdTGwsIk5iFRZHcddchmd9mdBDT4A9oqFlYKFmLhH9jY+AMWfoKkjGBj4c1mQVTUu+zMmTP3nLmXqzum8CRjzyGlrb2jsyvcHenp7euPxgYGc55dcQ2eNWzTdgu65nFTWDwrhTR5wXG5VtZNntdLy837fJW7nrCtdXng8K2ytmeJXWFokqiUux2LswTzY+wnUAMQRxBrduwem9iBDQMVlMFhQRI2ocGjrwgVDA5xW6gR5xIS/j3HESKkrVAWpwyN2BKte3QqBqxF56an56sNesWk3yXlGCbYE7thDfbIbtkLe//Vq+Z7NGs5oF1vabmzHT0eybz9qyrTLrH/qfpDoVN2q7Jndkd11Temi/k/e5TYxbzfm6BeHZ9pdm20fKqH543MQnqiNsmuWJ36vSTnB3K2qq/GdYqnL2hc6vfh/AS5mYTKEmpqNp5cCgYXxijGMUXTmUMSK1hDll7lOMEpzpRhZUFZVJZbqUoo0AzhSyirH/1ElaE=</latexit> <latexit sha1_base64="l3XaGwPmE5Vm/Jhs1LG2xz0k9Bs=">AAACknichVG7SgNBFD1ZXzE+Eh+FYCMGRSzCrAiKVdTGwsIk5iFRZHcddchmd9mdBDT4A9oqFlYKFmLhH9jY+AMWfoKkjGBj4c1mQVTUu+zMmTP3nLmXqzum8CRjzyGlrb2jsyvcHenp7euPxgYGc55dcQ2eNWzTdgu65nFTWDwrhTR5wXG5VtZNntdLy837fJW7nrCtdXng8K2ytmeJXWFokqiUux2LswTzY+wnUAMQRxBrduwem9iBDQMVlMFhQRI2ocGjrwgVDA5xW6gR5xIS/j3HESKkrVAWpwyN2BKte3QqBqxF56an56sNesWk3yXlGCbYE7thDfbIbtkLe//Vq+Z7NGs5oF1vabmzHT0eybz9qyrTLrH/qfpDoVN2q7Jndkd11Temi/k/e5TYxbzfm6BeHZ9pdm20fKqH543MQnqiNsmuWJ36vSTnB3K2qq/GdYqnL2hc6vfh/AS5mYTKEmpqNp5cCgYXxijGMUXTmUMSK1hDll7lOMEpzpRhZUFZVJZbqUoo0AzhSyirH/1ElaE=</latexit> <latexit sha1_base64="l3XaGwPmE5Vm/Jhs1LG2xz0k9Bs=">AAACknichVG7SgNBFD1ZXzE+Eh+FYCMGRSzCrAiKVdTGwsIk5iFRZHcddchmd9mdBDT4A9oqFlYKFmLhH9jY+AMWfoKkjGBj4c1mQVTUu+zMmTP3nLmXqzum8CRjzyGlrb2jsyvcHenp7euPxgYGc55dcQ2eNWzTdgu65nFTWDwrhTR5wXG5VtZNntdLy837fJW7nrCtdXng8K2ytmeJXWFokqiUux2LswTzY+wnUAMQRxBrduwem9iBDQMVlMFhQRI2ocGjrwgVDA5xW6gR5xIS/j3HESKkrVAWpwyN2BKte3QqBqxF56an56sNesWk3yXlGCbYE7thDfbIbtkLe//Vq+Z7NGs5oF1vabmzHT0eybz9qyrTLrH/qfpDoVN2q7Jndkd11Temi/k/e5TYxbzfm6BeHZ9pdm20fKqH543MQnqiNsmuWJ36vSTnB3K2qq/GdYqnL2hc6vfh/AS5mYTKEmpqNp5cCgYXxijGMUXTmUMSK1hDll7lOMEpzpRhZUFZVJZbqUoo0AzhSyirH/1ElaE=</latexit> e <latexit sha1_base64="JucbY3V7iD4pyjirU4Kr9PWwJLM=">AAACknichVG7SgNBFD1Z3/EVH4VgIwZFLMKsCEoqNY2FRR7GKFFkdx11cLO77E4CMfgD2ioWVgoWYuEf2Nj4AxZ+glgq2Fh49wGiot5lZ86cuefMvVzdMYUnGXuMKU3NLa1t7R3xzq7unt5EX/+KZ1ddgxcN27TdVV3zuCksXpRCmnzVcblW0U1e0vcy/n2pxl1P2NayrDt8o6LtWGJbGJokKsc3E0mWYkGM/ARqBJKIImsnbrGOLdgwUEUFHBYkYRMaPPrKUMHgELeBBnEuIRHccxwgTtoqZXHK0Ijdo3WHTuWItejse3qB2qBXTPpdUo5gjD2wK/bC7tk1e2Lvv3o1Ag+/ljrteqjlzmbv4VDh7V9VhXaJ3U/VHwqdssPKHtkN1fW8Nlku/dmjxDZmg94E9eoEjN+1EfrU9k9fCun8WGOcXbBn6vecnO/I2aq9Gpc5nj+jcanfh/MTrEylVJZSc9PJuYVocO0YxigmaDozmMMisijSqxxHOMaJMqiklXklE6YqsUgzgC+hLH0A4O6VlA==</latexit> <latexit sha1_base64="JucbY3V7iD4pyjirU4Kr9PWwJLM=">AAACknichVG7SgNBFD1Z3/EVH4VgIwZFLMKsCEoqNY2FRR7GKFFkdx11cLO77E4CMfgD2ioWVgoWYuEf2Nj4AxZ+glgq2Fh49wGiot5lZ86cuefMvVzdMYUnGXuMKU3NLa1t7R3xzq7unt5EX/+KZ1ddgxcN27TdVV3zuCksXpRCmnzVcblW0U1e0vcy/n2pxl1P2NayrDt8o6LtWGJbGJokKsc3E0mWYkGM/ARqBJKIImsnbrGOLdgwUEUFHBYkYRMaPPrKUMHgELeBBnEuIRHccxwgTtoqZXHK0Ijdo3WHTuWItejse3qB2qBXTPpdUo5gjD2wK/bC7tk1e2Lvv3o1Ag+/ljrteqjlzmbv4VDh7V9VhXaJ3U/VHwqdssPKHtkN1fW8Nlku/dmjxDZmg94E9eoEjN+1EfrU9k9fCun8WGOcXbBn6vecnO/I2aq9Gpc5nj+jcanfh/MTrEylVJZSc9PJuYVocO0YxigmaDozmMMisijSqxxHOMaJMqiklXklE6YqsUgzgC+hLH0A4O6VlA==</latexit> <latexit sha1_base64="JucbY3V7iD4pyjirU4Kr9PWwJLM=">AAACknichVG7SgNBFD1Z3/EVH4VgIwZFLMKsCEoqNY2FRR7GKFFkdx11cLO77E4CMfgD2ioWVgoWYuEf2Nj4AxZ+glgq2Fh49wGiot5lZ86cuefMvVzdMYUnGXuMKU3NLa1t7R3xzq7unt5EX/+KZ1ddgxcN27TdVV3zuCksXpRCmnzVcblW0U1e0vcy/n2pxl1P2NayrDt8o6LtWGJbGJokKsc3E0mWYkGM/ARqBJKIImsnbrGOLdgwUEUFHBYkYRMaPPrKUMHgELeBBnEuIRHccxwgTtoqZXHK0Ijdo3WHTuWItejse3qB2qBXTPpdUo5gjD2wK/bC7tk1e2Lvv3o1Ag+/ljrteqjlzmbv4VDh7V9VhXaJ3U/VHwqdssPKHtkN1fW8Nlku/dmjxDZmg94E9eoEjN+1EfrU9k9fCun8WGOcXbBn6vecnO/I2aq9Gpc5nj+jcanfh/MTrEylVJZSc9PJuYVocO0YxigmaDozmMMisijSqxxHOMaJMqiklXklE6YqsUgzgC+hLH0A4O6VlA==</latexit> <latexit sha1_base64="JucbY3V7iD4pyjirU4Kr9PWwJLM=">AAACknichVG7SgNBFD1Z3/EVH4VgIwZFLMKsCEoqNY2FRR7GKFFkdx11cLO77E4CMfgD2ioWVgoWYuEf2Nj4AxZ+glgq2Fh49wGiot5lZ86cuefMvVzdMYUnGXuMKU3NLa1t7R3xzq7unt5EX/+KZ1ddgxcN27TdVV3zuCksXpRCmnzVcblW0U1e0vcy/n2pxl1P2NayrDt8o6LtWGJbGJokKsc3E0mWYkGM/ARqBJKIImsnbrGOLdgwUEUFHBYkYRMaPPrKUMHgELeBBnEuIRHccxwgTtoqZXHK0Ijdo3WHTuWItejse3qB2qBXTPpdUo5gjD2wK/bC7tk1e2Lvv3o1Ag+/ljrteqjlzmbv4VDh7V9VhXaJ3U/VHwqdssPKHtkN1fW8Nlku/dmjxDZmg94E9eoEjN+1EfrU9k9fCun8WGOcXbBn6vecnO/I2aq9Gpc5nj+jcanfh/MTrEylVJZSc9PJuYVocO0YxigmaDozmMMisijSqxxHOMaJMqiklXklE6YqsUgzgC+hLH0A4O6VlA==</latexit> 4ZTUFN  'FFECBDLQBUI *OQVU 0VUQVU
  25. 37 Dynamic Target Controller &SSPS 4FUQPJOU 1 * % r

    <latexit sha1_base64="l3XaGwPmE5Vm/Jhs1LG2xz0k9Bs=">AAACknichVG7SgNBFD1ZXzE+Eh+FYCMGRSzCrAiKVdTGwsIk5iFRZHcddchmd9mdBDT4A9oqFlYKFmLhH9jY+AMWfoKkjGBj4c1mQVTUu+zMmTP3nLmXqzum8CRjzyGlrb2jsyvcHenp7euPxgYGc55dcQ2eNWzTdgu65nFTWDwrhTR5wXG5VtZNntdLy837fJW7nrCtdXng8K2ytmeJXWFokqiUux2LswTzY+wnUAMQRxBrduwem9iBDQMVlMFhQRI2ocGjrwgVDA5xW6gR5xIS/j3HESKkrVAWpwyN2BKte3QqBqxF56an56sNesWk3yXlGCbYE7thDfbIbtkLe//Vq+Z7NGs5oF1vabmzHT0eybz9qyrTLrH/qfpDoVN2q7Jndkd11Temi/k/e5TYxbzfm6BeHZ9pdm20fKqH543MQnqiNsmuWJ36vSTnB3K2qq/GdYqnL2hc6vfh/AS5mYTKEmpqNp5cCgYXxijGMUXTmUMSK1hDll7lOMEpzpRhZUFZVJZbqUoo0AzhSyirH/1ElaE=</latexit> <latexit sha1_base64="l3XaGwPmE5Vm/Jhs1LG2xz0k9Bs=">AAACknichVG7SgNBFD1ZXzE+Eh+FYCMGRSzCrAiKVdTGwsIk5iFRZHcddchmd9mdBDT4A9oqFlYKFmLhH9jY+AMWfoKkjGBj4c1mQVTUu+zMmTP3nLmXqzum8CRjzyGlrb2jsyvcHenp7euPxgYGc55dcQ2eNWzTdgu65nFTWDwrhTR5wXG5VtZNntdLy837fJW7nrCtdXng8K2ytmeJXWFokqiUux2LswTzY+wnUAMQRxBrduwem9iBDQMVlMFhQRI2ocGjrwgVDA5xW6gR5xIS/j3HESKkrVAWpwyN2BKte3QqBqxF56an56sNesWk3yXlGCbYE7thDfbIbtkLe//Vq+Z7NGs5oF1vabmzHT0eybz9qyrTLrH/qfpDoVN2q7Jndkd11Temi/k/e5TYxbzfm6BeHZ9pdm20fKqH543MQnqiNsmuWJ36vSTnB3K2qq/GdYqnL2hc6vfh/AS5mYTKEmpqNp5cCgYXxijGMUXTmUMSK1hDll7lOMEpzpRhZUFZVJZbqUoo0AzhSyirH/1ElaE=</latexit> <latexit sha1_base64="l3XaGwPmE5Vm/Jhs1LG2xz0k9Bs=">AAACknichVG7SgNBFD1ZXzE+Eh+FYCMGRSzCrAiKVdTGwsIk5iFRZHcddchmd9mdBDT4A9oqFlYKFmLhH9jY+AMWfoKkjGBj4c1mQVTUu+zMmTP3nLmXqzum8CRjzyGlrb2jsyvcHenp7euPxgYGc55dcQ2eNWzTdgu65nFTWDwrhTR5wXG5VtZNntdLy837fJW7nrCtdXng8K2ytmeJXWFokqiUux2LswTzY+wnUAMQRxBrduwem9iBDQMVlMFhQRI2ocGjrwgVDA5xW6gR5xIS/j3HESKkrVAWpwyN2BKte3QqBqxF56an56sNesWk3yXlGCbYE7thDfbIbtkLe//Vq+Z7NGs5oF1vabmzHT0eybz9qyrTLrH/qfpDoVN2q7Jndkd11Temi/k/e5TYxbzfm6BeHZ9pdm20fKqH543MQnqiNsmuWJ36vSTnB3K2qq/GdYqnL2hc6vfh/AS5mYTKEmpqNp5cCgYXxijGMUXTmUMSK1hDll7lOMEpzpRhZUFZVJZbqUoo0AzhSyirH/1ElaE=</latexit> <latexit sha1_base64="l3XaGwPmE5Vm/Jhs1LG2xz0k9Bs=">AAACknichVG7SgNBFD1ZXzE+Eh+FYCMGRSzCrAiKVdTGwsIk5iFRZHcddchmd9mdBDT4A9oqFlYKFmLhH9jY+AMWfoKkjGBj4c1mQVTUu+zMmTP3nLmXqzum8CRjzyGlrb2jsyvcHenp7euPxgYGc55dcQ2eNWzTdgu65nFTWDwrhTR5wXG5VtZNntdLy837fJW7nrCtdXng8K2ytmeJXWFokqiUux2LswTzY+wnUAMQRxBrduwem9iBDQMVlMFhQRI2ocGjrwgVDA5xW6gR5xIS/j3HESKkrVAWpwyN2BKte3QqBqxF56an56sNesWk3yXlGCbYE7thDfbIbtkLe//Vq+Z7NGs5oF1vabmzHT0eybz9qyrTLrH/qfpDoVN2q7Jndkd11Temi/k/e5TYxbzfm6BeHZ9pdm20fKqH543MQnqiNsmuWJ36vSTnB3K2qq/GdYqnL2hc6vfh/AS5mYTKEmpqNp5cCgYXxijGMUXTmUMSK1hDll7lOMEpzpRhZUFZVJZbqUoo0AzhSyirH/1ElaE=</latexit> e <latexit sha1_base64="JucbY3V7iD4pyjirU4Kr9PWwJLM=">AAACknichVG7SgNBFD1Z3/EVH4VgIwZFLMKsCEoqNY2FRR7GKFFkdx11cLO77E4CMfgD2ioWVgoWYuEf2Nj4AxZ+glgq2Fh49wGiot5lZ86cuefMvVzdMYUnGXuMKU3NLa1t7R3xzq7unt5EX/+KZ1ddgxcN27TdVV3zuCksXpRCmnzVcblW0U1e0vcy/n2pxl1P2NayrDt8o6LtWGJbGJokKsc3E0mWYkGM/ARqBJKIImsnbrGOLdgwUEUFHBYkYRMaPPrKUMHgELeBBnEuIRHccxwgTtoqZXHK0Ijdo3WHTuWItejse3qB2qBXTPpdUo5gjD2wK/bC7tk1e2Lvv3o1Ag+/ljrteqjlzmbv4VDh7V9VhXaJ3U/VHwqdssPKHtkN1fW8Nlku/dmjxDZmg94E9eoEjN+1EfrU9k9fCun8WGOcXbBn6vecnO/I2aq9Gpc5nj+jcanfh/MTrEylVJZSc9PJuYVocO0YxigmaDozmMMisijSqxxHOMaJMqiklXklE6YqsUgzgC+hLH0A4O6VlA==</latexit> <latexit sha1_base64="JucbY3V7iD4pyjirU4Kr9PWwJLM=">AAACknichVG7SgNBFD1Z3/EVH4VgIwZFLMKsCEoqNY2FRR7GKFFkdx11cLO77E4CMfgD2ioWVgoWYuEf2Nj4AxZ+glgq2Fh49wGiot5lZ86cuefMvVzdMYUnGXuMKU3NLa1t7R3xzq7unt5EX/+KZ1ddgxcN27TdVV3zuCksXpRCmnzVcblW0U1e0vcy/n2pxl1P2NayrDt8o6LtWGJbGJokKsc3E0mWYkGM/ARqBJKIImsnbrGOLdgwUEUFHBYkYRMaPPrKUMHgELeBBnEuIRHccxwgTtoqZXHK0Ijdo3WHTuWItejse3qB2qBXTPpdUo5gjD2wK/bC7tk1e2Lvv3o1Ag+/ljrteqjlzmbv4VDh7V9VhXaJ3U/VHwqdssPKHtkN1fW8Nlku/dmjxDZmg94E9eoEjN+1EfrU9k9fCun8WGOcXbBn6vecnO/I2aq9Gpc5nj+jcanfh/MTrEylVJZSc9PJuYVocO0YxigmaDozmMMisijSqxxHOMaJMqiklXklE6YqsUgzgC+hLH0A4O6VlA==</latexit> <latexit sha1_base64="JucbY3V7iD4pyjirU4Kr9PWwJLM=">AAACknichVG7SgNBFD1Z3/EVH4VgIwZFLMKsCEoqNY2FRR7GKFFkdx11cLO77E4CMfgD2ioWVgoWYuEf2Nj4AxZ+glgq2Fh49wGiot5lZ86cuefMvVzdMYUnGXuMKU3NLa1t7R3xzq7unt5EX/+KZ1ddgxcN27TdVV3zuCksXpRCmnzVcblW0U1e0vcy/n2pxl1P2NayrDt8o6LtWGJbGJokKsc3E0mWYkGM/ARqBJKIImsnbrGOLdgwUEUFHBYkYRMaPPrKUMHgELeBBnEuIRHccxwgTtoqZXHK0Ijdo3WHTuWItejse3qB2qBXTPpdUo5gjD2wK/bC7tk1e2Lvv3o1Ag+/ljrteqjlzmbv4VDh7V9VhXaJ3U/VHwqdssPKHtkN1fW8Nlku/dmjxDZmg94E9eoEjN+1EfrU9k9fCun8WGOcXbBn6vecnO/I2aq9Gpc5nj+jcanfh/MTrEylVJZSc9PJuYVocO0YxigmaDozmMMisijSqxxHOMaJMqiklXklE6YqsUgzgC+hLH0A4O6VlA==</latexit> <latexit sha1_base64="JucbY3V7iD4pyjirU4Kr9PWwJLM=">AAACknichVG7SgNBFD1Z3/EVH4VgIwZFLMKsCEoqNY2FRR7GKFFkdx11cLO77E4CMfgD2ioWVgoWYuEf2Nj4AxZ+glgq2Fh49wGiot5lZ86cuefMvVzdMYUnGXuMKU3NLa1t7R3xzq7unt5EX/+KZ1ddgxcN27TdVV3zuCksXpRCmnzVcblW0U1e0vcy/n2pxl1P2NayrDt8o6LtWGJbGJokKsc3E0mWYkGM/ARqBJKIImsnbrGOLdgwUEUFHBYkYRMaPPrKUMHgELeBBnEuIRHccxwgTtoqZXHK0Ijdo3WHTuWItejse3qB2qBXTPpdUo5gjD2wK/bC7tk1e2Lvv3o1Ag+/ljrteqjlzmbv4VDh7V9VhXaJ3U/VHwqdssPKHtkN1fW8Nlku/dmjxDZmg94E9eoEjN+1EfrU9k9fCun8WGOcXbBn6vecnO/I2aq9Gpc5nj+jcanfh/MTrEylVJZSc9PJuYVocO0YxigmaDozmMMisijSqxxHOMaJMqiklXklE6YqsUgzgC+hLH0A4O6VlA==</latexit> (PBQQ  'FFECBDLQBUI 4FNBQIPSF $166TBHF $IBOHFSBUF • Having two nested control loops to change set-point on running. • The outer loop sets new set-points of the inner loop when CPU usage reaches the upper limit. $POUSPMMFS
  26. Bounding concurrency using buffered channel as semaphore 39 To bound

    concurrency in Go func main() { var wg sync.WaitGroup sem := make(chan struct{}, 3) for i := 0; i < 10; i++ { wg.Add(1) sem <- struct{}{} go func() { defer wg.Done() defer func() { <-sem }() task() }() } wg.Wait() close(sem) }
  27. 40 Elastic semaphore DIBO &MBTUJDTFNBQIPSF $PVOUFS XBJU TJHOBM JODSFNFOU-JNJU 

     $PVOUFS $PVOUFS 8BJUJOH /PUXBJUJOH  • wait() & signal() as same as semaphore • incrementLimit() changes upper of semaphore variable • Ensure atomicity of above operations
  28. Architecture of kaburaya 42 • The optimal number of concurrency

    to be determined dynamically and be controlled rapidly • Use CPU usage upper limit as performance metrics. • Use feedback control to determine the number of goroutines continuosly. • Use elastic semaphore to bound concurrency. • Kaburaya is “ధ໼” which is a Japanese arrow with a whistle. • https://github.com/monochromegane/kaburaya
  29. • Kaburaya optimizes the number of goroutines by feedback control.

    43 monochromegane/kaburaya func main() { sem := kaburaya.NewSem(100 * time.Millisecond) var wg sync.WaitGroup for i := 0; i < 10; i++ { wg.Add(1) sem.Wait() go func() { defer sem.Signal() defer wg.Done() task() }() } wg.Wait() sem.Stop() } Create semaphore and specify time span for update the semaphore and CPU target. Wait for decrement operation. Increment operation. Stop update the semaphore and CPU target.
  30. • Apply kaburaya to some tasks to evaluate efficacy. •

    Measure speedup, CPU usage, and limit of semaphore. • Find good and not so good cases for the kaburaya controller. 45 Evaluation
  31. Task: pt_mac-scan 46 • pt runs on macOS with realtime

    virus scan • CPU 8, Memory 16GB • GOMAXPROCS=8 • Better concurrent are from 16 to 64. • More degradation of performance due to increased goroutines.
  32. Task: pt_mac-scan (Good) 47 • Kaburaya found good set points

    and adjust numbers within ideal range. Duration: 50ms, Gain: 0.1, Change rate: ±0.3 Median
  33. Task: pt_mac-no-scan 48 • Runs pt on Mac without realtime

    virus scan • CPU 8, Memory 16GB • GOMAXPROCS=8 • Better concurrent is 4 • Less degradation of performance due to increased goroutines.
  34. Task: pt_mac-no-scan (Good) 49 • Kaburaya found good set points

    and adjust numbers within ideal range. Duration: 50ms, Gain: 0.1, Change rate: ±0.3 Median
  35. Task: pt_mac-no-scan (Bad) 50 • Find set points and adjust

    numbers within ideal range. • Increase the number due to failure to reset the set point at end of period. Duration: 50ms, Gain: 0.1, Change rate: ±0.3 Median
  36. Task: pt_linux 51 • pt runs on Linux • CPU

    4, Memory 4GB • GOMAXPROCS=4 • Better concurrent are from 4 to 32. • Less degradation of performance due to increased goroutines.
  37. Task: pt_linux (Not so bad) 52 • Kaburaya found good

    set points and adjust numbers within ideal range. • Increase the number due to failure to reset the set point at end of period. Duration: 50ms, Gain: 0.1, Change rate: ±0.3 Median
  38. Task: mem_4096000_10000 53 • Runs task uses memory. • 4MB/task

    x 10,000 • CPU 4, Memory 4GB • GOMAXPROCS=4 • Better concurrent is 512. • More degradation of performance (Swap out) due to increased goroutines.
  39. Task: mem_4096000_10000 (Good) 54 • Kaburaya found set points and

    adjust numbers within ideal range. • Avoid starvation of resource Duration: 500ms, Gain: 0.3, Change rate: ±0.3 Median
  40. Task: mem_40960_1000000 55 • Runs task uses memory. • 40kB/task

    x 1,000,000 • CPU 4, Memory 4GB • GOMAXPROCS=4 • Better concurrent are from 4 to 64. • Less degradation of performance due to increased goroutines.
  41. Task: mem_40960_1000000 (Good) 56 • Kaburaya found set points and

    adjust numbers within ideal range. Duration: 500ms, Gain: 0.3, Change rate: ±0.3 Median
  42. Task: mem_409600_100000 57 • Runs task uses memory. • 400kB/task

    x 100,000 • CPU 4, Memory 4GB • GOMAXPROCS=4 • Better concurrent is over 1024? • Less degradation of performance due to increased goroutines.
  43. Task: mem_409600_100000 (Not so good) 58 • Kaburaya found set

    points and adjust numbers. • Does not reach the limit of performance metrics… Duration: 500ms, Gain: 0.3, Change rate: ±0.3 Median
  44. • Kaburaya finds set point and adjusts numbers within ideal

    range by feedback control and elastic semaphore. • Towards to improve the controller • prediction accuracy of set point • Optimal parameters (K{p,i,d} and range of change rate) • Solve them in the related researches about feedback control as the next step. 59 Experimental results
  45. Conclusion 61 • I proposed kaburaya to control the number

    of goroutines without depending on platform, runtime and current load. • Experimental results show possible and some issues of kaburaya. • In particular improvement detecting performance upper limit of task is important. • In the future, I will also consider the application to auto-scaling of cloud computing. • Go
  46. • Implementation of Kaburaya • https://github.com/monochromegane/kaburaya • Study of Kaburaya

    (In Japanese) • https://blog.monochromegane.com/blog/2018/11/25/wsa_3_kaburaya/ • https://gist.github.com/monochromegane/5e9fc94371a9b2385eedd38d98341e6b • https://blog.monochromegane.com/blog/2019/04/14/wsa_4_ebira/ • Learn more goroutine • https://speakerdeck.com/monochromegane/road-to-your-goroutines 63 Reference
  47. 64 Go gopher • The Go gopher was designed by

    Renee French. • The gopher illustrations were drawn by Keita Kawamoto. • Licensed under the Creative Commons 3.0 Attributions license.