Optimization for Number of goroutines Using Feedback Control

Optimization for Number of goroutines Using Feedback Control

GopherCon 2019 - Marriott Marquis San Diego Marina, California
https://www.gophercon.com/

Optimization for Number of goroutines Using Feedback Control
https://www.gophercon.com/agenda/speakers/442434

The design for the number of concurrency is important to achieve both speed and stability. To give a good performance without depending on platform and load conditions, it’s desirable for the number to be dynamic and rapidly controlled. In this talk, I will propose an architecture to solve this by utilizing feedback control.

Cd3d2cb2dadf5488935fe0ddaea7938a?s=128

monochromegane

July 25, 2019
Tweet

Transcript

  1. Yusuke MIYAKE / Pepabo R&D Institute, GMO Pepabo, Inc. 2019.07.25

    GopherCon 2019 Optimization for number of goroutines using feedback control
  2. 1SJODJQBMFOHJOFFS :VTVLF.*:",&!NPOPDISPNFHBOF 1FQBCP3%*OTUJUVUF (.01FQBCP *OD IUUQTCMPHNPOPDISPNFHBOFDPN

  3. Go and I 3 • A Japanese gopher who loves

    writing OSS using Go. • monochromegane/the_platinum_searcher • An organizer of the local Go community in Fukuoka, Japan. • Fukuoka.go (https://fukuokago.dev) • Held Go Conference’19 summer in Fukuoka on 7/13. • https://fukuoka.gocon.jp 5IF(PHPQIFSXBTEFTJHOFECZ3FOFF'SFODI 5IFHPQIFSJMMVTUSBUJPOXBTESBXOCZ,FJUB,BXBNPUP -JDFOTFEVOEFSUIF$SFBUJWF$PNNPOT"UUSJCVUJPOTMJDFOTF
  4. 1. Introduction 2. Background 3. Proposal 4. Evaluation 5. Conclusion

    4 Agenda
  5. 1. Introduction

  6. How many is the optimal number of goroutines?

  7. Performance tuning in case of pt 7 'JOE (PSPVUJOFT $IBO

    $IBO 4FBSDI .BUDI QU 1SJOU ʜ HPSPVUJOFT /VN$16  -JOVY $16 (0."9130$4
  8. Performance tuning in case of pt 8 'JOE (PSPVUJOFT $IBO

    $IBO 4FBSDI .BUDI QU 1SJOU ʜ 3FBMUJNFWJSVTTDBOQSPDFTT 6OTUBCMF HPSPVUJOFT /VN$16  NBD04 $16 (0."9130$4
  9. Performance tuning in case of pt 9 'JOE (PSPVUJOFT $IBO

    $IBO 4FBSDI .BUDI QU 1SJOU ʜ 3FBMUJNFWJSVTTDBOQSPDFTT NBD04 $16 (0."9130$4 ❌ HPSPVUJOFT /VN$16
  10. Performance tuning in case of pt 10 • I chose

    stability rather than speed…
  11. How many is the optimal number of goroutines in each

    case?
  12. 2. Background

  13. 1. Concurrency and complexity 2. Concurrency and Go 3. Concurrency

    and application
  14. • Concurrency brings our application good performance and complexity. •

    We design a task to have concurrency and request runtime to run them in parallel. 14 Concurrency and complexity %FTJHO *NQMFNFOUBUJPO 3VOUJNF $PODVSSFOU 1BSBMMFM w 5ISFBENBOBHFNFOU w 3BDFDPOEJUJPOT w .FNPSZBDDFTTTZOD w %FBEMPDLT w 4UBSWBUJPO w (SBOVMBSJUZ w /VNCFSPG DPODVSSFODZ
  15. 1. Concurrency and complexity 2. Concurrency and Go 3. Concurrency

    and application
  16. Concurrency and Go 16 • Go hides the complexity by

    smart runtime scheduler and features. 3VOUJNFTDIFEVMFS $IBOOFM %FTJHO *NQMFNFOUBUJPO 3VOUJNF $PODVSSFOU 1BSBMMFM w 5ISFBENBOBHFNFOU w (SBOVMBSJUZ w /VNCFSPG DPODVSSFODZ w 3BDFDPOEJUJPOT w .FNPSZBDDFTTTZOD w %FBEMPDLT w 4UBSWBUJPO
  17. Runtime scheduler of Go 17 ( . 1 ( (

    -PDBM3VO2VFVF (MPCBM3VO2VFVF ( ( SVOUJNFTZTNPO . TDIFEVMF FYFDVUF (GO HPFYJU • G: Goroutine • M: OS thread • P: Processor (Scheduling context) • It provides M:N Scheduler (Some goroutines:Some threads). • It has local run queue. • We can think of Goroutines as application-level threads.
 OS thread behaves like a worker for goroutine using a run queue. 3FGIUUQTTQFBLFSEFDLDPNSFUFSWJTJPOHPSVOUJNFTDIFEVMFS
  18. New “g” 18 YD YD YD TUBDLHVBSE )JHI -PX YDE

    TIFETQ YDF TULUPQTQ Stack (2048bytes) TULUPQ 4UBDL 4UBDLHVBSE   Sched (type gobuf) type gobuf struct { sp uintptr // 824633919448 -> 0xc0000307d8 pc uintptr // 4359616 -> 0x4285c0 <runtime.main> g guintptr // newg ctxt unsafe.Pointer // 0x4c96c0 <runtime.mainPC> ret sys.Uintreg. // 0 lr uintptr // 0 bp uintptr // 0 } • Runtime allocates 2kB stack to “g”. • “g” has gobuf named “sched”. • “sched” has goroutines state. (e.g.: stack pointer, program counter…)
  19. 1. Concurrency and complexity 2. Concurrency and Go 3. Concurrency

    and application
  20. Concurrency and application 20 • The design for the number

    of concurrency is important to achieve both speed and stability. • Running many tasks tend to cause starvation. %FTJHO *NQMFNFOUBUJPO 3VOUJNF $PODVSSFOU 1BSBMMFM w 5ISFBENBOBHFNFOU w (SBOVMBSJUZ w /VNCFSPG DPODVSSFODZ w 3BDFDPOEJUJPOT w .FNPSZBDDFTTTZOD w %FBEMPDLT w 4UBSWBUJPO
  21. Concurrency and application 21 • The design for the number

    of concurrency is difficult. • The optimal number depends on the app, environments and load condition. • The environment in which the program is tuned and the environment in which it will be executed is different. • It's desirable for the number of concurrency to be determined dynamically and be controlled rapidly by detection the bottleneck on running program.
  22. 3. Proposal

  23. Goal 23 • The optimal number of concurrency to be

    determined dynamically and be controlled rapidly.
  24. Basic idea 24 1. Increase the number of goroutines to

    performance target 1FSGPSNBODF      /VNCFSPGHPSPVUJOFT      4UFQ           (PSPVUJOFT 1FSGPSNBODF 1FSGPSNBODFUBSHFU
  25. Basic idea 25 1. Increase the number of goroutines to

    performance target 2. Stop increasing the number of goroutines if it meets the performance target 1FSGPSNBODF      /VNCFSPGHPSPVUJOFT      4UFQ           (PSPVUJOFT 1FSGPSNBODF 1FSGPSNBODFUBSHFU
  26. Basic idea 26 1. Increase the number of goroutines to

    performance target 2. Stop increasing the number of goroutines if it meets the performance target 3. Attempt to decrease the number of goroutines 1FSGPSNBODF      /VNCFSPGHPSPVUJOFT      4UFQ           (PSPVUJOFT 1FSGPSNBODF 1FSGPSNBODFUBSHFU
  27. Issues to solve for the realization 27 1. Selection of

    performance metrics 2. Finding how to control rapidly and continuously
  28. Performance metrics

  29. Performance metrics 29 • Independent resource type which task use.

    • e.g.: CPU usage, Throughput. • Go's scheduler turns blocking tasks into as CPU bound as possible by switching tasks continuously.
  30. Performance metrics 30 • The metrics upper limit is calculated

    on running. 1. Set the target value high 1FSGPSNBODF       /VNCFSPGHPSPVUJOFT       4UFQ           (PSPVUJOFT 1FSGPSNBODF 1FSGPSNBODFUBSHFU
  31. Performance metrics 31 • The metrics upper limit is calculated

    on running. 1. Set the target value high 2. gradually adjust it to the upper limit value. 1FSGPSNBODF       /VNCFSPGHPSPVUJOFT       4UFQ           (PSPVUJOFT 1FSGPSNBODF 1FSGPSNBODFUBSHFU
  32. Determining

  33. Determine the number of goroutines 33 • Determine the number

    of goroutines based on the metrics • Continuously • Rapidly • Accurately • Feedback control meets these conditions.
  34. • Tracking a given set-point using errors from the set-point.

    • Applying an automatic correction continuously. • Identifying suitable set-point and input/output is important. • Dealing with the system as a black box. Feedback control $POUSPMMFS 4ZTUFN *OQVU 0VUQVU &SSPS 'FFECBDLQBUI 4FUQPJOU
  35. • Applying a correction rapidly and accurately. • The output

    is a combination of its {Proportional, Integral, Derivative} control. 35 PID Controller (1/2) $POUSPMMFS &SSPS 4FUQPJOU 1 Kpe(t) <latexit sha1_base64="WSac9fWZ+QE538KWn4+gz2K9Jv4=">AAACmHichVG7SgNBFL1ZXzE+ErURbYIhEi3CrAiKVdBCxSYP85AYwu46iUv2xe4kEIM/YGclmErBQiz8Axsbf8AinyApI9hYePcBosF4l9m5c+aeM/dwRUORLUZIx8cNDY+MjvnHAxOTU9PB0MxsztLrpkSzkq7oZkEULKrIGs0ymSm0YJhUUEWF5sXajn2fb1DTknXtkDUNWlKFqiZXZElgCOUPygaNsZVyKELixIlwf8J7SQS8SOqhJziGE9BBgjqoQEEDhrkCAlj4FYEHAgZiJWghZmImO/cUziGA3DpWUawQEK3hv4qnoodqeLY1LYct4SsKLhOZYYiSV3JPeuSFPJA38vmnVsvRsHtp4i66XGqUgxfzmY9/WSruDE6/WQMYIla7nXXII/bVPVot5gd6ZFCBTcebjF4NB7FdS65O4+yql9lKR1vL5JZ00e8NKj+jstZ4l+5SNN3GcfG/h9Of5NbiPInzqfVIYtsbnB8WYQliOJ0NSMAeJCHr9HYJ19DmFrgEt8vtu6Wcz+PMwY/g0l/eDZev</latexit> <latexit sha1_base64="WSac9fWZ+QE538KWn4+gz2K9Jv4=">AAACmHichVG7SgNBFL1ZXzE+ErURbYIhEi3CrAiKVdBCxSYP85AYwu46iUv2xe4kEIM/YGclmErBQiz8Axsbf8AinyApI9hYePcBosF4l9m5c+aeM/dwRUORLUZIx8cNDY+MjvnHAxOTU9PB0MxsztLrpkSzkq7oZkEULKrIGs0ymSm0YJhUUEWF5sXajn2fb1DTknXtkDUNWlKFqiZXZElgCOUPygaNsZVyKELixIlwf8J7SQS8SOqhJziGE9BBgjqoQEEDhrkCAlj4FYEHAgZiJWghZmImO/cUziGA3DpWUawQEK3hv4qnoodqeLY1LYct4SsKLhOZYYiSV3JPeuSFPJA38vmnVsvRsHtp4i66XGqUgxfzmY9/WSruDE6/WQMYIla7nXXII/bVPVot5gd6ZFCBTcebjF4NB7FdS65O4+yql9lKR1vL5JZ00e8NKj+jstZ4l+5SNN3GcfG/h9Of5NbiPInzqfVIYtsbnB8WYQliOJ0NSMAeJCHr9HYJ19DmFrgEt8vtu6Wcz+PMwY/g0l/eDZev</latexit> <latexit sha1_base64="WSac9fWZ+QE538KWn4+gz2K9Jv4=">AAACmHichVG7SgNBFL1ZXzE+ErURbYIhEi3CrAiKVdBCxSYP85AYwu46iUv2xe4kEIM/YGclmErBQiz8Axsbf8AinyApI9hYePcBosF4l9m5c+aeM/dwRUORLUZIx8cNDY+MjvnHAxOTU9PB0MxsztLrpkSzkq7oZkEULKrIGs0ymSm0YJhUUEWF5sXajn2fb1DTknXtkDUNWlKFqiZXZElgCOUPygaNsZVyKELixIlwf8J7SQS8SOqhJziGE9BBgjqoQEEDhrkCAlj4FYEHAgZiJWghZmImO/cUziGA3DpWUawQEK3hv4qnoodqeLY1LYct4SsKLhOZYYiSV3JPeuSFPJA38vmnVsvRsHtp4i66XGqUgxfzmY9/WSruDE6/WQMYIla7nXXII/bVPVot5gd6ZFCBTcebjF4NB7FdS65O4+yql9lKR1vL5JZ00e8NKj+jstZ4l+5SNN3GcfG/h9Of5NbiPInzqfVIYtsbnB8WYQliOJ0NSMAeJCHr9HYJ19DmFrgEt8vtu6Wcz+PMwY/g0l/eDZev</latexit> <latexit sha1_base64="WSac9fWZ+QE538KWn4+gz2K9Jv4=">AAACmHichVG7SgNBFL1ZXzE+ErURbYIhEi3CrAiKVdBCxSYP85AYwu46iUv2xe4kEIM/YGclmErBQiz8Axsbf8AinyApI9hYePcBosF4l9m5c+aeM/dwRUORLUZIx8cNDY+MjvnHAxOTU9PB0MxsztLrpkSzkq7oZkEULKrIGs0ymSm0YJhUUEWF5sXajn2fb1DTknXtkDUNWlKFqiZXZElgCOUPygaNsZVyKELixIlwf8J7SQS8SOqhJziGE9BBgjqoQEEDhrkCAlj4FYEHAgZiJWghZmImO/cUziGA3DpWUawQEK3hv4qnoodqeLY1LYct4SsKLhOZYYiSV3JPeuSFPJA38vmnVsvRsHtp4i66XGqUgxfzmY9/WSruDE6/WQMYIla7nXXII/bVPVot5gd6ZFCBTcebjF4NB7FdS65O4+yql9lKR1vL5JZ00e8NKj+jstZ4l+5SNN3GcfG/h9Of5NbiPInzqfVIYtsbnB8WYQliOJ0NSMAeJCHr9HYJ19DmFrgEt8vtu6Wcz+PMwY/g0l/eDZev</latexit> * Ki Z t 0 e(r)dr <latexit sha1_base64="aDkxGx5q/sb9YJIQEipbOLXS4BA=">AAACpnichVFNS+NQFD3Gz6lf1W4G3JQpSp1FuRFBcSW6EQakrVOrVM0k8amPpklIXgu1+Af8Ay5mMYwgKC78B7OZzfyBWfgTxKWCm1nMbRoQFfWGvHfeefecdy/X8h0ZKqLrLq27p7evf+BDYnBoeGQ0OTa+Hnr1wBYl23O8YMMyQ+FIV5SUVI7Y8ANh1ixHlK3qcvu+3BBBKD33q2r6Yrtm7rtyT9qmYspIpr4Ycku6yqAdlRbZYDq9GxjJDOUoivRLoMcggzjyXvIXtrALDzbqqEHAhWLswETIXwU6CD5z22gxFzCS0b3AERKsrXOW4AyT2Sqv+3yqxKzL57ZnGKltfsXhP2BlGpP0ly7ojv7QJd3Qv1e9WpFHu5Ym71ZHK3xj9Pjj2sO7qhrvCgePqjcUFmd3KrumK67rdvNzpfxmjwp7mI96k9yrHzHtru2OT+Pw5G5toTjZmqJTuuV+f7Lzb3Z2G/f2WUEUv/O49OfDeQnWZ3I65fTCbGZxKR7cACbwCVmezhwWsYI8SvxqEz9wjgstq61qJa3cSdW6Yk0KT0L79h9J15xt</latexit> <latexit sha1_base64="aDkxGx5q/sb9YJIQEipbOLXS4BA=">AAACpnichVFNS+NQFD3Gz6lf1W4G3JQpSp1FuRFBcSW6EQakrVOrVM0k8amPpklIXgu1+Af8Ay5mMYwgKC78B7OZzfyBWfgTxKWCm1nMbRoQFfWGvHfeefecdy/X8h0ZKqLrLq27p7evf+BDYnBoeGQ0OTa+Hnr1wBYl23O8YMMyQ+FIV5SUVI7Y8ANh1ixHlK3qcvu+3BBBKD33q2r6Yrtm7rtyT9qmYspIpr4Ycku6yqAdlRbZYDq9GxjJDOUoivRLoMcggzjyXvIXtrALDzbqqEHAhWLswETIXwU6CD5z22gxFzCS0b3AERKsrXOW4AyT2Sqv+3yqxKzL57ZnGKltfsXhP2BlGpP0ly7ojv7QJd3Qv1e9WpFHu5Ym71ZHK3xj9Pjj2sO7qhrvCgePqjcUFmd3KrumK67rdvNzpfxmjwp7mI96k9yrHzHtru2OT+Pw5G5toTjZmqJTuuV+f7Lzb3Z2G/f2WUEUv/O49OfDeQnWZ3I65fTCbGZxKR7cACbwCVmezhwWsYI8SvxqEz9wjgstq61qJa3cSdW6Yk0KT0L79h9J15xt</latexit> <latexit sha1_base64="aDkxGx5q/sb9YJIQEipbOLXS4BA=">AAACpnichVFNS+NQFD3Gz6lf1W4G3JQpSp1FuRFBcSW6EQakrVOrVM0k8amPpklIXgu1+Af8Ay5mMYwgKC78B7OZzfyBWfgTxKWCm1nMbRoQFfWGvHfeefecdy/X8h0ZKqLrLq27p7evf+BDYnBoeGQ0OTa+Hnr1wBYl23O8YMMyQ+FIV5SUVI7Y8ANh1ixHlK3qcvu+3BBBKD33q2r6Yrtm7rtyT9qmYspIpr4Ycku6yqAdlRbZYDq9GxjJDOUoivRLoMcggzjyXvIXtrALDzbqqEHAhWLswETIXwU6CD5z22gxFzCS0b3AERKsrXOW4AyT2Sqv+3yqxKzL57ZnGKltfsXhP2BlGpP0ly7ojv7QJd3Qv1e9WpFHu5Ym71ZHK3xj9Pjj2sO7qhrvCgePqjcUFmd3KrumK67rdvNzpfxmjwp7mI96k9yrHzHtru2OT+Pw5G5toTjZmqJTuuV+f7Lzb3Z2G/f2WUEUv/O49OfDeQnWZ3I65fTCbGZxKR7cACbwCVmezhwWsYI8SvxqEz9wjgstq61qJa3cSdW6Yk0KT0L79h9J15xt</latexit> <latexit sha1_base64="aDkxGx5q/sb9YJIQEipbOLXS4BA=">AAACpnichVFNS+NQFD3Gz6lf1W4G3JQpSp1FuRFBcSW6EQakrVOrVM0k8amPpklIXgu1+Af8Ay5mMYwgKC78B7OZzfyBWfgTxKWCm1nMbRoQFfWGvHfeefecdy/X8h0ZKqLrLq27p7evf+BDYnBoeGQ0OTa+Hnr1wBYl23O8YMMyQ+FIV5SUVI7Y8ANh1ixHlK3qcvu+3BBBKD33q2r6Yrtm7rtyT9qmYspIpr4Ycku6yqAdlRbZYDq9GxjJDOUoivRLoMcggzjyXvIXtrALDzbqqEHAhWLswETIXwU6CD5z22gxFzCS0b3AERKsrXOW4AyT2Sqv+3yqxKzL57ZnGKltfsXhP2BlGpP0ly7ojv7QJd3Qv1e9WpFHu5Ym71ZHK3xj9Pjj2sO7qhrvCgePqjcUFmd3KrumK67rdvNzpfxmjwp7mI96k9yrHzHtru2OT+Pw5G5toTjZmqJTuuV+f7Lzb3Z2G/f2WUEUv/O49OfDeQnWZ3I65fTCbGZxKR7cACbwCVmezhwWsYI8SvxqEz9wjgstq61qJa3cSdW6Yk0KT0L79h9J15xt</latexit> % Kd de(t) dt <latexit sha1_base64="pK4ns8fNxecVndhwIulsD6E0PEo=">AAACp3ichVHLSsNAFD2Nr/quj4XgplgUdVEmIiiuim4EF9rWPqRKSdKphqZJSKYFDf0Bf8CFGxVcVBf+gRs3/oALP0FcKrhx4W0aEBX1hsycOXPPmXu5qm3ormDsMSR1dHZ194R7+/oHBoeGIyOjWdeqORrPaJZhOXlVcbmhmzwjdGHwvO1wpaoaPKdW1lr3uTp3XN0yt8Whzfeqyr6pl3VNEUQVI+MbxVJ0t+womlfis2Ku4ZVEoxiJsTjzI/oTyAGIIYgtK3KLXZRgQUMNVXCYEIQNKHDpK0AGg03cHjziHEK6f8/RQB9pa5TFKUMhtkLrPp0KAWvSueXp+mqNXjHod0gZxTR7YE32wu7ZNXti7796eb5Hq5ZD2tW2ltvF4eOJ9Nu/qirtAgefqj8UKmW3K3tkN1TX8858IfdnjwJlLPu96dSr7TOtrrW2T/3o5CW9kpr2ZtgFe6Z+z8n5jpzN+qt2meSpUxqX/H04P0F2IS6zuJxcjCVWg8GFMYkpzNJ0lpDAOraQoVePcIYmrqQ5aVPKSvl2qhQKNGP4EpLyAfEdnZo=</latexit> <latexit sha1_base64="pK4ns8fNxecVndhwIulsD6E0PEo=">AAACp3ichVHLSsNAFD2Nr/quj4XgplgUdVEmIiiuim4EF9rWPqRKSdKphqZJSKYFDf0Bf8CFGxVcVBf+gRs3/oALP0FcKrhx4W0aEBX1hsycOXPPmXu5qm3ormDsMSR1dHZ194R7+/oHBoeGIyOjWdeqORrPaJZhOXlVcbmhmzwjdGHwvO1wpaoaPKdW1lr3uTp3XN0yt8Whzfeqyr6pl3VNEUQVI+MbxVJ0t+womlfis2Ku4ZVEoxiJsTjzI/oTyAGIIYgtK3KLXZRgQUMNVXCYEIQNKHDpK0AGg03cHjziHEK6f8/RQB9pa5TFKUMhtkLrPp0KAWvSueXp+mqNXjHod0gZxTR7YE32wu7ZNXti7796eb5Hq5ZD2tW2ltvF4eOJ9Nu/qirtAgefqj8UKmW3K3tkN1TX8858IfdnjwJlLPu96dSr7TOtrrW2T/3o5CW9kpr2ZtgFe6Z+z8n5jpzN+qt2meSpUxqX/H04P0F2IS6zuJxcjCVWg8GFMYkpzNJ0lpDAOraQoVePcIYmrqQ5aVPKSvl2qhQKNGP4EpLyAfEdnZo=</latexit> <latexit sha1_base64="pK4ns8fNxecVndhwIulsD6E0PEo=">AAACp3ichVHLSsNAFD2Nr/quj4XgplgUdVEmIiiuim4EF9rWPqRKSdKphqZJSKYFDf0Bf8CFGxVcVBf+gRs3/oALP0FcKrhx4W0aEBX1hsycOXPPmXu5qm3ormDsMSR1dHZ194R7+/oHBoeGIyOjWdeqORrPaJZhOXlVcbmhmzwjdGHwvO1wpaoaPKdW1lr3uTp3XN0yt8Whzfeqyr6pl3VNEUQVI+MbxVJ0t+womlfis2Ku4ZVEoxiJsTjzI/oTyAGIIYgtK3KLXZRgQUMNVXCYEIQNKHDpK0AGg03cHjziHEK6f8/RQB9pa5TFKUMhtkLrPp0KAWvSueXp+mqNXjHod0gZxTR7YE32wu7ZNXti7796eb5Hq5ZD2tW2ltvF4eOJ9Nu/qirtAgefqj8UKmW3K3tkN1TX8858IfdnjwJlLPu96dSr7TOtrrW2T/3o5CW9kpr2ZtgFe6Z+z8n5jpzN+qt2meSpUxqX/H04P0F2IS6zuJxcjCVWg8GFMYkpzNJ0lpDAOraQoVePcIYmrqQ5aVPKSvl2qhQKNGP4EpLyAfEdnZo=</latexit> <latexit sha1_base64="pK4ns8fNxecVndhwIulsD6E0PEo=">AAACp3ichVHLSsNAFD2Nr/quj4XgplgUdVEmIiiuim4EF9rWPqRKSdKphqZJSKYFDf0Bf8CFGxVcVBf+gRs3/oALP0FcKrhx4W0aEBX1hsycOXPPmXu5qm3ormDsMSR1dHZ194R7+/oHBoeGIyOjWdeqORrPaJZhOXlVcbmhmzwjdGHwvO1wpaoaPKdW1lr3uTp3XN0yt8Whzfeqyr6pl3VNEUQVI+MbxVJ0t+womlfis2Ku4ZVEoxiJsTjzI/oTyAGIIYgtK3KLXZRgQUMNVXCYEIQNKHDpK0AGg03cHjziHEK6f8/RQB9pa5TFKUMhtkLrPp0KAWvSueXp+mqNXjHod0gZxTR7YE32wu7ZNXti7796eb5Hq5ZD2tW2ltvF4eOJ9Nu/qirtAgefqj8UKmW3K3tkN1TX8858IfdnjwJlLPu96dSr7TOtrrW2T/3o5CW9kpr2ZtgFe6Z+z8n5jpzN+qt2meSpUxqX/H04P0F2IS6zuJxcjCVWg8GFMYkpzNJ0lpDAOraQoVePcIYmrqQ5aVPKSvl2qhQKNGP4EpLyAfEdnZo=</latexit> r <latexit sha1_base64="l3XaGwPmE5Vm/Jhs1LG2xz0k9Bs=">AAACknichVG7SgNBFD1ZXzE+Eh+FYCMGRSzCrAiKVdTGwsIk5iFRZHcddchmd9mdBDT4A9oqFlYKFmLhH9jY+AMWfoKkjGBj4c1mQVTUu+zMmTP3nLmXqzum8CRjzyGlrb2jsyvcHenp7euPxgYGc55dcQ2eNWzTdgu65nFTWDwrhTR5wXG5VtZNntdLy837fJW7nrCtdXng8K2ytmeJXWFokqiUux2LswTzY+wnUAMQRxBrduwem9iBDQMVlMFhQRI2ocGjrwgVDA5xW6gR5xIS/j3HESKkrVAWpwyN2BKte3QqBqxF56an56sNesWk3yXlGCbYE7thDfbIbtkLe//Vq+Z7NGs5oF1vabmzHT0eybz9qyrTLrH/qfpDoVN2q7Jndkd11Temi/k/e5TYxbzfm6BeHZ9pdm20fKqH543MQnqiNsmuWJ36vSTnB3K2qq/GdYqnL2hc6vfh/AS5mYTKEmpqNp5cCgYXxijGMUXTmUMSK1hDll7lOMEpzpRhZUFZVJZbqUoo0AzhSyirH/1ElaE=</latexit> <latexit sha1_base64="l3XaGwPmE5Vm/Jhs1LG2xz0k9Bs=">AAACknichVG7SgNBFD1ZXzE+Eh+FYCMGRSzCrAiKVdTGwsIk5iFRZHcddchmd9mdBDT4A9oqFlYKFmLhH9jY+AMWfoKkjGBj4c1mQVTUu+zMmTP3nLmXqzum8CRjzyGlrb2jsyvcHenp7euPxgYGc55dcQ2eNWzTdgu65nFTWDwrhTR5wXG5VtZNntdLy837fJW7nrCtdXng8K2ytmeJXWFokqiUux2LswTzY+wnUAMQRxBrduwem9iBDQMVlMFhQRI2ocGjrwgVDA5xW6gR5xIS/j3HESKkrVAWpwyN2BKte3QqBqxF56an56sNesWk3yXlGCbYE7thDfbIbtkLe//Vq+Z7NGs5oF1vabmzHT0eybz9qyrTLrH/qfpDoVN2q7Jndkd11Temi/k/e5TYxbzfm6BeHZ9pdm20fKqH543MQnqiNsmuWJ36vSTnB3K2qq/GdYqnL2hc6vfh/AS5mYTKEmpqNp5cCgYXxijGMUXTmUMSK1hDll7lOMEpzpRhZUFZVJZbqUoo0AzhSyirH/1ElaE=</latexit> <latexit sha1_base64="l3XaGwPmE5Vm/Jhs1LG2xz0k9Bs=">AAACknichVG7SgNBFD1ZXzE+Eh+FYCMGRSzCrAiKVdTGwsIk5iFRZHcddchmd9mdBDT4A9oqFlYKFmLhH9jY+AMWfoKkjGBj4c1mQVTUu+zMmTP3nLmXqzum8CRjzyGlrb2jsyvcHenp7euPxgYGc55dcQ2eNWzTdgu65nFTWDwrhTR5wXG5VtZNntdLy837fJW7nrCtdXng8K2ytmeJXWFokqiUux2LswTzY+wnUAMQRxBrduwem9iBDQMVlMFhQRI2ocGjrwgVDA5xW6gR5xIS/j3HESKkrVAWpwyN2BKte3QqBqxF56an56sNesWk3yXlGCbYE7thDfbIbtkLe//Vq+Z7NGs5oF1vabmzHT0eybz9qyrTLrH/qfpDoVN2q7Jndkd11Temi/k/e5TYxbzfm6BeHZ9pdm20fKqH543MQnqiNsmuWJ36vSTnB3K2qq/GdYqnL2hc6vfh/AS5mYTKEmpqNp5cCgYXxijGMUXTmUMSK1hDll7lOMEpzpRhZUFZVJZbqUoo0AzhSyirH/1ElaE=</latexit> <latexit sha1_base64="l3XaGwPmE5Vm/Jhs1LG2xz0k9Bs=">AAACknichVG7SgNBFD1ZXzE+Eh+FYCMGRSzCrAiKVdTGwsIk5iFRZHcddchmd9mdBDT4A9oqFlYKFmLhH9jY+AMWfoKkjGBj4c1mQVTUu+zMmTP3nLmXqzum8CRjzyGlrb2jsyvcHenp7euPxgYGc55dcQ2eNWzTdgu65nFTWDwrhTR5wXG5VtZNntdLy837fJW7nrCtdXng8K2ytmeJXWFokqiUux2LswTzY+wnUAMQRxBrduwem9iBDQMVlMFhQRI2ocGjrwgVDA5xW6gR5xIS/j3HESKkrVAWpwyN2BKte3QqBqxF56an56sNesWk3yXlGCbYE7thDfbIbtkLe//Vq+Z7NGs5oF1vabmzHT0eybz9qyrTLrH/qfpDoVN2q7Jndkd11Temi/k/e5TYxbzfm6BeHZ9pdm20fKqH543MQnqiNsmuWJ36vSTnB3K2qq/GdYqnL2hc6vfh/AS5mYTKEmpqNp5cCgYXxijGMUXTmUMSK1hDll7lOMEpzpRhZUFZVJZbqUoo0AzhSyirH/1ElaE=</latexit> e <latexit sha1_base64="JucbY3V7iD4pyjirU4Kr9PWwJLM=">AAACknichVG7SgNBFD1Z3/EVH4VgIwZFLMKsCEoqNY2FRR7GKFFkdx11cLO77E4CMfgD2ioWVgoWYuEf2Nj4AxZ+glgq2Fh49wGiot5lZ86cuefMvVzdMYUnGXuMKU3NLa1t7R3xzq7unt5EX/+KZ1ddgxcN27TdVV3zuCksXpRCmnzVcblW0U1e0vcy/n2pxl1P2NayrDt8o6LtWGJbGJokKsc3E0mWYkGM/ARqBJKIImsnbrGOLdgwUEUFHBYkYRMaPPrKUMHgELeBBnEuIRHccxwgTtoqZXHK0Ijdo3WHTuWItejse3qB2qBXTPpdUo5gjD2wK/bC7tk1e2Lvv3o1Ag+/ljrteqjlzmbv4VDh7V9VhXaJ3U/VHwqdssPKHtkN1fW8Nlku/dmjxDZmg94E9eoEjN+1EfrU9k9fCun8WGOcXbBn6vecnO/I2aq9Gpc5nj+jcanfh/MTrEylVJZSc9PJuYVocO0YxigmaDozmMMisijSqxxHOMaJMqiklXklE6YqsUgzgC+hLH0A4O6VlA==</latexit> <latexit sha1_base64="JucbY3V7iD4pyjirU4Kr9PWwJLM=">AAACknichVG7SgNBFD1Z3/EVH4VgIwZFLMKsCEoqNY2FRR7GKFFkdx11cLO77E4CMfgD2ioWVgoWYuEf2Nj4AxZ+glgq2Fh49wGiot5lZ86cuefMvVzdMYUnGXuMKU3NLa1t7R3xzq7unt5EX/+KZ1ddgxcN27TdVV3zuCksXpRCmnzVcblW0U1e0vcy/n2pxl1P2NayrDt8o6LtWGJbGJokKsc3E0mWYkGM/ARqBJKIImsnbrGOLdgwUEUFHBYkYRMaPPrKUMHgELeBBnEuIRHccxwgTtoqZXHK0Ijdo3WHTuWItejse3qB2qBXTPpdUo5gjD2wK/bC7tk1e2Lvv3o1Ag+/ljrteqjlzmbv4VDh7V9VhXaJ3U/VHwqdssPKHtkN1fW8Nlku/dmjxDZmg94E9eoEjN+1EfrU9k9fCun8WGOcXbBn6vecnO/I2aq9Gpc5nj+jcanfh/MTrEylVJZSc9PJuYVocO0YxigmaDozmMMisijSqxxHOMaJMqiklXklE6YqsUgzgC+hLH0A4O6VlA==</latexit> <latexit sha1_base64="JucbY3V7iD4pyjirU4Kr9PWwJLM=">AAACknichVG7SgNBFD1Z3/EVH4VgIwZFLMKsCEoqNY2FRR7GKFFkdx11cLO77E4CMfgD2ioWVgoWYuEf2Nj4AxZ+glgq2Fh49wGiot5lZ86cuefMvVzdMYUnGXuMKU3NLa1t7R3xzq7unt5EX/+KZ1ddgxcN27TdVV3zuCksXpRCmnzVcblW0U1e0vcy/n2pxl1P2NayrDt8o6LtWGJbGJokKsc3E0mWYkGM/ARqBJKIImsnbrGOLdgwUEUFHBYkYRMaPPrKUMHgELeBBnEuIRHccxwgTtoqZXHK0Ijdo3WHTuWItejse3qB2qBXTPpdUo5gjD2wK/bC7tk1e2Lvv3o1Ag+/ljrteqjlzmbv4VDh7V9VhXaJ3U/VHwqdssPKHtkN1fW8Nlku/dmjxDZmg94E9eoEjN+1EfrU9k9fCun8WGOcXbBn6vecnO/I2aq9Gpc5nj+jcanfh/MTrEylVJZSc9PJuYVocO0YxigmaDozmMMisijSqxxHOMaJMqiklXklE6YqsUgzgC+hLH0A4O6VlA==</latexit> <latexit sha1_base64="JucbY3V7iD4pyjirU4Kr9PWwJLM=">AAACknichVG7SgNBFD1Z3/EVH4VgIwZFLMKsCEoqNY2FRR7GKFFkdx11cLO77E4CMfgD2ioWVgoWYuEf2Nj4AxZ+glgq2Fh49wGiot5lZ86cuefMvVzdMYUnGXuMKU3NLa1t7R3xzq7unt5EX/+KZ1ddgxcN27TdVV3zuCksXpRCmnzVcblW0U1e0vcy/n2pxl1P2NayrDt8o6LtWGJbGJokKsc3E0mWYkGM/ARqBJKIImsnbrGOLdgwUEUFHBYkYRMaPPrKUMHgELeBBnEuIRHccxwgTtoqZXHK0Ijdo3WHTuWItejse3qB2qBXTPpdUo5gjD2wK/bC7tk1e2Lvv3o1Ag+/ljrteqjlzmbv4VDh7V9VhXaJ3U/VHwqdssPKHtkN1fW8Nlku/dmjxDZmg94E9eoEjN+1EfrU9k9fCun8WGOcXbBn6vecnO/I2aq9Gpc5nj+jcanfh/MTrEylVJZSc9PJuYVocO0YxigmaDozmMMisijSqxxHOMaJMqiklXklE6YqsUgzgC+hLH0A4O6VlA==</latexit> 4ZTUFN  'FFECBDLQBUI *OQVU 0VUQVU
  36. • Proportional control(P) output is proportional to the error. •

    Integral control(I) output is proportional to the integral of the error over time. • Derivative control(D) output is proportional to the derivative of the error. 36 PID Controller (2/2) $POUSPMMFS &SSPS 4FUQPJOU 1 Kpe(t) <latexit sha1_base64="WSac9fWZ+QE538KWn4+gz2K9Jv4=">AAACmHichVG7SgNBFL1ZXzE+ErURbYIhEi3CrAiKVdBCxSYP85AYwu46iUv2xe4kEIM/YGclmErBQiz8Axsbf8AinyApI9hYePcBosF4l9m5c+aeM/dwRUORLUZIx8cNDY+MjvnHAxOTU9PB0MxsztLrpkSzkq7oZkEULKrIGs0ymSm0YJhUUEWF5sXajn2fb1DTknXtkDUNWlKFqiZXZElgCOUPygaNsZVyKELixIlwf8J7SQS8SOqhJziGE9BBgjqoQEEDhrkCAlj4FYEHAgZiJWghZmImO/cUziGA3DpWUawQEK3hv4qnoodqeLY1LYct4SsKLhOZYYiSV3JPeuSFPJA38vmnVsvRsHtp4i66XGqUgxfzmY9/WSruDE6/WQMYIla7nXXII/bVPVot5gd6ZFCBTcebjF4NB7FdS65O4+yql9lKR1vL5JZ00e8NKj+jstZ4l+5SNN3GcfG/h9Of5NbiPInzqfVIYtsbnB8WYQliOJ0NSMAeJCHr9HYJ19DmFrgEt8vtu6Wcz+PMwY/g0l/eDZev</latexit> <latexit sha1_base64="WSac9fWZ+QE538KWn4+gz2K9Jv4=">AAACmHichVG7SgNBFL1ZXzE+ErURbYIhEi3CrAiKVdBCxSYP85AYwu46iUv2xe4kEIM/YGclmErBQiz8Axsbf8AinyApI9hYePcBosF4l9m5c+aeM/dwRUORLUZIx8cNDY+MjvnHAxOTU9PB0MxsztLrpkSzkq7oZkEULKrIGs0ymSm0YJhUUEWF5sXajn2fb1DTknXtkDUNWlKFqiZXZElgCOUPygaNsZVyKELixIlwf8J7SQS8SOqhJziGE9BBgjqoQEEDhrkCAlj4FYEHAgZiJWghZmImO/cUziGA3DpWUawQEK3hv4qnoodqeLY1LYct4SsKLhOZYYiSV3JPeuSFPJA38vmnVsvRsHtp4i66XGqUgxfzmY9/WSruDE6/WQMYIla7nXXII/bVPVot5gd6ZFCBTcebjF4NB7FdS65O4+yql9lKR1vL5JZ00e8NKj+jstZ4l+5SNN3GcfG/h9Of5NbiPInzqfVIYtsbnB8WYQliOJ0NSMAeJCHr9HYJ19DmFrgEt8vtu6Wcz+PMwY/g0l/eDZev</latexit> <latexit sha1_base64="WSac9fWZ+QE538KWn4+gz2K9Jv4=">AAACmHichVG7SgNBFL1ZXzE+ErURbYIhEi3CrAiKVdBCxSYP85AYwu46iUv2xe4kEIM/YGclmErBQiz8Axsbf8AinyApI9hYePcBosF4l9m5c+aeM/dwRUORLUZIx8cNDY+MjvnHAxOTU9PB0MxsztLrpkSzkq7oZkEULKrIGs0ymSm0YJhUUEWF5sXajn2fb1DTknXtkDUNWlKFqiZXZElgCOUPygaNsZVyKELixIlwf8J7SQS8SOqhJziGE9BBgjqoQEEDhrkCAlj4FYEHAgZiJWghZmImO/cUziGA3DpWUawQEK3hv4qnoodqeLY1LYct4SsKLhOZYYiSV3JPeuSFPJA38vmnVsvRsHtp4i66XGqUgxfzmY9/WSruDE6/WQMYIla7nXXII/bVPVot5gd6ZFCBTcebjF4NB7FdS65O4+yql9lKR1vL5JZ00e8NKj+jstZ4l+5SNN3GcfG/h9Of5NbiPInzqfVIYtsbnB8WYQliOJ0NSMAeJCHr9HYJ19DmFrgEt8vtu6Wcz+PMwY/g0l/eDZev</latexit> <latexit sha1_base64="WSac9fWZ+QE538KWn4+gz2K9Jv4=">AAACmHichVG7SgNBFL1ZXzE+ErURbYIhEi3CrAiKVdBCxSYP85AYwu46iUv2xe4kEIM/YGclmErBQiz8Axsbf8AinyApI9hYePcBosF4l9m5c+aeM/dwRUORLUZIx8cNDY+MjvnHAxOTU9PB0MxsztLrpkSzkq7oZkEULKrIGs0ymSm0YJhUUEWF5sXajn2fb1DTknXtkDUNWlKFqiZXZElgCOUPygaNsZVyKELixIlwf8J7SQS8SOqhJziGE9BBgjqoQEEDhrkCAlj4FYEHAgZiJWghZmImO/cUziGA3DpWUawQEK3hv4qnoodqeLY1LYct4SsKLhOZYYiSV3JPeuSFPJA38vmnVsvRsHtp4i66XGqUgxfzmY9/WSruDE6/WQMYIla7nXXII/bVPVot5gd6ZFCBTcebjF4NB7FdS65O4+yql9lKR1vL5JZ00e8NKj+jstZ4l+5SNN3GcfG/h9Of5NbiPInzqfVIYtsbnB8WYQliOJ0NSMAeJCHr9HYJ19DmFrgEt8vtu6Wcz+PMwY/g0l/eDZev</latexit> * Ki Z t 0 e(r)dr <latexit sha1_base64="aDkxGx5q/sb9YJIQEipbOLXS4BA=">AAACpnichVFNS+NQFD3Gz6lf1W4G3JQpSp1FuRFBcSW6EQakrVOrVM0k8amPpklIXgu1+Af8Ay5mMYwgKC78B7OZzfyBWfgTxKWCm1nMbRoQFfWGvHfeefecdy/X8h0ZKqLrLq27p7evf+BDYnBoeGQ0OTa+Hnr1wBYl23O8YMMyQ+FIV5SUVI7Y8ANh1ixHlK3qcvu+3BBBKD33q2r6Yrtm7rtyT9qmYspIpr4Ycku6yqAdlRbZYDq9GxjJDOUoivRLoMcggzjyXvIXtrALDzbqqEHAhWLswETIXwU6CD5z22gxFzCS0b3AERKsrXOW4AyT2Sqv+3yqxKzL57ZnGKltfsXhP2BlGpP0ly7ojv7QJd3Qv1e9WpFHu5Ym71ZHK3xj9Pjj2sO7qhrvCgePqjcUFmd3KrumK67rdvNzpfxmjwp7mI96k9yrHzHtru2OT+Pw5G5toTjZmqJTuuV+f7Lzb3Z2G/f2WUEUv/O49OfDeQnWZ3I65fTCbGZxKR7cACbwCVmezhwWsYI8SvxqEz9wjgstq61qJa3cSdW6Yk0KT0L79h9J15xt</latexit> <latexit sha1_base64="aDkxGx5q/sb9YJIQEipbOLXS4BA=">AAACpnichVFNS+NQFD3Gz6lf1W4G3JQpSp1FuRFBcSW6EQakrVOrVM0k8amPpklIXgu1+Af8Ay5mMYwgKC78B7OZzfyBWfgTxKWCm1nMbRoQFfWGvHfeefecdy/X8h0ZKqLrLq27p7evf+BDYnBoeGQ0OTa+Hnr1wBYl23O8YMMyQ+FIV5SUVI7Y8ANh1ixHlK3qcvu+3BBBKD33q2r6Yrtm7rtyT9qmYspIpr4Ycku6yqAdlRbZYDq9GxjJDOUoivRLoMcggzjyXvIXtrALDzbqqEHAhWLswETIXwU6CD5z22gxFzCS0b3AERKsrXOW4AyT2Sqv+3yqxKzL57ZnGKltfsXhP2BlGpP0ly7ojv7QJd3Qv1e9WpFHu5Ym71ZHK3xj9Pjj2sO7qhrvCgePqjcUFmd3KrumK67rdvNzpfxmjwp7mI96k9yrHzHtru2OT+Pw5G5toTjZmqJTuuV+f7Lzb3Z2G/f2WUEUv/O49OfDeQnWZ3I65fTCbGZxKR7cACbwCVmezhwWsYI8SvxqEz9wjgstq61qJa3cSdW6Yk0KT0L79h9J15xt</latexit> <latexit sha1_base64="aDkxGx5q/sb9YJIQEipbOLXS4BA=">AAACpnichVFNS+NQFD3Gz6lf1W4G3JQpSp1FuRFBcSW6EQakrVOrVM0k8amPpklIXgu1+Af8Ay5mMYwgKC78B7OZzfyBWfgTxKWCm1nMbRoQFfWGvHfeefecdy/X8h0ZKqLrLq27p7evf+BDYnBoeGQ0OTa+Hnr1wBYl23O8YMMyQ+FIV5SUVI7Y8ANh1ixHlK3qcvu+3BBBKD33q2r6Yrtm7rtyT9qmYspIpr4Ycku6yqAdlRbZYDq9GxjJDOUoivRLoMcggzjyXvIXtrALDzbqqEHAhWLswETIXwU6CD5z22gxFzCS0b3AERKsrXOW4AyT2Sqv+3yqxKzL57ZnGKltfsXhP2BlGpP0ly7ojv7QJd3Qv1e9WpFHu5Ym71ZHK3xj9Pjj2sO7qhrvCgePqjcUFmd3KrumK67rdvNzpfxmjwp7mI96k9yrHzHtru2OT+Pw5G5toTjZmqJTuuV+f7Lzb3Z2G/f2WUEUv/O49OfDeQnWZ3I65fTCbGZxKR7cACbwCVmezhwWsYI8SvxqEz9wjgstq61qJa3cSdW6Yk0KT0L79h9J15xt</latexit> <latexit sha1_base64="aDkxGx5q/sb9YJIQEipbOLXS4BA=">AAACpnichVFNS+NQFD3Gz6lf1W4G3JQpSp1FuRFBcSW6EQakrVOrVM0k8amPpklIXgu1+Af8Ay5mMYwgKC78B7OZzfyBWfgTxKWCm1nMbRoQFfWGvHfeefecdy/X8h0ZKqLrLq27p7evf+BDYnBoeGQ0OTa+Hnr1wBYl23O8YMMyQ+FIV5SUVI7Y8ANh1ixHlK3qcvu+3BBBKD33q2r6Yrtm7rtyT9qmYspIpr4Ycku6yqAdlRbZYDq9GxjJDOUoivRLoMcggzjyXvIXtrALDzbqqEHAhWLswETIXwU6CD5z22gxFzCS0b3AERKsrXOW4AyT2Sqv+3yqxKzL57ZnGKltfsXhP2BlGpP0ly7ojv7QJd3Qv1e9WpFHu5Ym71ZHK3xj9Pjj2sO7qhrvCgePqjcUFmd3KrumK67rdvNzpfxmjwp7mI96k9yrHzHtru2OT+Pw5G5toTjZmqJTuuV+f7Lzb3Z2G/f2WUEUv/O49OfDeQnWZ3I65fTCbGZxKR7cACbwCVmezhwWsYI8SvxqEz9wjgstq61qJa3cSdW6Yk0KT0L79h9J15xt</latexit> % Kd de(t) dt <latexit sha1_base64="pK4ns8fNxecVndhwIulsD6E0PEo=">AAACp3ichVHLSsNAFD2Nr/quj4XgplgUdVEmIiiuim4EF9rWPqRKSdKphqZJSKYFDf0Bf8CFGxVcVBf+gRs3/oALP0FcKrhx4W0aEBX1hsycOXPPmXu5qm3ormDsMSR1dHZ194R7+/oHBoeGIyOjWdeqORrPaJZhOXlVcbmhmzwjdGHwvO1wpaoaPKdW1lr3uTp3XN0yt8Whzfeqyr6pl3VNEUQVI+MbxVJ0t+womlfis2Ku4ZVEoxiJsTjzI/oTyAGIIYgtK3KLXZRgQUMNVXCYEIQNKHDpK0AGg03cHjziHEK6f8/RQB9pa5TFKUMhtkLrPp0KAWvSueXp+mqNXjHod0gZxTR7YE32wu7ZNXti7796eb5Hq5ZD2tW2ltvF4eOJ9Nu/qirtAgefqj8UKmW3K3tkN1TX8858IfdnjwJlLPu96dSr7TOtrrW2T/3o5CW9kpr2ZtgFe6Z+z8n5jpzN+qt2meSpUxqX/H04P0F2IS6zuJxcjCVWg8GFMYkpzNJ0lpDAOraQoVePcIYmrqQ5aVPKSvl2qhQKNGP4EpLyAfEdnZo=</latexit> <latexit sha1_base64="pK4ns8fNxecVndhwIulsD6E0PEo=">AAACp3ichVHLSsNAFD2Nr/quj4XgplgUdVEmIiiuim4EF9rWPqRKSdKphqZJSKYFDf0Bf8CFGxVcVBf+gRs3/oALP0FcKrhx4W0aEBX1hsycOXPPmXu5qm3ormDsMSR1dHZ194R7+/oHBoeGIyOjWdeqORrPaJZhOXlVcbmhmzwjdGHwvO1wpaoaPKdW1lr3uTp3XN0yt8Whzfeqyr6pl3VNEUQVI+MbxVJ0t+womlfis2Ku4ZVEoxiJsTjzI/oTyAGIIYgtK3KLXZRgQUMNVXCYEIQNKHDpK0AGg03cHjziHEK6f8/RQB9pa5TFKUMhtkLrPp0KAWvSueXp+mqNXjHod0gZxTR7YE32wu7ZNXti7796eb5Hq5ZD2tW2ltvF4eOJ9Nu/qirtAgefqj8UKmW3K3tkN1TX8858IfdnjwJlLPu96dSr7TOtrrW2T/3o5CW9kpr2ZtgFe6Z+z8n5jpzN+qt2meSpUxqX/H04P0F2IS6zuJxcjCVWg8GFMYkpzNJ0lpDAOraQoVePcIYmrqQ5aVPKSvl2qhQKNGP4EpLyAfEdnZo=</latexit> <latexit sha1_base64="pK4ns8fNxecVndhwIulsD6E0PEo=">AAACp3ichVHLSsNAFD2Nr/quj4XgplgUdVEmIiiuim4EF9rWPqRKSdKphqZJSKYFDf0Bf8CFGxVcVBf+gRs3/oALP0FcKrhx4W0aEBX1hsycOXPPmXu5qm3ormDsMSR1dHZ194R7+/oHBoeGIyOjWdeqORrPaJZhOXlVcbmhmzwjdGHwvO1wpaoaPKdW1lr3uTp3XN0yt8Whzfeqyr6pl3VNEUQVI+MbxVJ0t+womlfis2Ku4ZVEoxiJsTjzI/oTyAGIIYgtK3KLXZRgQUMNVXCYEIQNKHDpK0AGg03cHjziHEK6f8/RQB9pa5TFKUMhtkLrPp0KAWvSueXp+mqNXjHod0gZxTR7YE32wu7ZNXti7796eb5Hq5ZD2tW2ltvF4eOJ9Nu/qirtAgefqj8UKmW3K3tkN1TX8858IfdnjwJlLPu96dSr7TOtrrW2T/3o5CW9kpr2ZtgFe6Z+z8n5jpzN+qt2meSpUxqX/H04P0F2IS6zuJxcjCVWg8GFMYkpzNJ0lpDAOraQoVePcIYmrqQ5aVPKSvl2qhQKNGP4EpLyAfEdnZo=</latexit> <latexit sha1_base64="pK4ns8fNxecVndhwIulsD6E0PEo=">AAACp3ichVHLSsNAFD2Nr/quj4XgplgUdVEmIiiuim4EF9rWPqRKSdKphqZJSKYFDf0Bf8CFGxVcVBf+gRs3/oALP0FcKrhx4W0aEBX1hsycOXPPmXu5qm3ormDsMSR1dHZ194R7+/oHBoeGIyOjWdeqORrPaJZhOXlVcbmhmzwjdGHwvO1wpaoaPKdW1lr3uTp3XN0yt8Whzfeqyr6pl3VNEUQVI+MbxVJ0t+womlfis2Ku4ZVEoxiJsTjzI/oTyAGIIYgtK3KLXZRgQUMNVXCYEIQNKHDpK0AGg03cHjziHEK6f8/RQB9pa5TFKUMhtkLrPp0KAWvSueXp+mqNXjHod0gZxTR7YE32wu7ZNXti7796eb5Hq5ZD2tW2ltvF4eOJ9Nu/qirtAgefqj8UKmW3K3tkN1TX8858IfdnjwJlLPu96dSr7TOtrrW2T/3o5CW9kpr2ZtgFe6Z+z8n5jpzN+qt2meSpUxqX/H04P0F2IS6zuJxcjCVWg8GFMYkpzNJ0lpDAOraQoVePcIYmrqQ5aVPKSvl2qhQKNGP4EpLyAfEdnZo=</latexit> r <latexit sha1_base64="l3XaGwPmE5Vm/Jhs1LG2xz0k9Bs=">AAACknichVG7SgNBFD1ZXzE+Eh+FYCMGRSzCrAiKVdTGwsIk5iFRZHcddchmd9mdBDT4A9oqFlYKFmLhH9jY+AMWfoKkjGBj4c1mQVTUu+zMmTP3nLmXqzum8CRjzyGlrb2jsyvcHenp7euPxgYGc55dcQ2eNWzTdgu65nFTWDwrhTR5wXG5VtZNntdLy837fJW7nrCtdXng8K2ytmeJXWFokqiUux2LswTzY+wnUAMQRxBrduwem9iBDQMVlMFhQRI2ocGjrwgVDA5xW6gR5xIS/j3HESKkrVAWpwyN2BKte3QqBqxF56an56sNesWk3yXlGCbYE7thDfbIbtkLe//Vq+Z7NGs5oF1vabmzHT0eybz9qyrTLrH/qfpDoVN2q7Jndkd11Temi/k/e5TYxbzfm6BeHZ9pdm20fKqH543MQnqiNsmuWJ36vSTnB3K2qq/GdYqnL2hc6vfh/AS5mYTKEmpqNp5cCgYXxijGMUXTmUMSK1hDll7lOMEpzpRhZUFZVJZbqUoo0AzhSyirH/1ElaE=</latexit> <latexit sha1_base64="l3XaGwPmE5Vm/Jhs1LG2xz0k9Bs=">AAACknichVG7SgNBFD1ZXzE+Eh+FYCMGRSzCrAiKVdTGwsIk5iFRZHcddchmd9mdBDT4A9oqFlYKFmLhH9jY+AMWfoKkjGBj4c1mQVTUu+zMmTP3nLmXqzum8CRjzyGlrb2jsyvcHenp7euPxgYGc55dcQ2eNWzTdgu65nFTWDwrhTR5wXG5VtZNntdLy837fJW7nrCtdXng8K2ytmeJXWFokqiUux2LswTzY+wnUAMQRxBrduwem9iBDQMVlMFhQRI2ocGjrwgVDA5xW6gR5xIS/j3HESKkrVAWpwyN2BKte3QqBqxF56an56sNesWk3yXlGCbYE7thDfbIbtkLe//Vq+Z7NGs5oF1vabmzHT0eybz9qyrTLrH/qfpDoVN2q7Jndkd11Temi/k/e5TYxbzfm6BeHZ9pdm20fKqH543MQnqiNsmuWJ36vSTnB3K2qq/GdYqnL2hc6vfh/AS5mYTKEmpqNp5cCgYXxijGMUXTmUMSK1hDll7lOMEpzpRhZUFZVJZbqUoo0AzhSyirH/1ElaE=</latexit> <latexit sha1_base64="l3XaGwPmE5Vm/Jhs1LG2xz0k9Bs=">AAACknichVG7SgNBFD1ZXzE+Eh+FYCMGRSzCrAiKVdTGwsIk5iFRZHcddchmd9mdBDT4A9oqFlYKFmLhH9jY+AMWfoKkjGBj4c1mQVTUu+zMmTP3nLmXqzum8CRjzyGlrb2jsyvcHenp7euPxgYGc55dcQ2eNWzTdgu65nFTWDwrhTR5wXG5VtZNntdLy837fJW7nrCtdXng8K2ytmeJXWFokqiUux2LswTzY+wnUAMQRxBrduwem9iBDQMVlMFhQRI2ocGjrwgVDA5xW6gR5xIS/j3HESKkrVAWpwyN2BKte3QqBqxF56an56sNesWk3yXlGCbYE7thDfbIbtkLe//Vq+Z7NGs5oF1vabmzHT0eybz9qyrTLrH/qfpDoVN2q7Jndkd11Temi/k/e5TYxbzfm6BeHZ9pdm20fKqH543MQnqiNsmuWJ36vSTnB3K2qq/GdYqnL2hc6vfh/AS5mYTKEmpqNp5cCgYXxijGMUXTmUMSK1hDll7lOMEpzpRhZUFZVJZbqUoo0AzhSyirH/1ElaE=</latexit> <latexit sha1_base64="l3XaGwPmE5Vm/Jhs1LG2xz0k9Bs=">AAACknichVG7SgNBFD1ZXzE+Eh+FYCMGRSzCrAiKVdTGwsIk5iFRZHcddchmd9mdBDT4A9oqFlYKFmLhH9jY+AMWfoKkjGBj4c1mQVTUu+zMmTP3nLmXqzum8CRjzyGlrb2jsyvcHenp7euPxgYGc55dcQ2eNWzTdgu65nFTWDwrhTR5wXG5VtZNntdLy837fJW7nrCtdXng8K2ytmeJXWFokqiUux2LswTzY+wnUAMQRxBrduwem9iBDQMVlMFhQRI2ocGjrwgVDA5xW6gR5xIS/j3HESKkrVAWpwyN2BKte3QqBqxF56an56sNesWk3yXlGCbYE7thDfbIbtkLe//Vq+Z7NGs5oF1vabmzHT0eybz9qyrTLrH/qfpDoVN2q7Jndkd11Temi/k/e5TYxbzfm6BeHZ9pdm20fKqH543MQnqiNsmuWJ36vSTnB3K2qq/GdYqnL2hc6vfh/AS5mYTKEmpqNp5cCgYXxijGMUXTmUMSK1hDll7lOMEpzpRhZUFZVJZbqUoo0AzhSyirH/1ElaE=</latexit> e <latexit sha1_base64="JucbY3V7iD4pyjirU4Kr9PWwJLM=">AAACknichVG7SgNBFD1Z3/EVH4VgIwZFLMKsCEoqNY2FRR7GKFFkdx11cLO77E4CMfgD2ioWVgoWYuEf2Nj4AxZ+glgq2Fh49wGiot5lZ86cuefMvVzdMYUnGXuMKU3NLa1t7R3xzq7unt5EX/+KZ1ddgxcN27TdVV3zuCksXpRCmnzVcblW0U1e0vcy/n2pxl1P2NayrDt8o6LtWGJbGJokKsc3E0mWYkGM/ARqBJKIImsnbrGOLdgwUEUFHBYkYRMaPPrKUMHgELeBBnEuIRHccxwgTtoqZXHK0Ijdo3WHTuWItejse3qB2qBXTPpdUo5gjD2wK/bC7tk1e2Lvv3o1Ag+/ljrteqjlzmbv4VDh7V9VhXaJ3U/VHwqdssPKHtkN1fW8Nlku/dmjxDZmg94E9eoEjN+1EfrU9k9fCun8WGOcXbBn6vecnO/I2aq9Gpc5nj+jcanfh/MTrEylVJZSc9PJuYVocO0YxigmaDozmMMisijSqxxHOMaJMqiklXklE6YqsUgzgC+hLH0A4O6VlA==</latexit> <latexit sha1_base64="JucbY3V7iD4pyjirU4Kr9PWwJLM=">AAACknichVG7SgNBFD1Z3/EVH4VgIwZFLMKsCEoqNY2FRR7GKFFkdx11cLO77E4CMfgD2ioWVgoWYuEf2Nj4AxZ+glgq2Fh49wGiot5lZ86cuefMvVzdMYUnGXuMKU3NLa1t7R3xzq7unt5EX/+KZ1ddgxcN27TdVV3zuCksXpRCmnzVcblW0U1e0vcy/n2pxl1P2NayrDt8o6LtWGJbGJokKsc3E0mWYkGM/ARqBJKIImsnbrGOLdgwUEUFHBYkYRMaPPrKUMHgELeBBnEuIRHccxwgTtoqZXHK0Ijdo3WHTuWItejse3qB2qBXTPpdUo5gjD2wK/bC7tk1e2Lvv3o1Ag+/ljrteqjlzmbv4VDh7V9VhXaJ3U/VHwqdssPKHtkN1fW8Nlku/dmjxDZmg94E9eoEjN+1EfrU9k9fCun8WGOcXbBn6vecnO/I2aq9Gpc5nj+jcanfh/MTrEylVJZSc9PJuYVocO0YxigmaDozmMMisijSqxxHOMaJMqiklXklE6YqsUgzgC+hLH0A4O6VlA==</latexit> <latexit sha1_base64="JucbY3V7iD4pyjirU4Kr9PWwJLM=">AAACknichVG7SgNBFD1Z3/EVH4VgIwZFLMKsCEoqNY2FRR7GKFFkdx11cLO77E4CMfgD2ioWVgoWYuEf2Nj4AxZ+glgq2Fh49wGiot5lZ86cuefMvVzdMYUnGXuMKU3NLa1t7R3xzq7unt5EX/+KZ1ddgxcN27TdVV3zuCksXpRCmnzVcblW0U1e0vcy/n2pxl1P2NayrDt8o6LtWGJbGJokKsc3E0mWYkGM/ARqBJKIImsnbrGOLdgwUEUFHBYkYRMaPPrKUMHgELeBBnEuIRHccxwgTtoqZXHK0Ijdo3WHTuWItejse3qB2qBXTPpdUo5gjD2wK/bC7tk1e2Lvv3o1Ag+/ljrteqjlzmbv4VDh7V9VhXaJ3U/VHwqdssPKHtkN1fW8Nlku/dmjxDZmg94E9eoEjN+1EfrU9k9fCun8WGOcXbBn6vecnO/I2aq9Gpc5nj+jcanfh/MTrEylVJZSc9PJuYVocO0YxigmaDozmMMisijSqxxHOMaJMqiklXklE6YqsUgzgC+hLH0A4O6VlA==</latexit> <latexit sha1_base64="JucbY3V7iD4pyjirU4Kr9PWwJLM=">AAACknichVG7SgNBFD1Z3/EVH4VgIwZFLMKsCEoqNY2FRR7GKFFkdx11cLO77E4CMfgD2ioWVgoWYuEf2Nj4AxZ+glgq2Fh49wGiot5lZ86cuefMvVzdMYUnGXuMKU3NLa1t7R3xzq7unt5EX/+KZ1ddgxcN27TdVV3zuCksXpRCmnzVcblW0U1e0vcy/n2pxl1P2NayrDt8o6LtWGJbGJokKsc3E0mWYkGM/ARqBJKIImsnbrGOLdgwUEUFHBYkYRMaPPrKUMHgELeBBnEuIRHccxwgTtoqZXHK0Ijdo3WHTuWItejse3qB2qBXTPpdUo5gjD2wK/bC7tk1e2Lvv3o1Ag+/ljrteqjlzmbv4VDh7V9VhXaJ3U/VHwqdssPKHtkN1fW8Nlku/dmjxDZmg94E9eoEjN+1EfrU9k9fCun8WGOcXbBn6vecnO/I2aq9Gpc5nj+jcanfh/MTrEylVJZSc9PJuYVocO0YxigmaDozmMMisijSqxxHOMaJMqiklXklE6YqsUgzgC+hLH0A4O6VlA==</latexit> 4ZTUFN  'FFECBDLQBUI *OQVU 0VUQVU
  37. 37 Dynamic Target Controller &SSPS 4FUQPJOU 1 * % r

    <latexit sha1_base64="l3XaGwPmE5Vm/Jhs1LG2xz0k9Bs=">AAACknichVG7SgNBFD1ZXzE+Eh+FYCMGRSzCrAiKVdTGwsIk5iFRZHcddchmd9mdBDT4A9oqFlYKFmLhH9jY+AMWfoKkjGBj4c1mQVTUu+zMmTP3nLmXqzum8CRjzyGlrb2jsyvcHenp7euPxgYGc55dcQ2eNWzTdgu65nFTWDwrhTR5wXG5VtZNntdLy837fJW7nrCtdXng8K2ytmeJXWFokqiUux2LswTzY+wnUAMQRxBrduwem9iBDQMVlMFhQRI2ocGjrwgVDA5xW6gR5xIS/j3HESKkrVAWpwyN2BKte3QqBqxF56an56sNesWk3yXlGCbYE7thDfbIbtkLe//Vq+Z7NGs5oF1vabmzHT0eybz9qyrTLrH/qfpDoVN2q7Jndkd11Temi/k/e5TYxbzfm6BeHZ9pdm20fKqH543MQnqiNsmuWJ36vSTnB3K2qq/GdYqnL2hc6vfh/AS5mYTKEmpqNp5cCgYXxijGMUXTmUMSK1hDll7lOMEpzpRhZUFZVJZbqUoo0AzhSyirH/1ElaE=</latexit> <latexit sha1_base64="l3XaGwPmE5Vm/Jhs1LG2xz0k9Bs=">AAACknichVG7SgNBFD1ZXzE+Eh+FYCMGRSzCrAiKVdTGwsIk5iFRZHcddchmd9mdBDT4A9oqFlYKFmLhH9jY+AMWfoKkjGBj4c1mQVTUu+zMmTP3nLmXqzum8CRjzyGlrb2jsyvcHenp7euPxgYGc55dcQ2eNWzTdgu65nFTWDwrhTR5wXG5VtZNntdLy837fJW7nrCtdXng8K2ytmeJXWFokqiUux2LswTzY+wnUAMQRxBrduwem9iBDQMVlMFhQRI2ocGjrwgVDA5xW6gR5xIS/j3HESKkrVAWpwyN2BKte3QqBqxF56an56sNesWk3yXlGCbYE7thDfbIbtkLe//Vq+Z7NGs5oF1vabmzHT0eybz9qyrTLrH/qfpDoVN2q7Jndkd11Temi/k/e5TYxbzfm6BeHZ9pdm20fKqH543MQnqiNsmuWJ36vSTnB3K2qq/GdYqnL2hc6vfh/AS5mYTKEmpqNp5cCgYXxijGMUXTmUMSK1hDll7lOMEpzpRhZUFZVJZbqUoo0AzhSyirH/1ElaE=</latexit> <latexit sha1_base64="l3XaGwPmE5Vm/Jhs1LG2xz0k9Bs=">AAACknichVG7SgNBFD1ZXzE+Eh+FYCMGRSzCrAiKVdTGwsIk5iFRZHcddchmd9mdBDT4A9oqFlYKFmLhH9jY+AMWfoKkjGBj4c1mQVTUu+zMmTP3nLmXqzum8CRjzyGlrb2jsyvcHenp7euPxgYGc55dcQ2eNWzTdgu65nFTWDwrhTR5wXG5VtZNntdLy837fJW7nrCtdXng8K2ytmeJXWFokqiUux2LswTzY+wnUAMQRxBrduwem9iBDQMVlMFhQRI2ocGjrwgVDA5xW6gR5xIS/j3HESKkrVAWpwyN2BKte3QqBqxF56an56sNesWk3yXlGCbYE7thDfbIbtkLe//Vq+Z7NGs5oF1vabmzHT0eybz9qyrTLrH/qfpDoVN2q7Jndkd11Temi/k/e5TYxbzfm6BeHZ9pdm20fKqH543MQnqiNsmuWJ36vSTnB3K2qq/GdYqnL2hc6vfh/AS5mYTKEmpqNp5cCgYXxijGMUXTmUMSK1hDll7lOMEpzpRhZUFZVJZbqUoo0AzhSyirH/1ElaE=</latexit> <latexit sha1_base64="l3XaGwPmE5Vm/Jhs1LG2xz0k9Bs=">AAACknichVG7SgNBFD1ZXzE+Eh+FYCMGRSzCrAiKVdTGwsIk5iFRZHcddchmd9mdBDT4A9oqFlYKFmLhH9jY+AMWfoKkjGBj4c1mQVTUu+zMmTP3nLmXqzum8CRjzyGlrb2jsyvcHenp7euPxgYGc55dcQ2eNWzTdgu65nFTWDwrhTR5wXG5VtZNntdLy837fJW7nrCtdXng8K2ytmeJXWFokqiUux2LswTzY+wnUAMQRxBrduwem9iBDQMVlMFhQRI2ocGjrwgVDA5xW6gR5xIS/j3HESKkrVAWpwyN2BKte3QqBqxF56an56sNesWk3yXlGCbYE7thDfbIbtkLe//Vq+Z7NGs5oF1vabmzHT0eybz9qyrTLrH/qfpDoVN2q7Jndkd11Temi/k/e5TYxbzfm6BeHZ9pdm20fKqH543MQnqiNsmuWJ36vSTnB3K2qq/GdYqnL2hc6vfh/AS5mYTKEmpqNp5cCgYXxijGMUXTmUMSK1hDll7lOMEpzpRhZUFZVJZbqUoo0AzhSyirH/1ElaE=</latexit> e <latexit sha1_base64="JucbY3V7iD4pyjirU4Kr9PWwJLM=">AAACknichVG7SgNBFD1Z3/EVH4VgIwZFLMKsCEoqNY2FRR7GKFFkdx11cLO77E4CMfgD2ioWVgoWYuEf2Nj4AxZ+glgq2Fh49wGiot5lZ86cuefMvVzdMYUnGXuMKU3NLa1t7R3xzq7unt5EX/+KZ1ddgxcN27TdVV3zuCksXpRCmnzVcblW0U1e0vcy/n2pxl1P2NayrDt8o6LtWGJbGJokKsc3E0mWYkGM/ARqBJKIImsnbrGOLdgwUEUFHBYkYRMaPPrKUMHgELeBBnEuIRHccxwgTtoqZXHK0Ijdo3WHTuWItejse3qB2qBXTPpdUo5gjD2wK/bC7tk1e2Lvv3o1Ag+/ljrteqjlzmbv4VDh7V9VhXaJ3U/VHwqdssPKHtkN1fW8Nlku/dmjxDZmg94E9eoEjN+1EfrU9k9fCun8WGOcXbBn6vecnO/I2aq9Gpc5nj+jcanfh/MTrEylVJZSc9PJuYVocO0YxigmaDozmMMisijSqxxHOMaJMqiklXklE6YqsUgzgC+hLH0A4O6VlA==</latexit> <latexit sha1_base64="JucbY3V7iD4pyjirU4Kr9PWwJLM=">AAACknichVG7SgNBFD1Z3/EVH4VgIwZFLMKsCEoqNY2FRR7GKFFkdx11cLO77E4CMfgD2ioWVgoWYuEf2Nj4AxZ+glgq2Fh49wGiot5lZ86cuefMvVzdMYUnGXuMKU3NLa1t7R3xzq7unt5EX/+KZ1ddgxcN27TdVV3zuCksXpRCmnzVcblW0U1e0vcy/n2pxl1P2NayrDt8o6LtWGJbGJokKsc3E0mWYkGM/ARqBJKIImsnbrGOLdgwUEUFHBYkYRMaPPrKUMHgELeBBnEuIRHccxwgTtoqZXHK0Ijdo3WHTuWItejse3qB2qBXTPpdUo5gjD2wK/bC7tk1e2Lvv3o1Ag+/ljrteqjlzmbv4VDh7V9VhXaJ3U/VHwqdssPKHtkN1fW8Nlku/dmjxDZmg94E9eoEjN+1EfrU9k9fCun8WGOcXbBn6vecnO/I2aq9Gpc5nj+jcanfh/MTrEylVJZSc9PJuYVocO0YxigmaDozmMMisijSqxxHOMaJMqiklXklE6YqsUgzgC+hLH0A4O6VlA==</latexit> <latexit sha1_base64="JucbY3V7iD4pyjirU4Kr9PWwJLM=">AAACknichVG7SgNBFD1Z3/EVH4VgIwZFLMKsCEoqNY2FRR7GKFFkdx11cLO77E4CMfgD2ioWVgoWYuEf2Nj4AxZ+glgq2Fh49wGiot5lZ86cuefMvVzdMYUnGXuMKU3NLa1t7R3xzq7unt5EX/+KZ1ddgxcN27TdVV3zuCksXpRCmnzVcblW0U1e0vcy/n2pxl1P2NayrDt8o6LtWGJbGJokKsc3E0mWYkGM/ARqBJKIImsnbrGOLdgwUEUFHBYkYRMaPPrKUMHgELeBBnEuIRHccxwgTtoqZXHK0Ijdo3WHTuWItejse3qB2qBXTPpdUo5gjD2wK/bC7tk1e2Lvv3o1Ag+/ljrteqjlzmbv4VDh7V9VhXaJ3U/VHwqdssPKHtkN1fW8Nlku/dmjxDZmg94E9eoEjN+1EfrU9k9fCun8WGOcXbBn6vecnO/I2aq9Gpc5nj+jcanfh/MTrEylVJZSc9PJuYVocO0YxigmaDozmMMisijSqxxHOMaJMqiklXklE6YqsUgzgC+hLH0A4O6VlA==</latexit> <latexit sha1_base64="JucbY3V7iD4pyjirU4Kr9PWwJLM=">AAACknichVG7SgNBFD1Z3/EVH4VgIwZFLMKsCEoqNY2FRR7GKFFkdx11cLO77E4CMfgD2ioWVgoWYuEf2Nj4AxZ+glgq2Fh49wGiot5lZ86cuefMvVzdMYUnGXuMKU3NLa1t7R3xzq7unt5EX/+KZ1ddgxcN27TdVV3zuCksXpRCmnzVcblW0U1e0vcy/n2pxl1P2NayrDt8o6LtWGJbGJokKsc3E0mWYkGM/ARqBJKIImsnbrGOLdgwUEUFHBYkYRMaPPrKUMHgELeBBnEuIRHccxwgTtoqZXHK0Ijdo3WHTuWItejse3qB2qBXTPpdUo5gjD2wK/bC7tk1e2Lvv3o1Ag+/ljrteqjlzmbv4VDh7V9VhXaJ3U/VHwqdssPKHtkN1fW8Nlku/dmjxDZmg94E9eoEjN+1EfrU9k9fCun8WGOcXbBn6vecnO/I2aq9Gpc5nj+jcanfh/MTrEylVJZSc9PJuYVocO0YxigmaDozmMMisijSqxxHOMaJMqiklXklE6YqsUgzgC+hLH0A4O6VlA==</latexit> (PBQQ  'FFECBDLQBUI 4FNBQIPSF $166TBHF $IBOHFSBUF • Having two nested control loops to change set-point on running. • The outer loop sets new set-points of the inner loop when CPU usage reaches the upper limit. $POUSPMMFS
  38. Bounding

  39. Bounding concurrency using buffered channel as semaphore 39 To bound

    concurrency in Go func main() { var wg sync.WaitGroup sem := make(chan struct{}, 3) for i := 0; i < 10; i++ { wg.Add(1) sem <- struct{}{} go func() { defer wg.Done() defer func() { <-sem }() task() }() } wg.Wait() close(sem) }
  40. 40 Elastic semaphore DIBO &MBTUJDTFNBQIPSF $PVOUFS XBJU TJHOBM JODSFNFOU-JNJU 

     $PVOUFS $PVOUFS 8BJUJOH /PUXBJUJOH  • wait() & signal() as same as semaphore • incrementLimit() changes upper of semaphore variable • Ensure atomicity of above operations
  41. Kaburaya

  42. Architecture of kaburaya 42 • The optimal number of concurrency

    to be determined dynamically and be controlled rapidly • Use CPU usage upper limit as performance metrics. • Use feedback control to determine the number of goroutines continuosly. • Use elastic semaphore to bound concurrency. • Kaburaya is “ధ໼” which is a Japanese arrow with a whistle. • https://github.com/monochromegane/kaburaya
  43. • Kaburaya optimizes the number of goroutines by feedback control.

    43 monochromegane/kaburaya func main() { sem := kaburaya.NewSem(100 * time.Millisecond) var wg sync.WaitGroup for i := 0; i < 10; i++ { wg.Add(1) sem.Wait() go func() { defer sem.Signal() defer wg.Done() task() }() } wg.Wait() sem.Stop() } Create semaphore and specify time span for update the semaphore and CPU target. Wait for decrement operation. Increment operation. Stop update the semaphore and CPU target.
  44. 4. Evaluation

  45. • Apply kaburaya to some tasks to evaluate efficacy. •

    Measure speedup, CPU usage, and limit of semaphore. • Find good and not so good cases for the kaburaya controller. 45 Evaluation
  46. Task: pt_mac-scan 46 • pt runs on macOS with realtime

    virus scan • CPU 8, Memory 16GB • GOMAXPROCS=8 • Better concurrent are from 16 to 64. • More degradation of performance due to increased goroutines.
  47. Task: pt_mac-scan (Good) 47 • Kaburaya found good set points

    and adjust numbers within ideal range. Duration: 50ms, Gain: 0.1, Change rate: ±0.3 Median
  48. Task: pt_mac-no-scan 48 • Runs pt on Mac without realtime

    virus scan • CPU 8, Memory 16GB • GOMAXPROCS=8 • Better concurrent is 4 • Less degradation of performance due to increased goroutines.
  49. Task: pt_mac-no-scan (Good) 49 • Kaburaya found good set points

    and adjust numbers within ideal range. Duration: 50ms, Gain: 0.1, Change rate: ±0.3 Median
  50. Task: pt_mac-no-scan (Bad) 50 • Find set points and adjust

    numbers within ideal range. • Increase the number due to failure to reset the set point at end of period. Duration: 50ms, Gain: 0.1, Change rate: ±0.3 Median
  51. Task: pt_linux 51 • pt runs on Linux • CPU

    4, Memory 4GB • GOMAXPROCS=4 • Better concurrent are from 4 to 32. • Less degradation of performance due to increased goroutines.
  52. Task: pt_linux (Not so bad) 52 • Kaburaya found good

    set points and adjust numbers within ideal range. • Increase the number due to failure to reset the set point at end of period. Duration: 50ms, Gain: 0.1, Change rate: ±0.3 Median
  53. Task: mem_4096000_10000 53 • Runs task uses memory. • 4MB/task

    x 10,000 • CPU 4, Memory 4GB • GOMAXPROCS=4 • Better concurrent is 512. • More degradation of performance (Swap out) due to increased goroutines.
  54. Task: mem_4096000_10000 (Good) 54 • Kaburaya found set points and

    adjust numbers within ideal range. • Avoid starvation of resource Duration: 500ms, Gain: 0.3, Change rate: ±0.3 Median
  55. Task: mem_40960_1000000 55 • Runs task uses memory. • 40kB/task

    x 1,000,000 • CPU 4, Memory 4GB • GOMAXPROCS=4 • Better concurrent are from 4 to 64. • Less degradation of performance due to increased goroutines.
  56. Task: mem_40960_1000000 (Good) 56 • Kaburaya found set points and

    adjust numbers within ideal range. Duration: 500ms, Gain: 0.3, Change rate: ±0.3 Median
  57. Task: mem_409600_100000 57 • Runs task uses memory. • 400kB/task

    x 100,000 • CPU 4, Memory 4GB • GOMAXPROCS=4 • Better concurrent is over 1024? • Less degradation of performance due to increased goroutines.
  58. Task: mem_409600_100000 (Not so good) 58 • Kaburaya found set

    points and adjust numbers. • Does not reach the limit of performance metrics… Duration: 500ms, Gain: 0.3, Change rate: ±0.3 Median
  59. • Kaburaya finds set point and adjusts numbers within ideal

    range by feedback control and elastic semaphore. • Towards to improve the controller • prediction accuracy of set point • Optimal parameters (K{p,i,d} and range of change rate) • Solve them in the related researches about feedback control as the next step. 59 Experimental results
  60. 5. Conclusion

  61. Conclusion 61 • I proposed kaburaya to control the number

    of goroutines without depending on platform, runtime and current load. • Experimental results show possible and some issues of kaburaya. • In particular improvement detecting performance upper limit of task is important. • In the future, I will also consider the application to auto-scaling of cloud computing. • Go
  62. Appendix

  63. • Implementation of Kaburaya • https://github.com/monochromegane/kaburaya • Study of Kaburaya

    (In Japanese) • https://blog.monochromegane.com/blog/2018/11/25/wsa_3_kaburaya/ • https://gist.github.com/monochromegane/5e9fc94371a9b2385eedd38d98341e6b • https://blog.monochromegane.com/blog/2019/04/14/wsa_4_ebira/ • Learn more goroutine • https://speakerdeck.com/monochromegane/road-to-your-goroutines 63 Reference
  64. 64 Go gopher • The Go gopher was designed by

    Renee French. • The gopher illustrations were drawn by Keita Kawamoto. • Licensed under the Creative Commons 3.0 Attributions license.
  65. Thank you!

  66. None