PyCon India 2018 - Executing scripts in a few milliseconds with MicroPython

PyCon India 2018 - Executing scripts in a few milliseconds with MicroPython

Command execution time can become important in a number of applications. Commands executed in command-line completion need to execute in less then 100ms or users will perceive a delay. In Shell scripting one might want to execute commands repeatedly in a for loop and fast execution times makes this more feasible.

Python is a very powerful language but has a much slower startup time compared to other interpreted languages like Perl, Lua and Bash. It can take up to 10 times longer to startup then some of these other languages.

MicroPython was written as a lean implementation of Python 3 with a small subset of the standard library mainly intended to run on microcontrollers. But it happily runs on Unix systems with excellent startup performance, making it an ideal candidate for implementing certain time sensitive commands.

Ce17126d6065d975c34973f3e55dc51b?s=128

Marwan Alsabbagh

October 07, 2018
Tweet

Transcript

  1. Executing scripts in milliseconds with MicroPython Marwan AlSabbagh

  2. In 2006 I fell in love

  3. With Python

  4. We did everything together... Command line scripting Web applications Graphical

    applications Network programing Coroutines
  5. Until one fateful day. Something horrible happened.

  6. CONKY

  7. None
  8. • Over 300 built-in widgets • Supports custom scripts •

    I wanted 5 of them • Running each second • 60*60*24*5 = 432,000 Conky Setup
  9. Python 15% Bash 0.5% vs CPU Usage

  10. • Text editor lint check on save Flake8 takes >300ms

    • Bash completion pip completion can take >1s Beyond Conky
  11. • Elapsed time • Clock cycles Measure Script Execution

  12. add.py print(1 + 1) add.sh echo $((1 + 1)) add.pl

    print 1 + 1 . "\n"; add.lua print(1 + 1) add.awk BEGIN {print 1+1; exit} Test scripts
  13. $ time python3.7 add.py 2 real 0m0.010s user 0m0.004s sys

    0m0.007s Bash time command
  14. $ perf stat -r 10 /usr/bin/python3.7 /opt/py/add.py Performance counter stats

    (10 runs): 9.146528 task-clock (msec) # 0.981 CPUs utilized 0 context-switches # 0.000 K/sec 0 cpu-migrations # 0.000 K/sec 804 page-faults # 0.088 M/sec 31,926,587 cycles # 3.491 GHz 39,586,183 instructions # 1.24 insn per cycle 8,926,391 branches # 975.932 M/sec 325,273 branch-misses # 3.64% of all branches 0.009324090 seconds time elapsed ( +- 0.25% ) Linux Performance Tool
  15. January 15, 2017 $9 USD ESP8266 80 MHz CPU MicroPython

  16. MicroPython • Lean implementation of Python 3 • Includes selection

    of core Python libraries • Supports Unix platforms • usocket (e.g. HTTP client/server) • uos (has uos.system) • ujson, ure, utime, uzlib
  17. Hardware • Model: system76 - Galago Pro • OS: Ubuntu

    Linux 16.04 • CPU: 3.5 GHz i7-7500U • RAM: 16 GB • DISK: 250 GB NVMe SSD • Shipped: July-2017 Software • Python 3.7.0 • MicroPython v1.9.4 (2018-10-04) Hardware/Software Used
  18. Execution Time

  19. CPU Cycles

  20. Part 2 pip bash completion

  21. $ time python -c 'import pip' real 0m0.743s user 0m0.696s

    sys 0m0.048s pip built-in completion • Usability needs < 100ms • Does not complete package names
  22. export PIP_NO_INDEX=1 export PIP_FIND_LINKS="http://localhost/wheels/" export PIP_TRUSTED_HOST='localhost' python3 -m virtualenv -p

    python3 /opt/py/venv/main Demo Details • Nginx hosting over 400 wheels • Auto complete sub-commands and packages • live list of packages on every complete (over http) • Completion includes package name and versions
  23. _pip_completion() { compopt -o filenames COMPREPLY=($(compgen -W "$(pipcomp.py $COMP_CWORD)" --

    \ "${COMP_WORDS[COMP_CWORD]}")) } complete -F _pip_completion pip bashrc
  24. #!/usr/bin/env micropython import urequests, uos, sys def get_packages(url): html =

    urequests.get(url).text wheels = [i for i in html.split('"') if i.endswith('.whl')] return ['{0}=={1}'.format(*i.split('-')) for i in wheels] def main(): if sys.argv[1] == '1': print('install uninstall freeze list show search help') if sys.argv[1] == '2': url = uos.getenv('PIP_FIND_LINKS') packages = get_packages(url) print('\n'.join(packages)) if __name__ == "__main__": main() pipcomp.py
  25. $ perf stat -r 10 /bin/micropython /opt/py/pipcomp.py 1 0.001258725 seconds

    time elapsed pipcomp Performance Sub-command completion: 1.3 ms $ perf stat -r 10 /bin/micropython /opt/py/pipcomp.py 2 0.009136741 seconds time elapsed Package name completion: 9.1 ms
  26. Live Demo!

  27. The End https://marwano.com