How Slow Are Your Loops?

Some basic techniques to measure and improve the speed of your code.

Transcript

1. HOW SLOW ARE YOUR LOOPS? M. Emre Aydın Python in Astronomy, Leiden - 2017
2. HOW FAST ARE YOUR LOOPS?
3. HOW SLOW ARE YOUR LOOPS?
5. THE ROOT OF ALL EVIL • If you process live data and your code can’t keep up… it’s slow. • If you find yourself squeezing in new hobbies while your code is “working”, it’s probably slow. xkcd: 303
6. WAYS TO IMPROVE SPEED: BETTER ALGORITHMS
7. WAYS TO IMPROVE SPEED: MATHEMATICAL APPROXIMATIONS • The MorletWavelet • = %&'( )' − %+ ,& ⁄ • When c < 0.02, %+ ,& ⁄ is negligible, a.k.a. TheAbbreviated MorletWavelet (Foster, 1996) • () = )'%&'(
8. WAYS TO IMPROVE SPEED: OPTIMIZED COMPILERS • Cython • Numba • PyPy
9. WAYS TO IMPROVE SPEED: OPTIMIZED COMPILERS • https://github.com/sanand0 • https://youtu.be/ylBslijJexw
10. WAYS TO IMPROVE SPEED: OPTIMIZED COMPILERS https://eng.climate.com/2015/04/09/numba-vs-cython-how-to-choose/
11. WAYS TO IMPROVE SPEED: OS & HARDWARE
12. CONVERTING CANON RAW FILES TO FITS FILES: CR2FITS Dcraw: Dave Coffin NetPbmFile: Christoph Gohlke
13. CONVERTING CANON RAW FILES TO FITS FILES: CR2FITS Dcraw: Dave Coffin NetPbmFile: Christoph Gohlke Meredith Rawls • @merrdiff
14. mount -t tmpfs -o size=512m tmpfs /mnt/tempfiles
15. COUNTING LINES OF A TEXT FILE
18. MEASURING TIME PYTHON 3.5.1 IPYTHON 5.1.0 xkcd: 1190, frame 403
19. TIMEIT MAGIC
20. LINE PROFILER
21. LINE PROFILER
24. MEMORY PROFILER
25. CHECK WHAT’S UNDER THE HOOD
26. #1 USE BUILT IN FUNCTIONS https://docs.python.org/3/library/functions.html
27. #2 FUNCTION CALL OVERHEAD https://www.python.org/doc/essays/list2str/ 1.72 s per loop 651 ms per loop “ […] function call overhead in Python is bigger than for loop overhead.” – Guido van Rossum
28. BYTECODE DISASSEMBLER
29. BYTECODE DISASSEMBLER http://akaptur.com/blog/2013/08/14/python- bytecode-fun-with-dis/ Allison Kaptur
30. #3 LIST COMPREHENSION
31. #3 LIST COMPREHENSION 15% speed up http://blog.cdleary.com/2010/04/efficiency-of-list-comprehensions/
32. #4 USE GENERATORS
33. #4 USE GENERATORS https://youtu.be/EnSu9hHGq5o Loop like a native: while, for, iterators, generators Ned Batchelder
34. #5 DATE PARSING
35. #5 DATE PARSING
36. #5 DATE PARSING %2 speed up!
37. #5 DATE PARSING %2 speed up! Let’s use datetime?
38. #5 DATE PARSING
39. #5 DATE PARSING %910 speed up!
40. #5 DATE PARSING %910 speed up! Can we get rid of strptime?
41. #5 DATE PARSING %854 (further) speed up! 105 ms to 1.31 ms is more than 80x faster…
42. #6 IS TRUE?
43. #7 TRY / EXCEPT
44. #7 TRY / EXCEPT
45. #7 TRY / EXCEPT
46. #8 GO LOCAL Sebastian Witowski : https://youtu.be/YjHsOrOOSuI
47. #9 USE NUMPY / PANDAS https://youtu.be/EEUXKG97YRw
48. #9 USE NUMPY / PANDAS BUT TEST IT! https://youtu.be/ylBslijJexw
49. WRAP UP • Don’t over-optimize! • Time it and profile it • Start with the most inner loop • Consider list comprehension, generator, and numpy arrays • Take leverage of your OS: On the shoulders of giants