Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Making Python computations fast

Russel Winder
September 18, 2015

Making Python computations fast

Slides from my PyConUK 2015 session

Russel Winder

September 18, 2015
Tweet

More Decks by Russel Winder

Other Decks in Technology

Transcript

  1. Copyright © 2015 Russel Winder 1 Making Python computations fast

    Dr Russel Winder email: [email protected] twitter: @russel_winder Web: http://www.russel.org.uk
  2. Copyright © 2007–2015 Russel Winder 7 The Maths The equation:

    Leads to the approximation formula: ∑ i=0 n 4 (1+x i 2) Δ x≈π ∫ 0 1 4 (1+x2) dx=π ¿
  3. Copyright © 2007–2015 Russel Winder 8 The Conception Calculate an

    approximation to the area under a conic section by filling the area with a known number of rectangles of known size and then sum the areas of the rectangles. The narrower the rectangles, the more accurate the approximation. This is a data parallel problem and embarrassingly parallel.
  4. Copyright © 2007–2015 Russel Winder 9 The Parallelism • Addition

    is commutative and associative so we can partition the overall summation in any way we want: a + b + c + d + e + f ( a + b ) + ( c + d ) + ( e + f ) Can partition the overall sum into bits to be done on different processors. This should make things faster.
  5. Copyright © 2007–2015 Russel Winder 10 Adding the Parallelism •

    Instead of doing all the calculations sequentially, partition the problem into a number of chunks. problem subproblem result subproblem subproblem subproblem . . . scatter gather fork join reduce Data parallel map
  6. Copyright © 2007–2015 Russel Winder 11 Some Code • Sequential.

    • Use threads to obtain parallelism…not. • Use process to obtain parallelism. Friends don't let friends use Python 2 instead of Python 3.
  7. Copyright © 2007–2015 Russel Winder 13 More Code • Use

    Cython. • Use Numba. cf. lightning talk at PyConUK 2014.
  8. Copyright © 2007–2015 Russel Winder 14 A lot faster but…

    is controlling parallelism in Python the thing to do…
  9. Copyright © 2007–2015 Russel Winder 15 Even More Code •

    Use NumPy. • Use C. • Use C++/Boost. • Use D/PyD. • Use Chapel/PyChapel. Offload the whole computation, Leaving Python code as the coordination code. https://github.com/russel/Pi_Quadrature/tree/master/Python http://dlang.org/ http://pyd.readthedocs.org/ http://chapel.cray.com/ http://pychapel.readthedocs.org/
  10. Copyright © 2007–2015 Russel Winder 16 Parallelism and computation is

    not Python's forte. (Even though Python is a great language.)
  11. Copyright © 2015 Russel Winder 20 Making Python computations fast

    Dr Russel Winder email: [email protected] twitter: @russel_winder Web: http://www.russel.org.uk
  12. Copyright © 2015 Russel Winder 21 John, you were a

    truly Pythonic hero, I miss your physical presence here.