Sprinting Pandas at ODSC 2020

Sprinting Pandas
Sometimes our Python Pandas code feels slow and sometimes we can't fit enough data into RAM. Based on recent updates to the 2nd edition of Ian's High Performance Python book and his public training classes come and learn how to get more into RAM (reducing your need for other technologies like Spark), how to quickly compile for significant speedups, how to run in parallel and which libraries you're missing that unlock additional performance benefits. You'll leave with new techniques to make your DataFrames smaller and many ideas for processing your data faster.
This talk is inspired by Ian's work updating his O'Reilly book High Performance Python to the 2nd edition for 2020. With over 10 years of evolution the Pandas DataFrame library has gained a huge amount of functionality and it is used by millions of Pythonistas - but the most obvious way to solve a task isn't always the fastest or most RAM efficient. This talk will help any Pandas user (beginner or beyond) process more data faster, making them more effective at their jobs.

See related talks at https://ianozsvald.com

Transcript

1. Sprinting Pandas (live from London) @IanOzsvald – ianozsvald.com Ian Ozsvald ODSC 2020
2.  Interim Chief Data Scientist  19+ years experience  Team coaching & public courses –I’m sharing from my Higher Performance Python course Introductions By [ian]@ianozsvald[.com] Ian Ozsvald 2nd Edition!
3.  Pandas – Saving RAM to fit in more data – Calculating faster by dropping to Numpy  Advice for “being highly performant”  Has Covid 19 affected UK Company Registrations? Today’s goal By [ian]@ianozsvald[.com] Ian Ozsvald
4. Strings are expensive and slow By [ian]@ianozsvald[.com] Ian Ozsvald
5. Categoricals are cheap and fast! By [ian]@ianozsvald[.com] Ian Ozsvald Circa 1% of previous memory cost
6. Categoricals “.cat” accessor By [ian]@ianozsvald[.com] Ian Ozsvald
7. Categoricals – over 10x speed up (on this data)! By [ian]@ianozsvald[.com] Ian Ozsvald
8. Categoricals – index queries faster! By [ian]@ianozsvald[.com] Ian Ozsvald Circa 500x speed-up!
9. float64 is default and a bit expensive By [ian]@ianozsvald[.com] Ian Ozsvald
10. float32 “half-price” and a bit faster By [ian]@ianozsvald[.com] Ian Ozsvald
11. Make choices to save RAM By [ian]@ianozsvald[.com] Ian Ozsvald Including the index (previously we ignored it) we still save circa 50% RAM so you can fit in more rows of data
12. “dtype_diet” gives you advice By [ian]@ianozsvald[.com] Ian Ozsvald
13. Drop to NumPy if you know you can By [ian]@ianozsvald[.com] Ian Ozsvald Caveat – Pandas mean is not np mean, the fair comparison is to np nanmean which is slower – see my blog or PyDataAmsterdam 2020 talk for details
14. NumPy vs Pandas overhead (ser.sum()) By [ian]@ianozsvald[.com] Ian Ozsvald 25 files, 83 functions Very few NumPy calls! Thanks! https://github.com/ianozsvald/callgraph_james_powell
15. Overhead... By [ian]@ianozsvald[.com] Ian Ozsvald
16. Overhead with ser.values.sum() By [ian]@ianozsvald[.com] Ian Ozsvald 18 files, 51 functions Many fewer Pandas calls (but still a lot!)
17. Is Pandas unnecessarily slow – NO! By [ian]@ianozsvald[.com] Ian Ozsvald https://github.com/pandas-dev/pandas/issues/34773 - the truth is a bit complicated!
18.  Install optional (but great!) Pandas dependencies – bottleneck – numexpr  Investigate https://github.com/ianozsvald/dtype_diet  Investigate my ipython_memory_usage (PyPI/Conda) Being highly performant By [ian]@ianozsvald[.com] Ian Ozsvald https://pandas.pydata.org/pandas-docs/stable/user_guide/enhancingperf.html
19. Pure Python is “slow” and expressive By [ian]@ianozsvald[.com] Ian Ozsvald Deliberately poor function – pretend this is clever but slow!
20. Compile to Numba judiciously By [ian]@ianozsvald[.com] Ian Ozsvald Near 10x speed-up!
21. Parallelise with Dask for multi-core By [ian]@ianozsvald[.com] Ian Ozsvald  Make plain-Python code multi-core  Note I had to drop text index column due to speed-hit  Data copy cost can overwhelm any benefits so (always) profile & time
22.  Mistakes slow us down (PAY ATTENTION!) – Try nullable Int64 & boolean, forthcoming Float64 – Write tests (unit & end-to-end) – Lots more material & my newsletter on my blog IanOzsvald.com – Time saving docs: Being highly performant By [ian]@ianozsvald[.com] Ian Ozsvald
23.  Memory mapped & lazy computation – New string dtype (RAM efficient)  Modin sits on Pandas, new “algebra” for dfs – Drop in replacement, easy to try Vaex / Modin By [ian]@ianozsvald[.com] Ian Ozsvald See talks on my blog:
24.  You have a huge dataset on a single harddrive  Memory mapped files (HDF5) are best  Numpy types and simpler Pandas-like functions  Investment – similar but different API to Pandas When to try Vaex By [ian]@ianozsvald[.com] Ian Ozsvald https://github.com/vaexio/vaex/issues/968
25.  You want Pandas but ran out of RAM on 1 machine  You want multi-machine cluster scalability  You want multi-core support for operations like groupby on parallelisable datasets  Investment – quick start then a learning curve When to try Dask By [ian]@ianozsvald[.com] Ian Ozsvald
26.  You want all of Pandas  You have lots of RAM and many CPUs  You’re doing groupby operations on many columns  Investment – easy to try When to try Modin By [ian]@ianozsvald[.com] Ian Ozsvald https://github.com/modin-project/modin/issues/1390
27. Covid 19’s effect on UK Economy? By [ian]@ianozsvald[.com] Ian Ozsvald Sharp decline in corporate registration after Lockdown – then apparent surge (perhaps just backed-up paperwork?). Will the recovery “last”? All open data, you can do similar things!
28.  Make it right then make it fast  Think about being performant  See blog for my classes  I’d love a postcard if you learned something new! Summary By [ian]@ianozsvald[.com] Ian Ozsvald
29. Be faster by learning new approaches By [ian]@ianozsvald[.com] Ian Ozsvald