Thriving with Python: Navigate the pitfalls in a polyglot world

This keynote was shared at PyTexas 2024. The talk shares how and when to combine Python with other programming languages. It offers tips for handling uncertainty and technology changes. It points out three pitfalls that developers face on their path to success.

Transcript

1. Carol Willing Thriving with Python Navigate the pitfalls in a polyglot world April 21, 2024 @willingc@hachyderm.io https://speakerdeck.com/willingc 1
3. polyglot knowing, using, or written in multiple languages
4. Hi!
5. Carol Willing Python Core Developer Python Steering Council (three terms) Python Software Foundation Fellow Jupyter Core Developer 2017 ACM Software System Award Papermill Maintainer pyOpenSci Advisory Board Chan Zuckerberg Open Science Board Quansight Labs Board VP of Engineering, Noteable
6. How do we succeed in a polyglot world?
7. Python and ??? Which languages to choose? • Rust, Go, C, C++ • JavaScript, TypeScript • Java, C# • Haskell, Clojure, Scala • Lua, Perl • Swift and others
8. Watch your step How do we avoid the pitfalls? Swooning at shiny Chasing speed Hype and FOMO
10. What should I do?
11. https://thenewstack.io/a-conversation-with-the-creators-behind-python-java-typescript-and-perl/ Guido van Rossum (Python) James Gosling (Java) Larry Wall (Perl) Anders Hejlsberg (Pascal, C#, TypeScript) me (Jupyter)
13. 1970s BASIC • Bell Labs mainframe BASIC • TRS-80 BASIC • Apple BASIC Don Slepian, CC BY-SA 4.0 <https://creativecommons.org/licenses/by-sa/ 4.0>, via Wikimedia Commons Early personal computers By FozzTexx - Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/ index.php?curid=79580939
14. 1980s Fortran, Pascal, C, LISP, SQL • Mobile phones emerging • No smartphones • No web, no google • No Linux • No Python https://archive.org/details/byte-magazine-1985-04/ Yes - Arti fi cial Intelligence 😱
15. My Journey
16. Paradigm Shi ft s Transistors and mainframes Personal Computers Worldwide Web, Smartphones, Cloud computing and data science Generative AI and LLMs Cellular service Time
17. How?
18. Follow your North Star
19. Embrace a new musician's mistakes mindset Loud and proud
20. Take a step, then SCAMPER https://www.designorate.com/a-guide-to-the-scamper-technique-for-creative-thinking/
21. Understand your
22. Let's go tackle some pitfalls
23. Pitfall #1 Chasing speed
24. Questions to ponder Speed • What to measure? • How to measure? • What are my goals? • Is it fast enough? • What are the tradeo ff s? Performance Reliability Flexibility Security
25. 3.11, 3.12, 3.13 Use a recent Python • Try the latest version • Alternatively, try pypy • 3.13 • Experimental freethreading without the gil (global interpreter lock) • JIT compiler • Release date: Tuesday, 2024-10-01 https://docs.python.org/3/whatsnew/index.html
26. Measure, then optimize Pro fi le • timeit or magic %timeit • cPro fi le (deterministic pro fi ler) • Pyinstrument (statistical pro fi ler) • py-spy (sampling pro fi ler) • pyperf (https://github.com/psf/pyperf) py-spy basics https://docs.ray.io/en/latest/cluster/kubernetes/k8s-ecosystem/pyspy.html py-spy Search fit (implicit/als.py:159) calculate_similar_artists (lastfm.py:79) fit (implicit/als.py:163) _get_similarity_score (implic.. _get_similarity_score (implicit/recommender_.. argpartition (numpy/core/fromnumeric.py:757) _wrapfunc (numpy/core/fromnumeric.py:51) calculate_similar_artists (lastfm.py:95) similar_items (implicit/recommender_base.py:203) <module> (lastfm.py:161) fl amegraphs to visualize
27. cPro fi le, py-spy Pro fi le & Optimize • Data structures • Concurrency / Asynchronous • Vectorization and numpy • Accelerate using best practices in Python • Substitute C, Rust, Go, C++ https://tinkering.xyz/fmo-optimization-story/
28. Itamar Turner-Trauring and pythonspeed.com Best practices Vectorization: How slow Python runs fast code | PyData Global 2022 Vectorization: How slow Python runs fast code | PyData Global 2022 Timestamps for video https://pythonspeed.com/datascience/ Faster Data Science: Speed up your data science and scienti fi c computing code
29. GPUs and JITs • CUDA • numba • cupy Use to test with and without the decorator: %timeit monte_carlo_pi(1000)
30. Complements Python and Rust Prototype in Python and move performance bottlenecks to Rust. - Moshe Zadka • Use PyO3 to wrap Rust code for Python • https://pyo3.rs/v0.21.2/getting-started • maturin develop https://opensource.com/article/23/3/python-loves-rust
31. Development Speed • Ru ff • Playwright https://bloomberg.github.io/memray/index.html https://bloomberg.github.io/pystack/ Bloomberg 1/2 million fi les, 100 million lines of code, funding PSF https://www.bloomberg.com/company/values/tech-at-bloomberg/python/
32. Up and running Prototyping Speed • Iterative • 5 lines of code or less can do meaningful work • Interactive with Jupyter notebooks https://web.mit.edu/music21/
33. Pitfall #2 Swooning at shiny
35. Grass is greener in another language
36. https://cheezburger.com/9380419584/1-ring-to-rule-them-all pip poetry pdm conda hatch fl it mamba uv pixi • Rust - cargo, fast resolver, one stop distribution • JS - lock fi les, tools update everything packaging Maybe
37. Python: a decade of improvements Maybe not • Pure Python packages • Thank you PyPA volunteers • Reach out and thank a maintainer • Watch Bernat Gabor's PyTexas talk on helping maintainers • Outreach e ff orts
38. Science, Python, Community pyOpenSci • Python Package Guide • Tutorials • Peer reviewed scienti fi c packages • Inclusive open science
39. Next gen conda pixi
40. Rust, TypeScript, Python play.ru ff .rs https://github.com/astral-sh/ruff/tree/main/playground Playground for formatting, AST, tokens, formatter IR
41. play.ru ff .rs https://github.com/astral-sh/ruff/tree/main/playground AST Abstract Syntax Tree
42. play.ru ff .rs https://github.com/astral-sh/ruff/tree/main/playground AST Abstract Syntax Tree
43. Rust and Python in the real world Py03
44. Python, JavaScript, p5.js JupyterLite, pyodide, WebAssembly https://www.willingconsulting.com/pysplashlive.html JupyterLite runs entirely in the browser.
46. Testing Playwright for Python • Performant and built using Rust • Web focus • JS, Python • Examples, docs, and YouTube updates https://playwright.dev/python/
47. import re from playwright.sync_api import Page, expect def test_has_title(page: Page): page.goto("https://playwright.dev/") # Expect a title "to contain" a substring. expect(page).to_have_title(re.compile("Playwright")) def test_get_started_link(page: Page): page.goto("https://playwright.dev/") # Click the get started link. page.get_by_role("link", name="Get started").click() # Expects page to have a heading with the name of Installation. expect(page.get_by_role("heading", name="Installation")).to_be_visible()
48. PWDEBUG=1 pytest -s test_example.py Launch Playwright debugger
49. Load page and test for link
50. Execute another test and step
51. Pitfall #3 Hype and FOMO
52. import antigravity Use Python. It's remarkable, and you can skip the TSA line. Wow!
56. Interview with Scott Hanselmann Dr. Brandeis Marshall We all are data people.
57. Tech Trends Source: Forrester Critical Global Issues Source: UN
58. Timeline to a new technology What if I try... now basic grounding understand from the beginning 5 years solve a problem best practices automation 6 months more examples peers and their experiences 1 year better tooling better docs emerging packages
60. Re fl ections from our polyglot journey
61. Awareness Pitfalls Swooning at shiny Chasing speed Hype and FOMO
62. Keys to success
63. Expect to make mistakes 1
64. Take a step, then SCAMPER 2
65. Understand your "why" 3
66. Treasure your time and energy 4
67. Fun builds resilience 5
68. What's next?
70. Python and the polyglots
71. Introducing their hit song Thriving at Python and the polyglots
72. Python and the polyglots Thank you
73. Python and the polyglots Thank you
74. Carol Willing Thriving with Python Navigate the pitfalls in a polyglot world April 21, 2024 The End