Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Thriving with Python: Navigate the pitfalls in a polyglot world

Thriving with Python: Navigate the pitfalls in a polyglot world

This keynote was shared at PyTexas 2024. The talk shares how and when to combine Python with other programming languages. It offers tips for handling uncertainty and technology changes. It points out three pitfalls that developers face on their path to success.

Carol Willing

April 21, 2024

More Decks by Carol Willing

Other Decks in Programming


  1. Carol Willing Thriving with Python Navigate the pitfalls in a

    polyglot world April 21, 2024 @[email protected] https://speakerdeck.com/willingc 1
  2. Hi!

  3. Carol Willing Python Core Developer Python Steering Council (three terms)

    Python Software Foundation Fellow Jupyter Core Developer 2017 ACM Software System Award Papermill Maintainer pyOpenSci Advisory Board Chan Zuckerberg Open Science Board Quansight Labs Board VP of Engineering, Noteable
  4. Python and ??? Which languages to choose? • Rust, Go,

    C, C++ • JavaScript, TypeScript • Java, C# • Haskell, Clojure, Scala • Lua, Perl • Swift and others
  5. Watch your step How do we avoid the pitfalls? Swooning

    at shiny Chasing speed Hype and FOMO
  6. 1970s BASIC • Bell Labs mainframe BASIC • TRS-80 BASIC

    • Apple BASIC Don Slepian, CC BY-SA 4.0 <https://creativecommons.org/licenses/by-sa/ 4.0>, via Wikimedia Commons Early personal computers By FozzTexx - Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/ index.php?curid=79580939
  7. 1980s Fortran, Pascal, C, LISP, SQL • Mobile phones emerging

    • No smartphones • No web, no google • No Linux • No Python https://archive.org/details/byte-magazine-1985-04/ Yes - Arti fi cial Intelligence 😱
  8. Paradigm Shi ft s Transistors and mainframes Personal Computers Worldwide

    Web, Smartphones, Cloud computing and data science Generative AI and LLMs Cellular service Time
  9. Questions to ponder Speed • What to measure? • How

    to measure? • What are my goals? • Is it fast enough? • What are the tradeo ff s? Performance Reliability Flexibility Security
  10. 3.11, 3.12, 3.13 Use a recent Python • Try the

    latest version • Alternatively, try pypy • 3.13 • Experimental freethreading without the gil (global interpreter lock) • JIT compiler • Release date: Tuesday, 2024-10-01 https://docs.python.org/3/whatsnew/index.html
  11. Measure, then optimize Pro fi le • timeit or magic

    %timeit • cPro fi le (deterministic pro fi ler) • Pyinstrument (statistical pro fi ler) • py-spy (sampling pro fi ler) • pyperf (https://github.com/psf/pyperf) py-spy basics https://docs.ray.io/en/latest/cluster/kubernetes/k8s-ecosystem/pyspy.html py-spy Search fit (implicit/als.py:159) calculate_similar_artists (lastfm.py:79) fit (implicit/als.py:163) _get_similarity_score (implic.. _get_similarity_score (implicit/recommender_.. argpartition (numpy/core/fromnumeric.py:757) _wrapfunc (numpy/core/fromnumeric.py:51) calculate_similar_artists (lastfm.py:95) similar_items (implicit/recommender_base.py:203) <module> (lastfm.py:161) fl amegraphs to visualize
  12. cPro fi le, py-spy Pro fi le & Optimize •

    Data structures • Concurrency / Asynchronous • Vectorization and numpy • Accelerate using best practices in Python • Substitute C, Rust, Go, C++ https://tinkering.xyz/fmo-optimization-story/
  13. Itamar Turner-Trauring and pythonspeed.com Best practices Vectorization: How slow Python

    runs fast code | PyData Global 2022 Vectorization: How slow Python runs fast code | PyData Global 2022 Timestamps for video https://pythonspeed.com/datascience/ Faster Data Science: Speed up your data science and scienti fi c computing code
  14. GPUs and JITs • CUDA • numba • cupy Use

    to test with and without the decorator: %timeit monte_carlo_pi(1000)
  15. Complements Python and Rust Prototype in Python and move performance

    bottlenecks to Rust. - Moshe Zadka • Use PyO3 to wrap Rust code for Python • https://pyo3.rs/v0.21.2/getting-started • maturin develop https://opensource.com/article/23/3/python-loves-rust
  16. Development Speed • Ru ff • Playwright https://bloomberg.github.io/memray/index.html https://bloomberg.github.io/pystack/ Bloomberg

    1/2 million fi les, 100 million lines of code, funding PSF https://www.bloomberg.com/company/values/tech-at-bloomberg/python/
  17. Up and running Prototyping Speed • Iterative • 5 lines

    of code or less can do meaningful work • Interactive with Jupyter notebooks https://web.mit.edu/music21/
  18. https://cheezburger.com/9380419584/1-ring-to-rule-them-all pip poetry pdm conda hatch fl it mamba uv

    pixi • Rust - cargo, fast resolver, one stop distribution • JS - lock fi les, tools update everything packaging Maybe
  19. Python: a decade of improvements Maybe not • Pure Python

    packages • Thank you PyPA volunteers • Reach out and thank a maintainer • Watch Bernat Gabor's PyTexas talk on helping maintainers • Outreach e ff orts
  20. Science, Python, Community pyOpenSci • Python Package Guide • Tutorials

    • Peer reviewed scienti fi c packages • Inclusive open science
  21. Testing Playwright for Python • Performant and built using Rust

    • Web focus • JS, Python • Examples, docs, and YouTube updates https://playwright.dev/python/
  22. import re from playwright.sync_api import Page, expect def test_has_title(page: Page):

    page.goto("https://playwright.dev/") # Expect a title "to contain" a substring. expect(page).to_have_title(re.compile("Playwright")) def test_get_started_link(page: Page): page.goto("https://playwright.dev/") # Click the get started link. page.get_by_role("link", name="Get started").click() # Expects page to have a heading with the name of Installation. expect(page.get_by_role("heading", name="Installation")).to_be_visible()
  23. Timeline to a new technology What if I try... now

    basic grounding understand from the beginning 5 years solve a problem best practices automation 6 months more examples peers and their experiences 1 year better tooling better docs emerging packages