Thriving with Python: Navigate the pitfalls in a polyglot world

Slide 1

Slide 1 text

Carol Willing Thriving with Python Navigate the pitfalls in a polyglot world April 21, 2024 @[email protected] https://speakerdeck.com/willingc 1

Slide 2

Slide 2 text

No content

Slide 3

Slide 3 text

polyglot knowing, using, or written in multiple languages

Slide 4

Slide 4 text

Hi!

Slide 5

Slide 5 text

Carol Willing Python Core Developer Python Steering Council (three terms) Python Software Foundation Fellow Jupyter Core Developer 2017 ACM Software System Award Papermill Maintainer pyOpenSci Advisory Board Chan Zuckerberg Open Science Board Quansight Labs Board VP of Engineering, Noteable

Slide 6

Slide 6 text

How do we succeed in a polyglot world?

Slide 7

Slide 7 text

Python and ??? Which languages to choose? • Rust, Go, C, C++ • JavaScript, TypeScript • Java, C# • Haskell, Clojure, Scala • Lua, Perl • Swift and others

Slide 8

Slide 8 text

Watch your step How do we avoid the pitfalls? Swooning at shiny Chasing speed Hype and FOMO

Slide 9

Slide 9 text

No content

Slide 10

Slide 10 text

What should I do?

Slide 11

Slide 11 text

https://thenewstack.io/a-conversation-with-the-creators-behind-python-java-typescript-and-perl/ Guido van Rossum (Python) James Gosling (Java) Larry Wall (Perl) Anders Hejlsberg (Pascal, C#, TypeScript) me (Jupyter)

Slide 12

Slide 12 text

No content

Slide 13

Slide 13 text

1970s BASIC • Bell Labs mainframe BASIC • TRS-80 BASIC • Apple BASIC Don Slepian, CC BY-SA 4.0 , via Wikimedia Commons Early personal computers By FozzTexx - Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/ index.php?curid=79580939

Slide 14

Slide 14 text

1980s Fortran, Pascal, C, LISP, SQL • Mobile phones emerging • No smartphones • No web, no google • No Linux • No Python https://archive.org/details/byte-magazine-1985-04/ Yes - Arti fi cial Intelligence 😱

Slide 15

Slide 15 text

My Journey

Slide 16

Slide 16 text

Paradigm Shi ft s Transistors and mainframes Personal Computers Worldwide Web, Smartphones, Cloud computing and data science Generative AI and LLMs Cellular service Time

Slide 17

Slide 17 text

How?

Slide 18

Slide 18 text

Follow your North Star

Slide 19

Slide 19 text

Embrace a new musician's mistakes mindset Loud and proud

Slide 20

Slide 20 text

Take a step, then SCAMPER https://www.designorate.com/a-guide-to-the-scamper-technique-for-creative-thinking/

Slide 21

Slide 21 text

Understand your

Slide 22

Slide 22 text

Let's go tackle some pitfalls

Slide 23

Slide 23 text

Pitfall #1 Chasing speed

Slide 24

Slide 24 text

Questions to ponder Speed • What to measure? • How to measure? • What are my goals? • Is it fast enough? • What are the tradeo ff s? Performance Reliability Flexibility Security

Slide 25

Slide 25 text

3.11, 3.12, 3.13 Use a recent Python • Try the latest version • Alternatively, try pypy • 3.13 • Experimental freethreading without the gil (global interpreter lock) • JIT compiler • Release date: Tuesday, 2024-10-01 https://docs.python.org/3/whatsnew/index.html

Slide 26

Slide 26 text

Measure, then optimize Pro fi le • timeit or magic %timeit • cPro fi le (deterministic pro fi ler) • Pyinstrument (statistical pro fi ler) • py-spy (sampling pro fi ler) • pyperf (https://github.com/psf/pyperf) py-spy basics https://docs.ray.io/en/latest/cluster/kubernetes/k8s-ecosystem/pyspy.html py-spy Search fit (implicit/als.py:159) calculate_similar_artists (lastfm.py:79) fit (implicit/als.py:163) _get_similarity_score (implic.. _get_similarity_score (implicit/recommender_.. argpartition (numpy/core/fromnumeric.py:757) _wrapfunc (numpy/core/fromnumeric.py:51) calculate_similar_artists (lastfm.py:95) similar_items (implicit/recommender_base.py:203) (lastfm.py:161) fl amegraphs to visualize

Slide 27

Slide 27 text

cPro fi le, py-spy Pro fi le & Optimize • Data structures • Concurrency / Asynchronous • Vectorization and numpy • Accelerate using best practices in Python • Substitute C, Rust, Go, C++ https://tinkering.xyz/fmo-optimization-story/

Slide 28

Slide 28 text

Itamar Turner-Trauring and pythonspeed.com Best practices Vectorization: How slow Python runs fast code | PyData Global 2022 Vectorization: How slow Python runs fast code | PyData Global 2022 Timestamps for video https://pythonspeed.com/datascience/ Faster Data Science: Speed up your data science and scienti fi c computing code

Slide 29

Slide 29 text

GPUs and JITs • CUDA • numba • cupy Use to test with and without the decorator: %timeit monte_carlo_pi(1000)

Slide 30

Slide 30 text

Complements Python and Rust Prototype in Python and move performance bottlenecks to Rust. - Moshe Zadka • Use PyO3 to wrap Rust code for Python • https://pyo3.rs/v0.21.2/getting-started • maturin develop https://opensource.com/article/23/3/python-loves-rust

Slide 31

Slide 31 text

Development Speed • Ru ff • Playwright https://bloomberg.github.io/memray/index.html https://bloomberg.github.io/pystack/ Bloomberg 1/2 million fi les, 100 million lines of code, funding PSF https://www.bloomberg.com/company/values/tech-at-bloomberg/python/

Slide 32

Slide 32 text

Up and running Prototyping Speed • Iterative • 5 lines of code or less can do meaningful work • Interactive with Jupyter notebooks https://web.mit.edu/music21/

Slide 33

Slide 33 text

Pitfall #2 Swooning at shiny

Slide 34

Slide 34 text

No content

Slide 35

Slide 35 text

Grass is greener in another language

Slide 36

Slide 36 text

https://cheezburger.com/9380419584/1-ring-to-rule-them-all pip poetry pdm conda hatch fl it mamba uv pixi • Rust - cargo, fast resolver, one stop distribution • JS - lock fi les, tools update everything packaging Maybe

Slide 37

Slide 37 text

Python: a decade of improvements Maybe not • Pure Python packages • Thank you PyPA volunteers • Reach out and thank a maintainer • Watch Bernat Gabor's PyTexas talk on helping maintainers • Outreach e ff orts

Slide 38

Slide 38 text

Science, Python, Community pyOpenSci • Python Package Guide • Tutorials • Peer reviewed scienti fi c packages • Inclusive open science

Slide 39

Slide 39 text

Next gen conda pixi

Slide 40

Slide 40 text

Rust, TypeScript, Python play.ru ff .rs https://github.com/astral-sh/ruff/tree/main/playground Playground for formatting, AST, tokens, formatter IR

Slide 41

Slide 41 text

play.ru ff .rs https://github.com/astral-sh/ruff/tree/main/playground AST Abstract Syntax Tree

Slide 42

Slide 42 text

play.ru ff .rs https://github.com/astral-sh/ruff/tree/main/playground AST Abstract Syntax Tree

Slide 43

Slide 43 text

Rust and Python in the real world Py03

Slide 44

Slide 44 text

Python, JavaScript, p5.js JupyterLite, pyodide, WebAssembly https://www.willingconsulting.com/pysplashlive.html JupyterLite runs entirely in the browser.

Slide 45

Slide 45 text

No content

Slide 46

Slide 46 text

Testing Playwright for Python • Performant and built using Rust • Web focus • JS, Python • Examples, docs, and YouTube updates https://playwright.dev/python/

Slide 47

Slide 47 text

import re from playwright.sync_api import Page, expect def test_has_title(page: Page): page.goto("https://playwright.dev/") # Expect a title "to contain" a substring. expect(page).to_have_title(re.compile("Playwright")) def test_get_started_link(page: Page): page.goto("https://playwright.dev/") # Click the get started link. page.get_by_role("link", name="Get started").click() # Expects page to have a heading with the name of Installation. expect(page.get_by_role("heading", name="Installation")).to_be_visible()

Slide 48

Slide 48 text

PWDEBUG=1 pytest -s test_example.py Launch Playwright debugger

Slide 49

Slide 49 text

Load page and test for link

Slide 50

Slide 50 text

Execute another test and step

Slide 51

Slide 51 text

Pitfall #3 Hype and FOMO

Slide 52

Slide 52 text

import antigravity Use Python. It's remarkable, and you can skip the TSA line. Wow!

Slide 53

Slide 53 text

No content

Slide 54

Slide 54 text

No content

Slide 55

Slide 55 text

No content

Slide 56

Slide 56 text

Interview with Scott Hanselmann Dr. Brandeis Marshall We all are data people.

Slide 57

Slide 57 text

Tech Trends Source: Forrester Critical Global Issues Source: UN

Slide 58

Slide 58 text

Timeline to a new technology What if I try... now basic grounding understand from the beginning 5 years solve a problem best practices automation 6 months more examples peers and their experiences 1 year better tooling better docs emerging packages

Slide 59

Slide 59 text

No content

Slide 60

Slide 60 text

Re fl ections from our polyglot journey

Slide 61

Slide 61 text

Awareness Pitfalls Swooning at shiny Chasing speed Hype and FOMO

Slide 62

Slide 62 text

Keys to success

Slide 63

Slide 63 text

Expect to make mistakes 1

Slide 64

Slide 64 text

Take a step, then SCAMPER 2

Slide 65

Slide 65 text

Understand your "why" 3

Slide 66

Slide 66 text

Treasure your time and energy 4

Slide 67

Slide 67 text

Fun builds resilience 5

Slide 68

Slide 68 text

What's next?

Slide 69

Slide 69 text

No content

Slide 70

Slide 70 text

Python and the polyglots

Slide 71

Slide 71 text

Introducing their hit song Thriving at Python and the polyglots

Slide 72

Slide 72 text

Python and the polyglots Thank you

Slide 73

Slide 73 text

Python and the polyglots Thank you

Slide 74

Slide 74 text

Carol Willing Thriving with Python Navigate the pitfalls in a polyglot world April 21, 2024 The End