The art of building and maintaining Tania Allard , Director Quansight Labs PyData London 2024 successful and sustainable open source projects

The art of building and maintaining Tania Allard , Director Quansight Labs PyData London 2024 successful and sustainable open source projects

Get the slides at https://bit.ly/oss-success-24

What & why this talk? Reflect on lessons from contributing to and maintaining OSS projects and communities for +15 years Recognise the challenges and opportunities ahead to continue to serve our community This topic has been in my head non-stop for a while

01. SUCCESS

What are your favourite tools or libraries?

What makes them special?

What does success look like for an open source project?

Adoption Stability Robustness Reliability Inclusion and diversity Quality Security Governance Accessibility Discoverability Usability Perhaps… Documentation

Success means different things to different people

Success means different things to different people Do you support an end-user community, developer community*, or many different groups? *data scientists, researchers, research software engineers, hobbyists… choose your group

• Enable users to do stuff • Do not strive to do the job for the user • Find their niche and stick to it* • Do not reinvent the wheel & preserve context But good tools share some traits * Unix philosophy: Do one thing and do it well

• Help users to experience their first “AHA moment” early on • Work with other tools (are composable and extensible) • Are intuitive But good tools share some traits

But… don’t these principles contradict each other? Avoid monolithic ownership and allow the user to self-serve Find your niche (do one thing)

Leverage the existing ecosystem Be judiciously extensible Be extensible and composable But… don’t these principles contradict each other?

Remember, your users define success and you find ways to empower them and be successful You do not need (or want) to own all the pieces and tools!

“ … the situation of putting together jigsaw puzzles with hammers, scissors, and tape, instead of having finely-crafted pieces that are designed to fit together. https://rachelbythebay.com/w/2020/08/14/jobs/ Remember the cloud migration and what followed?

“ We say “Machine Learning is the new electricity” I’d like to offer an alternative metaphor: machine learning has become alchemy. - Ben Rich and Ali Rahimi NeurIps Keynote 2017 What about today?

02. Design

“ Design is concerned with how things work, how they are controlled, and the nature of the interaction between people and technology. - Don Norman, The design of everyday things

As developers we sometimes think we are just making technical implementation decisions

We are making design decisions

We are deciding on and building pathways for success for our users

We are deciding on and building pathways for success for our users User-first development/design != assuming what users want and how they want it

Example: packaging and environment management

At the beginning there was one… and life was good: pip Then there were two… conda and pip and life was still good

And now there are many more… and life is just confusing https://alpopkes.com/posts/python/packaging_tools/

Let’s meet a user… • Ravi: an investigative journalist focused on data- driven, factual narratives detailing the corruption of local governments in the global South • Needs to: analyse data (csv, json, etc.), identify patterns, and create narratives and visualisations to inform the general public • Has Python installed

How do we even get Ravi the rest of the tools he needs? https://alpopkes.com/posts/python/packaging_tools/ pixi uv

Looking through a usability and discoverability lens: pip, conda, pixi, uv

Affordances Represent the possibilities of how the agent can interact with something

Signifiers A perceivable signal of the affordance (relationship between the agent and the object)

$ pip --help Usage: pip [options] Commands: install Install packages. download Download packages. uninstall Uninstall packages. freeze Output installed packages in requirements format. inspect Inspect the python environment. list List installed packages. show Show information about installed packages. check Verify installed packages have compatible dependencies. config Manage local and global configuration. search Search PyPI for packages. cache Inspect and manage pip's wheel cache. pip is the package installer for Python. You can use pip to install packages from the Python Package Index and other indexes. Affordances and signifiers

Affordances and signifiers $ conda --help usage: conda [-h] [-v] [--no-plugins] [-V] COMMAND ... conda is a tool for managing and deploying applications, environments and packages. commands: The following built-in and plugins subcommands are available. COMMAND activate Activate a conda environment. build Build conda packages from a conda recipe. clean Remove unused packages and caches. compare Compare packages between conda environments. config Modify configuration values in .condarc. convert Convert pure Python packages to other platforms (a.k.a., subdirs). create Create a new conda environment from a list of specified packages. deactivate Deactivate the current active conda environment. install Install a list of packages into a specified conda environment. develop Install a Python package in 'development mode'. Similar to `pip install --editable`

$ uv --help Usage: uv [OPTIONS] Commands: pip Resolve and install Python packages tool Run and manage executable Python packages venv Create a virtual environment cache Manage the cache version Display uv's version help Print this message or the help of the given subcommand(s) An extremely fast Python package installer and resolver, written in Rust. Designed as a drop-in replacement for common pip and pip- tools workflows. Affordances and signifiers

$ pixie --help Pixi is a versatile developer workflow tool designed to streamline the management of your project's dependencies, tasks, and environments. Built on top of the Conda ecosystem, Pixi offers seamless integration with the PyPI ecosystem. Basic Usage: Initialize pixi for a project: $ pixi init $ pixi add python numpy pytest Run a task: $ pixi add task test 'pytest -s' $ pixi run test Usage: pixi [OPTIONS] Commands: init Creates a new project add Adds dependencies to the project [aliases: a] remove Removes dependencies from the project [aliases: rm] install Install all dependencies [aliases: i] Affordances and signifiers

Consistency Preserve mental maps and follow patterns that already exist

Consistency across tools $ conda —help $ conda install # Install a list of packages into a specified conda environment. $ uv —help $ uv pip # Resolve and install Python packages $ pixie —help $ pixie install # Install all dependencies [aliases: I] $ pixie add # Adds dependencies to the project [aliases: a] $ pip —help $ pip install # Install packages.

Consistency and complexity $ conda —help $ conda install # Install a list of packages into a specified conda environment. $ uv —help $ uv pip # Resolve and install Python packages $ uv pip install —help Install packages into the current environment Usage: uv pip install [OPTIONS] |--editable

Consistency within the tool $ uv pip # Resolve and install Python packages Commands: compile Compile a `requirements.in` file to a `requirements.txt` file sync Sync dependencies from a `requirements.txt` file install Install packages into the current environment $ uv pip install —help Install packages into the current environment Usage: uv pip install [OPTIONS] |--editable $ uv venv --help Create a virtual environment Usage: uv venv [OPTIONS] [NAME]

Making it discoverable • Comprehensive help texts • Provide lots of examples • Suggest what command to run next • Suggest what to do when there is an error (feedback) Provide additional information and signals through manuals or documentation

Discoverability reduces the time to a user’s first AHA moment

03. TLD;R

Complexity is not going anywhere “How does one package and share large language models? — Real question

As someone building the tools • As a maintainer (or developer) a lot of our work is about designing pathways for success (users, contributors, the project itself) • We will have to make trade offs • We will have to say no

As someone building the tools • You need to meet people where they are • Choose your poison: maintain, enhance or extend, do not support, sunset • Listen to and adapt to feedback

Intuitive Flexible yet robust Simple and consistent Empowering Aim to build tools and software that are

Intuitive Flexible yet robust Simple and consistent Empowering Aim to adopt governance that is

“ Abandon a standard when it is demonstrably harmful to productivity or user satisfaction. - Jef Raskin, The Humane Interface

Why this matters User Your tool Success Advocate Sponsor Contributor Maintainer

We are deciding on pathways for success for our contributors

We are deciding on pathways for success for our stakeholders

We are deciding on pathways for success for our project

Thank you @ixek @trallard.chaos.social