my talk is Modernizing development workflow for a 7-year old 74K LoC Python project using Pantsbuild. My presentation will be in English. The presentation materials are in English. I will publish the presentation materials. I agree to having my picture taken during my presentation. I will comply with the PyCon JP Code of Conduct.
Inc. since 2015 Has been developing Backend.AI for 7+ years An open-source enthusiast Domain: backend engineering, systems programming, distributed & accelerated computing Interests: writing codes manageable 3 years later • Recent status (updating my codes written 5 years ago...) • Talked in PyCon KR for 8 years in serial... Mostly about asyncio-related topics 🤯 😵💫
problem • Problems with the prior art in my team • Short introduction to Pantsbuild • Mono-repo migration process using Pantsbuild • Customization to adapt with our cases • Experience after migration • Recap
SW engineering is all about keeping the pace of upgradability. The key of upgradability is dependency management. Backend.AI is passing this point (7-years old, ~74K LoC) ref) https://abseil.io/resources/swe-book/html/ch01.html#time_and_change 😵
cargo, poetry, pipenv, go get, ... Diamond ( ) dependency conflicts ✓ Sandboxing all n-th order dependencies vs. full resolution (NP-complete[1]) • Two axes of dependency management Internal: Cross-dependency between components written by us External: Depedency to components written by others Most existing build systems take care of external dependencies only! • Managing internal dependencies Multi-repo: multiple per-component repositories Mono-repo: single merged repository of all components [1] https://research.swtch.com/version-sat
answer! How do your component teams collaborate? ✓ Mono-repo may be better if a single team develops multiple components. How synchronous are the release cycles of related components? How closely coupled are the components? (e.g., direct type refs, unversioned APIs) ✗ Scalability issues if the repo becomes very large ✗ Difficult to fork individual components ✗ Difficult to set up CI/CD workflows ✓ Less pain for refactoring across components ✓ Sharing the same development culture & process ✓ Easier onboarding with a unified view of systems ✓ Independent release cycle & versioning ✓ Per-repository team access control ✓ Taking advantage of existing build systems ✗ Team fragmentation by repo boundaries ✗ Sync overheads of internal dependencies ✗ Difficult to have a holistic view Multi-Repo Mono-Repo ref) https://kinsta.com/blog/monorepo-vs-multi-repo/
mgmt. Easy to find duplications and refactor all API usage occurrences at once e.g., "One version rule" • Mono-repo at large scale needs a "modern" build system. Support both: a single unified build vs. per-component builds Reproducible builds Minimize human errors & mistakes ✓ Declarative dependency configurations ✓ Automatic dependency resolution & inference Speed up builds and CI/CD pipelines ✓ Detecting the affected modules for a changeset in CI workflows ✓ Parallelized & distributed execution with artifact caching
one repository Release each package independently using the standard setuptools/pip toolchain • Setting up Backend.AI[1] development env. The minimum set of components for the server-side (6 repositories) ✓ manager, agent, common, client-py, webserver, storage-proxy ✓ (There is yet another long story for the frontend...) A single-line installation script (install-dev.sh) ✓ Installs database containers using docker-compose ("halfstack") ✓ Clones multiple repositories, creates venvs, runs "editable-install" in each venv, and populates database schema with fixtures [1] https://github.com/lablup/backend.ai
review multiple PRs for a single issue A single issue often consists of multiple PRs to multiple repositories Often we forget to switch git branches same for multi-repo clones ✓ There is an implicit rule to match the PR branch names in different repos, and new contributors often forget this, breaking CI/CD. Difficult to keep our mind contexts when switching repositories ✓ e.g., Forgeting to add corresponding client function for a new server API ✓ Often reviewers forget things as well. 🤯 Not very compatible with GitHub ✓ The issue resolution from multiple linked PRs: "OR" instead of "AND" ✓ GitHub Codespace works for a single repository only. ✓ GitHub Project v2 is still missing cross-repo label, milestone configurations.
Reducing the maintenance points > Splitting components by clear purpose & semantics No way to specify explicit internal dependencies • Time-consuming release process Painful to repeat the same release workflow for 6 repositories... (error-prone) ✓ Need to repeat updating CI/CD configs for 6 times... (reduced motivation to improve) Waiting for dependee packages to get released when there are internal dependencies • Difficult to keep track of compatible set of component version combinations Often a minor patch release makes it incompatible because different components are closed coupled. Causing headaches for on-site engineering staffs when upgrading and applying custom patches for individual customer sites
& improve dev process (cynicism) Too high context switching overheads for managing issues & PRs • Let's migrate to (semi-)mono-repo! Backend.AI is not yet as big as Google's repository — don't need to worry about the extreme scalability issues. Target repos: open-source core components that shares the same release cycle and has internal cross-dependencies • Challenges How to automate internal dependency management? (e.g., parsing/generating setup.cfg?) How to run tests against only changed modules on commits? (e.g., git sparse checkout?) We need a modern build system tailored for mono-repo!
static analysis First-class support for the Python ecosystem Graph-based parallel & async task execution Extensible with a plugin subsystem • Overview https://www.pantsbuild.org/docs/how-does-pants-work https://blog.pantsbuild.org/pycon-us-2022-talk/ https://blog.pantsbuild.org/pants-vs-bazel/ Pants 2 is a fast, scalable, user-friendly build system for codebases of all sizes. It's currently focused on Python, Go, Java, Scala, Shell, and Docker, with support for other languages and frameworks coming soon. ref) https://www.pantsbuild.org/
script (download from https://static.pantsbuild.org/setup/pants) pants.toml & pants.ci.toml **/BUILD files ✓ What to build (including source & resource files), what they depends on others • ./pants [global-options] {goal} [goal-options] [targets] What it does: ✓ Self-bootstrap Pants itself at ~/.cache/pants/ & ./.pants.d ✓ Generate a task DAG from BUILD files ✓ Asynchronously run the DAG with parallelization when possible Refer our team's cheatsheet how it works with daily development workflows[2] [1] https://www.pantsbuild.org/docs/installation [2] https://docs.backend.ai/en/latest/dev/daily-workflows.html
Merged: May 31 (168 commits) / many follow-up PRs afterwards ✓ The initial plan was two weeks, but as always... 😅 More than 60 times of Q&A in the Pantsbuild community Slack Pants: 5 bug reports (all fixed now), 2 feature requests, 2 doc patches Pex: 3 bug reports triggering new releases ✓ Afternoon KST: bug report / Dinner KST: talk with developers / Night-morning KST: developers fix the issue and release / Next morning KST: apply the release • The size of mono-repo Backend.AI Core LoC: 74K+ LoC including all external dependencies: 1.5M+
Single-source the version number from the root's VERSION file Change long_description_type depending on the extension of README (.md, .rst) Change trove classifer depending on the version number suffix (a, b, rc) Add the license type argument so that each wheel package may have different licenses
different resource files (pre-built executables) for Backend.AI Agent by the target platform argument Needed to rewrite the code upon Pants minor version updates as dependency management implemention in Pants has frequent updates (expecting to be stabilized soon)
& parses BUILD files using AST Backend.AI largely depends on entry_points of the package metadata for plugin and replacible module discoveries. https://github.com/lablup/backend.ai/blob/main/src/ai/backend/plugin/entrypoint.py • Use ./pants export :: and ./py wrapper script instead of ./pants run ... In PEX envs, there is neither BUILD files nor the package metadata.
release (hours → ~10 min.) Automated release-related workflows e.g., Generate GitHub's release note by extracting the latest section of CHANGELOG.md • Reduced code review burdens & context switching overheads One issue completes with one PR! (single file tree & single diff) Review all things together, including documentation Now we can utilize GitHub better: Projects v2 & Codespace • Minimized the impact to CI/CD execution times by taking diffs ./pants test --changed-since=main
Pants requires Python 3.9 on Apple Silicon Macs / Backend.AI requires Python 3.10 macOS Monterey (via XCode CLI tools) provides Python 3.8 by default Depending on when & how you have used Homebrew or pyenv, Python 3.9 may be missing! ✓ Fastest workaround: brew install [email protected] or pyenv local 3.9.13 For new contributors with less experience on managing multiple Python versions, this becomes a huge hurdle!
PYTHONPATH environment variable Subprocesses should also take pants.toml's source_roots into account • ImportError due to dynamic imports importlib.import_module() SQLAlchemy selects which engine module to import based on the database URL Such dependencies should be manually specified in BUILD files.
the test suite Port number conflicts of database containers created as a test fixture ✓ pants.toml: [pytest].execution_slot_var = "BACKEND_TEST_EXEC_SLOT" ✓ Use emphemeral port numbers and/or add the slot number to a fixed constant • Ubuntu 22.04 + Snap + Docker + /tmp Snap enforces Docker to use a private /tmp instead of the host /tmp. Mounting a /tmp sub-directory to fixture containers → unexpected failure It is not Pants-own problem, but test parallelization would lead to this pitfall in many cases. ✓ Workaround by using ./.tmp instead of /tmp
of development process after migration, thanks to Pants! ✓ Introduced black + isort + git hook, automation of release note generation Opened many ways to exploit new features of GitHub ✓ Larger action runner to speed up CI, Projects v2, and Codespace • New concerns: unifying tracking of public & private issues • Friendly technical support from the Pantsbuild community Slack The core members and contributors are welcome to questions. I'm trying to contribute back as well! (bug reports, PyCon talks, etc.) • Pants is a highly recommended option if you consider Python-based mono-repo!
platform to develop and operate AI services" • It is an open-source project with enterprise plugins. https://github.com/lablup/backend.ai Contributed to & created many open-source libraries to support this project ✓ aiodocker, aiohttp, aiomonitor-ng, aiotools, aiotusclient, async-timeout, callosum (async RPC), click, etcetra (async etcd3 client), janus, pyzmq, ...
achimnol
j5ik2o
har1101
ternbusty
mizchi
kn_to_maxpno
jinwatanabe
seosoft
nrslib
javiergs
andpad
yusukebe
hirokunimaeta
thoeni
eileencodes
chriscoyier
techseoconnect
jonoalderson
lindahogenes
lara
cassininazir
.png){kind=avatar}
helenjbeal
lauravandoore
codingconduct
![Pants: Basic Usage • Requirements to start using Pants[1] ./pants](https://files.speakerdeck.com/presentations/15c003d21cfd49e099301d4f8b0052f9/slide_18.jpg){kind=link}
