機械学習チームのモノレポ移行

Slide 1

Slide 1 text

Koki Nishihara

Slide 2

Slide 2 text

Koki Nishihara (github ) CADDi AI Team, Tech Lead MLOps Complicated-subsystem team OSS pydantic, Pants 10 27 @nishikoh PyCon APAC 2023

Slide 3

Slide 3 text

LT CADDi Tech Blog

Slide 4

Slide 4 text

CI Pull Request

Slide 5

Slide 5 text

CI Pull Request

Slide 6

Slide 6 text

Bazel Pants : 3rd party plugin : Python Python Pants Bazel Pants

Slide 7

Slide 7 text

polyrepo monorepo + dir BUILD polyrepo monorepo app 1 ├── src 2 │ └── main.py 3 ├── Dockerfile 4 ├── poetry.lock 5 ├── pyproject.toml 6 └── tests 7 └── test_main.py 8 pants.toml 1 projects 2 ├── app_1 3 │ ├── src 4 │ │ ├── BUILD 5 │ │ └── main.py 6 │ ├── BUILD 7 │ ├── Dockerfile 8 │ ├── poetry.lock 9 │ ├── pyproject.toml 10 │ └── tests 11 │ ├── BUILD 12 │ └── test_main.py 13 └── app_N 14 ├── ... 15

Slide 8

Slide 8 text

pants.toml [GLOBAL] 1 pants_version = "2.17.0" 2 backend_packages = [ 3 "pants.backend.python", 4 "pants.backend.python.lint.black", 5 "pants.backend.python.lint.flake8", 6 "pants.backend.python.lint.isort", 7 "pants.backend.python.lint.docformatter", 8 "pants.backend.python.lint.bandit", 9 "pants.backend.python.lint.autoflake", 10 "pants.backend.python.lint.pyupgrade", 11 "pants.backend.python.lint.pylint", 12 "pants.backend.experimental.python.lint.ruff", 13 "pants.backend.experimental.python.lint.add_trailing_comma", 14 "pants.backend.python.typecheck.mypy", 15 ] 16 17

Slide 9

Slide 9 text

--changed-since --changed-dependees=transitive CI p50 30 , p75 90 , max 20 format lint test pants fmt --changed-since=origin/main --changed-dependees=transitive 1

Slide 10

Slide 10 text

Python CADDi Tech Blog

Slide 11

Slide 11 text

WebAPI Python

Slide 12

Slide 12 text

3rd party package : 300 : 30 3rd party package 300 packege 5

Slide 13

Slide 13 text

Good CI/CD format 1

Slide 14

Slide 14 text

More must git clone git lfs .git/ GitHub Actions disk git lfs 300 3rd party

Slide 15

Slide 15 text

More Pants model registry git lfs GitHub Actions hosted runner Cloud Build GitHub Actions larger runners