Upgrade to Pro — share decks privately, control downloads, hide ads and more …

機械学習チームのモノレポ移行

 機械学習チームのモノレポ移行

「モノレポへの移行 LT -生産性の高いアーキテクチャに向けた第一歩-」の発表資料

https://findy.connpass.com/event/296339/

[email protected]

October 06, 2023
Tweet

Other Decks in Programming

Transcript

  1. Koki Nishihara

    View full-size slide

  2. Koki Nishihara (github )
    CADDi AI Team, Tech Lead
    MLOps
    Complicated-subsystem team
    OSS
    pydantic, Pants
    10 27
    @nishikoh
    PyCon APAC 2023

    View full-size slide

  3. LT
    CADDi Tech Blog

    View full-size slide

  4. CI
    Pull Request

    View full-size slide

  5. CI
    Pull Request

    View full-size slide

  6. Bazel Pants
    :
    3rd party plugin
    : Python
    Python
    Pants
    Bazel
    Pants

    View full-size slide

  7. polyrepo
    monorepo + dir BUILD
    polyrepo monorepo
    app
    1
    ├── src
    2
    │ └── main.py
    3
    ├── Dockerfile
    4
    ├── poetry.lock
    5
    ├── pyproject.toml
    6
    └── tests
    7
    └── test_main.py
    8
    pants.toml
    1
    projects
    2
    ├── app_1
    3
    │ ├── src
    4
    │ │ ├── BUILD
    5
    │ │ └── main.py
    6
    │ ├── BUILD
    7
    │ ├── Dockerfile
    8
    │ ├── poetry.lock
    9
    │ ├── pyproject.toml
    10
    │ └── tests
    11
    │ ├── BUILD
    12
    │ └── test_main.py
    13
    └── app_N
    14
    ├── ...
    15

    View full-size slide

  8. pants.toml
    [GLOBAL]
    1
    pants_version = "2.17.0"
    2
    backend_packages = [
    3
    "pants.backend.python",
    4
    "pants.backend.python.lint.black",
    5
    "pants.backend.python.lint.flake8",
    6
    "pants.backend.python.lint.isort",
    7
    "pants.backend.python.lint.docformatter",
    8
    "pants.backend.python.lint.bandit",
    9
    "pants.backend.python.lint.autoflake",
    10
    "pants.backend.python.lint.pyupgrade",
    11
    "pants.backend.python.lint.pylint",
    12
    "pants.backend.experimental.python.lint.ruff",
    13
    "pants.backend.experimental.python.lint.add_trailing_comma",
    14
    "pants.backend.python.typecheck.mypy",
    15
    ]
    16
    17

    View full-size slide

  9. --changed-since
    --changed-dependees=transitive
    CI p50 30 , p75 90 , max 20
    format lint test
    pants fmt --changed-since=origin/main --changed-dependees=transitive
    1

    View full-size slide

  10. Python
    CADDi Tech Blog

    View full-size slide

  11. WebAPI
    Python

    View full-size slide

  12. 3rd party package : 300
    : 30
    3rd party package
    300 packege
    5

    View full-size slide

  13. Good
    CI/CD
    format
    1

    View full-size slide

  14. More
    must
    git clone
    git lfs
    .git/
    GitHub Actions disk
    git lfs 300 3rd party

    View full-size slide

  15. More
    Pants
    model registry git lfs
    GitHub Actions hosted runner
    Cloud Build GitHub Actions
    larger runners

    View full-size slide

  16. CADDi Tech Blog

    View full-size slide

  17. Software Engineering at Google
    Build Systems and Build Philosophy

    View full-size slide

  18. PyCon APAC 2023 “Pants
    Python Monorepo” Python, Pants

    View full-size slide

  19. CADDi
    https://recruit.caddi.tech

    View full-size slide