KRSH: 선언형 Kubeflow, Terraform처럼 파이프라인 관리하기(김완수)

Slide 1

Slide 1 text

ML Engineer, AIOps HyperOpt, Maintainer

Slide 2

Slide 2 text

KubeFlow Pipeline Declarative Kubeflow, KRSH

Slide 3

Slide 3 text

KubeFlow Pipeline

Slide 4

Slide 4 text

“ Kubernetes 위에서 Container 기반으로 ML Workflow를 구축하고 배포하기 위한 도구 ”

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

No content

Slide 7

Slide 7 text

조합가능성 이식성 확장성

Slide 8

Slide 8 text

조합가능성 공통 컴포넌트 컴포넌트 컴포넌트 컴포넌트 컴포넌트 PIPE1 PIPE2

Slide 9

Slide 9 text

이식성 Kubernetes KubeFlow

Slide 10

Slide 10 text

Big Storage High Memory High CPU Many GPU Big Storage High Memory High CPU No GPU Big Storage High Memory High CPU No GPU Big Storage High Memory High CPU Many GPU Small Storage Low Memory Low CPU No GPU Big Storage Low Memory Low CPU No GPU Data Ingestion Data Process Data Validate Model Training Model Validate Model Deploy 확장성

Slide 11

Slide 11 text

Container Op Container Op Container Op Container Op Pipeline VolumeOp VolumeOp VolumeOp Artifact User Input

Slide 12

Slide 12 text

@kfp.dsl.pipeline PIPELINE COMPONENT kfp.dsl.BaseOp DOCKER IMAGE OPTIONAL VOLUME OTHER COMPONENTS

Slide 13

Slide 13 text

Pipeline 작성 DSL Compile Pipeline Version 작성 KF UI에서 Upload

Slide 14

Slide 14 text

No content

Slide 15

Slide 15 text

첫번째 문제, 형상관리가 어려움

Slide 16

Slide 16 text

두번째 문제, 업로드 과정이 불편함

Slide 17

Slide 17 text

세번째 문제, CI/CD가 불편함

Slide 18

Slide 18 text

Declarative KubeFlow, KRSH

Slide 19

Slide 19 text

Write Plan Apply

Slide 20

Slide 20 text

“ KRSH는 Kubeflow Pipeline을 선언적으로 관리하며, Pipeline의 개발, 배포 주기를 단축시킬 수 있다 ”

Slide 21

Slide 21 text

No content

Slide 22

Slide 22 text

$ pip install krsh

Slide 23

Slide 23 text

components/ dockerfiles/ pipelines/

Slide 24

Slide 24 text

No content

Slide 25

Slide 25 text

No content

Slide 26

Slide 26 text

No content

Slide 27

Slide 27 text

No content

Slide 28

Slide 28 text

No content

Slide 29

Slide 29 text

No content

Slide 30

Slide 30 text

No content

Slide 31

Slide 31 text

No content

Slide 32

Slide 32 text

첫번째 문제, 형상관리가 어려움 Repository를 기반으로 동작하기 때문에 Git으로 형상관 리를 할 수 있게됨

Slide 33

Slide 33 text

두번째 문제, 업로드 과정이 불편함 Apply를 기반으로 자동으로 Repository에 기반하여 업로드

Slide 34

Slide 34 text

세번째 문제, CI/CD가 불편함 GitHub Action에 KRSH CLI를 통합하여 쉽게 CI/CD 구축 가능

Slide 35

Slide 35 text

github.com/riiid/krsh Release 1.0.0-alpha

Slide 36

Slide 36 text

Declarative Continuous Training Declarative AutoML Schedule Automated Docker Build Roadmap

Slide 1

Slide 1 text

Slide 2

Slide 2 text

Slide 3

Slide 3 text

Slide 4

Slide 4 text

Slide 5

Slide 5 text

Slide 6

Slide 6 text

Slide 7

Slide 7 text

Slide 8

Slide 8 text

Slide 9

Slide 9 text

Slide 10

Slide 10 text

Slide 11

Slide 11 text

Slide 12

Slide 12 text

Slide 13

Slide 13 text

Slide 14

Slide 14 text

Slide 15

Slide 15 text

Slide 16

Slide 16 text

Slide 17

Slide 17 text

Slide 18

Slide 18 text

Slide 19

Slide 19 text

Slide 20

Slide 20 text

Slide 21

Slide 21 text

Slide 22

Slide 22 text

Slide 23

Slide 23 text

Slide 24

Slide 24 text

Slide 25

Slide 25 text

Slide 26

Slide 26 text

Slide 27

Slide 27 text

Slide 28

Slide 28 text

Slide 29

Slide 29 text

Slide 30

Slide 30 text

Slide 31

Slide 31 text

Slide 32

Slide 32 text

Slide 33

Slide 33 text

Slide 34

Slide 34 text

Slide 35

Slide 35 text

Slide 36

Slide 36 text

Slide 37

Slide 37 text

Slide 38

Slide 38 text

Slide 39

Slide 39 text