RPC Library for Distributed Processing Used for Machine Learning

Teppei Daito
LINE / Machine Learning Infrastructure / Manager

https://linedevday.linecorp.com/2021/ja/sessions/70
https://linedevday.linecorp.com/2021/en/sessions/70
https://linedevday.linecorp.com/2021/ko/sessions/70

Transcript

2. Machine Learning Job with Kubernetes - Machine learning jobs run on multiple nodes that communicate with each other. - CPU nodes fetch data from storage. - After preprocessing, sends it to the GPU node. - Pods can have access credentials for the storage using Secret. - Resource limits (CPU, Mem, GPU). - Collect and read logs from each Pod. $16
   /PEF $16
   /PEF (16
   /PEF 1PE 5SBJO
   1PE 1PE 1PE 1PE data storage (16
   /PEF 5SBJO
   1PE
3. Machine Learning Job with Kubernetes - Machine learning jobs run on multiple nodes that communicate with each other. - CPU nodes fetch data from storage. - After preprocessing, sends it to the GPU node. - Pods can have access credentials for the storage using Secret. - Resource limits (CPU, Mem, GPU). - Collect and read logs from each Pod. $16
   /PEF $16
   /PEF (16
   /PEF 1PE 5SBJO
   1PE 1PE 1PE 1PE data storage (16
   /PEF 5SBJO
   1PE
4. apiVersion: batch/v1 kind: Job metadata: generateName: hello- spec: backoffLimit: 0 ttlSecondsAfterFinished: 10 template: spec: restartPolicy: Never containers: - name: job image: bash args: - echo - Hello, world! resources: limits: cpu: 200m memory: 200Mi
5. apiVersion: batch/v1 kind: Job metadata: generateName: hello- spec: backoffLimit: 0 ttlSecondsAfterFinished: 10 template: spec: restartPolicy: Never containers: - name: job image: bash args: - echo - Hello, world! resources: limits: cpu: 200m memory: 200Mi Add suffixes to make it re-runnable
6. apiVersion: batch/v1 kind: Job metadata: generateName: hello- spec: backoffLimit: 0 ttlSecondsAfterFinished: 10 template: spec: restartPolicy: Never containers: - name: job image: bash args: - echo - Hello, world! resources: limits: cpu: 200m memory: 200Mi Set for all jobs to avoid retry
7. apiVersion: batch/v1 kind: Job metadata: generateName: hello- spec: backoffLimit: 0 ttlSecondsAfterFinished: 10 template: spec: restartPolicy: Never containers: - name: job image: bash args: - echo - Hello, world! resources: limits: cpu: 200m memory: 200Mi cleaned up TTL seconds after the resource has finished
8. apiVersion: batch/v1 kind: Job metadata: generateName: hello- spec: backoffLimit: 0 ttlSecondsAfterFinished: 10 template: spec: restartPolicy: Never containers: - name: job image: bash args: - echo - Hello, world! resources: limits: cpu: 200m memory: 200Mi Image name and command arguments
9. apiVersion: batch/v1 kind: Job metadata: generateName: hello- spec: backoffLimit: 0 ttlSecondsAfterFinished: 10 template: spec: restartPolicy: Never containers: - name: job image: bash args: - echo - Hello, world! resources: limits: cpu: 200m memory: 200Mi CPU and memory usage can be limited
10. apiVersion: batch/v1 kind: Job metadata: generateName: hello- spec: backoffLimit: 0 ttlSecondsAfterFinished: 10 template: spec: restartPolicy: Never containers: - name: job image: bash args: - echo - Hello, world! resources: limits: cpu: 200m memory: 200Mi kubectl create -f example.yaml kubectl logs -f -l job-name=hello-xxxxx
11. apiVersion: batch/v1 kind: Job metadata: generateName: ${job_prefix}- spec: backoffLimit: 0 template: spec: restartPolicy: Never nodeSelector: example/gpu: "v100" tolerations: - key: nvidia.com/gpu effect: NoSchedule operator: Exists volumes: - name: dshm emptyDir: medium: Memory containers: - name: job image: ${IMAGE} args: [bash, -c, "${args}"] env: - name: ACCESS_ID valueFrom: secretKeyRef: name: my-storage key: access_id - name: SECRET_KEY valueFrom: secretKeyRef: name: my-storage key: secret_key resources: limits: nvidia.com/gpu: 1 cpu: 30 memory: 100Gi volumeMounts: - mountPath: /dev/shm name: dshm set -eux export template_yaml=$1 export job_prefix=$2 echo IMAGE: $IMAGE if [ "$#" -gt 3 ]; then shift 2 export args="$@" fi basedir=$(dirname "$0") jobyaml=$job_prefix.yaml tmp_job=$(mktemp) envsubst < $basedir/$template_yaml > $jobyaml cat $jobyaml trap 'kubectl delete job $job_name; rm -f $tmp_job' EXIT kubectl create -f $jobyaml -o json > $tmp_job namespace=$(jq -r .metadata.namespace $tmp_job) job_name=$(jq -r .metadata.name $tmp_job) for ((k = 0; k < 20; ++k)); do # wait until pod is ready if kubectl wait --for=condition=ready pod -l job-name=$job_name --timeout=10s -n ${namespace}; then kubectl logs -f -n ${namespace} -l job-name=$job_name break else # check init status initStatus=$(kubectl get po -o go-template='{{range .items}}{{range .status.containerStatuses}}\ {{if .state.terminated}}{{"Exit"}}{{end}}{{end}}{{"\n"}}{{end}}' -n ${namespace} -l job-name=${job_name}) if [ X"${initStatus}" != X ]; then break fi fi done errorStatus=$(kubectl get po -o go-template='{{range .items}}{{range .status.containerStatuses}}\ {{.state.terminated.exitCode}}{{end}}{{end}}' -n ${namespace} -l job-name=${job_name} | sed 's/0\.//g') if [ -n "${errorStatus}" ]; then echo "${errorStatus}" exit 1 fi
12. apiVersion: batch/v1 kind: Job metadata: generateName: ${job_prefix}- spec: backoffLimit: 0 template: spec: restartPolicy: Never nodeSelector: example/gpu: "v100" tolerations: - key: nvidia.com/gpu effect: NoSchedule operator: Exists volumes: - name: dshm emptyDir: medium: Memory containers: - name: job image: ${IMAGE} args: [bash, -c, "${args}"] env: - name: ACCESS_ID valueFrom: secretKeyRef: name: my-storage key: access_id - name: SECRET_KEY valueFrom: secretKeyRef: name: my-storage key: secret_key resources: limits: nvidia.com/gpu: 1 cpu: 30 memory: 100Gi volumeMounts: - mountPath: /dev/shm name: dshm set -eux export template_yaml=$1 export job_prefix=$2 echo IMAGE: $IMAGE if [ "$#" -gt 3 ]; then shift 2 export args="$@" fi basedir=$(dirname "$0") jobyaml=$job_prefix.yaml tmp_job=$(mktemp) envsubst < $basedir/$template_yaml > $jobyaml cat $jobyaml trap 'kubectl delete job $job_name; rm -f $tmp_job' EXIT kubectl create -f $jobyaml -o json > $tmp_job namespace=$(jq -r .metadata.namespace $tmp_job) job_name=$(jq -r .metadata.name $tmp_job) for ((k = 0; k < 20; ++k)); do # wait until pod is ready if kubectl wait --for=condition=ready pod -l job-name=$job_name --timeout=10s -n ${namespace}; then kubectl logs -f -n ${namespace} -l job-name=$job_name break else # check init status initStatus=$(kubectl get po -o go-template='{{range .items}}{{range .status.containerStatuses}}\ {{if .state.terminated}}{{"Exit"}}{{end}}{{end}}{{"\n"}}{{end}}' -n ${namespace} -l job-name=${job_name}) if [ X"${initStatus}" != X ]; then break fi fi done errorStatus=$(kubectl get po -o go-template='{{range .items}}{{range .status.containerStatuses}}\ {{.state.terminated.exitCode}}{{end}}{{end}}' -n ${namespace} -l job-name=${job_name} | sed 's/0\.//g') if [ -n "${errorStatus}" ]; then echo "${errorStatus}" exit 1 fi An image and args can be replaced
13. apiVersion: batch/v1 kind: Job metadata: generateName: ${job_prefix}- spec: backoffLimit: 0 template: spec: restartPolicy: Never nodeSelector: example/gpu: "v100" tolerations: - key: nvidia.com/gpu effect: NoSchedule operator: Exists volumes: - name: dshm emptyDir: medium: Memory containers: - name: job image: ${IMAGE} args: [bash, -c, "${args}"] env: - name: ACCESS_ID valueFrom: secretKeyRef: name: my-storage key: access_id - name: SECRET_KEY valueFrom: secretKeyRef: name: my-storage key: secret_key resources: limits: nvidia.com/gpu: 1 cpu: 30 memory: 100Gi volumeMounts: - mountPath: /dev/shm name: dshm set -eux export template_yaml=$1 export job_prefix=$2 echo IMAGE: $IMAGE if [ "$#" -gt 3 ]; then shift 2 export args="$@" fi basedir=$(dirname "$0") jobyaml=$job_prefix.yaml tmp_job=$(mktemp) envsubst < $basedir/$template_yaml > $jobyaml cat $jobyaml trap 'kubectl delete job $job_name; rm -f $tmp_job' EXIT kubectl create -f $jobyaml -o json > $tmp_job namespace=$(jq -r .metadata.namespace $tmp_job) job_name=$(jq -r .metadata.name $tmp_job) for ((k = 0; k < 20; ++k)); do # wait until pod is ready if kubectl wait --for=condition=ready pod -l job-name=$job_name --timeout=10s -n ${namespace}; then kubectl logs -f -n ${namespace} -l job-name=$job_name break else # check init status initStatus=$(kubectl get po -o go-template='{{range .items}}{{range .status.containerStatuses}}\ {{if .state.terminated}}{{"Exit"}}{{end}}{{end}}{{"\n"}}{{end}}' -n ${namespace} -l job-name=${job_name}) if [ X"${initStatus}" != X ]; then break fi fi done errorStatus=$(kubectl get po -o go-template='{{range .items}}{{range .status.containerStatuses}}\ {{.state.terminated.exitCode}}{{end}}{{end}}' -n ${namespace} -l job-name=${job_name} | sed 's/0\.//g') if [ -n "${errorStatus}" ]; then echo "${errorStatus}" exit 1 fi common configurations for GPU
14. apiVersion: batch/v1 kind: Job metadata: generateName: ${job_prefix}- spec: backoffLimit: 0 template: spec: restartPolicy: Never nodeSelector: example/gpu: "v100" tolerations: - key: nvidia.com/gpu effect: NoSchedule operator: Exists volumes: - name: dshm emptyDir: medium: Memory containers: - name: job image: ${IMAGE} args: [bash, -c, "${args}"] env: - name: ACCESS_ID valueFrom: secretKeyRef: name: my-storage key: access_id - name: SECRET_KEY valueFrom: secretKeyRef: name: my-storage key: secret_key resources: limits: nvidia.com/gpu: 1 cpu: 30 memory: 100Gi volumeMounts: - mountPath: /dev/shm name: dshm set -eux export template_yaml=$1 export job_prefix=$2 echo IMAGE: $IMAGE if [ "$#" -gt 3 ]; then shift 2 export args="$@" fi basedir=$(dirname "$0") jobyaml=$job_prefix.yaml tmp_job=$(mktemp) envsubst < $basedir/$template_yaml > $jobyaml cat $jobyaml trap 'kubectl delete job $job_name; rm -f $tmp_job' EXIT kubectl create -f $jobyaml -o json > $tmp_job namespace=$(jq -r .metadata.namespace $tmp_job) job_name=$(jq -r .metadata.name $tmp_job) for ((k = 0; k < 20; ++k)); do # wait until pod is ready if kubectl wait --for=condition=ready pod -l job-name=$job_name --timeout=10s -n ${namespace}; then kubectl logs -f -n ${namespace} -l job-name=$job_name break else # check init status initStatus=$(kubectl get po -o go-template='{{range .items}}{{range .status.containerStatuses}}\ {{if .state.terminated}}{{"Exit"}}{{end}}{{end}}{{"\n"}}{{end}}' -n ${namespace} -l job-name=${job_name}) if [ X"${initStatus}" != X ]; then break fi fi done errorStatus=$(kubectl get po -o go-template='{{range .items}}{{range .status.containerStatuses}}\ {{.state.terminated.exitCode}}{{end}}{{end}}' -n ${namespace} -l job-name=${job_name} | sed 's/0\.//g') if [ -n "${errorStatus}" ]; then echo "${errorStatus}" exit 1 fi For ObjectStorage
15. apiVersion: batch/v1 kind: Job metadata: generateName: ${job_prefix}- spec: backoffLimit: 0 template: spec: restartPolicy: Never nodeSelector: example/gpu: "v100" tolerations: - key: nvidia.com/gpu effect: NoSchedule operator: Exists volumes: - name: dshm emptyDir: medium: Memory containers: - name: job image: ${IMAGE} args: [bash, -c, "${args}"] env: - name: ACCESS_ID valueFrom: secretKeyRef: name: my-storage key: access_id - name: SECRET_KEY valueFrom: secretKeyRef: name: my-storage key: secret_key resources: limits: nvidia.com/gpu: 1 cpu: 30 memory: 100Gi volumeMounts: - mountPath: /dev/shm name: dshm set -eux export template_yaml=$1 export job_prefix=$2 echo IMAGE: $IMAGE if [ "$#" -gt 3 ]; then shift 2 export args="$@" fi basedir=$(dirname "$0") jobyaml=$job_prefix.yaml tmp_job=$(mktemp) envsubst < $basedir/$template_yaml > $jobyaml cat $jobyaml trap 'kubectl delete job $job_name; rm -f $tmp_job' EXIT kubectl create -f $jobyaml -o json > $tmp_job namespace=$(jq -r .metadata.namespace $tmp_job) job_name=$(jq -r .metadata.name $tmp_job) for ((k = 0; k < 20; ++k)); do # wait until pod is ready if kubectl wait --for=condition=ready pod -l job-name=$job_name --timeout=10s -n ${namespace}; then kubectl logs -f -n ${namespace} -l job-name=$job_name break else # check init status initStatus=$(kubectl get po -o go-template='{{range .items}}{{range .status.containerStatuses}}\ {{if .state.terminated}}{{"Exit"}}{{end}}{{end}}{{"\n"}}{{end}}' -n ${namespace} -l job-name=${job_name}) if [ X"${initStatus}" != X ]; then break fi fi done errorStatus=$(kubectl get po -o go-template='{{range .items}}{{range .status.containerStatuses}}\ {{.state.terminated.exitCode}}{{end}}{{end}}' -n ${namespace} -l job-name=${job_name} | sed 's/0\.//g') if [ -n "${errorStatus}" ]; then echo "${errorStatus}" exit 1 fi "logs" will fail if called before the pod is started. "wait" waits until the pod goes running. It means that if the pod fails immediately after created, It will wait forever. 1SFQBSJOH 'JOJTIFE 3VOOJOH logs wait
16. apiVersion: batch/v1 kind: Job metadata: generateName: ${job_prefix}- spec: backoffLimit: 0 template: spec: restartPolicy: Never nodeSelector: example/gpu: "v100" tolerations: - key: nvidia.com/gpu effect: NoSchedule operator: Exists volumes: - name: dshm emptyDir: medium: Memory containers: - name: job image: ${IMAGE} args: [bash, -c, "${args}"] env: - name: ACCESS_ID valueFrom: secretKeyRef: name: my-storage key: access_id - name: SECRET_KEY valueFrom: secretKeyRef: name: my-storage key: secret_key resources: limits: nvidia.com/gpu: 1 cpu: 30 memory: 100Gi volumeMounts: - mountPath: /dev/shm name: dshm set -eux export template_yaml=$1 export job_prefix=$2 echo IMAGE: $IMAGE if [ "$#" -gt 3 ]; then shift 2 export args="$@" fi basedir=$(dirname "$0") jobyaml=$job_prefix.yaml tmp_job=$(mktemp) envsubst < $basedir/$template_yaml > $jobyaml cat $jobyaml trap 'kubectl delete job $job_name; rm -f $tmp_job' EXIT kubectl create -f $jobyaml -o json > $tmp_job namespace=$(jq -r .metadata.namespace $tmp_job) job_name=$(jq -r .metadata.name $tmp_job) for ((k = 0; k < 20; ++k)); do # wait until pod is ready if kubectl wait --for=condition=ready pod -l job-name=$job_name --timeout=10s -n ${namespace}; then kubectl logs -f -n ${namespace} -l job-name=$job_name break else # check init status initStatus=$(kubectl get po -o go-template='{{range .items}}{{range .status.containerStatuses}}\ {{if .state.terminated}}{{"Exit"}}{{end}}{{end}}{{"\n"}}{{end}}' -n ${namespace} -l job-name=${job_name}) if [ X"${initStatus}" != X ]; then break fi fi done errorStatus=$(kubectl get po -o go-template='{{range .items}}{{range .status.containerStatuses}}\ {{.state.terminated.exitCode}}{{end}}{{end}}' -n ${namespace} -l job-name=${job_name} | sed 's/0\.//g') if [ -n "${errorStatus}" ]; then echo "${errorStatus}" exit 1 fi "logs" will fail if called before the pod is started. "wait" waits until the pod goes running. It means that if the pod fails immediately after created, It will wait forever. 1SFQBSJOH 'BJMFE wait forever
17. apiVersion: batch/v1 kind: Job metadata: generateName: ${job_prefix}- spec: backoffLimit: 0 template: spec: restartPolicy: Never nodeSelector: example/gpu: "v100" tolerations: - key: nvidia.com/gpu effect: NoSchedule operator: Exists volumes: - name: dshm emptyDir: medium: Memory containers: - name: job image: ${IMAGE} args: [bash, -c, "${args}"] env: - name: ACCESS_ID valueFrom: secretKeyRef: name: my-storage key: access_id - name: SECRET_KEY valueFrom: secretKeyRef: name: my-storage key: secret_key resources: limits: nvidia.com/gpu: 1 cpu: 30 memory: 100Gi volumeMounts: - mountPath: /dev/shm name: dshm set -eux export template_yaml=$1 export job_prefix=$2 echo IMAGE: $IMAGE if [ "$#" -gt 3 ]; then shift 2 export args="$@" fi basedir=$(dirname "$0") jobyaml=$job_prefix.yaml tmp_job=$(mktemp) envsubst < $basedir/$template_yaml > $jobyaml cat $jobyaml trap 'kubectl delete job $job_name; rm -f $tmp_job' EXIT kubectl create -f $jobyaml -o json > $tmp_job namespace=$(jq -r .metadata.namespace $tmp_job) job_name=$(jq -r .metadata.name $tmp_job) for ((k = 0; k < 20; ++k)); do # wait until pod is ready if kubectl wait --for=condition=ready pod -l job-name=$job_name --timeout=10s -n ${namespace}; then kubectl logs -f -n ${namespace} -l job-name=$job_name break else # check init status initStatus=$(kubectl get po -o go-template='{{range .items}}{{range .status.containerStatuses}}\ {{if .state.terminated}}{{"Exit"}}{{end}}{{end}}{{"\n"}}{{end}}' -n ${namespace} -l job-name=${job_name}) if [ X"${initStatus}" != X ]; then break fi fi done errorStatus=$(kubectl get po -o go-template='{{range .items}}{{range .status.containerStatuses}}\ {{.state.terminated.exitCode}}{{end}}{{end}}' -n ${namespace} -l job-name=${job_name} | sed 's/0\.//g') if [ -n "${errorStatus}" ]; then echo "${errorStatus}" exit 1 fi "logs" will fail if called before the pod is started. "wait" waits until the pod goes running. It means that if the pod fails immediately after created, It will wait forever. 1SFQBSJOH 'JOJTIFE 3VOOJOH logs failed
18. apiVersion: batch/v1 kind: Job metadata: generateName: ${job_prefix}- spec: backoffLimit: 0 template: spec: restartPolicy: Never nodeSelector: example/gpu: "v100" tolerations: - key: nvidia.com/gpu effect: NoSchedule operator: Exists volumes: - name: dshm emptyDir: medium: Memory containers: - name: job image: ${IMAGE} args: [bash, -c, "${args}"] env: - name: ACCESS_ID valueFrom: secretKeyRef: name: my-storage key: access_id - name: SECRET_KEY valueFrom: secretKeyRef: name: my-storage key: secret_key resources: limits: nvidia.com/gpu: 1 cpu: 30 memory: 100Gi volumeMounts: - mountPath: /dev/shm name: dshm set -eux export template_yaml=$1 export job_prefix=$2 echo IMAGE: $IMAGE if [ "$#" -gt 3 ]; then shift 2 export args="$@" fi basedir=$(dirname "$0") jobyaml=$job_prefix.yaml tmp_job=$(mktemp) envsubst < $basedir/$template_yaml > $jobyaml cat $jobyaml trap 'kubectl delete job $job_name; rm -f $tmp_job' EXIT kubectl create -f $jobyaml -o json > $tmp_job namespace=$(jq -r .metadata.namespace $tmp_job) job_name=$(jq -r .metadata.name $tmp_job) for ((k = 0; k < 20; ++k)); do # wait until pod is ready if kubectl wait --for=condition=ready pod -l job-name=$job_name --timeout=10s -n ${namespace}; then kubectl logs -f -n ${namespace} -l job-name=$job_name break else # check init status initStatus=$(kubectl get po -o go-template='{{range .items}}{{range .status.containerStatuses}}\ {{if .state.terminated}}{{"Exit"}}{{end}}{{end}}{{"\n"}}{{end}}' -n ${namespace} -l job-name=${job_name}) if [ X"${initStatus}" != X ]; then break fi fi done errorStatus=$(kubectl get po -o go-template='{{range .items}}{{range .status.containerStatuses}}\ {{.state.terminated.exitCode}}{{end}}{{end}}' -n ${namespace} -l job-name=${job_name} | sed 's/0\.//g') if [ -n "${errorStatus}" ]; then echo "${errorStatus}" exit 1 fi Depending on the kubectl version, the exitCode can be 0 or 0.0.
19. swimmy.cmd python -m swimmy.cmd --name example-predict --image example.com/predictor python run-predict.py --all - No need to Deploy anything before - Execute command on the image - Print pod logs and events -
20. swimmy.cmd python -m swimmy.cmd --name example-predict --image example.com/predictor --cpu 4000m --mem 4Gi python run-predict.py --all - No need to Deploy anything before - Execute command on the image - Print pod logs and events - Resource limits -
21. swimmy.cmd python -m swimmy.cmd --name example-predict --image example.com/predictor --cpu 4000m --mem 4Gi -e KEY_SECRET -e STAGE=dev python run-predict.py --all - No need to Deploy anything before - Execute command on the image - Print pod logs and events - Resource limits - Environment variable -
22. swimmy.cmd python -m swimmy.cmd --name example-predict --image example.com/predictor --cpu 4000m --mem 4Gi -e KEY_SECRET -e STAGE=dev --template object-storage.yaml python run-predict.py --all - No need to Deploy anything before - Execute command on the image - Print pod logs and events - Resource limits - Environment variable - Pod template system - containers: - env: - name: ACCESS_ID valueFrom: secretKeyRef: name: my-storage key: access_id - name: SECRET_KEY valueFrom: secretKeyRef: name: my-storage key: secret_key
23. swimmy.cmd python -m swimmy.cmd --name example-predict --image example.com/predictor --cpu 4000m --mem 4Gi -e KEY_SECRET -e STAGE=dev --template object-storage.yaml --template gpu-v100.yaml python run-predict.py --all - No need to Deploy anything before - Execute command on the image - Print pod logs and events - Resource limits - Environment variable - Pod template system - nodeSelector: example/gpu: "v100" tolerations: - key: nvidia.com/gpu effect: NoSchedule operator: Exists containers: - resources: limits: nvidia.com/gpu: 1 volumeMounts: - mountPath: /dev/shm name: dshm
24. swimmy.cmd python -m swimmy.cmd --name example-predict --image example.com/predictor --cpu 4000m --mem 4Gi -e KEY_SECRET -e STAGE=dev --template object-storage --template configmap:mycm/gpu-v100.yaml python run-predict.py --all - No need to Deploy anything before - Execute command on the image - Print pod logs and events - Resource limits - Environment variable - Pod template system - 5FNQMBUF
    MPDBUJPO
    QSJPSJUZ
    
    QSFpY
    XJUI
    DPOpHNBQ
    
    MPDBM
    pMF
    
    SFBE
    BT
    DPOpHNBQTXJNNZUFNQMBUFT\UFNQMBUF^ZBNM
25. swimmy.cmd python -m swimmy.cmd --name example-predict --image example.com/predictor --cpu 4000m --mem 4Gi -e KEY_SECRET -e STAGE=dev --template object-storage.yaml --template gpu-v100.yaml --files run-predict.py python run-predict.py --all - No need to Deploy anything before - Execute command on the image - Print pod logs and events - Resource limits - Environment variable - Pod template system - Send any files -
26. swimmy.cmd python -m swimmy.cmd --name example-predict --image example.com/predictor --cpu 4000m --mem 4Gi -e KEY_SECRET -e STAGE=dev --template object-storage.yaml --template gpu-v100.yaml --files run-predict.py --connect-timeout 120 --ttl-seconds 3600 python run-predict.py --all - No need to Deploy anything before - Execute command on the image - Print pod logs and events - Resource limits - Environment variable - Pod template system - Send any files - TTL and startup time limit
27. swimmy.cmd python -m swimmy.cmd --name example-predict --image example.com/predictor --cpu 4000m --mem 4Gi -e KEY_SECRET -e STAGE=dev --template object-storage.yaml --template gpu-v100.yaml --files run-predict.py --connect-timeout 120 --ttl-seconds 3600 python run-predict.py --all - No need to Deploy anything before - Execute command on the image - Print pod logs and events - Resource limits - Environment variable - Pod template system - Send any files - TTL and startup time limit 4BNF
    WFSTJPO
    PG
    1ZUIPO
    JT
    SFRVJSFE
    JO
    UIF
    JNBHF
28. Hello, Swimmy! import asyncio import logging import swimmy import foo @swimmy.remotefn def hello(name: str) -> str: return foo.bang(f"Hello, {name}") async def main(): async with swimmy.KubeCluster(name="swimmy-test") as cluster: x = await cluster.run( swimmy.PodSpec( image="example.com/py37", cpu="200m", mem="200Mi", count=2, pyfiles=["foo.py"], ) ) print(await x.remote(hello("Swimmy"))) logging.basicConfig(level=logging.INFO) asyncio.run(main())
29. import asyncio import logging import swimmy import foo @swimmy.remotefn def hello(name: str) -> str: return foo.bang(f"Hello, {name}") async def main(): async with swimmy.KubeCluster(name="swimmy-test") as cluster: x = await cluster.run( swimmy.PodSpec( image="example.com/py37", cpu="200m", mem="200Mi", count=2, pyfiles=["foo.py"], ) ) print(await x.remote(hello("Swimmy"))) logging.basicConfig(level=logging.INFO) asyncio.run(main()) %SJWFS *OUFSOBMT Hello, Swimmy! ,VCFSOFUFT
    "1*
    4FSWFS $SFBUF
    +PC kind: Role rules: - apiGroups: - "" resources: - configmaps - pods - pods/status - pods/log - events verbs: - get - list - watch - apiGroups: - batch resources: - jobs verbs: - create - patch
30. import asyncio import logging import swimmy import foo @swimmy.remotefn def hello(name: str) -> str: return foo.bang(f"Hello, {name}") async def main(): async with swimmy.KubeCluster(name="swimmy-test") as cluster: x = await cluster.run( swimmy.PodSpec( image="example.com/py37", cpu="200m", mem="200Mi", count=2, pyfiles=["foo.py"], ) ) print(await x.remote(hello("Swimmy"))) logging.basicConfig(level=logging.INFO) asyncio.run(main()) %SJWFS *OUFSOBMT Hello, Swimmy! ,VCFSOFUFT
    "1*
    4FSWFS $SFBUF
    +PC kind: Role rules: - apiGroups: - "" resources: - configmaps - pods - pods/status - pods/log - events verbs: - get - list - watch - apiGroups: - batch resources: - jobs verbs: - create - patch
31. import asyncio import logging import swimmy import foo @swimmy.remotefn def hello(name: str) -> str: return foo.bang(f"Hello, {name}") async def main(): async with swimmy.KubeCluster(name="swimmy-test") as cluster: x = await cluster.run( swimmy.PodSpec( image="example.com/py37", cpu="200m", mem="200Mi", count=2, pyfiles=["foo.py"], ) ) print(await x.remote(hello("Swimmy"))) logging.basicConfig(level=logging.INFO) asyncio.run(main()) %SJWFS Hello, Swimmy! ,VCFSOFUFT
    "1*
    4FSWFS +PC %SJWFS 1PE *OUFSOBMT
32. import asyncio import logging import swimmy import foo @swimmy.remotefn def hello(name: str) -> str: return foo.bang(f"Hello, {name}") async def main(): async with swimmy.KubeCluster(name="swimmy-test") as cluster: x = await cluster.run( swimmy.PodSpec( image="example.com/py37", cpu="200m", mem="200Mi", count=2, pyfiles=["foo.py"], ) ) print(await x.remote(hello("Swimmy"))) logging.basicConfig(level=logging.INFO) asyncio.run(main()) %SJWFS Hello, Swimmy! ,VCFSOFUFT
    "1*
    4FSWFS +PC %SJWFS "HFOU 1:1* 'FUDI
    4XJNNZ *OUFSOBMT
33. import asyncio import logging import swimmy import foo @swimmy.remotefn def hello(name: str) -> str: return foo.bang(f"Hello, {name}") async def main(): async with swimmy.KubeCluster(name="swimmy-test") as cluster: x = await cluster.run( swimmy.PodSpec( image="example.com/py37", cpu="200m", mem="200Mi", count=2, pyfiles=["foo.py"], ) ) print(await x.remote(hello("Swimmy"))) logging.basicConfig(level=logging.INFO) asyncio.run(main()) %SJWFS Hello, Swimmy! ,VCFSOFUFT
    "1*
    4FSWFS +PC %SJWFS "HFOU 1:1* *OUFSOBMT )BEPPQ ,BGLB %BUB'SBNF 3FEJT /VN1Z ,FFQ
    .JOJNBM
    %FQFOEFODJFT
34. import asyncio import logging import swimmy import foo @swimmy.remotefn def hello(name: str) -> str: return foo.bang(f"Hello, {name}") async def main(): async with swimmy.KubeCluster(name="swimmy-test") as cluster: x = await cluster.run( swimmy.PodSpec( image="example.com/py37", cpu="200m", mem="200Mi", count=2, pyfiles=["foo.py"], ) ) print(await x.remote(hello("Swimmy"))) logging.basicConfig(level=logging.INFO) asyncio.run(main()) %SJWFS Hello, Swimmy! ,VCFSOFUFT
    "1*
    4FSWFS +PC %SJWFS "HFOU 1:1* 0QFO
    ;.2
    $POOFDUJPO *OUFSOBMT
35. import asyncio import logging import swimmy import foo @swimmy.remotefn def hello(name: str) -> str: return foo.bang(f"Hello, {name}") async def main(): async with swimmy.KubeCluster(name="swimmy-test") as cluster: x = await cluster.run( swimmy.PodSpec( image="example.com/py37", cpu="200m", mem="200Mi", count=2, pyfiles=["foo.py"], ) ) print(await x.remote(hello("Swimmy"))) logging.basicConfig(level=logging.INFO) asyncio.run(main()) %SJWFS Hello, Swimmy! ,VCFSOFUFT
    "1*
    4FSWFS +PC %SJWFS "HFOU 1:1* 0QFO
    ;.2
    $POOFDUJPO 8IZ
    ;.2
    'BTU
    4NBMM
    'MFYJCMF )JHITQFFE
    BTZODISPOPVT
    *0
    5JOZ
    TJOHMF
    XIFFM
    'JOF
    HSBJOFE
    qPX
    DPOUSPM *OUFSOBMT
36. import asyncio import logging import swimmy import foo @swimmy.remotefn def hello(name: str) -> str: return foo.bang(f"Hello, {name}") async def main(): async with swimmy.KubeCluster(name="swimmy-test") as cluster: x = await cluster.run( swimmy.PodSpec( image="example.com/py37", cpu="200m", mem="200Mi", count=2, pyfiles=["foo.py"], ) ) print(await x.remote(hello("Swimmy"))) logging.basicConfig(level=logging.INFO) asyncio.run(main()) %SJWFS Hello, Swimmy! ,VCFSOFUFT
    "1*
    4FSWFS +PC %SJWFS "HFOU 1:1* )551
    GPS
    pMFT *OUFSOBMT
37. import asyncio import logging import swimmy import foo @swimmy.remotefn def hello(name: str) -> str: return foo.bang(f"Hello, {name}") async def main(): async with swimmy.KubeCluster(name="swimmy-test") as cluster: x = await cluster.run( swimmy.PodSpec( image="example.com/py37", cpu="200m", mem="200Mi", count=2, pyfiles=["foo.py"], ) ) print(await x.remote(hello("Swimmy"))) logging.basicConfig(level=logging.INFO) asyncio.run(main()) %SJWFS Hello, Swimmy! ,VCFSOFUFT
    "1*
    4FSWFS +PC %SJWFS "HFOU 1:1* )FBMUI
    $IFDL *OUFSOBMT
38. import asyncio import logging import swimmy import foo @swimmy.remotefn def hello(name: str) -> str: return foo.bang(f"Hello, {name}") async def main(): async with swimmy.KubeCluster(name="swimmy-test") as cluster: x = await cluster.run( swimmy.PodSpec( image="example.com/py37", cpu="200m", mem="200Mi", count=2, pyfiles=["foo.py"], ) ) print(await x.remote(hello("Swimmy"))) logging.basicConfig(level=logging.INFO) asyncio.run(main()) %SJWFS Hello, Swimmy! ,VCFSOFUFT
    "1*
    4FSWFS +PC %SJWFS "HFOU 1:1* &YUFOE
    BDUJWF%FBEMJOF4FDPOET *OUFSOBMT kind: Role rules: - apiGroups: - "" resources: - configmaps - pods - pods/status - pods/log - events verbs: - get - list - watch - apiGroups: - batch resources: - jobs verbs: - create - patch
39. import asyncio import logging import swimmy import foo @swimmy.remotefn def hello(name: str) -> str: return foo.bang(f"Hello, {name}") async def main(): async with swimmy.KubeCluster(name="swimmy-test") as cluster: x = await cluster.run( swimmy.PodSpec( image="example.com/py37", cpu="200m", mem="200Mi", count=2, pyfiles=["foo.py"], ) ) print(await x.remote(hello("Swimmy"))) logging.basicConfig(level=logging.INFO) asyncio.run(main()) %SJWFS Hello, Swimmy! ,VCFSOFUFT
    "1*
    4FSWFS +PC %SJWFS "HFOU 1:1* 3FBE
    QPE
    MPHT
    BOE
    FWFOUT *OUFSOBMT kind: Role rules: - apiGroups: - "" resources: - configmaps - pods - pods/status - pods/log - events verbs: - get - list - watch - apiGroups: - batch resources: - jobs verbs: - create - patch
40. import asyncio import logging import swimmy import foo @swimmy.remotefn def hello(name: str) -> str: return foo.bang(f"Hello, {name}") async def main(): async with swimmy.KubeCluster(name="swimmy-test") as cluster: x = await cluster.run( swimmy.PodSpec( image="example.com/py37", cpu="200m", mem="200Mi", count=2, pyfiles=["foo.py"], ) ) print(await x.remote(hello("Swimmy"))) logging.basicConfig(level=logging.INFO) asyncio.run(main()) %SJWFS Hello, Swimmy! ,VCFSOFUFT
    "1*
    4FSWFS +PC %SJWFS "HFOU 1:1* 31$ *OUFSOBMT
41. What is it used for? - Distributed Machine Learning Library - Redis Monitoring - Scalable Load Testing
42. async def mpi_train(train, preprocess): async with swimmy.KubeCluster(name="mpi-train") as cluster: cp = await cluster.run(preprocess.podspec, name='preprocess') ct = await cluster.run(train.podspec, name='train') conn = connection_manager(cp, ct) fp = cp.remote(mpi_preprocess(preprocess.func)) ft = ct.remote(mpi_fit(train.func)) tasks = [asyncio.ensure_future(t) for t in [conn, fp, ft]] await asyncio.gather(*tasks) Ghee $16
    /PEF $16
    /PEF (16
    /PEF 1PE 5SBJO
    1PE 1PE 1PE 1PE HDFS (16
    /PEF 5SBJO
    1PE 1Z"SSPX .1* ;.2
43. Redis Monitoring Redis Redis Redis Redis Redis Redis 4XJNNZ
    %SJWFS $MJFOU "HFOU "HFOU "HFOU 10x performance Scanning 100M keys from the Redis-cluster, it took more than an hour
44. Load Testing Script 5BSHFU
    TFSWFST 4XJNNZ
    %SJWFS XSL 1ZUIPO 4DFOBSJP
    CBTFE
    UFTUJOH
    BOE
    WBMJEBUJPO -PBE
    5FTUJOH
    XJUI
    -VB
45. Thank you! - Python library for Kubernetes Job - Send function and files - Minimal dependencies - Pod template - Health check