Slide 1

Slide 1 text

R&D Center Base System Development Department Copyright 2019 Sony Corporation Container Tracer using OCI hooks on Kubernetes Kubernetes Invitational Meetup Tokyo #4 Kenta Tada R&D Center Sony Corporation

Slide 2

Slide 2 text

About me ⚫System Software Engineer, Sony ⚫OSS Contributor • runC • Docker • containerd and so on

Slide 3

Slide 3 text

Agenda ⚫OCI hooks recap ⚫Current status of Kubernetes integration ⚫Case Study • Tracer using OCI hooks for Tracer for ROS (Robot Operating System)

Slide 4

Slide 4 text

OCI hooks recap

Slide 5

Slide 5 text

OCI hooks ⚫OCI hooks are executed by container runtime ⚫You can set up the hooks at • Prestart • Poststart • Poststop ⚫From OCI runtime spec, the state of the container which includes container initial PID must be passed to hooks over stdin ⚫More hooks are currently proposed. • https://github.com/opencontainers/runtime-spec/pull/1008

Slide 6

Slide 6 text

Current status of Kubernetes integration

Slide 7

Slide 7 text

How to set up prestart hook in Kubernetes ⚫Kubernetes Pod Lifecycle and related hook ⚫Kubernetes did not provide prestart hook • https://github.com/kubernetes/kubernetes/issues/140 process lifetime process start process stop prestop poststart

Slide 8

Slide 8 text

How to operate OCI hooks from Kubernetes 1. Set up OCI hooks on high level runtime 2. Control OCI hooks per Pod using Kubernetes annotations

Slide 9

Slide 9 text

Current status of containerd 1. Set up OCI hooks on high level runtime • containerd is currently developing option for setting OCI hook – https://github.com/containerd/cri/pull/1248 – https://github.com/containerd/cri/issues/405 2. Control OCI hooks per Pod using Kubernetes annotations • containerd supported for passing annotations to low level runtime – https://github.com/containerd/cri/pull/1084 ○ ×

Slide 10

Slide 10 text

Current status of CRI-O 1. Set up OCI hooks on high level runtime • CRI-O and Podman have already provided their own solution "oci- hooks“ 2. Control OCI hooks per Pod using Kubernetes annotations • CRI-O is currently developing passing annotations to low level runtime – https://github.com/cri-o/cri-o/issues/2402 ○ ×

Slide 11

Slide 11 text

Case Study

Slide 12

Slide 12 text

Recap : Container Tracer ⚫We are developing an experimental tool to trace system calls inside containers without any debug tools in Pod. • https://speakerdeck.com/kentatada/debug-application-inside- kubernetes-using-linux-kernel-tools • https://github.com/KentaTada/oci-ftrace-syscall-analyzer ⚫ This tool uses OCI hooks to trace system calls from apps startup.

Slide 13

Slide 13 text

What is ROS (Robot Operating System)? ⚫ROS (Robot Operating System) provides libraries and tools to help software developers create robot applications. • http://wiki.ros.org/ ⚫ROS is the middleware. Hardware OS(Ex. Ubuntu) ROS Apps(Ex. path planning)

Slide 14

Slide 14 text

Tracer for ROS containers ⚫We are trying to launch ROS nodes on Kubernetes. • Ex. Using Docker with ROS –http://wiki.ros.org/docker ⚫We are also trying to use our container tracer to trace ROS nodes from apps startup.

Slide 15

Slide 15 text

Case Study : RT apps on the container environment ⚫RT apps on our rootless container failed to execute sched_setscheduler(2) during apps startup.

Slide 16

Slide 16 text

Logs

Slide 17

Slide 17 text

Tracer found two root causes ⚫Container side issue • With systemd, we needed to set up LimitRTPRIO in service file. –https://superuser.com/questions/403184/configuring-systemd-to- allow-daemon-to-set-rt-priority ⚫Kernel side issue • With systemd, we needed to consider CONFIG_RT_GROUP_SCHED. –https://blog.cybozu.io/entry/2018/06/22/080000 But these issues don’t depend on ROS.

Slide 18

Slide 18 text

Wrap up ⚫We cannot set up and control OCI hooks from Kubernetes for now. ⚫Container tracer is useful. • Contribute new facilities from our internal tool. –Ex. Linux capability checker –Ex. negative dentry snoop –https://qiita.com/kentaost/items/5f03ea32a2b2ef80270f • I want to discuss use cases of the container tracer among various industries.

Slide 19

Slide 19 text

SONYはソニー株式会社の登録商標または商標です。 各ソニー製品の商品名・サービス名はソニー株式会社またはグループ各社の登録商標または商標です。その他の製品および会社名は、各社の商号、登録商標または商標です。