Container Tracer using OCI hooks on Kubernetes

Transcript

1. R&D Center Base System Development Department
   Copyright 2019 Sony Corporation
   Container Tracer using OCI hooks on Kubernetes
   Kubernetes Invitational Meetup Tokyo #4
   Kenta Tada
   R&D Center
   Sony Corporation
   

   View Slide

2. About me
   ⚫System Software Engineer, Sony
   ⚫OSS Contributor
   • runC
   • Docker
   • containerd
   and so on
   

   View Slide

3. Agenda
   ⚫OCI hooks recap
   ⚫Current status of Kubernetes integration
   ⚫Case Study
   • Tracer using OCI hooks for Tracer for ROS (Robot Operating System)
   

   View Slide

4. OCI hooks recap
   

   View Slide

5. OCI hooks
   ⚫OCI hooks are executed by container runtime
   ⚫You can set up the hooks at
   • Prestart
   • Poststart
   • Poststop
   ⚫From OCI runtime spec, the state of the container which
   includes container initial PID must be passed to hooks over
   stdin
   ⚫More hooks are currently proposed.
   • https://github.com/opencontainers/runtime-spec/pull/1008
   

   View Slide

6. Current status of Kubernetes integration
   

   View Slide

7. How to set up prestart hook in Kubernetes
   ⚫Kubernetes Pod Lifecycle and related hook
   ⚫Kubernetes did not provide prestart hook
   • https://github.com/kubernetes/kubernetes/issues/140
   process lifetime
   process
   start
   process
   stop
   prestop
   poststart
   

   View Slide

8. How to operate OCI hooks from Kubernetes
   1. Set up OCI hooks on high level runtime
   2. Control OCI hooks per Pod using Kubernetes annotations
   

   View Slide

9. Current status of containerd
   1. Set up OCI hooks on high level runtime
   • containerd is currently developing option for setting OCI hook
   – https://github.com/containerd/cri/pull/1248
   – https://github.com/containerd/cri/issues/405
   2. Control OCI hooks per Pod using Kubernetes annotations
   • containerd supported for passing annotations to low level runtime
   – https://github.com/containerd/cri/pull/1084
   ○
   ×
   

   View Slide

10. Current status of CRI-O
    1. Set up OCI hooks on high level runtime
    • CRI-O and Podman have already provided their own solution "oci-
    hooks“
    2. Control OCI hooks per Pod using Kubernetes annotations
    • CRI-O is currently developing passing annotations to low level
    runtime
    – https://github.com/cri-o/cri-o/issues/2402
    ○
    ×
    

    View Slide

11. Case Study
    

    View Slide

12. Recap : Container Tracer
    ⚫We are developing an experimental tool to trace system
    calls inside containers without any debug tools in Pod.
    • https://speakerdeck.com/kentatada/debug-application-inside-
    kubernetes-using-linux-kernel-tools
    • https://github.com/KentaTada/oci-ftrace-syscall-analyzer
    ⚫ This tool uses OCI hooks to trace system calls from apps startup.
    

    View Slide

13. What is ROS (Robot Operating System)?
    ⚫ROS (Robot Operating System) provides libraries and tools
    to help software developers create robot applications.
    • http://wiki.ros.org/
    ⚫ROS is the middleware.
    Hardware
    OS(Ex. Ubuntu)
    ROS
    Apps(Ex. path planning)
    

    View Slide

14. Tracer for ROS containers
    ⚫We are trying to launch ROS nodes on Kubernetes.
    • Ex. Using Docker with ROS
    –http://wiki.ros.org/docker
    ⚫We are also trying to use our container tracer to trace ROS
    nodes from apps startup.
    

    View Slide

15. Case Study : RT apps on the container environment
    ⚫RT apps on our rootless container failed to execute
    sched_setscheduler(2) during apps startup.
    

    View Slide

16. Logs
    

    View Slide

17. Tracer found two root causes
    ⚫Container side issue
    • With systemd, we needed to set up LimitRTPRIO in service file.
    –https://superuser.com/questions/403184/configuring-systemd-to-
    allow-daemon-to-set-rt-priority
    ⚫Kernel side issue
    • With systemd, we needed to consider CONFIG_RT_GROUP_SCHED.
    –https://blog.cybozu.io/entry/2018/06/22/080000
    But these issues don’t depend on ROS.
    

    View Slide

18. Wrap up
    ⚫We cannot set up and control OCI hooks from Kubernetes
    for now.
    ⚫Container tracer is useful.
    • Contribute new facilities from our internal tool.
    –Ex. Linux capability checker
    –Ex. negative dentry snoop
    –https://qiita.com/kentaost/items/5f03ea32a2b2ef80270f
    • I want to discuss use cases of the container tracer among various
    industries.
    

    View Slide

19. SONYはソニー株式会社の登録商標または商標です。
    各ソニー製品の商品名・サービス名はソニー株式会社またはグループ各社の登録商標または商標です。その他の製品および会社名は、各社の商号、登録商標または商標です。
    

    View Slide