Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Container Tracer using OCI hooks on Kubernetes

Kenta Tada
December 18, 2019

Container Tracer using OCI hooks on Kubernetes

Kenta Tada

December 18, 2019
Tweet

More Decks by Kenta Tada

Other Decks in Programming

Transcript

  1. R&D Center Base System Development Department
    Copyright 2019 Sony Corporation
    Container Tracer using OCI hooks on Kubernetes
    Kubernetes Invitational Meetup Tokyo #4
    Kenta Tada
    R&D Center
    Sony Corporation

    View full-size slide

  2. About me
    ⚫System Software Engineer, Sony
    ⚫OSS Contributor
    • runC
    • Docker
    • containerd
    and so on

    View full-size slide

  3. Agenda
    ⚫OCI hooks recap
    ⚫Current status of Kubernetes integration
    ⚫Case Study
    • Tracer using OCI hooks for Tracer for ROS (Robot Operating System)

    View full-size slide

  4. OCI hooks recap

    View full-size slide

  5. OCI hooks
    ⚫OCI hooks are executed by container runtime
    ⚫You can set up the hooks at
    • Prestart
    • Poststart
    • Poststop
    ⚫From OCI runtime spec, the state of the container which
    includes container initial PID must be passed to hooks over
    stdin
    ⚫More hooks are currently proposed.
    • https://github.com/opencontainers/runtime-spec/pull/1008

    View full-size slide

  6. Current status of Kubernetes integration

    View full-size slide

  7. How to set up prestart hook in Kubernetes
    ⚫Kubernetes Pod Lifecycle and related hook
    ⚫Kubernetes did not provide prestart hook
    • https://github.com/kubernetes/kubernetes/issues/140
    process lifetime
    process
    start
    process
    stop
    prestop
    poststart

    View full-size slide

  8. How to operate OCI hooks from Kubernetes
    1. Set up OCI hooks on high level runtime
    2. Control OCI hooks per Pod using Kubernetes annotations

    View full-size slide

  9. Current status of containerd
    1. Set up OCI hooks on high level runtime
    • containerd is currently developing option for setting OCI hook
    – https://github.com/containerd/cri/pull/1248
    – https://github.com/containerd/cri/issues/405
    2. Control OCI hooks per Pod using Kubernetes annotations
    • containerd supported for passing annotations to low level runtime
    – https://github.com/containerd/cri/pull/1084

    ×

    View full-size slide

  10. Current status of CRI-O
    1. Set up OCI hooks on high level runtime
    • CRI-O and Podman have already provided their own solution "oci-
    hooks“
    2. Control OCI hooks per Pod using Kubernetes annotations
    • CRI-O is currently developing passing annotations to low level
    runtime
    – https://github.com/cri-o/cri-o/issues/2402

    ×

    View full-size slide

  11. Recap : Container Tracer
    ⚫We are developing an experimental tool to trace system
    calls inside containers without any debug tools in Pod.
    • https://speakerdeck.com/kentatada/debug-application-inside-
    kubernetes-using-linux-kernel-tools
    • https://github.com/KentaTada/oci-ftrace-syscall-analyzer
    ⚫ This tool uses OCI hooks to trace system calls from apps startup.

    View full-size slide

  12. What is ROS (Robot Operating System)?
    ⚫ROS (Robot Operating System) provides libraries and tools
    to help software developers create robot applications.
    • http://wiki.ros.org/
    ⚫ROS is the middleware.
    Hardware
    OS(Ex. Ubuntu)
    ROS
    Apps(Ex. path planning)

    View full-size slide

  13. Tracer for ROS containers
    ⚫We are trying to launch ROS nodes on Kubernetes.
    • Ex. Using Docker with ROS
    –http://wiki.ros.org/docker
    ⚫We are also trying to use our container tracer to trace ROS
    nodes from apps startup.

    View full-size slide

  14. Case Study : RT apps on the container environment
    ⚫RT apps on our rootless container failed to execute
    sched_setscheduler(2) during apps startup.

    View full-size slide

  15. Tracer found two root causes
    ⚫Container side issue
    • With systemd, we needed to set up LimitRTPRIO in service file.
    –https://superuser.com/questions/403184/configuring-systemd-to-
    allow-daemon-to-set-rt-priority
    ⚫Kernel side issue
    • With systemd, we needed to consider CONFIG_RT_GROUP_SCHED.
    –https://blog.cybozu.io/entry/2018/06/22/080000
    But these issues don’t depend on ROS.

    View full-size slide

  16. Wrap up
    ⚫We cannot set up and control OCI hooks from Kubernetes
    for now.
    ⚫Container tracer is useful.
    • Contribute new facilities from our internal tool.
    –Ex. Linux capability checker
    –Ex. negative dentry snoop
    –https://qiita.com/kentaost/items/5f03ea32a2b2ef80270f
    • I want to discuss use cases of the container tracer among various
    industries.

    View full-size slide

  17. SONYはソニー株式会社の登録商標または商標です。
    各ソニー製品の商品名・サービス名はソニー株式会社またはグループ各社の登録商標または商標です。その他の製品および会社名は、各社の商号、登録商標または商標です。

    View full-size slide