Rust-based secure and lightweight container runtime Introduction of a presentation at Cloud Native Rust Day 2021 libseccomp-rs: Native Rust crate for libseccomp library Development of a container runtime based on Kata Containers agent Activities of Kata Containers Introduction of runk, a container runtime based on the Kata Agent Secure containers by static-based system call analysis technique Activities of container security enhancement using Confine
systems have more restrictions than general-purpose systems Resource-constrained systems Small memory size Low-capacity storage Low-spec CPU Mission-critical systems Real-time application Critical functionality Longer life cycle
Kubernetes or Docker on embedded systems is difficult Include performance overhead and high resource usage Write operations by the daemon process shorten the lifespan of eMMC* We run a low-level container runtime alone on the systems *Embedded Multimedia Card
Security Linux Capabilities are not fine-grained access control e.g., Both ping and ARP spoofing need CAP_NET_RAW Rootless containers by user namespace are very strict for the systems The rootless containers cannot emulate all system calls However, some embedded applications need to access devices via mount(2), mknod(2), etc. Lightweight Container startup time is not fast enough for real-time systems The Go-based runtimes are not suitable for resource-constrained systems The application binary size is big Garbage Collection (GC) includes high CPU utilization
Lightweight container runtime (SL runtime) SL runtime is implemented fully in Rust with modern crates Minimal container runtime for embedded systems OCI-compatible runtime
great fit for embedded systems Performance is equivalent to C/C++ Memory safety without GC Small application binary size Awesome crates for developing the container runtime FFI (Foreign Function Interface) to bind Linux API Go is also a great programming language but has some limitations Problem interacting with namespaces by go-runtime The application binary size is big compared to Rust Overhead by GC
Many awesome crates for developing the container runtime capability : https://crates.io/crates/caps rlimit : https://crates.io/crates/rlimit cgroups : https://crates.io/crates/cgroups-rs libseccomp-rs : https://crates.io/crates/libseccomp passfd : https://crates.io/crates/passfd This crate is used for the fine-grained access control core_affinity : https://crates.io/crates/core_affinity This crate is used for the real-time support etc. clap : https://crates.io/crates/clap serde_json : https://crates.io/crates/serde_json anyhow : https://crates.io/crates/anyhow etc. Developing Runtime Creating Container
container speedily by leveraging a pre-created container Remove time for initializing the runtime and creating the container Replace only the execution process inside the container at startup Reuses the other configuration except for the execution process
enables the rootless containers to execute system calls safely FGAC server emulates the system calls in userspace on behalf of the container The rootless containers can access devices safely via mount(2), mknod(2), etc. FGAC mechanism is achieved using the new seccomp notify feature
FGAC server before starting a container The server is launched as root by only a system administrator Run the container using config.json that describes the seccomp notify OCI runtime specifications already support seccomp notify [1] [1] https://github.com/opencontainers/runtime-spec/pull/1074
time of the containers: Normal run and Fast startup Measure memory consumption of the container runtimes Environment Host: AMD Ryzen 9 3900X 12-Core (Ubuntu 20.04) Evaluated the runtimes*: SL runtime, runsc, singularity, runc, crun, and railcar Experimental Setup All the runtimes use a same config.json Execute /bin/true inside the container runtimes without any client tools *[Version of the evaluated runtimes] runsc:v20201208.0, singularity: v3.1.0, runc: v1.0.0-rc93, crun: v0.18, railcar: v1.0.4
is the fastest among the existing runtimes Normal run achieves a 7.4x speed-up compared to runc Fast startup achieves a 1.5x speed-up compared to the Normal run
libseccomp library Repository: https://github.com/ManaSugi/libseccomp-rs Anyone is welcome to join and contribute code, documentation, and use cases! A set of projects that enables developers to use the libseccomp API in Rust libseccomp: High-level safe API libseccomp-sys: Low-level unsafe API (automatically generated) tool: Tool for generating low-level bindings using bindgen Make libseccomp C library safe as mush as possible e.g., Use NonNull to ensure the return value of seccomp_init is non-null
have plan to put the libseccomp-rs into seccomp/libseccomp project https://github.com/seccomp/libseccomp/issues/323 Prepare for aggregating tests all in on tests directory of the libseccomp Get better functional code coverage for Golang and Rust There are 58 tests
Additional isolation with a lightweight VM and individual kernels The project is led by Intel, Apple, Red Hat and Ant Group Host Kernel Hardware Virtualization Hardware Virtualization Hardware Virtualization Virtual Machine A Guest Kernel Container Container Virtual Machine B Guest Kernel Container Container Virtual Machine C Guest Kernel Container Container
inside the VM Support many OCI behaviors The execution unit is the sandbox Rewritten in Rust for Kata Containers 2.0 release Reduces the memory footprint while keeping the memory safety Release kata-agent-ctl, a useful developer tool to aid in the agent API debugging Communicate with the other kata components over ttRPC
Kata Agent Support Seccomp (WIP) https://github.com/kata-containers/kata-containers/pull/1788 Make the Kata Containers more secure inside the Pod sandbox Use libseccomp-rs Support AppArmor (WIP) https://github.com/kata-containers/kata-containers/issues/2227 Use prestart OCI hooks for loading the AppArmor profile in the VM runk: Standard OCI container runtime based on Kata-agent (WIP)
Kata Containers The kata-agent is not a container runtime Receive requests from containerd-shim-kata-v2 using ttRPC Does not provide an OCI Command-Line Interface (CLI) The kata-agent has most of the features needed for the container runtime OCI compatibility Can we develop a standard container runtime based on the kata-agent? The kata-agent can be adopted in various systems Make the kata-agent easier to follow the latest OCI runtime specifications Make the kata-agent easier to test and debug
Rust-based standard container runtime based on the kata-agent The runk spawns and runs container on the host machine directly like runc Conform to the OCI Container Runtime specifications https://github.com/kata-containers/kata-containers/pull/2785
is faster than runc and has a lower memory footprint 1.4x speed up compared to runc runk runc crun time [msec] 39.01 53.09 35.35 memory footprint [MB] 4.223 16.54 3.157 This table shows the average of the elapsed time and the memory footprint (maximum resident set size) for running sequentially 500 containers, the containers run /bin/true with detaching mode; runk always runs containers with the mode currently.
isolation than VM Attackers can exploit kernel vulnerabilities to compromise the host Trusted computing base in container comprises the entire kernel The code base of the kernel has been expanding with the number of system calls Container Trusted Application Container Container Platform Operating System Hardware Container Runtime Untrusted Application × Container Trusted Application Exploit Compromise Compromise
Linux Kernel Reduce attack surface of the kernel Remove code that is inaccessible, or not needed for a given workload of configuration NIST container security guidelines [2] suggestion is to reduce the attack surface by limiting the functionality available to containers Seccomp limits system calls for containers Deny potentially dangerous system calls Reduce the kernel code available to each container few syscalls seccomp Application Linux Kernel [2] Karen Scarfone Murugiah Souppaya, John Morello. Application Container Security Guide, 2017. https://nvlpubs.nist.gov/nistpubs/Spec ialPublications/NIST.SP.800-190.pdf.
know all system calls issued by containers Seccomp filter requires system calls number to insert the deny or allow list Docker, containerd, and CRI-O drops potentially dangerous system calls by default Seccomp profiles e.g., pivot_root, ptrace, unshare, etc. How do we get all system calls inside an application binary? Static analysis Dynamic analysis
code analysis takes a lot of time Dynamic analysis does not capture exhaustively all the code Not capture parts of code that are executed rarely, such as error handling routines Combing static and dynamic analysis is important to analyze system calls in the application binary effectively
Automated system call policy generation for attack surface reduction [3] Confine takes a container image as input and generates a system call policy Reduce attack surface of the kernel by limiting system calls in containers Extract the system calls automatically using static and dynamic analysis Limit the extracted system calls by adopting Seccomp Results of the evaluation by authors Confine can disable 145 system calls (out of 326) using 150 Docker Images Confine can neutralize 51 previously disclosed kernel vulnerabilities [3] Seyedhamed Ghavamnia, Tapti Palit, Azzedine Benameur, and Michalis Polychronakis. Confine: Automated System Call Policy Generation for Container Attack Surface Reduction. In International Symposium on Research in Attacks, Intrusions and Defenses (RAID), 2020.
applications that may run on the container Identify all library functions imported by each application Map library functions to system calls Extract direct system call invocations from applications and libraries
high-level container runtimes are not accurate Include many system calls that are not used in the containers Recently, new static-based system call analysis techniques have been proposed in research papers We can analyze system calls accurately If container image developers can use Confine or other new static analysis tools to extract system calls that are used by container images, they can generate more accurate default profiles for the container images than the runtime default profiles
Spec Define a new SecurityContext media type in OCI Image Spec (WIP) Include Seccomp and Linux Capability configurations inside the image spec Issue: https://github.com/opencontainers/image-spec/issues/867 Goal Allow users to choose the image default SecurityContext including the default seccomp profiles from the container orchestration software such as Kubernetes OCI Image Image Default SecurityContext New! Runtime Default Profile Custom Profile Choose User
User side: Set default seccomp profiles for a container image in Kubernetes configuration Image developer side: Analyze a container image using system call analysis tools such as Confine Add information about Seccomp to SecurityContext in the image spec Push container image public or private registries spec: securityContext: seccompProfile: type: ImageDefault
Systems System Call Analysis Tool based on Confine Container Image Developer Run Confine with Docker and K8s Generate limited system calls name list Put the system calls list in SecurityContext We need to add the system call list to the OCI image spec https://github.com/opencontainers/image-spec/issues/867 . OCI Image containerd Pull the image Container Runtime OCI Secure Container Launch a container Image Registry Store the mage kubelet CRI User Generate Seccomp profiles in config.json “seccomp”: { .... } We need to develop and modify the systems and the specifications Container Runtime Secure Container Run Confine with “low-level” container runtime “seccomp”: { .... } We need to modify the K8s specification to be able to choose Seccomp profiles generated by Confine in the image spec Image Default SecurityContext User
great fit for embedded systems Small memory footprint and binary size for resource-constrained systems Memory safety without any overhead for mission-critical systems Awesome crates for developing container runtimes runk is a standard container runtime based on a modified version of the Kata Containers agent System call analysis tools make containers more secure Various state-of-the-art system call analysis techniques have been proposed in research papers
of Sony products and services are the registered trademarks and/or trademarks of Sony Group Corporation or its Group companies. Other company names and product names are registered trademarks and/or trademarks of the respective companies.
manabusugimoto
hiroramos4
techtekt
isaoshimizu
maf_0521
kaminashi
oomatomo
tamaiyutaro
sansantech
murabayashi
chanyou0311
chacco38
appleboy
tmiket
danielanewman
dugsong
paigeccino
dougneiner
sergeychernyshev
nikkihalliwell
lara
auna
holman
minecr
seathinner
![R&D Center Oct/9th/2021 37 Confine [Ghavamnia et al. RAID’20] ](https://files.speakerdeck.com/presentations/7687479c11364389ad1380e61260fab6/slide_36.jpg){kind=link}
