Container-related technologies supporting Gitpod

by うたもく

Slide 1

Slide 1 text

Container-related technologies supporting Gitpod Container Runtime Meetup #4 Toru Komatsu(@utam0k) 1

Slide 2

Slide 2 text

Toru(うたもく), Engineer at Gitpod utam0k utam0k 2

Slide 3

Slide 3 text

Today you will learn Ideas for applying container-related tech to your services Not just using Kubernetes and Docker 3

Slide 4

Slide 4 text

01 What is Gitpod? Table of contents 02 User workspace environments on Kubernetes pod 03 Faster image pulls with IPFS 04 Summary 05 Please give me your questions! 4

Slide 5

Slide 5 text

What is Gitpod? 01 5

Slide 6

Slide 6 text

https://gitpod.new Always ready to code. 6

Slide 7

Slide 7 text

Open Source We're free from big tech influence and integrate, not dictate. This makes our product development fast and close to our users & community. SaaS or Self-Hosted Use our SaaS solution running on the carbon neutral Google Cloud Platform or host Gitpod on your own cloud infrastructure using GKE, k3s, EKS or AKS. Secure by design Gitpod centralizes all source code and safely stores it in the cloud, never locally. Security is at the core of everything we do at Gitpod. 7

Slide 8

Slide 8 text

User workspace environments on Kubernetes pod 02 8

Slide 9

Slide 9 text

automated dev enviroments in Kubernetes Gitpod provides workspace pod workspace pod workspace pod workspace pod Node workspace pod 9

Slide 10

Slide 10 text

These are table stakes for developers. Granting root privileges to users? Isn't this dangerous? It can be done safely using rootless containers. no sudo, no apt-get, no Docker 10

Slide 11

Slide 11 text

a rough sketch Node DaemonSet workspace container How can we make it work? 11

Slide 12

Slide 12 text

a rough sketch Node DaemonSet workspace container ring0 ring1 PID 1 PID 2 User Namespace How can we make it work? ring2 PID 3 PID Namespace 12

Slide 13

Slide 13 text

a rough sketch Node DaemonSet workspace container ring0 ring1 PID 1 PID 2 User Namespace writeMapping(pid: 2) hostPID := translatePID(pid: 2) write(/proc/$hostPID/uid_map) write(/proc/$hostPID/gid_map) How can we make it work? ring2 PID 3 PID Namespace 13

Slide 14

Slide 14 text

workspace's user namespace node 0 → 10000 33333 → 43333 14

Slide 15

Slide 15 text

Slide 16

Slide 16 text

a rough sketch Node DaemonSet workspace container ring0 ring1 mount($mark, "/newroot/", "shiftfs") mount("proc", "/newroot/proc", "proc") … PID 1 PID 2 mount($containerRootFS, "shiftfs", "mark") User Namespace writeMapping(pid: 2) prepareUserNS() hostPID := translatePID(pid: 2) write(/proc/$hostPID/uid_map) write(/proc/$hostPID/gid_map) How can we make it work? ring2 PID 3 PID Namespace 16

Slide 17

Slide 17 text

workspace's user namespace node 0 → 10000 33333 → 43333 17

Slide 18

Slide 18 text

Slide 19

Slide 19 text

a rough sketch Node DaemonSet workspace container ring0 ring1 ring2 mount($mark, "/newroot/", "shiftfs") mount("proc", "/newroot/proc", "proc") … PID 1 PID 2 PID 3 pivot_root("/newroot") mount($containerRootFS, "shiftfs", "mark") User Namespace PID Namespace writeMapping(pid: 2) prepareUserNS() hostPID := translatePID(pid: 2) write(/proc/$hostPID/uid_map) write(/proc/$hostPID/gid_map) How can we make it work? 19

Slide 20

Slide 20 text

Linux Namespace layering 20

Slide 21

Slide 21 text

Linux Namespace layering 21

Slide 22

Slide 22 text

Ok, now the developers have root privileges, can Docker work on a workspace? No, why? mount proc, capabilities… 22

Slide 23

Slide 23 text

mount proc with seccomp notify Node DaemonSet workspace container User Namespace PID Namespace How can we make it work? mount -t proc proc /proc ring1 23

Slide 24

Slide 24 text

mount proc with seccomp notify Node DaemonSet workspace container User Namespace PID Namespace How can we make it work? seccomp agent mount -t proc proc /proc mountProc($pid) ring1 24

Slide 25

Slide 25 text

mount proc with seccomp notify Node DaemonSet workspace container User Namespace PID Namespace How can we make it work? seccomp agent mount -t proc proc /proc mountProc($pid) hostPID := translatePID(pid: $pid) target := filepath.Join($WsRoot, "/proc") unix.Mount("proc", target, "proc") ring1 25

Slide 26

Slide 26 text

Dynamic/Static Resource limiting for pods CPU cpu.max cpu.stat Process pids.max IO io.max And more… 26

Slide 27

Slide 27 text

workspace pod workspace pod Node Daemons at each node govern the workspace of that node. It periodically monitors the resource usage of the workspace and updates the cgroup values to ensure efficient resource utilization. Dynamic/Static resource limiting DaemonSet ① Monitor ① Usage ③ Change ③ cgroup values containerd ② Ask a cgroup root 27

Slide 28

Slide 28 text

Faster image pulls with IPFS 03 28

Slide 29

Slide 29 text

Gitpod users can use custom images. This means that a variety of images are built and pulled, which is unpredictable from our developer's point of view. When starting up a new workspace, a lot of time is spent on image pulls. Why did Gitpod need to speed up image pulls? 29

Slide 30

Slide 30 text

IPFS(InterPlanetary File System) enables us P2P and content-addressable data sharing 30

Slide 31

Slide 31 text

P2P image distribution by IPFS When pulling, images can be retrieved from containerds without a registry such as the Docker Registry What is IPFS with containerd? https://medium.com/nttlabs/nerdctl-ipfs-975569520e3d 31

Slide 32

Slide 32 text

Manifest for image not using IPFS IPFS urls are added to the urls field Pull image once to use IPFS 32

Slide 33

Slide 33 text

enables us to ship many different things in one OCI image. registry-facade 33

Slide 34

Slide 34 text

enables us to ship many different things in one OCI image. registry-facade user's workspace image 34

Slide 35

Slide 35 text

user's workspace image web IDE desktop IDE workspacekit supervisor docker-up, runc-facade Put Gitpod's tools on the user's image to build a workspace enables us to ship many different things in one OCI image. registry-facade 35

Slide 36

Slide 36 text

sha256:63c395644fe1767284082… sha256:3353847b14ebfc8fd1fd3… sha256:1bd9d3b7686a1c61c4d6d… sha256:a58c6b717b32ed3061fc6… sha256:b50a80767b4a9335fbb11… sha256:f82b95de3049e4a2fb6ab… sha256:dece7c53616378469e462… sha256:2bcb17063a1c100b4a6fd… sha256:e1caa81aa0d0cc390b573… sha256:3d92e3e7e06dce602725c… sha256:c186ba26dc778921dd494… sha256:7b193551031047df6cba1… sha256:1cf47bc524f71ccca20e9… sha256:08c01a0ec47e82ebe2bec… sha256:e9444ec2d0d74706e9f21… … sha256:695bc3e4c4edea7f91799… sha256:764d7a91d4d599a243d1a… Put Gitpod's tools on the user's image to build a workspace enables us to ship many different things in one OCI image. registry-facade 36

Slide 37

Slide 37 text

workspace manager kubelet containerd registry- facade registry workspace manager kubelet containerd registry- facade registry pull reg.gitpod.io/remote/ GET /v2/remote/manifests/ GetImageSpec(instanceId) GET /v2/workspace-image/manifest/latest GET /v2/workspacekit/manifest/some_version GET /v2/docker-up/manifest/some_version 37

Slide 38

Slide 38 text

Slide 39

Slide 39 text

Slide 40

Slide 40 text

registry- facade registry registry- facade registry … IPFS IPFS containerd GET /v2/remote/manifests/ containerd Redis Redis GET /2/workspace-image/manifest/latest 40

Slide 41

Slide 41 text

registry- facade registry registry- facade registry … IPFS IPFS containerd GET /v2/remote/manifests/ containerd Redis Redis GET /2/workspace-image/manifest/latest Does each layers exists on IPFS? Reply the CID if it exists on IPFS 41

Slide 42

Slide 42 text

registry- facade registry registry- facade registry … IPFS IPFS containerd GET /v2/remote/manifests/ containerd Redis Redis GET /2/workspace-image/manifest/latest modify the manifest for IPFS Reply the modified manifest GET Object from the urls field Does each layers exists on IPFS? Reply the CID if it exists on IPFS 42

Slide 43

Slide 43 text

Manifest for image not using IPFS IPFS urls are added to the urls field Pull image once to use IPFS 43

Slide 44

Slide 44 text

Fast Image Pulls Using IPFS And Opportunistic Caching @KubeCon US 2022 Christian Weichel & Manuel de Brito Fontes, Gitpod 44

Slide 45

Slide 45 text

Summary 04 45

Slide 46

Slide 46 text

Summary Thanks! ☑ ☑ ☑ ☑ ☑ ☑ User workspace environments on Kubernetes pod namespace - rings cgroup - dynamic resource limits seccomp notify - mount procfs Faster image pulls with IPFS contained on IPFS OCI Image spec Image pull intercept 46

Slide 47

Slide 47 text

Any questions? 05 47