Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Container-related technologies supporting Gitpod

Container-related technologies supporting Gitpod

うたもく

August 19, 2022
Tweet

More Decks by うたもく

Other Decks in Programming

Transcript

  1. Container-related technologies supporting Gitpod Container Runtime Meetup #4 Toru Komatsu(@utam0k)

    1
  2. Toru(うたもく), Engineer at Gitpod utam0k utam0k 2

  3. Today you will learn Ideas for applying container-related tech to

    your services Not just using Kubernetes and Docker 3
  4. 01 What is Gitpod? Table of contents 02 User workspace

    environments on Kubernetes pod 03 Faster image pulls with IPFS 04 Summary 05 Please give me your questions! 4
  5. What is Gitpod? 01 5

  6. https://gitpod.new Always ready to code. 6

  7. Open Source We're free from big tech influence and integrate,

    not dictate. This makes our product development fast and close to our users & community. SaaS or Self-Hosted Use our SaaS solution running on the carbon neutral Google Cloud Platform or host Gitpod on your own cloud infrastructure using GKE, k3s, EKS or AKS. Secure by design Gitpod centralizes all source code and safely stores it in the cloud, never locally. Security is at the core of everything we do at Gitpod. 7
  8. User workspace environments on Kubernetes pod 02 8

  9. automated dev enviroments in Kubernetes Gitpod provides workspace pod​ workspace

    pod​ workspace pod​ workspace pod​ Node workspace pod​ 9
  10. These are table stakes for developers. Granting root privileges to

    users? Isn't this dangerous? It can be done safely using rootless containers. no sudo, no apt-get, no Docker 10
  11. a rough sketch Node DaemonSet workspace container How can we

    make it work? 11
  12. a rough sketch Node DaemonSet workspace container ring0 ring1 PID

    1 PID 2 User Namespace How can we make it work? ring2 PID 3 PID Namespace 12
  13. a rough sketch Node DaemonSet workspace container ring0 ring1 PID

    1 PID 2 User Namespace writeMapping(pid: 2) hostPID := translatePID(pid: 2) write(/proc/$hostPID/uid_map) write(/proc/$hostPID/gid_map)​ How can we make it work? ring2 PID 3 PID Namespace 13
  14. workspace's user namespace node 0 → 10000 33333 → 43333

    14
  15. a rough sketch Node DaemonSet workspace container ring0 ring1 PID

    1 PID 2 User Namespace writeMapping(pid: 2) hostPID := translatePID(pid: 2) write(/proc/$hostPID/uid_map) write(/proc/$hostPID/gid_map)​ How can we make it work? ring2 PID 3 PID Namespace 15
  16. a rough sketch Node DaemonSet workspace container ring0 ring1 mount($mark,

    "/newroot/", "shiftfs") mount("proc", "/newroot/proc", "proc") … PID 1 PID 2 mount($containerRootFS, "shiftfs", "mark") User Namespace writeMapping(pid: 2) prepareUserNS() hostPID := translatePID(pid: 2) write(/proc/$hostPID/uid_map) write(/proc/$hostPID/gid_map)​ How can we make it work? ring2 PID 3 PID Namespace 16
  17. workspace's user namespace node 0 → 10000 33333 → 43333

    17
  18. a rough sketch Node DaemonSet workspace container ring0 ring1 mount($mark,

    "/newroot/", "shiftfs") mount("proc", "/newroot/proc", "proc") … PID 1 PID 2 mount($containerRootFS, "shiftfs", "mark") User Namespace writeMapping(pid: 2) prepareUserNS() hostPID := translatePID(pid: 2) write(/proc/$hostPID/uid_map) write(/proc/$hostPID/gid_map)​ How can we make it work? ring2 PID 3 PID Namespace 18
  19. a rough sketch Node DaemonSet workspace container ring0 ring1 ring2

    mount($mark, "/newroot/", "shiftfs") mount("proc", "/newroot/proc", "proc") … PID 1 PID 2 PID 3 pivot_root("/newroot") mount($containerRootFS, "shiftfs", "mark") User Namespace PID Namespace writeMapping(pid: 2) prepareUserNS() hostPID := translatePID(pid: 2) write(/proc/$hostPID/uid_map) write(/proc/$hostPID/gid_map)​ How can we make it work? 19
  20. Linux Namespace layering 20

  21. Linux Namespace layering 21

  22. Ok, now the developers have root privileges, can Docker work

    on a workspace? No, why? mount proc, capabilities… 22
  23. mount proc with seccomp notify Node DaemonSet workspace container User

    Namespace PID Namespace How can we make it work? mount -t proc proc /proc ring1 23
  24. mount proc with seccomp notify Node DaemonSet workspace container User

    Namespace PID Namespace How can we make it work? seccomp agent​ mount -t proc proc /proc mountProc($pid) ring1 24
  25. mount proc with seccomp notify Node DaemonSet workspace container User

    Namespace PID Namespace How can we make it work? seccomp agent​ mount -t proc proc /proc mountProc($pid) hostPID := translatePID(pid: $pid) target := filepath.Join($WsRoot, "/proc") unix.Mount("proc", target, "proc") ring1 25
  26. Dynamic/Static Resource limiting for pods CPU cpu.max cpu.stat Process pids.max

    IO io.max And more… 26
  27. workspace pod​ workspace pod​ Node Daemons at each node govern

    the workspace of that node. It periodically monitors the resource usage of the workspace and updates the cgroup values to ensure efficient resource utilization. Dynamic/Static resource limiting DaemonSet ① Monitor ① Usage ③ Change ③ cgroup values containerd ② Ask a cgroup root 27
  28. Faster image pulls with IPFS 03 28

  29. Gitpod users can use custom images. This means that a

    variety of images are built and pulled, which is unpredictable from our developer's point of view. When starting up a new workspace, a lot of time is spent on image pulls. Why did Gitpod need to speed up image pulls? 29
  30. IPFS(InterPlanetary File System) enables us P2P and content-addressable data sharing

    30
  31. P2P image distribution by IPFS When pulling, images can be

    retrieved from containerds without a registry such as the Docker Registry What is IPFS with containerd? https://medium.com/nttlabs/nerdctl-ipfs-975569520e3d 31
  32. Manifest for image not using IPFS IPFS urls are added

    to the urls field Pull image once to use IPFS 32
  33. enables us to ship many different things in one OCI

    image. registry-facade 33
  34. enables us to ship many different things in one OCI

    image. registry-facade user's workspace image 34
  35. user's workspace image web IDE desktop IDE workspacekit supervisor docker-up,

    runc-facade Put Gitpod's tools on the user's image to build a workspace enables us to ship many different things in one OCI image. registry-facade 35
  36. sha256:63c395644fe1767284082… sha256:3353847b14ebfc8fd1fd3…  sha256:1bd9d3b7686a1c61c4d6d… sha256:a58c6b717b32ed3061fc6… sha256:b50a80767b4a9335fbb11… sha256:f82b95de3049e4a2fb6ab… sha256:dece7c53616378469e462… sha256:2bcb17063a1c100b4a6fd… sha256:e1caa81aa0d0cc390b573…

    sha256:3d92e3e7e06dce602725c… sha256:c186ba26dc778921dd494… sha256:7b193551031047df6cba1… sha256:1cf47bc524f71ccca20e9… sha256:08c01a0ec47e82ebe2bec… sha256:e9444ec2d0d74706e9f21… … sha256:695bc3e4c4edea7f91799… sha256:764d7a91d4d599a243d1a… Put Gitpod's tools on the user's image to build a workspace enables us to ship many different things in one OCI image. registry-facade 36
  37. workspace manager kubelet containerd registry- facade registry workspace manager kubelet

    containerd registry- facade registry pull reg.gitpod.io/remote/<instanceId> GET /v2/remote/manifests/<instanceId> GetImageSpec(instanceId) GET /v2/workspace-image/manifest/latest GET /v2/workspacekit/manifest/some_version GET /v2/docker-up/manifest/some_version 37
  38. workspace manager kubelet containerd registry- facade registry workspace manager kubelet

    containerd registry- facade registry pull reg.gitpod.io/remote/<instanceId> GET /v2/remote/manifests/<instanceId> GetImageSpec(instanceId) GET /v2/workspace-image/manifest/latest GET /v2/workspacekit/manifest/some_version GET /v2/docker-up/manifest/some_version gitpod's original component 38
  39. workspace manager kubelet containerd registry- facade registry workspace manager kubelet

    containerd registry- facade registry pull reg.gitpod.io/remote/<instanceId> GET /v2/remote/manifests/<instanceId> GetImageSpec(instanceId) GET /v2/workspace-image/manifest/latest GET /v2/workspacekit/manifest/some_version GET /v2/docker-up/manifest/some_version OCI distribution spec 39
  40. registry- facade registry registry- facade registry … IPFS IPFS containerd

    GET /v2/remote/manifests/<instanceId> containerd Redis Redis GET /2/workspace-image/manifest/latest 40
  41. registry- facade registry registry- facade registry … IPFS IPFS containerd

    GET /v2/remote/manifests/<instanceId> containerd Redis Redis GET /2/workspace-image/manifest/latest Does each layers exists on IPFS? Reply the CID if it exists on IPFS 41
  42. registry- facade registry registry- facade registry … IPFS IPFS containerd

    GET /v2/remote/manifests/<instanceId> containerd Redis Redis GET /2/workspace-image/manifest/latest modify the manifest for IPFS Reply the modified manifest GET Object from the urls field Does each layers exists on IPFS? Reply the CID if it exists on IPFS 42
  43. Manifest for image not using IPFS IPFS urls are added

    to the urls field Pull image once to use IPFS 43
  44. Fast Image Pulls Using IPFS And Opportunistic Caching @KubeCon US

    2022 Christian Weichel & Manuel de Brito Fontes, Gitpod 44
  45. Summary 04 45

  46. Summary Thanks! ☑ ☑ ☑ ☑ ☑ ☑ User workspace

    environments on Kubernetes pod namespace - rings cgroup - dynamic resource limits seccomp notify - mount procfs Faster image pulls with IPFS contained on IPFS OCI Image spec Image pull intercept 46
  47. Any questions? 05 47