$30 off During Our Annual Pro Sale. View Details »

Container-related technologies supporting Gitpod

Container-related technologies supporting Gitpod

うたもく

August 19, 2022
Tweet

More Decks by うたもく

Other Decks in Programming

Transcript

  1. Container-related
    technologies
    supporting Gitpod
    Container Runtime Meetup #4
    Toru Komatsu(@utam0k)
    1

    View Slide

  2. Toru(うたもく), Engineer at Gitpod
    utam0k
    utam0k
    2

    View Slide

  3. Today you will learn
    Ideas for applying

    container-related tech to your services

    Not just using Kubernetes and Docker
    3

    View Slide

  4. 01 What is Gitpod?
    Table of contents
    02 User workspace environments on Kubernetes pod
    03 Faster image pulls with IPFS
    04 Summary
    05 Please give me your questions!
    4

    View Slide

  5. What is Gitpod?
    01
    5

    View Slide

  6. https://gitpod.new
    Always ready to code.
    6

    View Slide

  7. Open Source
    We're free from big tech
    influence and integrate, not
    dictate. This makes our
    product development fast
    and close to our users &
    community.
    SaaS or Self-Hosted
    Use our SaaS solution
    running on the carbon neutral
    Google Cloud Platform or
    host Gitpod on your own
    cloud infrastructure using
    GKE, k3s, EKS or AKS.
    Secure by design
    Gitpod centralizes all source
    code and safely stores it in
    the cloud, never locally.
    Security is at the core of
    everything we do at Gitpod.
    7

    View Slide

  8. User workspace environments
    on Kubernetes pod
    02
    8

    View Slide

  9. automated dev enviroments in Kubernetes
    Gitpod provides
    workspace

    pod​
    workspace

    pod​
    workspace

    pod​
    workspace

    pod​
    Node
    workspace

    pod​
    9

    View Slide

  10. These are table stakes for developers.

    Granting root privileges to users? Isn't this dangerous?

    It can be done safely using rootless containers.
    no sudo, no apt-get, no Docker
    10

    View Slide

  11. a rough sketch
    Node
    DaemonSet
    workspace
    container
    How can we make it work?
    11

    View Slide

  12. a rough sketch
    Node
    DaemonSet
    workspace
    container
    ring0
    ring1
    PID 1
    PID 2
    User Namespace
    How can we make it work?
    ring2 PID 3
    PID Namespace 12

    View Slide

  13. a rough sketch
    Node
    DaemonSet
    workspace
    container
    ring0
    ring1
    PID 1
    PID 2
    User Namespace
    writeMapping(pid: 2)
    hostPID := translatePID(pid: 2)

    write(/proc/$hostPID/uid_map)

    write(/proc/$hostPID/gid_map)​
    How can we make it work?
    ring2 PID 3
    PID Namespace 13

    View Slide

  14. workspace's user namespace node
    0 → 10000
    33333 → 43333
    14

    View Slide

  15. a rough sketch
    Node
    DaemonSet
    workspace
    container
    ring0
    ring1
    PID 1
    PID 2
    User Namespace
    writeMapping(pid: 2)
    hostPID := translatePID(pid: 2)

    write(/proc/$hostPID/uid_map)

    write(/proc/$hostPID/gid_map)​
    How can we make it work?
    ring2 PID 3
    PID Namespace 15

    View Slide

  16. a rough sketch
    Node
    DaemonSet
    workspace
    container
    ring0
    ring1
    mount($mark, "/newroot/", "shiftfs")
    mount("proc", "/newroot/proc", "proc")

    PID 1
    PID 2
    mount($containerRootFS, "shiftfs", "mark")
    User Namespace
    writeMapping(pid: 2)
    prepareUserNS()
    hostPID := translatePID(pid: 2)

    write(/proc/$hostPID/uid_map)

    write(/proc/$hostPID/gid_map)​
    How can we make it work?
    ring2 PID 3
    PID Namespace 16

    View Slide

  17. workspace's user namespace node
    0 → 10000
    33333 → 43333
    17

    View Slide

  18. a rough sketch
    Node
    DaemonSet
    workspace
    container
    ring0
    ring1
    mount($mark, "/newroot/", "shiftfs")
    mount("proc", "/newroot/proc", "proc")

    PID 1
    PID 2
    mount($containerRootFS, "shiftfs", "mark")
    User Namespace
    writeMapping(pid: 2)
    prepareUserNS()
    hostPID := translatePID(pid: 2)

    write(/proc/$hostPID/uid_map)

    write(/proc/$hostPID/gid_map)​
    How can we make it work?
    ring2 PID 3
    PID Namespace 18

    View Slide

  19. a rough sketch
    Node
    DaemonSet
    workspace
    container
    ring0
    ring1
    ring2
    mount($mark, "/newroot/", "shiftfs")
    mount("proc", "/newroot/proc", "proc")

    PID 1
    PID 2
    PID 3
    pivot_root("/newroot")
    mount($containerRootFS, "shiftfs", "mark")
    User Namespace
    PID Namespace
    writeMapping(pid: 2)
    prepareUserNS()
    hostPID := translatePID(pid: 2)

    write(/proc/$hostPID/uid_map)

    write(/proc/$hostPID/gid_map)​
    How can we make it work?
    19

    View Slide

  20. Linux Namespace layering
    20

    View Slide

  21. Linux Namespace layering
    21

    View Slide

  22. Ok, now

    the developers have

    root privileges,
    can Docker work

    on a workspace?
    No, why?
    mount proc, capabilities…
    22

    View Slide

  23. mount proc with seccomp notify
    Node
    DaemonSet
    workspace
    container
    User Namespace
    PID Namespace
    How can we make it work?
    mount -t proc proc /proc
    ring1
    23

    View Slide

  24. mount proc with seccomp notify
    Node
    DaemonSet
    workspace
    container
    User Namespace
    PID Namespace
    How can we make it work?
    seccomp

    agent​
    mount -t proc proc /proc
    mountProc($pid)
    ring1
    24

    View Slide

  25. mount proc with seccomp notify
    Node
    DaemonSet
    workspace
    container
    User Namespace
    PID Namespace
    How can we make it work?
    seccomp

    agent​
    mount -t proc proc /proc
    mountProc($pid)
    hostPID := translatePID(pid: $pid)
    target := filepath.Join($WsRoot, "/proc")

    unix.Mount("proc", target, "proc")
    ring1
    25

    View Slide

  26. Dynamic/Static Resource limiting for pods
    CPU
    cpu.max
    cpu.stat
    Process
    pids.max
    IO
    io.max
    And more…
    26

    View Slide

  27. workspace

    pod​
    workspace

    pod​
    Node
    Daemons at each node govern the workspace of that node.
    It periodically monitors the resource usage of the
    workspace and updates the cgroup values to ensure
    efficient resource utilization.
    Dynamic/Static resource limiting
    DaemonSet
    ① Monitor
    ① Usage
    ③ Change
    ③ cgroup values
    containerd
    ② Ask a cgroup root
    27

    View Slide

  28. Faster image pulls
    with IPFS
    03
    28

    View Slide

  29. Gitpod users can use custom images.

    This means that a variety of images are built and pulled, which is
    unpredictable from our developer's point of view.

    When starting up a new workspace, a lot of time is spent on image pulls.
    Why did Gitpod need to speed up image pulls?
    29

    View Slide

  30. IPFS(InterPlanetary File System)
    enables us P2P and

    content-addressable data sharing
    30

    View Slide

  31. P2P image distribution by IPFS
    When pulling, images can be retrieved from containerds without a registry
    such as the Docker Registry
    What is IPFS with containerd?
    https://medium.com/nttlabs/nerdctl-ipfs-975569520e3d
    31

    View Slide

  32. Manifest for image not using IPFS
    IPFS urls are added to the urls field
    Pull image once to use IPFS
    32

    View Slide

  33. enables us to ship many
    different things in one OCI
    image.
    registry-facade
    33

    View Slide

  34. enables us to ship many
    different things in one OCI
    image.
    registry-facade
    user's workspace image
    34

    View Slide

  35. user's workspace image
    web IDE
    desktop IDE
    workspacekit
    supervisor
    docker-up, runc-facade
    Put Gitpod's tools

    on the user's image

    to build a workspace
    enables us to ship many
    different things in one OCI
    image.
    registry-facade
    35

    View Slide

  36. sha256:63c395644fe1767284082…
    sha256:3353847b14ebfc8fd1fd3…
    
    sha256:1bd9d3b7686a1c61c4d6d…
    sha256:a58c6b717b32ed3061fc6…
    sha256:b50a80767b4a9335fbb11…
    sha256:f82b95de3049e4a2fb6ab…
    sha256:dece7c53616378469e462…
    sha256:2bcb17063a1c100b4a6fd…
    sha256:e1caa81aa0d0cc390b573…
    sha256:3d92e3e7e06dce602725c…
    sha256:c186ba26dc778921dd494…
    sha256:7b193551031047df6cba1…
    sha256:1cf47bc524f71ccca20e9…
    sha256:08c01a0ec47e82ebe2bec…
    sha256:e9444ec2d0d74706e9f21…

    sha256:695bc3e4c4edea7f91799…
    sha256:764d7a91d4d599a243d1a…
    Put Gitpod's tools

    on the user's image

    to build a workspace
    enables us to ship many
    different things in one OCI
    image.
    registry-facade
    36

    View Slide

  37. workspace

    manager
    kubelet containerd registry-
    facade registry
    workspace

    manager
    kubelet containerd registry-
    facade registry
    pull reg.gitpod.io/remote/
    GET /v2/remote/manifests/
    GetImageSpec(instanceId)
    GET /v2/workspace-image/manifest/latest
    GET /v2/workspacekit/manifest/some_version
    GET /v2/docker-up/manifest/some_version
    37

    View Slide

  38. workspace

    manager
    kubelet containerd registry-
    facade registry
    workspace

    manager
    kubelet containerd registry-
    facade registry
    pull reg.gitpod.io/remote/
    GET /v2/remote/manifests/
    GetImageSpec(instanceId)
    GET /v2/workspace-image/manifest/latest
    GET /v2/workspacekit/manifest/some_version
    GET /v2/docker-up/manifest/some_version
    gitpod's original component
    38

    View Slide

  39. workspace

    manager
    kubelet containerd registry-
    facade registry
    workspace

    manager
    kubelet containerd registry-
    facade registry
    pull reg.gitpod.io/remote/
    GET /v2/remote/manifests/
    GetImageSpec(instanceId)
    GET /v2/workspace-image/manifest/latest
    GET /v2/workspacekit/manifest/some_version
    GET /v2/docker-up/manifest/some_version
    OCI distribution spec
    39

    View Slide

  40. registry-
    facade registry
    registry-
    facade registry

    IPFS
    IPFS
    containerd
    GET /v2/remote/manifests/
    containerd Redis
    Redis
    GET /2/workspace-image/manifest/latest
    40

    View Slide

  41. registry-
    facade registry
    registry-
    facade registry

    IPFS
    IPFS
    containerd
    GET /v2/remote/manifests/
    containerd Redis
    Redis
    GET /2/workspace-image/manifest/latest
    Does each layers exists on IPFS?
    Reply the CID if it exists on IPFS
    41

    View Slide

  42. registry-
    facade registry
    registry-
    facade registry

    IPFS
    IPFS
    containerd
    GET /v2/remote/manifests/
    containerd Redis
    Redis
    GET /2/workspace-image/manifest/latest
    modify the manifest

    for IPFS
    Reply the modified manifest
    GET Object from the urls field
    Does each layers exists on IPFS?
    Reply the CID if it exists on IPFS
    42

    View Slide

  43. Manifest for image not using IPFS
    IPFS urls are added to the urls field
    Pull image once to use IPFS
    43

    View Slide

  44. Fast Image Pulls Using IPFS And Opportunistic Caching
    @KubeCon US 2022
    Christian Weichel & Manuel de Brito Fontes, Gitpod
    44

    View Slide

  45. Summary
    04
    45

    View Slide

  46. Summary
    Thanks! ☑





    User workspace environments on Kubernetes pod
    namespace - rings
    cgroup - dynamic resource limits
    seccomp notify - mount procfs
    Faster image pulls with IPFS
    contained on IPFS
    OCI Image spec
    Image pull intercept
    46

    View Slide

  47. Any questions?
    05
    47

    View Slide