Container-related technologies supporting Gitpod

https://runtime.connpass.com/event/253798/

Transcript

1. Container-related
   technologies
   supporting Gitpod
   Container Runtime Meetup #4
   Toru Komatsu(@utam0k)
   1
   

   View Slide

2. Toru(うたもく), Engineer at Gitpod
   utam0k
   utam0k
   2
   

   View Slide

3. Today you will learn
   Ideas for applying
   
   container-related tech to your services
   
   Not just using Kubernetes and Docker
   3
   

   View Slide

4. 01 What is Gitpod?
   Table of contents
   02 User workspace environments on Kubernetes pod
   03 Faster image pulls with IPFS
   04 Summary
   05 Please give me your questions!
   4
   

   View Slide

5. What is Gitpod?
   01
   5
   

   View Slide

6. https://gitpod.new
   Always ready to code.
   6
   

   View Slide

7. Open Source
   We're free from big tech
   influence and integrate, not
   dictate. This makes our
   product development fast
   and close to our users &
   community.
   SaaS or Self-Hosted
   Use our SaaS solution
   running on the carbon neutral
   Google Cloud Platform or
   host Gitpod on your own
   cloud infrastructure using
   GKE, k3s, EKS or AKS.
   Secure by design
   Gitpod centralizes all source
   code and safely stores it in
   the cloud, never locally.
   Security is at the core of
   everything we do at Gitpod.
   7
   

   View Slide

8. User workspace environments
   on Kubernetes pod
   02
   8
   

   View Slide

9. automated dev enviroments in Kubernetes
   Gitpod provides
   workspace
   
   pod​
   workspace
   
   pod​
   workspace
   
   pod​
   workspace
   
   pod​
   Node
   workspace
   
   pod​
   9
   

   View Slide

10. These are table stakes for developers.
    
    Granting root privileges to users? Isn't this dangerous?
    
    It can be done safely using rootless containers.
    no sudo, no apt-get, no Docker
    10
    

    View Slide

11. a rough sketch
    Node
    DaemonSet
    workspace
    container
    How can we make it work?
    11
    

    View Slide

12. a rough sketch
    Node
    DaemonSet
    workspace
    container
    ring0
    ring1
    PID 1
    PID 2
    User Namespace
    How can we make it work?
    ring2 PID 3
    PID Namespace 12
    

    View Slide

13. a rough sketch
    Node
    DaemonSet
    workspace
    container
    ring0
    ring1
    PID 1
    PID 2
    User Namespace
    writeMapping(pid: 2)
    hostPID := translatePID(pid: 2)
    
    write(/proc/$hostPID/uid_map)
    
    write(/proc/$hostPID/gid_map)​
    How can we make it work?
    ring2 PID 3
    PID Namespace 13
    

    View Slide

14. workspace's user namespace node
    0 → 10000
    33333 → 43333
    14
    

    View Slide

15. a rough sketch
    Node
    DaemonSet
    workspace
    container
    ring0
    ring1
    PID 1
    PID 2
    User Namespace
    writeMapping(pid: 2)
    hostPID := translatePID(pid: 2)
    
    write(/proc/$hostPID/uid_map)
    
    write(/proc/$hostPID/gid_map)​
    How can we make it work?
    ring2 PID 3
    PID Namespace 15
    

    View Slide

16. a rough sketch
    Node
    DaemonSet
    workspace
    container
    ring0
    ring1
    mount($mark, "/newroot/", "shiftfs")
    mount("proc", "/newroot/proc", "proc")
    …
    PID 1
    PID 2
    mount($containerRootFS, "shiftfs", "mark")
    User Namespace
    writeMapping(pid: 2)
    prepareUserNS()
    hostPID := translatePID(pid: 2)
    
    write(/proc/$hostPID/uid_map)
    
    write(/proc/$hostPID/gid_map)​
    How can we make it work?
    ring2 PID 3
    PID Namespace 16
    

    View Slide

17. workspace's user namespace node
    0 → 10000
    33333 → 43333
    17
    

    View Slide

18. a rough sketch
    Node
    DaemonSet
    workspace
    container
    ring0
    ring1
    mount($mark, "/newroot/", "shiftfs")
    mount("proc", "/newroot/proc", "proc")
    …
    PID 1
    PID 2
    mount($containerRootFS, "shiftfs", "mark")
    User Namespace
    writeMapping(pid: 2)
    prepareUserNS()
    hostPID := translatePID(pid: 2)
    
    write(/proc/$hostPID/uid_map)
    
    write(/proc/$hostPID/gid_map)​
    How can we make it work?
    ring2 PID 3
    PID Namespace 18
    

    View Slide

19. a rough sketch
    Node
    DaemonSet
    workspace
    container
    ring0
    ring1
    ring2
    mount($mark, "/newroot/", "shiftfs")
    mount("proc", "/newroot/proc", "proc")
    …
    PID 1
    PID 2
    PID 3
    pivot_root("/newroot")
    mount($containerRootFS, "shiftfs", "mark")
    User Namespace
    PID Namespace
    writeMapping(pid: 2)
    prepareUserNS()
    hostPID := translatePID(pid: 2)
    
    write(/proc/$hostPID/uid_map)
    
    write(/proc/$hostPID/gid_map)​
    How can we make it work?
    19
    

    View Slide

20. Linux Namespace layering
    20
    

    View Slide

21. Linux Namespace layering
    21
    

    View Slide

22. Ok, now
    
    the developers have
    
    root privileges,
    can Docker work
    
    on a workspace?
    No, why?
    mount proc, capabilities…
    22
    

    View Slide

23. mount proc with seccomp notify
    Node
    DaemonSet
    workspace
    container
    User Namespace
    PID Namespace
    How can we make it work?
    mount -t proc proc /proc
    ring1
    23
    

    View Slide

24. mount proc with seccomp notify
    Node
    DaemonSet
    workspace
    container
    User Namespace
    PID Namespace
    How can we make it work?
    seccomp
    
    agent​
    mount -t proc proc /proc
    mountProc($pid)
    ring1
    24
    

    View Slide

25. mount proc with seccomp notify
    Node
    DaemonSet
    workspace
    container
    User Namespace
    PID Namespace
    How can we make it work?
    seccomp
    
    agent​
    mount -t proc proc /proc
    mountProc($pid)
    hostPID := translatePID(pid: $pid)
    target := filepath.Join($WsRoot, "/proc")
    
    unix.Mount("proc", target, "proc")
    ring1
    25
    

    View Slide

26. Dynamic/Static Resource limiting for pods
    CPU
    cpu.max
    cpu.stat
    Process
    pids.max
    IO
    io.max
    And more…
    26
    

    View Slide

27. workspace
    
    pod​
    workspace
    
    pod​
    Node
    Daemons at each node govern the workspace of that node.
    It periodically monitors the resource usage of the
    workspace and updates the cgroup values to ensure
    efficient resource utilization.
    Dynamic/Static resource limiting
    DaemonSet
    ① Monitor
    ① Usage
    ③ Change
    ③ cgroup values
    containerd
    ② Ask a cgroup root
    27
    

    View Slide

28. Faster image pulls
    with IPFS
    03
    28
    

    View Slide

29. Gitpod users can use custom images.
    
    This means that a variety of images are built and pulled, which is
    unpredictable from our developer's point of view.
    
    When starting up a new workspace, a lot of time is spent on image pulls.
    Why did Gitpod need to speed up image pulls?
    29
    

    View Slide

30. IPFS(InterPlanetary File System)
    enables us P2P and
    
    content-addressable data sharing
    30
    

    View Slide

31. P2P image distribution by IPFS
    When pulling, images can be retrieved from containerds without a registry
    such as the Docker Registry
    What is IPFS with containerd?
    https://medium.com/nttlabs/nerdctl-ipfs-975569520e3d
    31
    

    View Slide

32. Manifest for image not using IPFS
    IPFS urls are added to the urls field
    Pull image once to use IPFS
    32
    

    View Slide

33. enables us to ship many
    different things in one OCI
    image.
    registry-facade
    33
    

    View Slide

34. enables us to ship many
    different things in one OCI
    image.
    registry-facade
    user's workspace image
    34
    

    View Slide

35. user's workspace image
    web IDE
    desktop IDE
    workspacekit
    supervisor
    docker-up, runc-facade
    Put Gitpod's tools
    
    on the user's image
    
    to build a workspace
    enables us to ship many
    different things in one OCI
    image.
    registry-facade
    35
    

    View Slide

36. sha256:63c395644fe1767284082…
    sha256:3353847b14ebfc8fd1fd3…
    
    sha256:1bd9d3b7686a1c61c4d6d…
    sha256:a58c6b717b32ed3061fc6…
    sha256:b50a80767b4a9335fbb11…
    sha256:f82b95de3049e4a2fb6ab…
    sha256:dece7c53616378469e462…
    sha256:2bcb17063a1c100b4a6fd…
    sha256:e1caa81aa0d0cc390b573…
    sha256:3d92e3e7e06dce602725c…
    sha256:c186ba26dc778921dd494…
    sha256:7b193551031047df6cba1…
    sha256:1cf47bc524f71ccca20e9…
    sha256:08c01a0ec47e82ebe2bec…
    sha256:e9444ec2d0d74706e9f21…
    …
    sha256:695bc3e4c4edea7f91799…
    sha256:764d7a91d4d599a243d1a…
    Put Gitpod's tools
    
    on the user's image
    
    to build a workspace
    enables us to ship many
    different things in one OCI
    image.
    registry-facade
    36
    

    View Slide

37. workspace
    
    manager
    kubelet containerd registry-
    facade registry
    workspace
    
    manager
    kubelet containerd registry-
    facade registry
    pull reg.gitpod.io/remote/
    GET /v2/remote/manifests/
    GetImageSpec(instanceId)
    GET /v2/workspace-image/manifest/latest
    GET /v2/workspacekit/manifest/some_version
    GET /v2/docker-up/manifest/some_version
    37
    

    View Slide

38. workspace
    
    manager
    kubelet containerd registry-
    facade registry
    workspace
    
    manager
    kubelet containerd registry-
    facade registry
    pull reg.gitpod.io/remote/
    GET /v2/remote/manifests/
    GetImageSpec(instanceId)
    GET /v2/workspace-image/manifest/latest
    GET /v2/workspacekit/manifest/some_version
    GET /v2/docker-up/manifest/some_version
    gitpod's original component
    38
    

    View Slide

39. workspace
    
    manager
    kubelet containerd registry-
    facade registry
    workspace
    
    manager
    kubelet containerd registry-
    facade registry
    pull reg.gitpod.io/remote/
    GET /v2/remote/manifests/
    GetImageSpec(instanceId)
    GET /v2/workspace-image/manifest/latest
    GET /v2/workspacekit/manifest/some_version
    GET /v2/docker-up/manifest/some_version
    OCI distribution spec
    39
    

    View Slide

40. registry-
    facade registry
    registry-
    facade registry
    …
    IPFS
    IPFS
    containerd
    GET /v2/remote/manifests/
    containerd Redis
    Redis
    GET /2/workspace-image/manifest/latest
    40
    

    View Slide

41. registry-
    facade registry
    registry-
    facade registry
    …
    IPFS
    IPFS
    containerd
    GET /v2/remote/manifests/
    containerd Redis
    Redis
    GET /2/workspace-image/manifest/latest
    Does each layers exists on IPFS?
    Reply the CID if it exists on IPFS
    41
    

    View Slide

42. registry-
    facade registry
    registry-
    facade registry
    …
    IPFS
    IPFS
    containerd
    GET /v2/remote/manifests/
    containerd Redis
    Redis
    GET /2/workspace-image/manifest/latest
    modify the manifest
    
    for IPFS
    Reply the modified manifest
    GET Object from the urls field
    Does each layers exists on IPFS?
    Reply the CID if it exists on IPFS
    42
    

    View Slide

43. Manifest for image not using IPFS
    IPFS urls are added to the urls field
    Pull image once to use IPFS
    43
    

    View Slide

44. Fast Image Pulls Using IPFS And Opportunistic Caching
    @KubeCon US 2022
    Christian Weichel & Manuel de Brito Fontes, Gitpod
    44
    

    View Slide

45. Summary
    04
    45
    

    View Slide

46. Summary
    Thanks! ☑
    ☑
    ☑
    ☑
    ☑
    ☑
    User workspace environments on Kubernetes pod
    namespace - rings
    cgroup - dynamic resource limits
    seccomp notify - mount procfs
    Faster image pulls with IPFS
    contained on IPFS
    OCI Image spec
    Image pull intercept
    46
    

    View Slide

47. Any questions?
    05
    47
    

    View Slide