firecracker-containerd and SOCI Snapshotter

Transcript

1. © 2022, Amazon Web Services, Inc. or its affiliates.
   © 2022, Amazon Web Services, Inc. or its affiliates.
   firecracker-containerd and
   SOCI Snapshotter
   Kazuyoshi Kato (he/him)
   Sr. Software Development Engineer
   Amazon Web Services
   

   View Slide

2. © 2022, Amazon Web Services, Inc. or its affiliates.
   Linux container primitives
   • Namespaces – Visibility restrictions
   • Control groups (cgroups) – Resource limits
   • Capabilities – Permission rules
   • Seccomp – Syscall allow/deny lists
   2
   

   View Slide

3. © 2022, Amazon Web Services, Inc. or its affiliates.
   Is this secure enough?
   3
   Your app Your sidecar Malicious app
   Linux kernel
   Container Container Container
   

   View Slide

4. © 2022, Amazon Web Services, Inc. or its affiliates.
   runc CVE-2019-5736
   • A malicious actor could overwrite the host runc binary through
   /proc/self/exe
   • https://aws.amazon.com/blogs/compute/anatomy-of-cve-2019-
   5736-a-runc-container-escape/
   4
   

   View Slide

5. © 2022, Amazon Web Services, Inc. or its affiliates.
   Firecracker
   • “Firecracker is an open source virtualization technology that is
   purpose-built for creating and managing secure, multi-tenant
   container and function-based services”
   • Open-source virtual machine monitor written in Rust
   • Utilizes hardware-assisted virtualization through Linux’s KVM
   • Minimalistic design to support only ”serverless” workloads
   • Not aware about Linux containers
   5
   

   View Slide

6. © 2022, Amazon Web Services, Inc. or its affiliates.
   Running containers with Firecracker
   6
   Your app Your sidecar Malicious app
   Linux kernel Linux kernel
   Firecracker
   Firecracker
   Linux kernel
   Container Container Container
   

   View Slide

7. © 2022, Amazon Web Services, Inc. or its affiliates.
   Firecracker + containerd = firecracker-containerd
   • Secure isolation through Firecracker’s virtualization
   • Convenience and familiarity of containers from containerd
   • https://github.com/firecracker-microvm/firecracker-containerd
   7
   

   View Slide

8. © 2022, Amazon Web Services, Inc. or its affiliates.
   firecracker-containerd implementation
   8
   fc-control
   (plugin)
   containerd-shim-
   aws-firecracker
   (runtime/shim)
   agent
   runc
   container
   client
   (e.g. ctr)
   firecracker-containerd
   (daemon)
   containerd
   Firecracker microVM
   

   View Slide

9. © 2022, Amazon Web Services, Inc. or its affiliates.
   Working with containerd community
   • Two maintainers from Amazon Web Services
   • OpenTelemetry tracing support
   • Device Mapper Snapshotter
   9
   

   View Slide

10. © 2022, Amazon Web Services, Inc. or its affiliates.
    What are snapshotters?
    • Snapshotter converts container images to filesystems
    • “Graph Driver” in Docker Engine
    • overlay (default)
    • devmapper (used by firecracker-containerd)
    • btrfs, aufs, zfs, …
    10
    

    View Slide

11. © 2022, Amazon Web Services, Inc. or its affiliates.
    Lazy-loading snapshotters
    • Stargz Snapshotter
    • Nydus Snapshotter
    • Downloading a container image and assembling the filesystem
    from the image is time-consuming
    • Containers don’t need the all files on the images to start doing
    useful work
    11
    

    View Slide

12. © 2022, Amazon Web Services, Inc. or its affiliates.
    Lazy-loading without conversion
    • Explicitly converting images and managing them is cumbersome
    • Implicitly converting images have negative security implications
    • For example, image signing wouldn’t work if AWS implicitly
    converts images
    12
    

    View Slide

13. © 2022, Amazon Web Services, Inc. or its affiliates.
    SOCI Snapshotter
    • SOCI Snapshotter is a new lazy-loading snapshotter, based on
    Stargz Snapshotter
    • Utilizes FUSE and HTTP’s ranged GET
    • No image conversion
    • Workload-specific load order optimization
    • https://github.com/awslabs/soci-snapshotter
    13
    

    View Slide

14. © 2022, Amazon Web Services, Inc. or its affiliates.
    SOCI: Seekable OCI
    14
    OCI Image
    Layer 1
    Layer 2
    Layer 3
    SOCI Index
    zTOC 1
    zTOC 2
    zTOC 3
    

    View Slide

15. © 2022, Amazon Web Services, Inc. or its affiliates.
    zTOC
    15
    Checkpoint M
    /bin/ls TOC entry
    Compressed Span
    M
    zTOC N Compressed Layer N Uncompressed Span M
    /bin/ls data
    

    View Slide

16. © 2022, Amazon Web Services, Inc. or its affiliates.
    Workload-specific load order optimization
    • The list of to-be-prefetched files wouldn’t be 1:1 to container
    images
    • Base images (e.g. Python 3) would have multiple possible prefetch
    lists, depending on upper application layers
    • The lists could be more dynamic than container images
    themselves.
    16
    

    View Slide

17. © 2022, Amazon Web Services, Inc. or its affiliates.
    Workload-specific load order optimization
    17
    Python 3
    ML training application Web application
    Load order
    document
    Load order
    document
    Python 3
    Debian Bullseye Debian Bullseye
    

    View Slide

18. © 2022, Amazon Web Services, Inc. or its affiliates.
    What’s next?
    • Better code sharing with Stargz Snapshotter
    • Support OCI Reference Types instead of ORAS
    • Finalize SOCI Index and zTOC format
    (e.g. getting rid of encoding/gob)
    • Load order optimization
    • https://github.com/awslabs/soci-snapshotter
    18
    

    View Slide

19. © 2022, Amazon Web Services, Inc. or its affiliates.
    Thank you!
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Kazuyoshi Kato
    @kzys
    katokazu@amazon.com
    

    View Slide