Slide 1

Slide 1 text

© 2019, Amazon Web Services, Inc. or its Affiliates. © 2019, Amazon Web Services, Inc. or its Affiliates. Samuel Karp and Noah Meyerhans April 28, 2019 at LinuxFest Northwest Deep Dive into firecracker-containerd Running container workloads with Firecracker VMM

Slide 2

Slide 2 text

© 2019, Amazon Web Services, Inc. or its Affiliates. Agenda • Overview of containers • What is a container runtime? What is containerd? • The Firecracker Virtual Machine Monitor (VMM) • Adapting containerd to Firecracker • Demo • Current status and roadmap • Q&A

Slide 3

Slide 3 text

© 2019, Amazon Web Services, Inc. or its Affiliates. A really brief overview of containers... • A mechanism for running software • … with some isolation • … with some repeatability • … with a standard format for distribution • … with common tooling

Slide 4

Slide 4 text

© 2019, Amazon Web Services, Inc. or its Affiliates. Containers enable… • Repeatability for deployment – Fully model local software dependencies – Efficient image-based deployment ● Minimize duplicate storage and network transfer • Expressive modeling of application – Shared network, communicate over loopback – Separate filesystem, bundled dependencies

Slide 5

Slide 5 text

© 2019, Amazon Web Services, Inc. or its Affiliates. Containers ease… • Deployment automation • Separation of purposes • Composability – use multiple containers together

Slide 6

Slide 6 text

© 2019, Amazon Web Services, Inc. or its Affiliates. Linux container primitives • Namespaces – Visibility restrictions • Control groups (cgroups) – Resource limits • Capabilities – Permission restrictions • Seccomp – Syscall allow/deny lists • Linux Security Modules – Resource access control • Union Filesystems

Slide 7

Slide 7 text

© 2019, Amazon Web Services, Inc. or its Affiliates. Containers and VMs Containers ● Use Linux primitives to isolate processes ● Share a Linux kernel ● Fast starts, minimal overhead ● Flexible isolation Virtual Machines ● Virtualize or emulate hardware components ● Completely separate kernels (maybe not Linux!) ● Slower starts, must boot kernel and set up hardware Hardware Linux Kernel namespaces cgroups ... Container Container Hardware Linux Kernel KVM Virt hardware Virt hardware VM Guest VM Guest

Slide 8

Slide 8 text

© 2019, Amazon Web Services, Inc. or its Affiliates. Why would I want a VM then? • Sharing a Linux kernel means sharing the kernel interfaces • Interface between VM and VMM (hypercalls) is defined by the hypervisor • Interface between kernel and hardware is well-understood • Good for defining trust and resource boundaries – Isolating multi-tenant workloads – Isolating non-trusted workloads

Slide 9

Slide 9 text

© 2019, Amazon Web Services, Inc. or its Affiliates. What do we mean by isolation? ● Prevent tenants from affecting one another – Unintentional and malicious ● Threat modeling with STRIDE – Spoofing – Tampering – Repudiation – Information Disclosure – Escalation of privilege ● Container security – seccomp – Linux security modules – capabilities ● Hypervisor

Slide 10

Slide 10 text

© 2019, Amazon Web Services, Inc. or its Affiliates. Common container tooling • Docker UX (docker build, docker run) • Images and registries for software distribution (docker push, docker pull) • Container orchestrators – Amazon ECS – Kubernetes – Mesos • Open Containers Initiative (OCI) – Image standard – Runtime standard

Slide 11

Slide 11 text

© 2019, Amazon Web Services, Inc. or its Affiliates. © 2019, Amazon Web Services, Inc. or its Affiliates. Container runtimes What are they? What is containerd?

Slide 12

Slide 12 text

© 2019, Amazon Web Services, Inc. or its Affiliates. How you run containers Cluster orchestrator Amazon ECS, Kubernetes, Mesos Local orchestrator Docker Local management containerd Container runtime runc or Firecracker

Slide 13

Slide 13 text

© 2019, Amazon Web Services, Inc. or its Affiliates. Container runtimes Container runtimes ● Mechanism for starting and managing container workloads ● (Linux containers) Set up cgroups, namespaces, filesystems, capabilities, etc ● OCI Runtime specification – CLI for setting up a container – On-disk “bundle” ● Root filesystem ● JSON file describing configuration ● runc – Reference implementation – Donated by Docker

Slide 14

Slide 14 text

© 2019, Amazon Web Services, Inc. or its Affiliates. containerd containerd ● Daemon for managing containers ● Modular framework for container lifecycle workflows ● Integrates with OCI runtimes and containerd v2 runtimes

Slide 15

Slide 15 text

© 2019, Amazon Web Services, Inc. or its Affiliates. The containerd stack ● gRPC API and Services ● Storage services – Content store – Snapshotters ● Runtime (runc, OCI, v2) ● In-process plugins ● Out-of-process gRPC plugins gRPC Metrics Storage Content Snapshot Diff Metadata Images Containers Tasks Events OS Runtimes Runtimes

Slide 16

Slide 16 text

© 2019, Amazon Web Services, Inc. or its Affiliates. The containerd stack ● gRPC API and Services ● Storage services – Content store – Snapshotters ● Runtime (runc, OCI, v2) ● In-process plugins ● Out-of-process gRPC plugins gRPC Metrics Storage Content Snapshot Diff Metadata Images Containers Tasks Events OS Runtimes Runtimes

Slide 17

Slide 17 text

© 2019, Amazon Web Services, Inc. or its Affiliates. The containerd stack ● gRPC API and Services ● Storage services – Content store – Snapshotters ● Runtime (runc, OCI, v2) ● In-process plugins ● Out-of-process gRPC plugins gRPC Metrics Storage Content Snapshot Diff Metadata Images Containers Tasks Events OS Runtimes Runtimes

Slide 18

Slide 18 text

© 2019, Amazon Web Services, Inc. or its Affiliates. The containerd stack ● gRPC API and Services ● Storage services – Content store – Snapshotters ● Runtime (runc, OCI, v2) ● In-process plugins ● Out-of-process gRPC plugins gRPC Metrics Storage Content Snapshot Diff Metadata Images Containers Tasks Events OS Runtimes Runtimes

Slide 19

Slide 19 text

© 2019, Amazon Web Services, Inc. or its Affiliates. © 2019, Amazon Web Services, Inc. or its Affiliates. Firecracker

Slide 20

Slide 20 text

© 2019, Amazon Web Services, Inc. or its Affiliates. Firecracker Virtual Machine Monitor (VMM) ● New KVM-based VMM in Rust ● Targeted at Lambda and Fargate workloads ● Small, limited device model ● Small, limited set of features ● Very fast boot ● Containers, but with hypervisor- provided isolation

Slide 21

Slide 21 text

© 2019, Amazon Web Services, Inc. or its Affiliates. Firecracker design philosophy Security ● Very limited device model ● Very limited feature set ● Eliminate guest interaction with host kernel ● Prohibit VMM syscalls ● Memory-safe programming language ● Single VM per Firecracker process Efficiency ● Fast boot time ● Low memory and CPU overhead ● API driven

Slide 22

Slide 22 text

© 2019, Amazon Web Services, Inc. or its Affiliates. firecracker-containerd goals Containers ● Compatible images ● Familiar tooling ● Support existing workflows ● Allow composable containers ● Integrate with orchestrators ● Minimal additional overhead Security ● Hypervisor-based isolation ● Limited access to the host

Slide 23

Slide 23 text

© 2019, Amazon Web Services, Inc. or its Affiliates. How do we make Firecracker like a container? Firecracker VMM considerations ● No filesystem sharing ● No hot device attachments ● Limited networking options (tap) ● Cross-boundary communication with vsock Adapting to containerd ● Block-device snapshotter ● API to manage VMM lifecycle ● Split the “shim” into two parts ● “Runtime” on the host ● “Agent” inside the VM ● ...that runs containers via runc ● Network ● Tap device ● usable with Container Network Interface (CNI) plugins

Slide 24

Slide 24 text

© 2019, Amazon Web Services, Inc. or its Affiliates. firecracker-containerd architecture microVM microVM Containerd Containerd FC Snapshotter FC Snapshotter Container Container Internal FC Agent Internal FC Agent runc runc Content Store Content Store FC Control Plugin FC Control Plugin VM Disk Image VM Disk Image Kernel Image Kernel Image Firecracker VMM Firecracker VMM FC Runtime FC Runtime

Slide 25

Slide 25 text

© 2019, Amazon Web Services, Inc. or its Affiliates. firecracker-containerd architecture microVM microVM Containerd Containerd FC Snapshotter FC Snapshotter Container Container Internal FC Agent Internal FC Agent runc runc Content Store Content Store FC Control Plugin FC Control Plugin VM Disk Image VM Disk Image Kernel Image Kernel Image Firecracker VMM Firecracker VMM FC Runtime FC Runtime

Slide 26

Slide 26 text

© 2019, Amazon Web Services, Inc. or its Affiliates. What is a “block device” snapshotter? ● VM treats the snapshots as devices, not as filesystems or directories ● Inside the VM, we mount the device to expose its filesystem ● Device behaviors and assumptions, including exclusive access

Slide 27

Slide 27 text

© 2019, Amazon Web Services, Inc. or its Affiliates. Design considerations and tradeoffs ● “Naive” – flat files and ahead-of-time copying ● “devicemapper” – like Docker’s “devicemapper” storage driver with thin provisioning (copy-on-write) ● “LVM” – sparse images with LVM (copy-on-write) ● “rawblock” – reflink for copy-on-write

Slide 28

Slide 28 text

© 2019, Amazon Web Services, Inc. or its Affiliates. firecracker-containerd architecture microVM microVM Containerd Containerd FC Snapshotter FC Snapshotter Container Container Internal FC Agent Internal FC Agent runc runc Content Store Content Store FC Control Plugin FC Control Plugin VM Disk Image VM Disk Image Kernel Image Kernel Image Firecracker VMM Firecracker VMM FC Runtime FC Runtime

Slide 29

Slide 29 text

© 2019, Amazon Web Services, Inc. or its Affiliates. What’s this “firecracker-control” plugin? ● First-class VM construct and API ● Specify VM-related parameters like the kernel and VM root filesystem ● Allocate and manage VM resources: block devices, network interfaces, etc ● Manages the VM lifecycle ● Compiled-in plugin ● gRPC API over the same socket ● Specific to Firecracker for now ● Hopefully leading to a more- generic “sandbox” API

Slide 30

Slide 30 text

© 2019, Amazon Web Services, Inc. or its Affiliates. firecracker-containerd architecture microVM microVM Containerd Containerd FC Snapshotter FC Snapshotter Container Container Internal FC Agent Internal FC Agent runc runc Content Store Content Store FC Control Plugin FC Control Plugin VM Disk Image VM Disk Image Kernel Image Kernel Image Firecracker VMM Firecracker VMM FC Runtime FC Runtime

Slide 31

Slide 31 text

© 2019, Amazon Web Services, Inc. or its Affiliates. What does the “runtime” do? ● Proxies container management commands to agent inside VM ● Proxies I/O streams ● Proxies events and metrics from inside the VM back out to containerd ● containerd’s V2 API ● gRPC API for communication ● Extended firecracker-containerd structs ● (Planned) Many containers per VM, but one runtime

Slide 32

Slide 32 text

© 2019, Amazon Web Services, Inc. or its Affiliates. firecracker-containerd architecture microVM microVM Containerd Containerd FC Snapshotter FC Snapshotter Container Container Internal FC Agent Internal FC Agent runc runc Content Store Content Store FC Control Plugin FC Control Plugin VM Disk Image VM Disk Image Kernel Image Kernel Image Firecracker VMM Firecracker VMM FC Runtime FC Runtime

Slide 33

Slide 33 text

© 2019, Amazon Web Services, Inc. or its Affiliates. What about the “agent”? ● Manages the lifecycle of the containers inside the VM ● Communicates over vsock ● Receives container management commands from the runtime ● Proxies I/O streams ● Proxies events and metrics from inside the VM back out to the runtime ● Associate each container with the appropriate block device ● Mounts container filesystems ● Uses runc to set up cgroups, namespaces, etc – Looks like a traditional container to the workload inside

Slide 34

Slide 34 text

© 2019, Amazon Web Services, Inc. or its Affiliates. How we run containers with firecracker-containerd orchestrator containerd snapshotter control plugin Firecracker runtime agent for each container container process for each container

Slide 35

Slide 35 text

© 2019, Amazon Web Services, Inc. or its Affiliates. How we run containers with firecracker-containerd orchestrator containerd snapshotter control plugin Firecracker runtime agent for each container container process for each container

Slide 36

Slide 36 text

© 2019, Amazon Web Services, Inc. or its Affiliates. How we run containers with firecracker-containerd orchestrator containerd snapshotter control plugin Firecracker runtime agent for each container prepare snapshot container process for each container prepare snapshot

Slide 37

Slide 37 text

© 2019, Amazon Web Services, Inc. or its Affiliates. How we run containers with firecracker-containerd orchestrator containerd snapshotter control plugin Firecracker runtime agent for each container prepare snapshot snapshot $fooN container process for each container prepare snapshot snapshot $fooN

Slide 38

Slide 38 text

© 2019, Amazon Web Services, Inc. or its Affiliates. How we run containers with firecracker-containerd orchestrator containerd snapshotter control plugin Firecracker runtime agent for each container prepare snapshot snapshot $fooN create VM launch & boot container process for each container prepare snapshot snapshot $fooN

Slide 39

Slide 39 text

© 2019, Amazon Web Services, Inc. or its Affiliates. How we run containers with firecracker-containerd orchestrator containerd snapshotter control plugin Firecracker runtime agent for each container prepare snapshot snapshot $fooN create VM launch & boot VM $id container process for each container prepare snapshot snapshot $fooN

Slide 40

Slide 40 text

© 2019, Amazon Web Services, Inc. or its Affiliates. How we run containers with firecracker-containerd orchestrator containerd snapshotter control plugin Firecracker runtime agent for each container prepare snapshot snapshot $fooN create VM launch & boot VM $id container process for each container prepare snapshot snapshot $fooN

Slide 41

Slide 41 text

© 2019, Amazon Web Services, Inc. or its Affiliates. How we run containers with firecracker-containerd orchestrator containerd snapshotter control plugin Firecracker runtime agent for each container prepare snapshot snapshot $fooN create VM launch & boot VM $id run container $barN with snapshot $fooN container process for each container run container $barN with snapshot $fooN prepare snapshot snapshot $fooN

Slide 42

Slide 42 text

© 2019, Amazon Web Services, Inc. or its Affiliates. How we run containers with firecracker-containerd orchestrator containerd snapshotter control plugin Firecracker runtime agent for each container prepare snapshot snapshot $fooN create VM launch & boot VM $id run container $barN with snapshot $fooN attach $fooN container process for each container run container $barN with snapshot $fooN prepare snapshot snapshot $fooN

Slide 43

Slide 43 text

© 2019, Amazon Web Services, Inc. or its Affiliates. How we run containers with firecracker-containerd orchestrator containerd snapshotter control plugin Firecracker runtime agent for each container prepare snapshot snapshot $fooN create VM launch & boot VM $id run container $barN with snapshot $fooN attach $fooN mount $fooN container process for each container run container $barN with snapshot $fooN prepare snapshot snapshot $fooN

Slide 44

Slide 44 text

© 2019, Amazon Web Services, Inc. or its Affiliates. How we run containers with firecracker-containerd orchestrator containerd snapshotter control plugin Firecracker runtime agent for each container prepare snapshot snapshot $fooN create VM launch & boot VM $id run container $barN with snapshot $fooN attach $fooN mount $fooN container process run container $barN start process for each container run container $barN with snapshot $fooN prepare snapshot snapshot $fooN

Slide 45

Slide 45 text

© 2019, Amazon Web Services, Inc. or its Affiliates. How we run containers with firecracker-containerd orchestrator containerd snapshotter control plugin Firecracker runtime agent for each container prepare snapshot snapshot $fooN create VM launch & boot VM $id run container $barN with snapshot $fooN attach $fooN mount $fooN container process run container $barN start process container $barN running container $barN running for each container run container $barN with snapshot $fooN prepare snapshot snapshot $fooN

Slide 46

Slide 46 text

© 2019, Amazon Web Services, Inc. or its Affiliates. How we run containers with firecracker-containerd orchestrator containerd snapshotter control plugin Firecracker runtime agent for each container prepare snapshot snapshot $fooN create VM launch & boot VM $id run container $barN with snapshot $fooN attach $fooN mount $fooN container process run container $barN start process subscribe events for $barN container $barN running container $barN running for each container run container $barN with snapshot $fooN prepare snapshot snapshot $fooN

Slide 47

Slide 47 text

© 2019, Amazon Web Services, Inc. or its Affiliates. How we run containers with firecracker-containerd orchestrator containerd snapshotter control plugin Firecracker runtime agent for each container prepare snapshot snapshot $fooN create VM launch & boot VM $id run container $barN with snapshot $fooN attach $fooN mount $fooN container process run container $barN start process After some time, container process exits subscribe events for $barN container $barN running container $barN running for each container run container $barN with snapshot $fooN prepare snapshot snapshot $fooN

Slide 48

Slide 48 text

© 2019, Amazon Web Services, Inc. or its Affiliates. How we run containers with firecracker-containerd orchestrator containerd snapshotter control plugin Firecracker runtime agent for each container prepare snapshot snapshot $fooN create VM launch & boot VM $id run container $barN with snapshot $fooN attach $fooN mount $fooN container process run container $barN start process After some time, container process exits subscribe events for $barN container $barN running container $barN running container $barN exited container $barN exited container $barN exited for each container run container $barN with snapshot $fooN prepare snapshot snapshot $fooN

Slide 49

Slide 49 text

© 2019, Amazon Web Services, Inc. or its Affiliates. How we run containers with firecracker-containerd orchestrator containerd snapshotter control plugin Firecracker runtime agent for each container prepare snapshot snapshot $fooN create VM launch & boot VM $id run container $barN with snapshot $fooN attach $fooN mount $fooN container process run container $barN start process After some time, container process exits subscribe events for $barN container $barN running container $barN running container $barN exited container $barN exited container $barN exited for each container stop VM $id stop VM run container $barN with snapshot $fooN prepare snapshot snapshot $fooN

Slide 50

Slide 50 text

© 2019, Amazon Web Services, Inc. or its Affiliates. How we run containers with firecracker-containerd orchestrator containerd snapshotter control plugin Firecracker runtime agent for each container prepare snapshot snapshot $fooN create VM launch & boot VM $id run container $barN with snapshot $fooN attach $fooN mount $fooN container process run container $barN start process After some time, container process exits subscribe events for $barN container $barN running container $barN running container $barN exited container $barN exited container $barN exited for each container stop VM $id stop VM run container $barN with snapshot $fooN prepare snapshot snapshot $fooN

Slide 51

Slide 51 text

© 2019, Amazon Web Services, Inc. or its Affiliates. © 2019, Amazon Web Services, Inc. or its Affiliates. Demo

Slide 52

Slide 52 text

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. $ sudo ctr run --tty --env CPU_THREADS=2 --env IO_THREADS=4 docker.io/nmeyerhans/stress:latest $(uuid)

Slide 53

Slide 53 text

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. $ sudo ctr run --tty --env CPU_THREADS=2 --env IO_THREADS=4 docker.io/nmeyerhans/stress:latest $(uuid) $ pgrep stress 2709 2710 2711 2712 2713 2714 2715

Slide 54

Slide 54 text

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. $ sudo ctr run --tty --env CPU_THREADS=2 --env IO_THREADS=4 docker.io/nmeyerhans/stress:latest $(uuid) $ pgrep stress 2709 2710 2711 2712 2713 2714 2715 $ pstree -pA -sS 2709 systemd(1)---containerd(1350)---containerd-shim(2666)---entrypoint(2685)---stress(2709)-+-stress(2710) |-stress(2711) |-stress(2712) |-stress(2713) |-stress(2714) `-stress(2715)

Slide 55

Slide 55 text

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. $ sudo ctr run \ --tty \ --env CPU_THREADS=2 \ --env IO_THREADS=4 \ --runtime aws.firecracker \ --snapshotter firecracker-naive \ docker.io/nmeyerhans/stress:latest $(uuid)

Slide 56

Slide 56 text

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. $ sudo ctr run \ --tty \ --env CPU_THREADS=2 \ --env IO_THREADS=4 \ --runtime aws.firecracker \ --snapshotter firecracker-naive \ docker.io/nmeyerhans/stress:latest $(uuid) $ pgrep stress $ pgrep firecracker 2501

Slide 57

Slide 57 text

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. $ sudo ctr run \ --tty \ --env CPU_THREADS=2 \ --env IO_THREADS=4 \ --runtime aws.firecracker \ --snapshotter firecracker-naive \ docker.io/nmeyerhans/stress:latest $(uuid) $ pgrep stress $ pgrep firecracker 2501 $ pstree -pA -sS 2501 systemd(1)---containerd-shim(2492)---firecracker(2501)-+-{fc_vcpu0}(2507) |-{fc_vcpu1}(2508) `-{fc_vmm}(2504)

Slide 58

Slide 58 text

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. $ sudo ctr run \ --tty \ --env CPU_THREADS=2 \ --env IO_THREADS=4 \ --runtime aws.firecracker \ --snapshotter firecracker-naive \ docker.io/nmeyerhans/stress:latest $(uuid) $ pgrep stress $ pgrep firecracker 2501 $ pstree -pA -sS 2501 systemd(1)---containerd-shim(2492)---firecracker(2501)-+-{fc_vcpu0}(2507) |-{fc_vcpu1}(2508) `-{fc_vmm}(2504) $ top -b -n1 | head -n40 top - 22:29:55 up 3:40, 5 users, load average: 2.51, 3.08, 2.98 Tasks: 659 total, 1 running, 328 sleeping, 0 stopped, 0 zombie %Cpu(s): 2.1 us, 0.2 sy, 0.0 ni, 97.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem : 52825036+total, 51928140+free, 867300 used, 8101676 buff/cache KiB Swap: 0 total, 0 free, 0 used. 52430393+avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 2787 root 20 0 270876 112660 112540 S 216.7 0.0 8:22.97 firecracker 515 root 20 0 0 0 0 I 5.6 0.0 0:03.58 kworker/56:1-ev 2838 admin 20 0 45400 4136 3076 R 5.6 0.0 0:00.03 top

Slide 59

Slide 59 text

© 2019, Amazon Web Services, Inc. or its Affiliates. Current status ● Everything is a prototype ● ...but it works! ● Runtime agent communication is evolving ↔ – Better vsock support in Firecracker ● Reliable identification for block devices ● firecracker-control VM API ● Next up: – Multiple containers per VM – Exec

Slide 60

Slide 60 text

© 2019, Amazon Web Services, Inc. or its Affiliates. An API for groups of containers • Required by CRI and implemented internally in cri-containerd • We want this for firecracker-containerd too!

Slide 61

Slide 61 text

© 2019, Amazon Web Services, Inc. or its Affiliates. Longer term ● Still things we’re not sure how to handle ● Container Runtime Interface (CRI) conformance for Kubernetes – Dynamic workload changes (new pods) – Host-based filesystem sharing (volumes) ● Networking with CNI

Slide 62

Slide 62 text

© 2019, Amazon Web Services, Inc. or its Affiliates. How to get involved ● GitHub: https://github.com/firecracker-microvm/firecracker-containerd ● Slack: https://tinyurl.com/firecracker-microvm ● … or come work with us!

Slide 63

Slide 63 text

© 2019, Amazon Web Services, Inc. or its Affiliates. © 2019, Amazon Web Services, Inc. or its Affiliates. Q&A More questions? Contact Sam: [email protected] or @samuelkarp on Twitter Contact Noah: [email protected] or @nmeyerhans on Twitter

Slide 64

Slide 64 text

© 2019, Amazon Web Services, Inc. or its Affiliates. A brief note before we finish — Session surveys provide valuable information to speakers Feedback that is very helpful: • Topics you were excited to learn about • Suggestions for improving understanding and clarity Feedback that is extremely unhelpful: • Comments unrelated to talk content (please refer to the LinuxFest Northwest Code of Conduct) The “hallway track” is always open! Feedback and questions welcome ([email protected], @samuelkarp, [email protected], @nmeyerhans) For support, use the AWS Forums or contact AWS Support

Slide 65

Slide 65 text

© 2019, Amazon Web Services, Inc. or its Affiliates. © 2019, Amazon Web Services, Inc. or its Affiliates. Thank you!