Slide 1

Slide 1 text

© 2019, Amazon Web Services, Inc. or its Affiliates. © 2020, Amazon Web Services, Inc. or its Affiliates. Samuel Karp Extending containerd Go plugins, gRPC proxies, and more March 6, 2020 at SCALE 18x

Slide 2

Slide 2 text

© 2020, Amazon Web Services, Inc. or its Affiliates. Table of contents • What is containerd? • Core modularity • Extension • Examples!

Slide 3

Slide 3 text

© 2019, Amazon Web Services, Inc. or its Affiliates. © 2020, Amazon Web Services, Inc. or its Affiliates. containerd

Slide 4

Slide 4 text

© 2020, Amazon Web Services, Inc. or its Affiliates. What is containerd? ● Small and focused container runtime ● Built on lessons from Docker ● Strict scope to limit features ● Modular, composable pieces ● Used by Docker

Slide 5

Slide 5 text

© 2020, Amazon Web Services, Inc. or its Affiliates. Container image layers ● A copy-on-write view of files ● New files exist in the top layer ● Modified files are “copied up” ● Unmodified files stay in original layer ● Deleted files are hidden, not removed Top layer (read-write) Intermediate layer (read-only) Base layer (read-only)

Slide 6

Slide 6 text

© 2020, Amazon Web Services, Inc. or its Affiliates. The containerd stack ● gRPC API and Services ● Storage services – Content store – Snapshotters ● Runtime (runc, OCI, v2) gRPC Metrics Storage Content Snapshot Diff Metadata Images Containers Tasks Events Runtimes Runtimes

Slide 7

Slide 7 text

© 2020, Amazon Web Services, Inc. or its Affiliates. The containerd stack ● gRPC API and Services ● Storage services – Content store – Snapshotters ● Runtime (runc, OCI, v2) gRPC Metrics Storage Content Snapshot Diff Metadata Images Containers Tasks Events Runtimes Runtimes

Slide 8

Slide 8 text

© 2020, Amazon Web Services, Inc. or its Affiliates. The containerd stack ● gRPC API and Services ● Storage services – Content store – Snapshotters ● Runtime (runc, OCI, v2) gRPC Metrics Storage Content Snapshot Diff Metadata Images Containers Tasks Events Runtimes Runtimes

Slide 9

Slide 9 text

© 2020, Amazon Web Services, Inc. or its Affiliates. The containerd stack ● gRPC API and Services ● Storage services – Content store – Snapshotters ● Runtime (runc, OCI, v2) gRPC Metrics Storage Content Snapshot Diff Metadata Images Containers Tasks Events Runtimes Runtimes

Slide 10

Slide 10 text

© 2020, Amazon Web Services, Inc. or its Affiliates. The containerd stack ● gRPC API and Services ● Storage services – Content store – Snapshotters ● Runtime (runc, OCI, v2) gRPC Metrics Storage Content Snapshot Diff Metadata Images Containers Tasks Events Runtimes Runtimes

Slide 11

Slide 11 text

© 2020, Amazon Web Services, Inc. or its Affiliates. Core modularity ● Small, separate services ● Use services together for higher-level functionality ● Services modeled with interfaces ● Services are implemented as plugins ● Client library to tie it all together

Slide 12

Slide 12 text

© 2019, Amazon Web Services, Inc. or its Affiliates. © 2020, Amazon Web Services, Inc. or its Affiliates. Extension

Slide 13

Slide 13 text

© 2020, Amazon Web Services, Inc. or its Affiliates. containerd extension points ● Client library extensions ● “CLI”/executable plugins ● gRPC proxy plugins ● Go plugins ● Built-in plugins

Slide 14

Slide 14 text

© 2020, Amazon Web Services, Inc. or its Affiliates. Client library extensions ● “Smart” client in Go provides interfaces ● Write your own implementations when you want something different! ● Requires that you control the client code ● Examples – Pulling images – I/O handling for containers

Slide 15

Slide 15 text

© 2020, Amazon Web Services, Inc. or its Affiliates. Client library extensions – Pulling images ● Pulling images happens in the client library ● Network access and protocol support ● Default implementation is Docker registry ● Examples – Distributed/peer-to-peer protocol like BitTorrent – Other registry protocols like Amazon ECR – Maybe you want to store images in git-lfs? – Anything you can think of!

Slide 16

Slide 16 text

© 2020, Amazon Web Services, Inc. or its Affiliates. Client library extension – default resolver img, err := client.Pull( namespaces.NamespaceFromEnv(ctx), "my.registry/myrepository:mytag", containerd.WithPullUnpack)

Slide 17

Slide 17 text

© 2020, Amazon Web Services, Inc. or its Affiliates. Client library extension – Amazon ECR resolver // import "github.com/awslabs/amazon-ecr-containerd-resolver" resolver, _ := ecr.NewResolver() img, err := client.Pull( namespaces.NamespaceFromEnv(ctx), "ecr.aws/arn:aws:ecr:us-west-2:123456789012:repository/myrepository:mytag", containerd.WithResolver(resolver), containerd.WithPullUnpack)

Slide 18

Slide 18 text

© 2020, Amazon Web Services, Inc. or its Affiliates. Client library extension – Resolver interface type Resolver interface { Resolve(ctx context.Context, ref string) (string, oci.Descriptor, error) Fetcher(ctx context.Context, ref string) (Fetcher, error) Pusher(ctx context.Context, ref string) (Pusher, error) }

Slide 19

Slide 19 text

© 2020, Amazon Web Services, Inc. or its Affiliates. Client library extension – Resolver interface type Resolver interface { Resolve(ctx context.Context, ref string) (string, oci.Descriptor, error) Fetcher(ctx context.Context, ref string) (Fetcher, error) Pusher(ctx context.Context, ref string) (Pusher, error) } type Fetcher interface { Fetch(ctx context.Context, desc oci.Descriptor) (io.ReadCloser, error) }

Slide 20

Slide 20 text

© 2020, Amazon Web Services, Inc. or its Affiliates. Client library extension – Resolver interface type Resolver interface { Resolve(ctx context.Context, ref string) (string, oci.Descriptor, error) Fetcher(ctx context.Context, ref string) (Fetcher, error) Pusher(ctx context.Context, ref string) (Pusher, error) } type Fetcher interface { Fetch(ctx context.Context, desc oci.Descriptor) (io.ReadCloser, error) } type Pusher interface { Push(ctx context.Context, desc oci.Descriptor) (content.Writer, error) }

Slide 21

Slide 21 text

© 2020, Amazon Web Services, Inc. or its Affiliates. “CLI”/executable plugins ● Command-line interface conventions ● Separate program from containerd ● containerd defines semantics for STDIO, flags, working directory, file names, etc ● Examples – Runtimes (OCI and “v2”) – Log forwarding – Stream processing/media transformation

Slide 22

Slide 22 text

© 2020, Amazon Web Services, Inc. or its Affiliates. “CLI”/executable plugins – Runtimes runc firecracker-containerd Default runtime Linux containers Alternative runtime Firecracker microVMs Adheres to OCI standard Adheres to containerd “v2” interface Specification covers: ● command-line arguments/flags ● working directory ● input files ● exit codes Specification covers: ● command-line arguments/flags ● working directory ● input files ● ttrpc on a Unix domain socket ● exit codes

Slide 23

Slide 23 text

© 2020, Amazon Web Services, Inc. or its Affiliates. “CLI”/executable plugins – “v2” runtimes ● Binary prefixes with containerd-shim-foo-bar

Slide 24

Slide 24 text

© 2020, Amazon Web Services, Inc. or its Affiliates. “CLI”/executable plugins – “v2” runtimes ● Binary prefixes with containerd-shim-foo-bar ● Be located within PATH

Slide 25

Slide 25 text

© 2020, Amazon Web Services, Inc. or its Affiliates. “CLI”/executable plugins – “v2” runtimes ● Binary prefixes with containerd-shim-foo-bar ● Be located within PATH ● Define program lifecycle through start and delete arguments

Slide 26

Slide 26 text

© 2020, Amazon Web Services, Inc. or its Affiliates. “CLI”/executable plugins – “v2” runtimes $ containerd-shim-foo-bar start /path/to/socket.sock $ containerd-shim-foo-bar delete

Slide 27

Slide 27 text

© 2020, Amazon Web Services, Inc. or its Affiliates. “CLI”/executable plugins – “v2” runtimes ● Binary prefixes with containerd-shim-foo-bar ● Be located within PATH ● Define program lifecycle through start and delete arguments ● Implement TaskService as a ttrpc service

Slide 28

Slide 28 text

© 2020, Amazon Web Services, Inc. or its Affiliates. “CLI”/executable plugins – “v2” runtimes type TaskService interface { State(context.Context, *StateRequest) (*StateResponse, error) Create(context.Context, *CreateTaskRequest) (*CreateTaskResponse, error) Start(context.Context, *StartRequest) (*StartResponse, error) Delete(context.Context, *DeleteRequest) (*DeleteResponse, error) Pids(context.Context, *PidsRequest) (*PidsResponse, error) Pause(context.Context, *PauseRequest) (*types1.Empty, error) Resume(context.Context, *ResumeRequest) (*types1.Empty, error) Kill(context.Context, *KillRequest) (*types1.Empty, error) Exec(context.Context, *ExecProcessRequest) (*types1.Empty, error) Update(context.Context, *UpdateTaskRequest) (*types1.Empty, error) Wait(context.Context, *WaitRequest) (*WaitResponse, error) … }

Slide 29

Slide 29 text

© 2020, Amazon Web Services, Inc. or its Affiliates. “CLI”/executable plugins – “v2” runtimes ● Binary prefixes with containerd-shim-foo-bar ● Be located within PATH ● Define program lifecycle through start and delete arguments ● Implement TaskService as a ttrpc service ● Can use containerd’s shim helpers

Slide 30

Slide 30 text

© 2020, Amazon Web Services, Inc. or its Affiliates. “CLI”/executable plugins – “v2” runtimes func main() { shim.Run("foo.bar", myShim) } func myShim( ctx context.Context, id string, publisher shim.Publisher, callback func(), ) (shim.Shim, error){ // my implementation here! }

Slide 31

Slide 31 text

© 2020, Amazon Web Services, Inc. or its Affiliates. “CLI”/executable plugins – “v2” runtimes ● Binary prefixes with containerd-shim-foo-bar ● Be located within PATH ● Define program lifecycle through start and delete arguments ● Implement TaskService as a ttrpc service ● Can use containerd’s shim helpers ● sudo ctr run \ --runtime foo.bar \ docker.io/library/hello-world:latest \ my-hello-world-container

Slide 32

Slide 32 text

© 2020, Amazon Web Services, Inc. or its Affiliates. gRPC proxy plugins ● Plugins run as separate processes ● Expose the service API over a Unix domain socket ● containerd acts as a pass-through ● Proxy plugin registered in containerd’s config file ● Snapshot and content services supported as proxy plugins

Slide 33

Slide 33 text

© 2020, Amazon Web Services, Inc. or its Affiliates. gRPC proxy plugins - Snapshotters ● Snapshotters provide image- and container-filesystems ● Many implement a form of copy-on-write ● Several built in to containerd ● Out-of-process gRPC proxy plugins enable new development ● Examples – Block-device snapshotters: devicemapper and lvm – Ongoing discussion about network-based snapshotters

Slide 34

Slide 34 text

© 2020, Amazon Web Services, Inc. or its Affiliates. gRPC proxy plugins - Snapshotters ● Implement Snapshotter as a gRPC service

Slide 35

Slide 35 text

© 2020, Amazon Web Services, Inc. or its Affiliates. gRPC proxy plugins - Snapshotters type Snapshotter interface { Stat(context.Context, string) (Info, error) Update(context.Context, Info, ...string) (Info, error) Usage(context.Context, string) (Usage, error) Mounts(context.Context, string) ([]mount.Mount, error) Prepare(context.Context, string, string, ...Opt) ([]mount.Mount, error) View(context.Context, string, string, ...Opt) ([]mount.Mount, error) Commit(context.Context, string, string, ...Opt) error Remove(context.Context, string) error Walk(context.Context, func(context.Context, Info) error) error Close() error }

Slide 36

Slide 36 text

© 2020, Amazon Web Services, Inc. or its Affiliates. gRPC proxy plugins - Snapshotters ● Implement Snapshotter as a gRPC service ● Registered in containerd configuration

Slide 37

Slide 37 text

© 2020, Amazon Web Services, Inc. or its Affiliates. gRPC proxy plugins - Snapshotters [proxy_plugins] [proxy_plugins.foo-snapshotter] type = "snapshot" Address = "/var/run/foo-snapshotter.sock"

Slide 38

Slide 38 text

© 2020, Amazon Web Services, Inc. or its Affiliates. gRPC proxy plugins - Snapshotters ● Implement Snapshotter as a gRPC service ● Registered in containerd configuration ● sudo ctr run \ --snapshotter foo-snapshotter \ docker.io/library/hello-world:latest \ my-hello-world-container

Slide 39

Slide 39 text

© 2020, Amazon Web Services, Inc. or its Affiliates. Go plugins ● Similar power/flexibility to built-in plugins ● Can add at runtime ● Loaded from containerd’s plugins folder (or configured folder) ● Name includes OS, architecture, and OS-specific extension: myplugin-linux-amd64.so ● Strongly tied to how containerd was built – OS, architecture – Version of Go – Versions of every common package ● You’re responsible for ensuring compatible build environment

Slide 40

Slide 40 text

© 2020, Amazon Web Services, Inc. or its Affiliates. Built-in plugins ● Default plugins are (mostly!) built-in ● In the source tree of containerd ● Can’t add at runtime ● Most powerful/flexible ● Most effort required ● Examples – Default snapshotters – Default content store – Default diff service – Default image service – Default container service – CRI plugin

Slide 41

Slide 41 text

© 2020, Amazon Web Services, Inc. or its Affiliates. Built-in plugins – Build your own ● Build in your own plugins ● ...by building your own containerd binary ● You don’t have to fork containerd! ● ...instead, use containerd as a library! ● You solve your own build environment and distribution ● You’re responsible for keeping up to date

Slide 42

Slide 42 text

© 2020, Amazon Web Services, Inc. or its Affiliates. Built-in plugins – Build your own ● Write your own main() function

Slide 43

Slide 43 text

© 2020, Amazon Web Services, Inc. or its Affiliates. Built-in plugins – Build your own func main() { app := command.App() if err := app.Run(os.Args); err != nil { fmt.Fprintf(os.Stderr, "containerd: %s\n", err) os.Exit(1) } }

Slide 44

Slide 44 text

© 2020, Amazon Web Services, Inc. or its Affiliates. Built-in plugins – Build your own ● Write your own main() function ● import the plugins you want

Slide 45

Slide 45 text

© 2020, Amazon Web Services, Inc. or its Affiliates. Built-in plugins – Build your own import ( // main function "github.com/containerd/containerd/cmd/containerd/command" // builtins, see // https://github.com/containerd/containerd/blob/master/cmd/containerd/builtins.go _ "github.com/containerd/containerd/diff/walking/plugin" _ "github.com/containerd/containerd/gc/scheduler" _ "github.com/containerd/containerd/runtime/restart/monitor" _ "github.com/containerd/containerd/services/containers" _ "github.com/containerd/containerd/services/content" _ "github.com/containerd/containerd/services/diff" _ "github.com/containerd/containerd/services/events" _ "github.com/containerd/containerd/services/healthcheck" _ "github.com/containerd/containerd/services/images" _ "github.com/containerd/containerd/services/introspection" _ "github.com/containerd/containerd/services/leases" _ "github.com/containerd/containerd/services/namespaces"

Slide 46

Slide 46 text

© 2020, Amazon Web Services, Inc. or its Affiliates. Built-in plugins – Build your own _ "github.com/containerd/containerd/services/opt" _ "github.com/containerd/containerd/services/snapshots" _ "github.com/containerd/containerd/services/tasks" _ "github.com/containerd/containerd/services/version" // Linux builtins, see // https://github.com/containerd/containerd/blob/master/cmd/containerd/builtins_linux.go _ "github.com/containerd/containerd/metrics/cgroups" _ "github.com/containerd/containerd/runtime/v1/linux" _ "github.com/containerd/containerd/runtime/v2" _ "github.com/containerd/containerd/runtime/v2/runc/options" // snapshotters _ "github.com/containerd/containerd/snapshots/devmapper" _ "github.com/containerd/containerd/snapshots/overlay" // Your plugin! _ "github.com/foobar/foobar/foobar-api” )

Slide 47

Slide 47 text

© 2020, Amazon Web Services, Inc. or its Affiliates. Built-in plugins – Build your own ● Write your own main() function ● import the plugins you want ● Register your plugin with init()

Slide 48

Slide 48 text

© 2020, Amazon Web Services, Inc. or its Affiliates. Built-in plugins – Build your own func init() { plugin.Register(&plugin.Registration{ Type: plugin.ServicePlugin, ID: "myPlugin.ID", Requires: []plugin.Type{ plugin.MetadataPlugin, }, InitFn: func(ic *plugin.InitContext) (interface{}, error) { // Init your plugin here }, }) }

Slide 49

Slide 49 text

© 2019, Amazon Web Services, Inc. or its Affiliates. © 2020, Amazon Web Services, Inc. or its Affiliates. Demo!

Slide 50

Slide 50 text

© 2020, Amazon Web Services, Inc. or its Affiliates. Demo summary ● Pull image from Amazon ECR with amazon-ecr-containerd-resolver client library extension ● Custom containerd binary with firecracker-control built-in plugin ● devmapper snapshotter (now embedded, former gRPC proxy plugin) ● containerd-shim-aws-firecracker runtime (executable plugin) to run Firecracker microVMs

Slide 51

Slide 51 text

© 2019, Amazon Web Services, Inc. or its Affiliates. © 2020, Amazon Web Services, Inc. or its Affiliates. Q&A Samuel Karp

Slide 52

Slide 52 text

© 2020, Amazon Web Services, Inc. or its Affiliates. A brief note before we finish — Session surveys provide valuable information to speakers Feedback that is very helpful: ● Topics you were excited to learn about ● Suggestions for improving understanding and clarity Feedback that is extremely unhelpful: ● Comments unrelated to talk content (please refer to the SCALE Code of Conduct) The “hallway track” is always open! Feedback and questions welcome – [email protected] or @samuelkarp For support, use the AWS Forums or contact AWS Support

Slide 53

Slide 53 text

© 2019, Amazon Web Services, Inc. or its Affiliates. © 2020, Amazon Web Services, Inc. or its Affiliates. Thank you! Samuel Karp (@samuelkarp)