Extending containerd (SCaLE 18x)

Extending containerd (SCaLE 18x)

containerd, a graduated CNCF project, is a widely used container runtime that provides core functionality for Docker. containerd was designed to be small and simple, but also very modular and extensible. This talk covers the architecture of containerd, explains the responsibilities of each component, and dives deep into containerd’s facility for extension. We’ll cover the individual gRPC services that make up containerd and show how they can be extended with proxy plugins, Go plugins, process interfaces (OCI runtimes and process-based logging), thick client implementations, and build-your-own containerd for compiled-in extension. These extension mechanisms can be shown with simple examples and real-world use in the firecracker-containerd project.

D3f5ebf1d4c147756c86ff6c8a83f4e0?s=128

Samuel Karp

March 06, 2020
Tweet

Transcript

  1. 1.

    © 2019, Amazon Web Services, Inc. or its Affiliates. ©

    2020, Amazon Web Services, Inc. or its Affiliates. Samuel Karp Extending containerd Go plugins, gRPC proxies, and more March 6, 2020 at SCALE 18x
  2. 2.

    © 2020, Amazon Web Services, Inc. or its Affiliates. Table

    of contents • What is containerd? • Core modularity • Extension • Examples!
  3. 3.

    © 2019, Amazon Web Services, Inc. or its Affiliates. ©

    2020, Amazon Web Services, Inc. or its Affiliates. containerd
  4. 4.

    © 2020, Amazon Web Services, Inc. or its Affiliates. What

    is containerd? • Small and focused container runtime • Built on lessons from Docker • Strict scope to limit features • Modular, composable pieces • Used by Docker
  5. 5.

    © 2020, Amazon Web Services, Inc. or its Affiliates. Container

    image layers • A copy-on-write view of files • New files exist in the top layer • Modified files are “copied up” • Unmodified files stay in original layer • Deleted files are hidden, not removed Top layer (read-write) Intermediate layer (read-only) Base layer (read-only)
  6. 6.

    © 2020, Amazon Web Services, Inc. or its Affiliates. The

    containerd stack • gRPC API and Services • Storage services – Content store – Snapshotters • Runtime (runc, OCI, v2) gRPC Metrics Storage Content Snapshot Diff Metadata Images Containers Tasks Events Runtimes Runtimes
  7. 7.

    © 2020, Amazon Web Services, Inc. or its Affiliates. The

    containerd stack • gRPC API and Services • Storage services – Content store – Snapshotters • Runtime (runc, OCI, v2) gRPC Metrics Storage Content Snapshot Diff Metadata Images Containers Tasks Events Runtimes Runtimes
  8. 8.

    © 2020, Amazon Web Services, Inc. or its Affiliates. The

    containerd stack • gRPC API and Services • Storage services – Content store – Snapshotters • Runtime (runc, OCI, v2) gRPC Metrics Storage Content Snapshot Diff Metadata Images Containers Tasks Events Runtimes Runtimes
  9. 9.

    © 2020, Amazon Web Services, Inc. or its Affiliates. The

    containerd stack • gRPC API and Services • Storage services – Content store – Snapshotters • Runtime (runc, OCI, v2) gRPC Metrics Storage Content Snapshot Diff Metadata Images Containers Tasks Events Runtimes Runtimes
  10. 10.

    © 2020, Amazon Web Services, Inc. or its Affiliates. The

    containerd stack • gRPC API and Services • Storage services – Content store – Snapshotters • Runtime (runc, OCI, v2) gRPC Metrics Storage Content Snapshot Diff Metadata Images Containers Tasks Events Runtimes Runtimes
  11. 11.

    © 2020, Amazon Web Services, Inc. or its Affiliates. Core

    modularity • Small, separate services • Use services together for higher-level functionality • Services modeled with interfaces • Services are implemented as plugins • Client library to tie it all together
  12. 12.

    © 2019, Amazon Web Services, Inc. or its Affiliates. ©

    2020, Amazon Web Services, Inc. or its Affiliates. Extension
  13. 13.

    © 2020, Amazon Web Services, Inc. or its Affiliates. containerd

    extension points • Client library extensions • “CLI”/executable plugins • gRPC proxy plugins • Go plugins • Built-in plugins
  14. 14.

    © 2020, Amazon Web Services, Inc. or its Affiliates. Client

    library extensions • “Smart” client in Go provides interfaces • Write your own implementations when you want something different! • Requires that you control the client code • Examples – Pulling images – I/O handling for containers
  15. 15.

    © 2020, Amazon Web Services, Inc. or its Affiliates. Client

    library extensions – Pulling images • Pulling images happens in the client library • Network access and protocol support • Default implementation is Docker registry • Examples – Distributed/peer-to-peer protocol like BitTorrent – Other registry protocols like Amazon ECR – Maybe you want to store images in git-lfs? – Anything you can think of!
  16. 16.

    © 2020, Amazon Web Services, Inc. or its Affiliates. Client

    library extension – default resolver img, err := client.Pull( namespaces.NamespaceFromEnv(ctx), "my.registry/myrepository:mytag", containerd.WithPullUnpack)
  17. 17.

    © 2020, Amazon Web Services, Inc. or its Affiliates. Client

    library extension – Amazon ECR resolver // import "github.com/awslabs/amazon-ecr-containerd-resolver" resolver, _ := ecr.NewResolver() img, err := client.Pull( namespaces.NamespaceFromEnv(ctx), "ecr.aws/arn:aws:ecr:us-west-2:123456789012:repository/myrepository:mytag", containerd.WithResolver(resolver), containerd.WithPullUnpack)
  18. 18.

    © 2020, Amazon Web Services, Inc. or its Affiliates. Client

    library extension – Resolver interface type Resolver interface { Resolve(ctx context.Context, ref string) (string, oci.Descriptor, error) Fetcher(ctx context.Context, ref string) (Fetcher, error) Pusher(ctx context.Context, ref string) (Pusher, error) }
  19. 19.

    © 2020, Amazon Web Services, Inc. or its Affiliates. Client

    library extension – Resolver interface type Resolver interface { Resolve(ctx context.Context, ref string) (string, oci.Descriptor, error) Fetcher(ctx context.Context, ref string) (Fetcher, error) Pusher(ctx context.Context, ref string) (Pusher, error) } type Fetcher interface { Fetch(ctx context.Context, desc oci.Descriptor) (io.ReadCloser, error) }
  20. 20.

    © 2020, Amazon Web Services, Inc. or its Affiliates. Client

    library extension – Resolver interface type Resolver interface { Resolve(ctx context.Context, ref string) (string, oci.Descriptor, error) Fetcher(ctx context.Context, ref string) (Fetcher, error) Pusher(ctx context.Context, ref string) (Pusher, error) } type Fetcher interface { Fetch(ctx context.Context, desc oci.Descriptor) (io.ReadCloser, error) } type Pusher interface { Push(ctx context.Context, desc oci.Descriptor) (content.Writer, error) }
  21. 21.

    © 2020, Amazon Web Services, Inc. or its Affiliates. “CLI”/executable

    plugins • Command-line interface conventions • Separate program from containerd • containerd defines semantics for STDIO, flags, working directory, file names, etc • Examples – Runtimes (OCI and “v2”) – Log forwarding – Stream processing/media transformation
  22. 22.

    © 2020, Amazon Web Services, Inc. or its Affiliates. “CLI”/executable

    plugins – Runtimes runc firecracker-containerd Default runtime Linux containers Alternative runtime Firecracker microVMs Adheres to OCI standard Adheres to containerd “v2” interface Specification covers: • command-line arguments/flags • working directory • input files • exit codes Specification covers: • command-line arguments/flags • working directory • input files • ttrpc on a Unix domain socket • exit codes
  23. 23.

    © 2020, Amazon Web Services, Inc. or its Affiliates. “CLI”/executable

    plugins – “v2” runtimes • Binary prefixes with containerd-shim-foo-bar
  24. 24.

    © 2020, Amazon Web Services, Inc. or its Affiliates. “CLI”/executable

    plugins – “v2” runtimes • Binary prefixes with containerd-shim-foo-bar • Be located within PATH
  25. 25.

    © 2020, Amazon Web Services, Inc. or its Affiliates. “CLI”/executable

    plugins – “v2” runtimes • Binary prefixes with containerd-shim-foo-bar • Be located within PATH • Define program lifecycle through start and delete arguments
  26. 26.

    © 2020, Amazon Web Services, Inc. or its Affiliates. “CLI”/executable

    plugins – “v2” runtimes $ containerd-shim-foo-bar start /path/to/socket.sock $ containerd-shim-foo-bar delete
  27. 27.

    © 2020, Amazon Web Services, Inc. or its Affiliates. “CLI”/executable

    plugins – “v2” runtimes • Binary prefixes with containerd-shim-foo-bar • Be located within PATH • Define program lifecycle through start and delete arguments • Implement TaskService as a ttrpc service
  28. 28.

    © 2020, Amazon Web Services, Inc. or its Affiliates. “CLI”/executable

    plugins – “v2” runtimes type TaskService interface { State(context.Context, *StateRequest) (*StateResponse, error) Create(context.Context, *CreateTaskRequest) (*CreateTaskResponse, error) Start(context.Context, *StartRequest) (*StartResponse, error) Delete(context.Context, *DeleteRequest) (*DeleteResponse, error) Pids(context.Context, *PidsRequest) (*PidsResponse, error) Pause(context.Context, *PauseRequest) (*types1.Empty, error) Resume(context.Context, *ResumeRequest) (*types1.Empty, error) Kill(context.Context, *KillRequest) (*types1.Empty, error) Exec(context.Context, *ExecProcessRequest) (*types1.Empty, error) Update(context.Context, *UpdateTaskRequest) (*types1.Empty, error) Wait(context.Context, *WaitRequest) (*WaitResponse, error) … }
  29. 29.

    © 2020, Amazon Web Services, Inc. or its Affiliates. “CLI”/executable

    plugins – “v2” runtimes • Binary prefixes with containerd-shim-foo-bar • Be located within PATH • Define program lifecycle through start and delete arguments • Implement TaskService as a ttrpc service • Can use containerd’s shim helpers
  30. 30.

    © 2020, Amazon Web Services, Inc. or its Affiliates. “CLI”/executable

    plugins – “v2” runtimes func main() { shim.Run("foo.bar", myShim) } func myShim( ctx context.Context, id string, publisher shim.Publisher, callback func(), ) (shim.Shim, error){ // my implementation here! }
  31. 31.

    © 2020, Amazon Web Services, Inc. or its Affiliates. “CLI”/executable

    plugins – “v2” runtimes • Binary prefixes with containerd-shim-foo-bar • Be located within PATH • Define program lifecycle through start and delete arguments • Implement TaskService as a ttrpc service • Can use containerd’s shim helpers • sudo ctr run \ --runtime foo.bar \ docker.io/library/hello-world:latest \ my-hello-world-container
  32. 32.

    © 2020, Amazon Web Services, Inc. or its Affiliates. gRPC

    proxy plugins • Plugins run as separate processes • Expose the service API over a Unix domain socket • containerd acts as a pass-through • Proxy plugin registered in containerd’s config file • Snapshot and content services supported as proxy plugins
  33. 33.

    © 2020, Amazon Web Services, Inc. or its Affiliates. gRPC

    proxy plugins - Snapshotters • Snapshotters provide image- and container-filesystems • Many implement a form of copy-on-write • Several built in to containerd • Out-of-process gRPC proxy plugins enable new development • Examples – Block-device snapshotters: devicemapper and lvm – Ongoing discussion about network-based snapshotters
  34. 34.

    © 2020, Amazon Web Services, Inc. or its Affiliates. gRPC

    proxy plugins - Snapshotters • Implement Snapshotter as a gRPC service
  35. 35.

    © 2020, Amazon Web Services, Inc. or its Affiliates. gRPC

    proxy plugins - Snapshotters type Snapshotter interface { Stat(context.Context, string) (Info, error) Update(context.Context, Info, ...string) (Info, error) Usage(context.Context, string) (Usage, error) Mounts(context.Context, string) ([]mount.Mount, error) Prepare(context.Context, string, string, ...Opt) ([]mount.Mount, error) View(context.Context, string, string, ...Opt) ([]mount.Mount, error) Commit(context.Context, string, string, ...Opt) error Remove(context.Context, string) error Walk(context.Context, func(context.Context, Info) error) error Close() error }
  36. 36.

    © 2020, Amazon Web Services, Inc. or its Affiliates. gRPC

    proxy plugins - Snapshotters • Implement Snapshotter as a gRPC service • Registered in containerd configuration
  37. 37.

    © 2020, Amazon Web Services, Inc. or its Affiliates. gRPC

    proxy plugins - Snapshotters [proxy_plugins] [proxy_plugins.foo-snapshotter] type = "snapshot" Address = "/var/run/foo-snapshotter.sock"
  38. 38.

    © 2020, Amazon Web Services, Inc. or its Affiliates. gRPC

    proxy plugins - Snapshotters • Implement Snapshotter as a gRPC service • Registered in containerd configuration • sudo ctr run \ --snapshotter foo-snapshotter \ docker.io/library/hello-world:latest \ my-hello-world-container
  39. 39.

    © 2020, Amazon Web Services, Inc. or its Affiliates. Go

    plugins • Similar power/flexibility to built-in plugins • Can add at runtime • Loaded from containerd’s plugins folder (or configured folder) • Name includes OS, architecture, and OS-specific extension: myplugin-linux-amd64.so • Strongly tied to how containerd was built – OS, architecture – Version of Go – Versions of every common package • You’re responsible for ensuring compatible build environment
  40. 40.

    © 2020, Amazon Web Services, Inc. or its Affiliates. Built-in

    plugins • Default plugins are (mostly!) built-in • In the source tree of containerd • Can’t add at runtime • Most powerful/flexible • Most effort required • Examples – Default snapshotters – Default content store – Default diff service – Default image service – Default container service – CRI plugin
  41. 41.

    © 2020, Amazon Web Services, Inc. or its Affiliates. Built-in

    plugins – Build your own • Build in your own plugins • ...by building your own containerd binary • You don’t have to fork containerd! • ...instead, use containerd as a library! • You solve your own build environment and distribution • You’re responsible for keeping up to date
  42. 42.

    © 2020, Amazon Web Services, Inc. or its Affiliates. Built-in

    plugins – Build your own • Write your own main() function
  43. 43.

    © 2020, Amazon Web Services, Inc. or its Affiliates. Built-in

    plugins – Build your own func main() { app := command.App() if err := app.Run(os.Args); err != nil { fmt.Fprintf(os.Stderr, "containerd: %s\n", err) os.Exit(1) } }
  44. 44.

    © 2020, Amazon Web Services, Inc. or its Affiliates. Built-in

    plugins – Build your own • Write your own main() function • import the plugins you want
  45. 45.

    © 2020, Amazon Web Services, Inc. or its Affiliates. Built-in

    plugins – Build your own import ( // main function "github.com/containerd/containerd/cmd/containerd/command" // builtins, see // https://github.com/containerd/containerd/blob/master/cmd/containerd/builtins.go _ "github.com/containerd/containerd/diff/walking/plugin" _ "github.com/containerd/containerd/gc/scheduler" _ "github.com/containerd/containerd/runtime/restart/monitor" _ "github.com/containerd/containerd/services/containers" _ "github.com/containerd/containerd/services/content" _ "github.com/containerd/containerd/services/diff" _ "github.com/containerd/containerd/services/events" _ "github.com/containerd/containerd/services/healthcheck" _ "github.com/containerd/containerd/services/images" _ "github.com/containerd/containerd/services/introspection" _ "github.com/containerd/containerd/services/leases" _ "github.com/containerd/containerd/services/namespaces"
  46. 46.

    © 2020, Amazon Web Services, Inc. or its Affiliates. Built-in

    plugins – Build your own _ "github.com/containerd/containerd/services/opt" _ "github.com/containerd/containerd/services/snapshots" _ "github.com/containerd/containerd/services/tasks" _ "github.com/containerd/containerd/services/version" // Linux builtins, see // https://github.com/containerd/containerd/blob/master/cmd/containerd/builtins_linux.go _ "github.com/containerd/containerd/metrics/cgroups" _ "github.com/containerd/containerd/runtime/v1/linux" _ "github.com/containerd/containerd/runtime/v2" _ "github.com/containerd/containerd/runtime/v2/runc/options" // snapshotters _ "github.com/containerd/containerd/snapshots/devmapper" _ "github.com/containerd/containerd/snapshots/overlay" // Your plugin! _ "github.com/foobar/foobar/foobar-api” )
  47. 47.

    © 2020, Amazon Web Services, Inc. or its Affiliates. Built-in

    plugins – Build your own • Write your own main() function • import the plugins you want • Register your plugin with init()
  48. 48.

    © 2020, Amazon Web Services, Inc. or its Affiliates. Built-in

    plugins – Build your own func init() { plugin.Register(&plugin.Registration{ Type: plugin.ServicePlugin, ID: "myPlugin.ID", Requires: []plugin.Type{ plugin.MetadataPlugin, }, InitFn: func(ic *plugin.InitContext) (interface{}, error) { // Init your plugin here }, }) }
  49. 49.

    © 2019, Amazon Web Services, Inc. or its Affiliates. ©

    2020, Amazon Web Services, Inc. or its Affiliates. Demo!
  50. 50.

    © 2020, Amazon Web Services, Inc. or its Affiliates. Demo

    summary • Pull image from Amazon ECR with amazon-ecr-containerd-resolver client library extension • Custom containerd binary with firecracker-control built-in plugin • devmapper snapshotter (now embedded, former gRPC proxy plugin) • containerd-shim-aws-firecracker runtime (executable plugin) to run Firecracker microVMs
  51. 51.

    © 2019, Amazon Web Services, Inc. or its Affiliates. ©

    2020, Amazon Web Services, Inc. or its Affiliates. Q&A Samuel Karp
  52. 52.

    © 2020, Amazon Web Services, Inc. or its Affiliates. A

    brief note before we finish — Session surveys provide valuable information to speakers Feedback that is very helpful: • Topics you were excited to learn about • Suggestions for improving understanding and clarity Feedback that is extremely unhelpful: • Comments unrelated to talk content (please refer to the SCALE Code of Conduct) The “hallway track” is always open! Feedback and questions welcome – skarp@amazon.com or @samuelkarp For support, use the AWS Forums or contact AWS Support
  53. 53.

    © 2019, Amazon Web Services, Inc. or its Affiliates. ©

    2020, Amazon Web Services, Inc. or its Affiliates. Thank you! Samuel Karp (@samuelkarp)