$30 off During Our Annual Pro Sale. View Details »

In a World of Ephemeral Containers, How Do We Keep Track of Things? @ OpenStack Summit Tokyo

Ian Lewis
October 29, 2015

In a World of Ephemeral Containers, How Do We Keep Track of Things? @ OpenStack Summit Tokyo

Ian Lewis

October 29, 2015

More Decks by Ian Lewis

Other Decks in Technology


  1. In a World of Ephemeral Containers, How Do We Keep

    Track of Things? Ian Lewis, Developer Advocate, Google Cloud Platform
  2. Ian Lewis Developer Advocate Google Cloud Platform google.com/+IanLewis-hoge @IanMLewis

  3. Clusters of ephemeral containers Keeping track of things (state!) Concrete

    example Secret bonus? 1 2 Agenda 3 4
  4. New Options and Tradeoffs no silver bullet

  5. Ephemeral containers immutability, storing state, container clusters, Kubernetes

  6. Moar Containers!

  7. Isolation: Keep jobs from interfering with each other Scheduling: Where

    should my job be run? Lifecycle: Keep my job running Discovery: Where is my job now? Constituency: Who is part of my job? Scale-up: Making my jobs bigger or smaller Auth{n,z}: Who can do things to my job? Monitoring: What’s happening with my job? Health: How is my job feeling? Now that we have containers...
  8. None
  9. Greek for “Helmsman”; also the root of the word “Governor”

    • Container orchestrator • Runs Docker containers • Supports multiple cloud and bare- metal environments • Inspired and informed by Google’s experiences and internal systems • Open source, written in Go Manage applications, not machines Kubernetes
  10. None
  11. A small group of tightly coupled containers Example: static site

    generator & web server A loop that drives current state towards desired state Example: replication controller A set of running pods that work together Example: load- balanced backends Arbitrary metadata to organize components Example: phase=production role=frontend Pod Replication Controller Service Labels
  12. declarative cluster management

  13. demo

  14. where's the state? (scaling stateless containers)

  15. None
  16. containers vs. images

  17. • An image is a stack of Read-Only file system

    layers. • Usual process: ◦ build ◦ push to repository ◦ pull to execution host ◦ start container from image Images Debian App PHP & Apache Libs
  18. Implemented on top of a number of (unrelated) Linux APIs:

    cgroups: Restrict resources a process can consume CPU, memory, disk IO, ... namespaces: Change a process’s view of the system Network interfaces, PIDs, users, mounts, ... capabilities: Limits what a user can do mount, kill, chown, etc They're processes! Containers
  19. Containers Read / Write Read / Write Debian App PHP

    & Apache Libs Read / Write • A container is a process ◦ started with kernel restrictions ◦ a stack of shared Read-Only file system layers ◦ plus a process specific Read- Write layer • Every new container gets a new Read-Write layer. All containers from the same image start from exactly the same state!
  20. • It's possible to mount host directories into a container's

    filesystem. • These are mutable and do outlive the container. • They're only available on that host. Mounting host directories Debian App PHP & Apache Libs Read / Write host dir
  21. in a cluster of ephemeral containers application state must exist

    outside of the container
  22. • Networked storage for the cluster • Exposed as a

    file system • These are mutable and do outlive the container. • Independent of host node! • Plugins: GCEPersistentDisk, AWSElasticBlockStore, NFS, iSCSI, RBD (Ceph Block Device), Glusterfs Kubernetes PersistentVolumes Debian App PHP & Apache Libs Read / Write
  23. Patterns for Keeping Track of Things

  24. Run software which stores state outside of the cluster. Manage

    it however you like. Connect to it over the network. ex: MySQL managed by DBAs or managed cloud services Patterns Most software expects access to filesystem state across process invocations. Use cluster tools to provide that across the whole cluster. ex: run a MySQL on a filesystem provided by the cluster outside the cluster adapt to run in the cluster cluster native Use software designed for running in clusters - where nodes come and go. Distributed systems. ex: run Cassandra or Riak inside the cluster
  25. Outside the cluster

  26. Aadapt to run in the cluster Volume

  27. Cluster native

  28. Cluster native Volume Volume Volume Volume Volume

  29. Concrete Example of Tradeoffs

  30. Determine your apps' data needs

  31. Outside the cluster - run MySQL the usual ways Manage

    it yourself • Depends on skills and availability of your admins • May already exist! Subscribe to a managed MySQL service • Automation of backups, replication failover, etc • Limited flexibility
  32. Adapt to run in the cluster - MySQL in the

    cluster Volume • MySQL running in the cluster with data stored in a PersistentVolume. • Data files available regardless of host node • Can restart process on any node (with a short downtime) • Test your workload!
  33. Cluster native - MySQL on Vitess • Open source MySQL

    scaling solution from YouTube • replication, sharding, caching and more • Designed for a distributed, containerized world • Kubernetes configs included • http://vitess.io/ Volume Volume Volume Volume Volume Vitess
  34. figure out your apps' data needs, then consider these patterns

  35. Kubernetes Service abstraction

  36. (you'll probably end up with multiple data services)

  37. Configuration and Secrets Configuration is Also State

  38. Using the same image for *almost* the exact same thing?

    Often need changes based on context: ◦ dev, test, prod ◦ GCP, AWS, your servers ◦ feature flags ◦ logging config ◦ etc, etc... Environment variables! Configuration
  39. what about passwords?

  40. secret.yaml apiVersion: "v1" kind: "Secret" id: "mysql-pw" metadata: name: "mysql-pw"

    data: password: "bXlzZWNyZXRwYXNzd29yZA==" Secrets pod.yaml <snip> volumes: - name: "password_volume" secret: secretName: "mysql-pw" containers: - name: "mysql" image: "gcr.io/google-samples/mysql:secret" volumeMounts: - name: "password_volume" mountPath: "/etc/mysql-password" readOnly: true bash (inside the container) $ cat /etc/mysql-password/password
  41. Wrap-up

  42. Watch John Wilkes talk • http://www.infoq.com/presentations/cluster-management-google Paper: Large-scale cluster management

    at Google with Borg • http://research.google.com/pubs/pub43438.html Kubernetes: http://kubernetes.io Try Kubernetes quickly and easily on Google Container Engine • https://cloud.google.com/container-engine/ Try these patterns in Kubernetes • https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/services. md#services-without-selectors • https://github.com/GoogleCloudPlatform/kubernetes/tree/master/examples/mysql- wordpress-pd • https://github.com/GoogleCloudPlatform/kubernetes/tree/master/examples/cassandra • https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/secrets.md • https://vitess.io/ Next steps (take a photo)
  43. Thank you! Thank you! kubernetes.io twitter.com/IanMLewis