Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Debunking the Myth: Kubernetes Storage is Hard [Kubecon Keynote]

Debunking the Myth: Kubernetes Storage is Hard [Kubecon Keynote]

"Debunking the Myth: Kubernetes Storage is Hard
Kubecon EU 2019 Barcelona Keynote
Abstract: https://sched.co/MQhi
Video: https://youtu.be/169w6QlWhmo
Summary Video: https://youtu.be/pAOkLdxilcI

515740717a9ba5d00fb79158c9071097?s=128

Saad Ali

May 22, 2019
Tweet

Transcript

  1. Saad Ali Senior Software Engineer, Google May 22, 2019 Debunking

    the Myth: Kubernetes Storage is Hard github.com/saad-ali twitter.com/the_saad_ali
  2. Magic of Containers Benefits of Kubernetes Self Healing Intelligent Scheduling

    Auto Scaling Service Discovery Load Balancing Safer Deployment App Portability
  3. What about stateful apps? Containers are inherently ephemeral Stateful apps

    need to persist bits across container restarts.
  4. Kubernetes Storage Myths It’s hard! Storage on Kubernetes is hard.

    Don’t do it! Don’t run stateful workloads on Kubernetes.
  5. Reality: Storage is complicated

  6. Reality: “Sprinkling Kubernetes” on a hard problem won’t make it

    go away.
  7. Separation of concerns Makes large problems manageable.

  8. 03 Integrate How do I make my deployed storage available

    in my cluster? 02 Deploy How do I deploy and manage my storage? Seperate Storage Problems 04 Consume How does my stateful app provision and use available storage? 01 Select What storage should I use?
  9. 03 Integrate How do I make my deployed storage available

    in my cluster? 02 Deploy How do I deploy and manage my storage? Seperate Storage Problems 04 Consume How does my stateful app provision and use available storage? 01 Select What storage should I use?
  10. 01 Select What storage should I use?

  11. NFS SMB GlusterFS CephFS InfluxDB Prometheus Graphite iSCSI Fibre Channel

    GCE Persistent Disks Amazon EBS Local Disks Apache Kafka RabbitMQ Google Cloud Pub/Sub Amazon SQS Amazon S3 Google Cloud Storage (GCS) MinIO What storage should I use? Object Stores SQL Databases NoSQL Databases MySQL PostgreSQL SQL Server Key-value or document based MongoDB Redis Cassandra File Storage Time series Databases Message Queues Block Storage 1 2 3 4 Understand your options Understand your requirements Weigh tradeoffs Make decision
  12. What storage should I use? Understand your options Understand your

    requirements Weigh tradeoffs What does your stateful app need? What kind of data are you storing? Where will you be accessing it? How frequently will you need to access it? What kind of data protection is required? 1 2 3 Make decision 4
  13. What storage should I use? Availability Durability Performance Cost Understand

    your options Understand your requirements Weigh tradeoffs 1 2 3 Make decision 4
  14. What storage should I use? Understand your options Understand your

    requirements Weigh tradeoffs 1 2 3 Make decision 4 “Sprinkling Kubernetes on it” doesn’t magically solve this. You have to do your homework. You have to make a decision about your architecture, based on your needs.
  15. 02 Deploy How do I deploy and manage my storage?

  16. You do not have to deploy your storage on Kubernetes

    to use it in Kubernetes. How do I deploy my storage?
  17. You deploy and manage. Someone else deploys and manages. Unmanaged

    Managed How do I deploy my storage?
  18. How do I deploy my storage? Storage deployed on top

    of Kubernetes is just another stateful application.
  19. Use an operator to deploy applications with complicated life cycles.

    How do I deploy my storage?
  20. How do I deploy my storage? Software defined storage deployed

    on Kubernetes is just another stateful application.
  21. How do I deploy my storage? Software defined storage deployed

    on Kubernetes is just another stateful application.
  22. 03 Integrate How do I make my deployed storage available

    in my cluster? 02 Deploy How do I deploy and manage my storage? Seperate Storage Problems 04 Consume How does my stateful app provision and use available storage? 01 Select What storage should I use?
  23. Data Services vs Block/File NFS SMB GlusterFS CephFS InfluxDB Prometheus

    Graphite iSCSI Fibre Channel GCE Persistent Disks Amazon EBS Local Disks Apache Kafka RabbitMQ Google Cloud Pub/Sub Amazon SQS Amazon S3 Google Cloud Storage (GCS) MinIO Object Stores SQL Databases NoSQL Databases MySQL PostgreSQL SQL Server Key-value or document based MongoDB Redis Cassandra File Storage Time series Databases Message Queues Block Storage
  24. Data Services vs Block/File NFS SMB GlusterFS CephFS InfluxDB Prometheus

    Graphite iSCSI Fibre Channel GCE Persistent Disks Amazon EBS Local Disks Apache Kafka RabbitMQ Google Cloud Pub/Sub Amazon SQS Amazon S3 Google Cloud Storage (GCS) MinIO Object Stores SQL Databases NoSQL Databases MySQL PostgreSQL SQL Server Key-value or document based MongoDB Redis Cassandra File Storage Time series Databases Message Queues Block Storage
  25. Data Service Block/File Storage Physical Storage Object Store, SQL/NoSQL DB,

    Message Queue, etc. NFS, iSCSI Fibre Channel, etc. SSD/Flash Disk Stateful App Your stateful app Data Services vs Block/File
  26. 03 Integrate How do I make my deployed storage available

    in my cluster?
  27. Use a Container Storage Interface (CSI) Driver. Your app must

    handle discovery and negotiation. Block/File Storage Data Service How do I integrate my storage with Kubernetes?
  28. Use a Container Storage Interface (CSI) Driver. Block/File Storage How

    do I integrate my storage with Kubernetes? Without Kubernetes With Kubernetes Swapping storage requires rewrite your app to handle provisioning, attaching, mounting. Swapping storage is as easy as deploying a new CSI Driver and creating a new StorageClass API object.
  29. 04 Consume How does my stateful app use storage?

  30. Use the Kubernetes Storage API (PVC, PV, etc.) Block/File Storage

    Your app must handle Data Service How does my stateful app use storage?
  31. Use the Kubernetes Storage API (PVC, PV, etc.) Block/File Storage

    How does my stateful app use storage? Without Kubernetes With Kubernetes Manual provision, manual attach, manual mount. Automatic (intelligent) provisioning. Intelligent scheduling based on storage. Storage automatically available to correct node and pod. Storage moved along with workload. Portable Kubernetes Storage API -- write once run anywhere
  32. 03 Integrate How do I make my deployed storage available

    in my cluster? 02 Deploy How do I deploy and manage my storage? Seperate Storage Problems 04 Consume How does my stateful app provision and use available storage? 01 Select What storage should I use?
  33. Storage is complicated... Kubernetes makes it manageable!

  34. Thank you! github.com/saad-ali twitter.com/the_saad_ali