Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Our journey to persistent Kubernetes storage with Rook

2745c78242fec52bd69ff276ce5ebe17?s=47 Andri Steiner
November 07, 2019

Our journey to persistent Kubernetes storage with Rook


Andri Steiner

November 07, 2019


  1. our journey to persistent Kubernetes storage with Rook our journey

    to persistent Kubernetes storage with Rook
  2. Hi! I'm Andri Hi! I'm Andri Ops One AG, Zurich

    as department in the market since 2007, spin-off 2016 Managed Server (PaaS) for PHP, Node, Python, Ruby, Java, ... Managed Applications (SaaS) Nextcloud, Matomo, Discourse, Gitlab, Atlassian Stack, ... Managed Kubernetes (CaaS) what we'll talking about soon :)
  3. Background Background

  4. Platform Overview Platform Overview Cockpit customer facing interface CRUD for

    servers and their config Hardware inhouse Nutanix cluster or: any cloud provider supporting Terraform or: manual on-premise installation Managed Server VM configuration managed by Puppet 7th evolution, perfected ever since
  5. Nutanix Nutanix hyperconverged infrastructure supermicro based appliance with service &

    support distributed compute & storage high availability concurrent to some Kubernetes features
  6. Docker Docker in production since 2015 images provided by vendors

    like Gitlab or Discourse own images for exotic needs tightly integrated into our Managed Servers
  7. Kubernetes Kubernetes started playing around in 2016 production since 2018

    beginner courses since 2019 see https://www.letsboot.com/kubernetes-docker-container-kurs
  8. Approach Approach started with simple and uncritical workloads played with

    distributions (Rancher, OpenShift, CoreOS) and other (NixOS) setups decided to get it running on our existing infrastructure started with single node clusters evaluated network, storage & load balancer learned the meaning behind "Kubernetes The Hard Way" Tutorial: https://github.com/kelseyhightower/kubernetes-the-hard-way
  9. Storage Evaluation Storage Evaluation Container Storage Interface (CSI) Driver available

    for Nutanix and all major cloud providers does not work with bare metal setups (on-premise or maybe our own someday) vendor lock-in Cloud-Native Storage Solution looked at Rook, Longhorn, StorageOS decided to go with Rook because the building blocks behind (Ceph, NFS) where proven already
  10. None
  11. Rook: Overview Rook: Overview cloud-native storage orchestrator framework for 3rd

    party storage providers and solutions « Rook turns distributed storage systems into self-managing, self-scaling, self- healing storage services. It automates the tasks of a storage administrator: deployment, bootstrapping, configuration, provisioning, scaling, upgrading, migration, disaster recovery, monitoring, and resource management. »
  12. Rook: Project Rook: Project open source (Apache 2.0) CNCF incubating

    project top contributors Cloudical, Nexenta, Red Hat, SUSE, Upbound
  13. Rook: Details Rook: Details extends Kubernetes with custom types and

    controllers controlled through existing operator pattern Rook Operator brain behind any storage CRUD, upgrades, rebalancing, health & monitoring not on the data path - can be offline for some time Storage Providers made available by their respective creators Ceph, EdgeFS, Cassandra, CockroachDB, Minio, NFS, YugabyteDB
  14. Rook: Ceph Dashboard Rook: Ceph Dashboard can be enabled in

    the cluster CRD will expose HTTP port as a service
  15. Rook: Ceph Monitoring Rook: Ceph Monitoring built in metrics collectors

    for Prometheus Grafana dashboards provided
  16. Ceph example Ceph example

  17. Rook: Ceph example Rook: Ceph example CustomResourceDefinition and operator git

    clone https://github.com/rook/rook.git cd rook/cluster/examples/kubernetes/ceph kubectl create -f common.yaml kubectl create -f operator.yaml
  18. Rook: Ceph example Rook: Ceph example define CephCluster apiVersion: ceph.rook.io/v1

    kind: CephCluster metadata: name: rook-ceph namespace: rook-ceph spec: cephVersion: image: ceph/ceph:v14.2.4-20190917 mon: count: 3 dashboard: enabled: true storage: useAllNodes: true useAllDevices: true
  19. Rook: Ceph example Rook: Ceph example Ceph agent, mgr, mon

    & osd running $ kubectl -n rook-ceph get pod NAME READY STATUS RESTARTS AGE rook-ceph-agent-4zkg8 1/1 Running 0 140s rook-ceph-mgr-a-d9dcf5748-5s9ft 1/1 Running 0 77s rook-ceph-mgr-a-dashboard-5s9ft 1/1 Running 0 77s rook-ceph-mon-a-7d8f675889-nw5pl 1/1 Running 0 105s rook-ceph-mon-b-856fdd5cb9-5h2qk 1/1 Running 0 94s rook-ceph-mon-c-57545897fc-j576h 1/1 Running 0 85s rook-ceph-operator-6c49994c4f-9csfz 1/1 Running 0 141s rook-ceph-osd-0-7cbbbf749f-j8fsd 1/1 Running 0 23s rook-ceph-osd-1-7f67f9646d-44p7v 1/1 Running 0 24s rook-ceph-osd-2-6cd4b776ff-v4d68 1/1 Running 0 25s rook-ceph-osd-prepare-node1-vx2rz 0/2 Completed 0 60s rook-ceph-osd-prepare-node2-ab3fd 0/2 Completed 0 60s rook-ceph-osd-prepare-node3-w4xyz 0/2 Completed 0 60s rook-discover-dhkb8 1/1 Running 0 140s
  20. Rook: Ceph example Rook: Ceph example CephBlockPool & StorageClass as

    of now, Rook will satisfy any PersistentVolumeClaims apiVersion: ceph.rook.io/v1 kind: CephBlockPool metadata: name: replicapool namespace: rook-ceph spec: replicated: size: 2 --- apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: rook-ceph-block provisioner: ceph.rook.io/block parameters: blockPool: replicapool clusterNamespace: rook-ceph
  21. Summary Summary Rook is a controller for existing software like

    Ceph today, we're using Rook with Ceph and NFS EdgeFS sounds promising for multi cloud setups intro: https://itnext.io/edgefs-cluster-with-rook-in-google-cloud-885227625b9b more than happy about the project and its process still, ponder over pros and cons of using CSI
  22. Thank you! Thank you! rook.io opsone.ch twitter.com/andristeiner