Deploying Production Ready Kubernetes Clusters: Lessons Learnt

Slide 1

Slide 1 text

Deploying production ready Kubernetes Clusters- Lessons Learnt Rahul Mahale, BigBinary #devopsdayscpt 2017

Slide 2

Slide 2 text

$whoami Rahul Mahale FOSS Enthusiast. Sr. DevOps Engineer at BigBinary https://www.bigbinary.com/ Kubernetes Partner. Rahul_Mahale

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

Agenda ● Kubernetes Cluster provisioning considerations for production. ● High Level requirements. ● Deploying applications on kubernetes cluster.

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

No content

Slide 7

Slide 7 text

Prerequisite ● Dockerization of application. ● Orchestration tools for docker.

Slide 8

Slide 8 text

No content

Slide 9

Slide 9 text

Container Orchestration ● Every orchestration s/w (Kubernetes, Docker swarm, Mesos- marathon) is opinionated when it comes to describing a containerized application. ● This session is focused on Kubernetes.

Slide 10

Slide 10 text

Kubernetes Architecture

Slide 11

Slide 11 text

Kubernetes terminologies ● Pods ● Deployments ● Services ● Configmaps ● Secrets ● Jobs ● Scheduled Jobs. ● HPA(Horizontal pod autoscaler)

Slide 12

Slide 12 text

Two Grounds ● Provisioning production ready k8s cluster. ● Deploying application on k8s cluster.

Slide 13

Slide 13 text

K8s cluster must be, ● Highly available. ● Secured (behind VPN, secured networking). ● Auto-Scalable.

Slide 14

Slide 14 text

Way to do it ? ● Provision HA cluster using Kops. ● Private networking using Calico/weave/flannel ● Use Cluster-autoscaler k8s addon. ● Other tools:- ○ kubeadm, kubespray. etc.

Slide 15

Slide 15 text

Kubernetes Storage. ● Persistent volumes. ● NFS storage. ● HostPath. ● Many more options with Kubernetes - Gluster, Ceph, OpenEBS, Rook.

Slide 16

Slide 16 text

Where to host database ? ● On kubernetes ? Check stateful sets. ● We host our DB on AWS RDS and on k8s using PVC. ● Pre-created on launch of the new application.

Slide 17

Slide 17 text

Kubernetes Networking ● Kubenet is default networking. ● Calico/Flannel/Weaveworks.

Slide 18

Slide 18 text

Image Building Base Images:- ● From Public Registries like DockerHub, Quay, GCR etc.. ● Self hosted. Generic Best Practice: ● One layer for base, one for user configuration and one for application ● Leverage ‘USER’ directive to run programs inside container as non-root. ● Ensure regular scanning of images. ● Use environment variables for runtime configuration.

Slide 19

Slide 19 text

Resource Management. Use kubernetes namespaces: ● One namespace per user or group ● Separate namespaces for Dev/Test/Staging/Build ● Specify Resource Quota (cpu, mem, #pods, #services, #RCs, #PersistentVolumeClaims) for each namespace

Slide 20

Slide 20 text

Authentication and Authorization. ● ABAC(Attribute-based access control) ● RBAC(Role-based access control) ● Webhooks. ● Custom modules.

Slide 21

Slide 21 text

Zero downtime deployments. Cluster Upgrades. - Kops rolling update App deployments - Healthcheck. - Readiness and liveness probes. - Container lifecycle hooks.

Slide 22

Slide 22 text

Ingress controller ➔ L7 load balancing - L7 Load-balancing - SSL termination - Path-based rules - Multiple host names

Slide 23

Slide 23 text

Auto-scaling applications on K8s. ● Horizontal Pod Autoscaler(CPU based). ● Memory based Autoscaler.(Own) ● Custom metrics autoscaling v1.7.2

Slide 24

Slide 24 text

Custom metrics autoscaling. ● Scale based on Request(Query) per second. ● Supported in v1.7

Slide 25

Slide 25 text

Kubernetes Jobs - Use it for custom and periodic unscheduled tasks

Slide 26

Slide 26 text

Kubernetes Cron Jobs ● Need to restart api server for enabling it with --runtime-config=batch/v2alpha1 ● Restart policy, restartPolicy: OnFailure

Slide 27

Slide 27 text

Known issues ● K8s Issue #42164 - Restart kubelet(docker) or drain and terminate node. - Requests must be less than limits. ● Pods with PVCs does not scale.

Slide 28

Slide 28 text

Kops rolling upgrade ● Kops rolling upgrade might break if you are using calico networking. ● Check this document[1]. ● Cross verify the kops supported version for k8s cluster. [1].https://github.com/kubernetes/kops/blob/master/docs/upgrade_from_kops_1.6_to_1.7_calico_cid r_migration.md

Slide 29

Slide 29 text

Monitoring Kubernetes cluster and apps. ● Heapster and influxdb addon is available. - kubectl top ● Cluster monitoring using Prometheus and Grafana. ● Configure Prometheus alerts to notify on slack/email etc. using alertmanager. - Prometheus nodeexporter and prometheus-core manifest must specify resources or might lead to consume more and more resources Other tools:- ● Datadog/sysdig/weave-scope etc.

Slide 30

Slide 30 text

Logging ● EFK [Elasticsearch-Fluentd-Kibana] addon from Kubernetes repository. ● Use Logtrail to see tail type logs from pod.

Slide 31

Slide 31 text

Automation ● Create artifacts using something like Ansible or your own tool, create database, secrets etc. ● Creates deployment templates. ● Helm is good tool from k8s community. ● kubectl or k8s API ● Label nodes script. Kops has artifact to specify in cluster.yml

Slide 32

Slide 32 text

Backup and restore of Kubernetes Cluster ● Kubernetes state is maintained in ETCD which is a distributed - key-value store ● Deployment considerations for ETCD - Fault-tolerant cluster - Storage for ETCD (Network and IO latency directly affects ETCD) - ETCD data backup and restore - Reshifter(https://github.com/mhausenblas/reshifter) - Ark ● Enable TLS

Slide 33

Slide 33 text

Don’t Forget ● Kubeval [1]. ● Kubernetes-dashboard. ● Heapster ● kube-state-metrics etc. [1](https://github.com/garethr/kubeval)

Slide 34

Slide 34 text

No content

Slide 35

Slide 35 text

Links http://kubernetes.io/docs/ http://github.com/kubernetes/kops/ http://blog.bigbinary.com/

Slide 36

Slide 36 text

Thanks! Contact us: Big Binary 203, Jewel Towers [email protected] https://www.bigbinary.com