SeaGL Kube101 Class

Transcript

1. Hands on Build, Deploy, and Debug Workshop JJ Asghar, Developer Advocate, IBM Cullen Taylor, Developer Advocate, IBM SeaGL 2018 Novemebr 8th, 2018 Kubernetes 101
2. Pre work for hands on portion of the talk IBM Cloud Kube Provisioning
3. Getting Started – Open Up these 3 URLs now • Setup Instructions – jjasghar.github.io/kube101-docs/ • Promo Codes: ibm.biz/BdYkQ3 • What you’ll be doing: – Creating an IBM Cloud Account – Adding Promo Code to enable Free Kubenetes cluster – Provisioning Kubernetes Cluster – this takes 20 – 30 minutes – Install IBM Cloud CLI and Kubectl for workshop – Git Clone code repo: github.com/ibm/kube101 • Please start this NOW if you want to do the hands on portion • If you run into issues raise your hand, HELPERS will come help
4. Learning Objectives for Today • Create an IBM Cloud Account and Kubernetes Cluster • Learn basic Conceptual Model of common Kubernetes Resources • Build an application image • Deploy application image to Kubernetes • Explore lifecycle of pods • Debug when things go wrong / break things to see common debug steps • Upgrade application Today's Application: System Status Page
5. Container Primer
6. Containers vs. Virtual Machines
7. Why Containers? • Fast startup time - only takes milliseconds to: – Create a new directory – Lay-down the container's filesystem – Setup the networks, mounts, … – Start the process • Better resource utilization – Can fit far more containers than VMs into a host
8. Why Kubernetes? • Baremetal → Virtualization increase OS deployments by factor of 10 • Virtualization → Containers has the potential to be even more than a factor of 10 • Container management can't just follow OS management strategies of the past • Kubernetes takes a definition of the desired state of the world – I need 3 of these, 5 of these, this application should always be running – Similar to config management approaches like Puppet & Chef • Many best practices baked into the platform • Can extend the platform from within the Platform using Custom Resource Definitions
9. Concepts first, YAML later Kubernetes Concepts
10. Kubernetes Concepts / Resources • Kube Execution – Pod – Deployment – Replica Set – Daemon Set • Kube Security – Secrets – Namespaces – Service Accounts • Kube Discovery – Labels / Selectors – Service • Kube Connectivity – NodePort – Load Balancer – Ingress Controller • Extending Kube – Kube API – Custom Resources
11. Pods / Nodes / Deployments Container Container Pod C Container Container Container Pod B Node Container Pod E Container Container Container Pod D Container Container Pod A1 Container Container Pod A2 Deployment Replicas = 2 Node Node A pod contains one or more containers A Deployment ensures a pod is always running on 1 or more nodes. A Deployment provides upgrade / rollback facilities. Pod is a unit of scheduling to a node
12. Failure Recover Container Container Pod C Container Container Container Pod B Node Container Pod E Container Container Pod A1 Container Container Pod A2 Node Node Container Container Container Pod B' Container Container Pod A2 If a pod is deleted or crashes it goes away If a pod in a deployment is deleted or crashes, a new version is started
13. Nodes Don't Matter Container Container Container Container Container Pod C Pod A Container Container Container Container Pod E Pod D Node Node Node For developers Nodes are generally hidden. It doesn't matter which Node applications are on.* *High Availability supported by affinity rules
14. Check point - how are your clusters coming?
15. Service Discovery Container Container Container Container Container Pod C Pod A Service name: foo-web 5000 80 Kube DNS Inside a pod, containers can talk to each other via localhost ports Pods must expose ports explicitly to the outside world Services create an internal DNS name and port mapping to allow other pods access Applications would connect to http://foo-web apiVersion: v1 kind: Service metadata: name: foo-web labels: app: foo-web spec: ports: - port: 80 targetPort: 5000 name: foo-web-port protocol: TCP selector: app: deploy-c apiVersion: apps/v1 kind: Deployment metadata: name: deploy-c labels: app: deploy-c spec: ... Container Container Pod C' 5000 Selectors can match more than one pod
16. NodePort Container Pod C Service name: foo-web spec: type: NodePort 5000 80 NodePort binds an external IP address to a single Node, with a port connection to the Pod. NodePort is the only external access solution available on Free Clusters. 169.60.78.157 Node
17. Load Balancer Container Pod C Service name: foo-web spec: type: Loadbalancer 5000 80 Loadbalancers bind an external IP address to a service. Free clusters do not have LoadBalancer access. Paid clusters have access to 5 external IPs by default. 169.60.78.157 Container Pod C' 5000
18. Ingress Container Pod C Container Container Container Container Pod E Pod D Service name: foo-web 5000 80 Ingress host: sdague-k001.us-east.containers.mybluemix.net http: paths: - path: / backend: serviceName: foo-web servicePort: 80 - path: /data-admin backend: serviceName: foo-data servicePort: 3000 Service name: foo-data Ingress controllers provide open internet access to http(s) services based on path. Multiple services may be accessed based on path matching from a single ingress controller. Free clusters do not have access to Ingress controllers. Ingress controllers are the recommended way to run real kube applications.
19. Kube Security Basics • Secrets – Stored in cluster – Exposed to containers via ENV vars, or filesystem • Service Accounts – By default applications act as the IAM user – Service Accounts can be allocated to drop permissions • Role / RoleAssignment – Allow access to Kube API / secrets – Can restrict per namespace / user / service account
20. Check point - how are your clusters coming?
21. Deploying Your First App
22. Step 1: Clone Repo > git clone https://github.com/sdague/kube101-lisa • Contains application content, and Kubernetes configuration for the workshop
23. Step 2: Build Image > cd kube101-lisa > ibmcloud login --sso > ibmcloud cr namespace-add status_page > ibmcloud cr build --tag \ registry.ng.bluemix.net/status_page/web:1 status_page • The Registry is Global, you’ll need to use something unique to YOU. • Creates a private namespace in IBM container registry • Builds the image for the application
24. Our Application FROM ubuntu:18.04 ENV DEBIAN_FRONTEND noninteractive ENV LANG C.UTF-8 ENV LC_ALL C.UTF-8 ENV FLASK_APP status_page ENV STATUS_PAGE_SETTINGS ../settings.cfg RUN apt-get update && apt-get install -y python3 python3-dev python3-pip && apt-get clean COPY ./ /var/www/status_page WORKDIR /var/www/status_page RUN pip3 install -U . CMD flask run -h 0.0.0.0
25. Step 3: Connect Kube to your cluster > ibmcloud ks cluster-config kubelisa OK The configuration for kubelisa was downloaded successfully. Export environment variables to start using Kubernetes. export KUBECONFIG=/home/YOU/.bluemix/plugins/container-service/clusters/kube lisa/kube-config-wdc06-kubelisa.yml • From here on out we will be using kubectl to interact with the cluster • Note: if cluster is not up yet, please raise hands. We'll try to address.
26. Step 4: Explore the Cluster > kubectl get all -o wide • What do you see?
27. Step 5: Deploy the Application > kubectl apply -f deploy/status-deployment.yaml … > kubectl get all -o wide • What do you see now?
28. Status Web Deployment - Dissected apiVersion: apps/v1 kind: Deployment metadata: name: status-web labels: app: status-web spec: replicas: 3 selector: matchLabels: app: status-web template: metadata: labels: app: status-web spec: containers: - name: status-web image: registry.ng.bluemix.net/status_page/web:1 imagePullPolicy: Always env: - name: REDIS_HOST value: "redis-leader" ports: - name: http containerPort: 5000 protocol: TCP readinessProbe: httpGet: path: /readiness port: 5000 Labels allow you to query by metadata Number of Pods to run Pass configuration by environment What ports to expose Test for when pod is ready to be added to a servic Pod
29. readinessProbe & livenessProbe • livenessProbe – When liveness fails, pod is restarted – Useful in detecting aberrant behavior (deadlocks, other non fatal errors) • readinessProbe – When readiness fails, pod is removed from any Services it's a part of – Provides facility to ensure pod is ready to accept traffic before it is exposed to the world • Every non-toy application should have both of these checks
30. Status Web Service - Dissected apiVersion: v1 kind: Service metadata: name: status-web labels: app: status-web spec: type: NodePort ports: - port: 5000 targetPort: 5000 name: status-web protocol: TCP selector: app: status-web Expose as NodePort (without this it would be internal only Mapping ports
31. Connect to Application > kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION EXTERNAL-IP OS-IMAGE KERNEL- VERSION CONTAINER-RUNTIME 10.188.103.229 Ready <none> 111d v1.11.3+IKS 169.60.97.54 Ubuntu 16.04.5 LTS 4.4.0-137- generic containerd://1.1.4 > kubectl get service -l app=status-web -o wide NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR status-web NodePort 172.21.242.132 <none> 5000:32358/TCP 23m app=status-web Navigate to http://169.60.97.54:32358 Does it work?
32. What's going on? > kubectl get pod -l app=status-web -o wide NAME READY STATUS RESTARTS AGE IP NODE status-web-5c47b9b67b-57ksk 0/1 Running 0 27m 172.30.47.116 10.188.103.229 status-web-5c47b9b67b-q7ln4 0/1 Running 0 27m 172.30.47.113 10.188.103.229 status-web-5c47b9b67b-srcbz 0/1 Running 0 27m 172.30.47.87 10.188.103.229 • No pods are ready? Why?
33. What's going on? > kubectl describe pod/status-web-5c47b9b67b-57ksk … Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 27m default-scheduler Successfully assigned default/ status-web-5c47b9b67b-57ksk to 10.188.103.229 Normal Pulling 27m kubelet, 10.188.103.229 pulling image "registry.ng..." Normal Pulled 27m kubelet, 10.188.103.229 Successfully pulled image "..." Normal Created 27m kubelet, 10.188.103.229 Created container Normal Started 27m kubelet, 10.188.103.229 Started container Warning Unhealthy 2m (x151 over 27m) kubelet, 10.188.103.229 Readiness probe failed: HTTP probe failed with statuscode: 500
34. Step 6: Deploy Datastore > kubectl apply -f deploy/redis-deployment.yaml • Look at the cluster with 'kubectl get all -o wide', see if you can see things go ready
35. Step 7: System Status Application Simple status tracking application for services
36. Step 8: Pod Lifecycle - Recovering from Failures > kubectl get pods -l app=status-web NAME READY STATUS RESTARTS AGE status-web-5c47b9b67b-57ksk 1/1 Running 0 1h status-web-5c47b9b67b-q7ln4 1/1 Running 0 1h status-web-5c47b9b67b-srcbz 1/1 Running 0 1h > kubectl delete pods status-web-5c47b9b67b-57ksk pod "status-web-5c47b9b67b-57ksk" deleted > kubectl get pods -l app=status-web NAME READY STATUS RESTARTS AGE status-web-5c47b9b67b-57ksk 1/1 Terminating 0 1h status-web-5c47b9b67b-cxk8c 0/1 ContainerCreating 0 2s status-web-5c47b9b67b-q7ln4 1/1 Running 0 1h status-web-5c47b9b67b-srcbz 1/1 Running 0 1h
37. Lessons Learned • Kuberenetes Deployments / Pods / Services • It's YAML all the way down • Readiness Checks ensure application is exposed to user only when ready • Deployments provide very basic HA/restart for services • kubectl describe provides events which are good for debugging
38. Upgrade and Debug
39. Upgrading Applications in Kubernetes • No software is "finished" • What facilities does Kubernetes provide for upgraded deployed software? • Also... Let's make this more interesting.... – What if we make a software change which actually doesn't start? – How would the environment react to that? – How would we figure out what's going on?
40. Step 1: Make a Bad Image • edit status_page/status_page/view.py • insert "import noimport" at top of file > ibmcloud cr build --tag registry.ng.bluemix.net/status_page/web:2 status_page
41. Step 2: Upgrade Image • edit deploy/status-deployment.yaml • change "image:" field to end in from :1 to :2 > kubectl apply -f deploy/status-deployment.yaml deployment.apps "status-web" configured service "status-web" unchanged > kubectl get pods -l app=status-web NAME READY STATUS RESTARTS AGE status-web-5c47b9b67b-cxk8c 1/1 Running 0 15m status-web-5c47b9b67b-q7ln4 1/1 Running 0 1h status-web-5c47b9b67b-srcbz 1/1 Running 0 1h status-web-5d6b4665dc-5t7c5 0/1 Error 3 56s
42. Upgrade Semantics • Deployments use a rolling upgrade mechanism by default • Attempt to start 1 new pod – If it succeeds, delete one of the old pods, start next pod – If it fails, keep existing applications running • You should still do testing before deploy, but some basic sanity checking in the application
43. Step 3: Finding out what's wrong - logs > kubectl logs status-web-5d6b4665dc-5t7c5 * Serving Flask app "status_page" * Environment: production WARNING: Do not use the development server in a production environment. Use a production WSGI server instead. * Debug mode: off Usage: flask run [OPTIONS] Error: While importing "status_page", an ImportError was raised: Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/flask/cli.py", line 235, in locate_app __import__(module_name) File "/var/www/status_page/status_page/__init__.py", line 17, in <module> import status_page.views File "/var/www/status_page/status_page/views.py", line 3, in <module> import noimport ModuleNotFoundError: No module named 'noimport'
44. Logging in Kubernetes • All stdout /stderr is captured and exposed as logs • Best practice for containers, log to standard out
45. Fix it, watch the upgrade complete • Fix the python code > ibmcloud cr build --tag registry.ng.bluemix.net/status-page/web:3 status_page • Edit the deployment file to reference image :3 > kubectl apply -f deploy/status-deployment.yaml > kubectl get pods -l app=status-web
46. Only works if you tag images correctly • Many times people use :latest on Docker images • This is an anti-pattern for kubernetes, as infrastructure doesn't know something changed • Even in development, use unique image versions
47. Interactive Debug • Some times the logs aren't enough • We can get inside a container and see whats going on > kubectl -it exec status-web-5c47b9b67b-cxk8c bash
48. Other important parts of Kube Ecosystem • Helm - "package manager", provides templating for kube yaml • Istio - service mesh, metrics and observability inside the cluster • Knative - emerging effort to make image build / serverless use cases simpler
49. What we learned today • Containers – How to bundle an application in a container – How to run that container in Kubernetes • Kubernetes – provides basic availability of applications – has built in health check mechanism for applications – provides rolling upgrade semantics out of the box, including basic failure detection – how to expose an application on the network* – how to debug when something goes wrong Thank You! • Feedback welcome ibm.biz/seagl2018-feedback