Developer Experience with Cloud Foundry and Kubernetes

What's the difference between these platforms, what do they have in common, and what does working with each of them look like from a developer perspective? Landing your code on the right platform will determine the quality of your developer experience. It's important, therefore, to understand what kinds of workloads are most suitable for each, the level of effort required to work with them, and what each platform does for you.

Do you let buildpacks create containers for you, or do you build your own? How much YAML do you need to author and maintain? What kind of security can your application expect from the platform?

You'll leave this session with a clear understanding of what two platforms do for developers.

Transcript

2. developer experience (DevX) with application runtime (cf) and container runtime (k8s) neven cvetkovic advisory platform architect, pivotal
3. introduction { "name": "neven cvetkovic", "role": "advisory platform architect", "company": "pivotal", "email": "nevenc@pivotal.io", "twitter": "@nevenc", "linkedin": "https://www.linkedin.com/in/neven", "github": "https://github.com/nevenc" }
4. “ developer experience (DevX) is user experience (UX) applied to developers, because developers are people too ... www.developerexperience.org
5. developers are users with experience using ... • tools • frameworks • libraries • APIs • platforms • in this talk we will look at devx using two platforms • application runtime / cloudfoundry (cf / PAS) • container runtime / kubernetes (k8s / PKS)
6. cloud platform abstractions ... Hardware IaaS CaaS PaaS FaaS Strategic goal: Push as many workloads as technically feasible to the top of the platform hierarchy Higher flexibility and less enforcement of standards Lower development complexity and higher operational efficiency Richard Seroter's Introduction to PCF 2.x @ S1P - December 2017
7. what do I care as a developer how do I … • run my application • prepare my application for deployment • deploy my application to production (dev, test, qa, uat…) • connect to external systems (e.g. databases, message queues) • update/patch my application with new versions (e.g. rolling upgrades) • observe my application (e.g. logging, monitoring) • make my application more resilient • and many other aspects of developer’s everyday life ...
8. what do I need to know as a developer how do I … • build and package my application • build a container image (e.g. docker) • optimize my container for best performance (e.g. jvm options) • configure application logging and monitoring • define container platform specific constructs • configure load balancing, routing, DNS, SSL/TLS, etc… • configure security and networking policies • … and many other things ...
9. some assumptions … you probably already know how to ... • use your dev tools to develop and build your application • connect your application to external systems (databases) • connect to your platform API (cf or k8s) • do a cf push (you are at cf summit)
10. as a developer how do I … • interact with a platform • deploy an application to production (dev, test, qa, uat…) • scale an application • connect to external systems (e.g. databases, message queues) • update/patch my application with new versions (e.g. rolling upgrades) • observe my application (e.g. logging, monitoring, distributed tracing) • interact with container • ...
11. developer experience with platform (cf and k8s) source code examples and demo screens github.com/nevenc/beer-service-cf github.com/nevenc/beer-service-k8s
12. “self-service platforms make my life easy ... a happy developer @ every enterprise appdev shop
13. interact with the platform
14. how do we interact with a platform? • interacting with cloudfoundry • cf CLI • single executable Go binary • windows, linux, macOS binaries • extensive plugin support • Java client available • interacting with kubernetes • kubectl CLI • single executable Go binary • windows, linux, macOS binaries • extensive plugin support • Java client available Cloud Foundry HTTP Rest cf Kubernetes (K8s) HTTP Rest kubectl
15. login to platform • cf cf login -a https://api.run.pivotal.io cf target -o my-org -s my-space cf target • k8s pks get-credentials my-cluster (to configure ~/.kube/config for cluster access) gcloud container clusters get-credentials my-cluster az aks get-credentials -n my-cluster aws eks update-kubeconfig --name my-cluster kubectl config use-context my-cluster kubectl cluster-info
16. deploying an application
17. example application: beer-service • simple spring boot (Java) application • actuators, lombok, rest repositories, jpa, h2, mysql • simple JPA entity, e.g. Beer • simple JPA repository, e.g. BeerRepository • expose actuator endpoints • add JPA DDL generation • add an initializer endpoint, e.g. BeerInitializer • add a kill switch (and a memory hog) https://github.com/nevenc/beer-service-cf https://github.com/nevenc/beer-service-k8s
18. deploying an application to cf • package the app ./mvnw clean package • deploy an app cf push beer-service -p target/beer-service-1.0.0.jar
19. deploying an application to k8s • package the app (edit Dockerfile) ./mvnw clean package • push the app to container registry (e.g. docker) ./mvnw dockerfile:push (or) docker push nevenc/beer-service:1.0.0 • deploy the app kubectl run beer-service \ --image=nevenc/beer-service-k8s:1.0.0 --port=8080 kubectl expose deployment beer-service \ --type=LoadBalancer --port=80 --target-port=8080
20. build, prepare and deploy • cf • build and package application • push application artifact to the platform (e.g. PWS, PAS) • k8s • build and package application • build OCI image (e.g. docker) • push image to container registry (e.g. docker hub, harbor, gcr, acr, ecr) • create kubernetes deployment • expose kubernetes service
21. build, prepare and deploy with manifests • cf ./mvnw clean package (edit manifest.yml) cf push • k8s ./mvnw clean package ./mvnw dockerfile:push (edit beer-service-k8s-deployment.yml, beer-service-k8s-service.yml) kubectl create -f beer-service-k8s-deployment.yml kubectl create -f beer-service-k8s-service.yml
22. manifest.yml (cf) --- applications: - name: beer-service memory: 1G path: target/beer-service-cf-1.0.0.jar
23. beer-service-k8s-deployment.yml apiVersion: extensions/v1beta1 kind: Deployment metadata: name: beer-service spec: replicas: 1 template: metadata: labels: app: beer-service spec: containers: - name: beer-service image: nevenc/beer-service-k8s:1.0.0 ports: - containerPort: 8080
24. beer-service-k8s-service.yml apiVersion: v1 kind: Service metadata: name: beer-service labels: app: beer-service spec: type: LoadBalancer ports: - port: 80 nodePort: 31000 targetPort: 8080 protocol: TCP selector: app: beer-service
25. scaling an application
26. scale an application • cf cf scale beer-service -i 3 cf app beer-service e.g. curl http://beer-service-cf.cfapps.io/actuator/env/vcap.application.instance_id curl https://beer-service-cf.cfapps.io/actuator/env/vcap.application.instance_index • k8s kubectl scale --replicas=3 deployment/beer-service kubectl get pods kubectl get svc e.g. curl http://35.242.147.24/actuator/env/HOSTNAME
27. connect to database (external system)
28. create a SQL database instance • cf • leverage service marketplace, through various service brokers cf create-service p-mysql 100mb pcf-mysql-database cf create-service cleardb spark cleardb-mysql-database cf create-service elephantsql turtle elephantsql-postgres-database cf create-user-provided-service k8s-mysql-database -p '{"uri" : "mysql://user:password@192.168.30.12/beers?reconnect=true"}' • k8s • can create our own database instance (e.g. using docker images, helm) • could leverage open service broker API for upcoming service offerings (edit mysql-database.yml, postgres-database.yml) kubectl create -f mysql-database.yml kubectl create -f postgres-database.yml
29. create a NoSQL database instance • cf • leverage service marketplace, through various service brokers cf create-service p-redis 100mb pcf-redis-database cf create-service rediscloud 100mb rediscloud-redis-database cf create-service mlab sandbox mlab-mongo-database cf create-user-provided-service k8s-mongo-database -p '{"uri" : "mongodb://user:password@192.168.30.13/beers"}' • k8s • can create our own database instance (e.g. using docker images, helm) • could leverage open service broker API for upcoming service offerings (edit redis-database.yml, mongo-database.yml) kubectl create -f redis-database.yml kubectl create -f mongo-database.yml
30. create an instance on native public clouds • cf • leverage service marketplace, through various service brokers cf create-service google-cloudsql-mysql small gcp-mysql-database cf create-service rds-mysql basic aws-mysql-database cf create-service azure-mysqldb basic50 azure-mysql-database ... cf create-service google-datastore default gcp-document-database cf create-service azure-documentdb standard azure-document-database ... • k8s • could leverage open service broker API for upcoming service offerings
31. update an application (blue/green)
32. zero downtime application updates • cf • traditionally, done as blue-green deployments cf rename beer-service beer-service-venerable cf push cf delete beer-service-venerable • often managed externally, through pipelines (e.g. Spinnaker1, Concourse, Jenkins) • CF CAPI team is working on the new cf push2 experience with cf (“better-push”) • k8s • update the deployment: (a) by editing deployment file (b) setting image manually (edit deployment.yml) kubectl apply -f deployment.yml kubectl set image deployment/beer-service beer-service=nevenc/beer-service:1.0.1 [1] https://www.youtube.com/watch?v=xcD4mWo_YHE [2] https://www.youtube.com/watch?v=uGG0WguYqFI
33. application update rollout • cf • current limitation of application runtime, will be addressed in near future • don’t delete previous version before you check the new version cf delete beer-service cf rename beer-service-venerable beer-service • new cf push1 will have access to previous versions to rollback updates • k8s • kubernetes has built-in rollout tools kubectl rollout status deployment/beer-service kubectl rollout history deployment/beer-service kubectl rollout undo deployment/beer-service --to-revision=2 kubectl rollout pause deployment/beer-service kubectl rollout resume deployment/beer-service [1] https://www.youtube.com/watch?v=uGG0WguYqFI
34. interacting with container
35. ssh to application containers • cf cf ssh beer-service -i 0 • k8s kubectl get pods kubectl exec -it beer-service-84b58c6656-9vwcm -- sh kubectl attach beer-service-84b58c6656-9vwcm -i
36. access application logs
37. access application logs • cf • all logs are aggregated and published to loggregator firehose cf logs beer-service cf logs beer-service --recent • k8s • each pod’s logs are streamed separately, no aggregate log pipeline • more versatile logging options (tail, since, timestamps, etc…) kubectl get pods kubectl logs beer-service-84b58c6656-9vwcm kubectl logs beer-service-84b58c6656-thddf kubectl logs beer-service-84b58c6656-thddf -f
38. distributed tracing
39. distributed tracing (zipkin) • cf • zipkin tracing built-in into gorouter (can be turned on/off in opsman) • spring cloud sleuth makes it very easy to add tracing information to logs • cf aggregates all logs into single loggregator stream • k8s • not trivial • depends on what kind of logging utility you use (e.g. Stackdriver Trace) • some ready-made zipkin proxy docker images that will get you jumpstarted • other plugins and add-ons that you could use to enable zipkin tracing
40. scratching the surface side by side comparison ...
41. what do I need to know as a developer cf • you build the application • platform builds container image • trust the selected buildpack • you leverage existing rich marketplace k8s • you build the application • you build container image • platform continuously scans your images • you should leverage base image strategy • you write k8s manifests • you learn k8s architecture / objects • understand specifics of your k8s cluster • you customize k8s manifests • you configure marketplace • you templatize k8s manifests
42. buildpacks • traditionally, reserved for heroku and cloudfoundry platforms • cloud native buildpacks - new CNCF incubator project • “The next generation of cloud native buildpacks will aid developers and operators in packaging applications into containers, allowing operators to efficiently manage the infrastructure necessary to keep application dependencies updated” • Buildpacks Anywhere - by Scott Sisil and Stephen Levine1 - S1P 2018 • increased developer productivity for every workload • OCI-compatible image, faster builds, local development • increased security posture • operational efficiency managing multiple images [1] https://www.youtube.com/watch?v=LW3oMBvM3cI
43. specifics of my k8s platform • every k8s platform is somewhat different from others • native cloud providers (GKE, EKS, AKS) • native cloud providers (PKS on GCP, AWS) • on-prem clouds (PKS on vSphere, openstack) • networking • load balancing • network policies • storage • storage classes make it a bit easier to configure storage for devs
44. how do I write k8s manifest for my app? • you have to be familiar with characteristics of your k8s cluster • need to understand k8s architecture and object constructs • pods, replicasets, deployments, statefulsets, services, storage classes … • manifest generating tools can help prepare for production • helm, gitkube, ksonnet, skaffold, metaparticle
45. how do I expose my application • you have to be familiar with characteristics of your k8s cluster • need to understand various k8s constructs • services, labels, deployments, loadbalancers ... • need to choose one of the ways to expose the service • ClusterIP, NodePort, LoadBalancer, ExternalName • consider other ingress controllers • nginx, haproxy, f5 bigip, istio ingress, etc. • add service description to your k8s manifest
46. what do I need to know as a developer cf • you build the application • platform builds container image • trust the selected buildpack • you leverage existing rich marketplace k8s • you build the application • you build container image • platform continuously scans your images • you should leverage base image strategy • you write k8s manifests • you learn k8s architecture / objects • understand specifics of your k8s cluster • you customize k8s manifests • you configure marketplace • you templatize k8s manifests
47. where should I run my apps/workloads? cf • custom-built software (linux/windows) • modern cloud-native apps • polyglot web applications • APIs • batch jobs • streaming apps • natural fit for spring boot apps k8s • stateful workloads • commercial (packaged) software • short-lived apps / workloads • software distributed via helm chart • apps using non-standard ports / networking • legacy apps (zero-factor) • data services (kafka, greenplum, mongodb)
48. thank you source code examples: github.com/nevenc/beer-service-cf github.com/nevenc/beer-service-k8s questions? @nevenc
49. references • Rolling Application Updates - Demo with Zach Robinson @ CF Summit 2018 • Continuous Delivery with Spinnaker by Jon Schneider @ S1P 2018 • P to V to C: The Value of Bringing ‘Everything’ to Containers by Cornelia Davis and Mukesh Ghadia @ S1P 2018 • Buildpacks Anywhere by Scott Sisil and Stephen Levine @ S1P 2018 • Choosing Software Abstractions by Dr. Dave Syer @ S1P September 2018 • Introduction to PCF 2.0 by Richard Seroter @ S1P 2017 (December 2017) • Comparing Kubernetes to Pivotal Cloud Foundry — A Developer’s Perspective by Oded Shopen • Kubernetes API Reference (v1.12) • Kubernetes Cheatsheet • GitHub source code examples and screencasts • https://github.com/nevenc/beer-service-cf • https://github.com/nevenc/beer-service-k8s