DEVX with Cloud Foundry and Kubernetes Neven Cvetkovic @nevenc

{ "name": "Neven Cvetkovic", "role": "Advisory Platform Architect", "company": "Pivotal", "email": "[email protected]", "twitter": "@nevenc", "linkedin": "https://www.linkedin.com/in/neven", "github": "https://github.com/nevenc-pivotal" } Introduction

“The only thing that will keep your developers happy is to make them feel productive.” Rob Mee http://nordic.businessinsider.com/pivotal-ceo-rob-mee-on-meetings-2016-7

Developer Experience (DEVX) • What do I care about as a developer or “How do I …” • run my application, • prepare my application for deployment, • deploy my application to production, • find out what my application is doing (e.g. logging, monitoring) • connect to external systems (e.g. databases, queues), • update my application with new versions (e.g. blue/green) • make my application more resilient (e.g. self-healing, scaling, auto-scaling) • ...

“Self-service platforms makes my life easy...” Happy Developer

Platform Hierarchy

CONTAINERS EVENT-DRIVEN FUNCTIONS DATA SERVICES Batches MONOLITHIC APPLICATIONS MICROSERVICES Most enterprises run a mix of workloads ...

Our goal: pick the best suited runtime for each workload IaaS Serverless Functions Runtime (FaaS) Container Orchestrator (CaaS) Application Platform (PaaS) CONTAINERS EVENT-DRIVEN FUNCTIONS DATA SERVICES Batches MONOLITHIC APPLICATIONS MICROSERVICES

Runtime Abstractions Container Orchestrator Container Scheduling Primitives for Network, Routing, Logs & Metrics CONTAINER Developer Provides Tool Provides Application Platform APPLICATION Container Orchestrator Serverless Functions FUNCTION Application Platform IaaS Container Image & build L7 Network & Routing Logs, Metrics, Monitoring Services Marketplace Team, Quotas & Usage Function scheduling Function exec services Container Orchestrator

Hardware IaaS Container Orchestrator Application Platform Serverless Functions Strategic goal: Push as many workloads as technically feasible to the top of the platform hierarchy Higher flexibility and less enforcement of standards Lower development complexity and higher operational efficiency

Cloud Foundry and Kubernetes

Application: Customer Service • Simple Spring Boot application • actuators, lombok, rest repositories, jpa, h2, mysql • Simple JPA Entity, e.g. Customer • Simple JPA Repository, e.g. CustomerRepository • Expose actuator endpoints • Add JPA DDL generation • Add a Kill Switch (and Memory Hog) https://github.com/nevenc-pivotal/customer-service-cf https://github.com/nevenc-pivotal/customer-service-k8s Let’s run this app locally first!

Developer Experience (DEVX) on CF and K8s • Deploy my application • Scale my application • Look at the application logs • Resiliency of the application • Connect to a database • Update my application with new version (blue/green deployment)

Deploying an Application

Deploying to Cloud Foundry • Interacting with Cloud Foundry • cf CLI • Single executable Go binary • Windows, Linux, MacOS binaries • Extensive plugin support • Java Client available • Deployment steps ./mvnw clean package cf push customer-service -p target/customer-service-1.0.0.jar Cloud Foundry HTTP Rest cf

Deploying to Kubernetes • Interacting with Kubernetes • kubectl CLI • Single executable Go binary • Windows, Linux, MacOS binaries • Extensive plugin support • Java client available • Deployment steps (edit Dockerfile) ./mvnw clean package ./mvnw dockerfile:push kubectl run customer-service --image=nevenc/customer-service-k8s:1.0.0 --port=8080 kubectl expose deployment customer-service --type=LoadBalancer --port=80 --target-port=8080 Kubernetes HTTP Rest kubectl

How do we interact with the platform? • Cloud Foundry • cf CLI • Single executable Go binary • Windows, Linux, MacOS binaries • Extensive plugin support • Java Client available • Kubernetes • kubectl CLI • Single executable Go binary • Windows, Linux, MacOS binaries • Extensive plugin support • Java client available Cloud Foundry Kubernetes HTTP Rest HTTP Rest kubectl cf

Build and Deploy an Application • Cloud Foundry • Build application • Push application artifact to the platform (e.g. PWS, PAS) • Kubernetes • Build application • Build Docker image • Push Docker image to container registry (e.g. Docker Hub, Google CR, Azure CR) • Create Kubernetes deployment • Expose Kubernetes service

Build and Deploy an Application (cont’d) • CloudFoundry ./mvnw clean package (edit manifest.yml) cf push • Kubernetes (edit Dockerfile) ./mvnw clean package ./mvnw dockerfile:push (edit deployment.yml, service.yml) kubectl create -f customer-service-k8s-deployment.yml kubectl create -f customer-service-k8s-service.yml

Scaling an Application

Scaling an Application • Cloud Foundry cf scale customer-service -i 3 cf app customer-service curl http://customer-service-cf.cfapps.io/actuator/env/cloud.application.instance_id • Kubernetes kubectl scale --replicas=3 deployment/customer-service kubectl get pods kubectl get svc curl http://104.155.73.62/actuator/env/HOSTNAME

Access Application Logs

Get Application Logs • Cloud Foundry • All logs are aggregated and published to loggregator firehose cf logs customer-service cf logs customer-service --recent • Kubernetes • Each pod’s logs are streamed separately, no aggregated logging pipeline • More versatile logging options (tail, since, timestamps, etc…) kubectl get pods kubectl logs customer-service-d569cd8cc-msgvj -f kubectl logs customer-service-d569cd8cc-msgvj ...

Connecting to Databases (external systems)

Create a SQL database instance • Cloud Foundry • We leverage service marketplace, through various service brokers cf create-service p-mysql 100mb pcf-mysql-database cf create-service cleardb spark cleardb-mysql-database cf create-service elephantsql turtle elephantsql-postgres-database cf create-user-provided-service k8s-mysql-service -p '{"uri" : "mysql://user:[email protected]/customers?reconnect=true"}' • Kubernetes • We can create our own database instance (e.g. using Docker images, Helm) • We could leverage open service broker API for upcoming service offerings (edit mysql-database.yml, postgres-database.yml) kubectl create -f mysql-database.yml kubectl create -f postgres-database.yml

Create a NoSQL database instance • Cloud Foundry • We leverage service marketplace, through various service brokers cf create-service p-redis 100mb pcf-redis-database cf create-service rediscloud 100mb rediscloud-redis-database cf create-service mlab sandbox mlab-mongo-database ... • Kubernetes • We can create our own database instance using existing Docker images or Helm • We could leverage open service broker API for upcoming service offerings (edit redis-database.yml, mongo-database.yml) kubectl create -f redis-database.yml kubectl create -f mongo-database.yml ...

Create a database instance on native public clouds • Cloud Foundry • We leverage service marketplace, through various service brokers cf create-service google-cloudsql-mysql small gcp-mysql-database cf create-service rds-mysql basic aws-mysql-database cf create-service azure-mysqldb basic50 azure-mysql-database ... cf create-service google-datastore default gcp-document-database cf create-service azure-documentdb standard azure-document-database ... • Kubernetes • We could leverage open service broker API for upcoming service offerings

Updating an Application (Blue/Green)

Zero Downtime Application Updates • Cloud Foundry • Traditionally, done as blue-green deployments cf rename customer-service customer-service-venerable cf push cf delete customer-service-venerable → only after you checked new version works properly! • CF CAPI team is working on the new cf push experience with Cloud Foundry • Kubernetes • Update the deployment: (a) by editing deployment file (b) setting image manually (edit deployment.yml) kubectl apply -f deployment.yml kubectl set image deployments/customer-service \ customer-service=nevenc/customer-service:1.0.1

Application Update Rollout • Cloud Foundry • Current limitation of application runtime, will be addressed in near future • Don’t delete previous version before you check the new version cf delete customer-service cf rename customer-service-venerable customer-service • New cf push will have access to previous versions to rollback updates • Kubernetes • K8s has built-in rollout tools kubernetes rollout status deployments/customer-service kubernetes rollout undo deployments/customer-service

SSH to Application Containers

SSH to Application Containers • Cloud Foundry cf ssh customer-service -i 0 • Kubernetes kubectl get pods kubectl exec -it customer-service-d569cd8cc-msgvj -- /bin/bash

Distributed Tracing

Distributed Tracing • Cloud Foundry • Zipkin tracing built-in into Gorouter (can be turned on/off in OpsManager) • Spring Cloud Sleuth makes it very easy to add tracing information to logs • Cloud Foundry aggregates all logs into single loggregator stream • Kubernetes • Not trivial • Depends on what kind of logging utility you use (e.g. Stackdriver Trace) - there might be ready-made Zipkin proxy Docker images that will get you jumpstarted • There are other plugins and add-ons that you could use to enable Zipkin tracing

CF and K8s

Run on Pivotal Application Service (CF) Run on VM Run on Pivotal Container Service (K8s) What kind of application is this? Is the application cloud ready? Do you have a docker container? Can you build a docker container? CUSTOM COTS YES NO YES NO YES NO Is the application worth refactoring? NO YES NO Do you need low level control? YES K8s Benefit: Run any non-cloud native applications CF Benefit: Lower development complexity How do we decide where to deploy our workloads ...

BOSH Other Broker Services Platform Services Logging Metrics Monitoring VMware GCP Azure Openstack AWS Spring Boot App PKS Controller GCP Service Broker Harbor NSX-T Kubernetes K8s Cluster K8s Cluster Spring Boot App Elastic Search Pivotal Application Service (PAS) Pivotal Container Service (PKS) Sample Use Case

Hardware IaaS Container Orchestrator Application Platform Serverless Functions Strategic goal: Push as many workloads as technically feasible to the top of the platform hierarchy Higher flexibility and less enforcement of standards Lower development complexity and higher operational efficiency

Pivotal Cloud Foundry

vSphere Openstack AWS Google Cloud Azure & Azure Stack Shared Services Shared Security Shared Networking Logging & Metrics / Services Brokers / API Management Credhub / UAA / Single Sign On VMWare NSX Embedded Operating System (Windows / Linux) Application Code & Frameworks Buildpacks / Spring Boot / Spring Cloud / Steeltoe PAS Pivotal Application Service PKS Pivotal Container Service PFS Pivotal Function Service Pivotal Services Marketplace Pivotal and Partner Products Concourse

Spring Goodies on Pivotal Application Service • Spring Boot apps integration with AppsManager • Spring deployment profiles (“cloud” profile) • Spring Cloud Connector for Cloud Foundry (VCAP_SERVICES, VCAP_APPLICATION) • Java Buildpack • Advanced JVM memory calculator • JVM heap dump histograms • 3rd party framework integration (APM) • Self-executable JARs • Spring Security integration with CF SSO • Spring integration with CredHub • Spring Cloud Services • Config Server, Service Registry, Circuit Breaker Dashboard • Spring Cloud Pipelines • Spring Cloud Dataflow for PCF • PCF Metrics

Thank You! Questions? Neven Cvetkovic @nevenc