@crichardson Deploying microservices: the path from laptop to production Chris Richardson Founder of Eventuate.io Founder of the original CloudFoundry.com Author of POJOs in Action and Microservices Patterns @crichardson [email protected] http://adopt.microservices.io Copyright © 2024. Chris Richardson Consulting, Inc. All rights reserved

@crichardson JFokus 2020: Geometry of microservices Process: Lean + DevOps Organization: Small, autonomous teams Microservice Architecture Deliver changes to long- lived applications rapidly, frequently and reliably Success Triangle Scale Cube Hexagonal Architecture API Iceberg services Channel Messaging Unit Integration Component End to End Testing Pyramid Microservice architecture has required -ilities Loose design-time coupling Loose run-time coupling Foundation of modern development The structure of each service The goal Key goal of microservices: accelerate software delivery

@crichardson Presentation goal Production (Kubernetes) Code repository Developer laptop Service Deployment pipeline An essential ingredient of rapid, frequent and reliable delivery 15 minutes How to design Fast, automated deployment pipeline

@crichardson About Chris http://adopt.microservices.io

@crichardson About Chris https://chrisrichardson.net/

@crichardson If you want to learn more: Thursday and Friday https://bit.ly/jfokus-2024-msa https://chrisrichardson.net/training.html

@crichardson Agenda The deployment pipeline: automating the path to production The test pyramid for microservices Designing a deployment pipeline for Kubernetes Deploying and releasing changes safely

@crichardson Let’s imagine you want to deploy the Customer Service Postgres ACID transaction Authorization: Bearer … GET /oauth2/jwks via Eventuate CDC service Apache Kafka Customer Service <> Customer Spring Boot/JPA-based service … … … CREDIT_LIMIT ID … CUSTOMER table MESSAGE (outbox) table … … … DESTINATION ID … Send message Update DB Reply channel POST /customer JPA Reply Publisher Command handler subscribes to Spring Authorization Server reserveCredit() releaseCredit() Customer Service command channel

@crichardson What’s the path from laptop to production Production Developer laptop Service ?

@crichardson The old way: slow, manual and unreliable Developers write code QA team test it Security team veri fi es it Ops team manually update production Slow and unreliable

@crichardson Modern development = fast automated deployment DevEx Work environment than minimizes interruptions, meaningful work, … Minimize - simplify environment, automate, good documentation, … Fast feedback from colleagues, tools, deployment pipeline, production, users, … Feedback Flow Cognitive load Great https://itrevolution.com/articles/the-three-ways-principles-underpinning-devops/

@crichardson New way: automated, fast and reliable Production Commit frequently to trunk Automated deployment pipeline Continuous integration Continuous testing (Git) repository for Code: e.g. Java Con fi guration: e.g. Kubernetes YAML, Terraform, … Dev Ops Continuous deployment 15 minutes

Pipeline Deployment pipeline: the path from laptop to production Pre commit tests Commit stage Tests Integration Tests Release/ Deploy Non- functional tests…. Component Tests Code analysis • Code quality • Security • Bugs • … Fast Slow

@crichardson Production Deployment pipeline per service Order Service Orders Team Automated deployment pipeline Source code repository Kitchen Service Kitchen Team Automated deployment pipeline Source code repository Delivery Service Delivery Team Automated deployment pipeline Source code repository

@crichardson Deployment pipeline overview Container Registry Production Helm Chart Container Image Service deployment pipeline Test in isolation Test Service Test Double Sole criteria for release Includes contract- testing to enforce API interoperability ServiceGit Repository Dev Flux K8s API Application configuration Git repository Service Test pyramid kind: HelmRelease version: XYZ … kind: HelmRelease version: XYZ … Defines testing strategy apply/delete Defines cluster state Reconciles Git commit/push GitOps git commit/push git commit/push Monitors for new chart versions 2 3 4 1

@crichardson Agenda The deployment pipeline: automating the path to production The test pyramid for microservices Designing a deployment pipeline for Kubernetes Deploying and releasing changes

@crichardson Automated testing is a foundation of modern software delivery Fast, reliable, scalable - humans are too slow Accelerates the edit-build-test work fl ow Use test-driven development to more easily write new code Rely on test suites when changing existing code

@crichardson Push tests down the test pyramid to shorten feedback loops Unit Integration Includes consumer-driven contract tests Component End to End Classes within service A services adapters An entire service Multiple services or application Services are independently deployable Brittle, Slow, Costly Reliable, Fast, Cheap

@crichardson Unit Integration Component End to End Goal of unit testing Unit Test Class/ classes Veri fi es behavior Test algorithms ./gradlew test Test Double

@crichardson Unit test selectively http://blog.stevensanderson.com/2009/11/04/selective-unit-testing-costs-and-bene fi ts/ Cost of unit testing = # dependencies Bene fi t of unit testing = non-obviousness High High Low Low Algorithms Trivial code Coordinators Overly complex code Refactor

@crichardson CustomerTest example class CustomerTest { @Test public void shouldHaveCreditLimitAvailable() { assertEquals(creditLimit, customer.availableCredit()); } @Test public void shouldHaveSomeAvailableCredit() { var expected = new Money("0.01"); var orderTotal = creditLimit.subtract(expected); customer.reserveCredit(orderId, orderTotal); assertEquals(expected, customer.availableCredit()); } … class Customer { Customer(String name, Money creditLimit) { … } Money availableCredit() { … } void reserveCredit(long orderId, Money orderTotal) { … } …

@crichardson Unit Integration Component End to End Goal of integration testing Integration Test Adapter Veri fi es behavior • Inbound requests: • HTTP controller • … • Outbound requests: • HTTP proxy • DAO • Messaging adapter • … ./gradlew integrationTest

@crichardson Integration tests - verify adapters for infrastructure services Other Service Proxy DAO Some Service Dao Tests Rest Controller Inbound Request Outbound Request Client Other Service Messaging Messaging Tests DB Broker

@crichardson CustomerRepositoriesTest @DataJpaTest … public class CustomerRepositoriesTest { public static EventuateDatabaseContainer> database = DatabaseContainerFactory.makeVanillaDatabaseContainer(); @DynamicPropertySource static void registerMySqlProperties(DynamicPropertyRegistry registry) { startAndProvideProperties(registry, database); } @Test public void shouldSaveAndLoadCustomer() { … var customerId = transactionTemplate.execute( ts -> { Customer c = new Customer(customerName, creditLimit); customerRepository.save(c); return c.getId(); }); … Testcontainers-provided database

@crichardson About integration testing with infrastructure services Testing adapters in isolation is more effective and easier than testing at a higher-level, e.g. via service API Launching containers is time-consuming but shorten feedback loops by: Reusing already running containers Reduce build-time coupling with module-per-adapter

@crichardson Integration tests - verify adapters for inter-service communication Other Service Proxy DAO Some Service Consumer- side Contract Tests Rest Controller Provider-side Contract Tests Prevents breaking changes Ensures agreement with provider Inbound Request Outbound Request Client Other Service Consumer-driven contract tests Messaging

@crichardson Consumer-driven contract testing Api Gateway Customer Service Proxy Customer Service Customer Controller POST /customers Provider Consumer • Written by consumer team • Published by provider team API de fi nition by example Customer Service ProxyTests Customer Controller Tests Standalone tests • Sends contract’s request • Veri fi es response matches • Con fi gures HTTP test double, e.g. Wiremock

@crichardson About contract testing Bene fi t: Provides fast feedback about service interoperability Prevents breaking changes Drawback: extra work BUT if it’s a burden then most likely: Excessively fi ne-grained architecture Unstable APIs

@crichardson Unit Integration Component End to End About component testing Component Test Service Veri fi es behavior ./gradlew componentTest • Don’t repeat • Focus: bean wiring

@crichardson Component testing with test containers public class CustomerServiceComponentTest { static EventuateZookeeperContainer zookeeper = … static EventuateKafkaContainer kafka = … static EventuateDatabaseContainer> database =… static AuthorizationServerContainer authorizationServer = … public static ServiceContainer service = new ServiceContainer(“./Dockerfile”, …) .withDatabase(database) .withKafka(kafka) @BeforeClass public static void startContainers() { Startables.deepStart(service, authorizationServer).join(); } @Before public void setup() { RestAssured.port = service.getFirstMappedPort(); RestAssured.authentication = oauth2(authorizationServer.getJwt()); } @Test public void shouldGetCustomers() { given() .when() .get("/customers") .then() .log().ifValidationFails() .statusCode(200); } Start containers Invoke API

@crichardson Unit Integration Component End to End About end-to-end testing: don’t! XBrittle, Slow, Costly https://microservices.io/post/architecture/2022/05/04/microservice-architecture-essentials-deployability.html

@crichardson Alternatives to end-to-end testing in the deployment pipeline Run end-to-end tests outside of deployment pipeline, e.g. nightly Run end-to-end tests continuously in production Test in production! Canary deployments - progressively route traf fi c to new version Traf fi c mirroring - route traf fi c to old and new versions and analyze Dynamic feature fl ags - e.g. only internal users access new version …

@crichardson Agenda The deployment pipeline: automating the path to production The test pyramid for microservices Designing a deployment pipeline for Kubernetes Deploying and releasing changes

@crichardson name: Build, test and publish (Java) … Test Helm Chart Publish image and chart Package format format for Kubernetes Deployment pipeline

Building and testing the service Runs unit, integration, and component tests Builds an executable Spring Boot JAR - name: Build run: ./gradlew build … slowTest.mustRunAfter(fastTest) … build.gradle

@crichardson Building and publishing container images ARG baseImageVersion FROM base-image:$baseImageVersion COPY build/libs/customer-service-main.jar service.jar docker buildx build —platform linux/amd64,linux/arm64 -t remoterepo/cust… —push … But publishing an image is insuf fi cient Docker fi le

@crichardson Pod Deploying a service on K8s = YAML ! kind: Deployment metadata: name: customer-service spec: replicas: 2 template: spec: containers: - image: ghcr.io/… env: - name: SPRING_DATASOURCE_… … Container Load balance kind: Service metadata: name: customer-service spec: type: ClusterIP ports: - port: 80 targetPort: http selector: … Virtual IP Domain name External LB kind: Ingress metadata: name: customer-service spec: rules: - host: … http: paths: - path: /customers … Container image Runtime

@crichardson …Deploying a service on K8s kind: Deployment metadata: name: customer-service spec: replicas: 2 template: spec: containers: - image: ghcr.io/../customer-service:0.1.0-SNAPSHOT env: - name: SPRING_DATASOURCE_URL value: jdbc:postgresql://customer-service-postgres/customer_service - name: SPRING_DATASOURCE_USERNAME value… - name: SPRING_DATASOURCE_PASSWORD value… Need environment-speci fi c values Spring Boot externalized con fi g

@crichardson Packaging services as Helm charts… Parameterized template fi les apiVersion: v2 name: customer-service description: A Helm chart for Kubernetes type: application version: 0.1.0 appVersion: "0.1.0-SNAPSHOT" dependencies: - name: postgres version: v0.4.0 Subcharts Part of service image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart … - name: SPRING_DATASOURCE_USERNAME {{- toYaml .Values.postgresConnection.username | nindent 12 }} - name: SPRING_DATASOURCE_PASSWORD {{- toYaml .Values.postgresConnection.password | nindent 12 }} … Default values Reference value

@crichardson Installing/upgrading a Helm chart $ helm upgrade —install \ customer-service \ oci://ghcr.io/…/customer-service \ —values values.yaml \ —set-string ingress.enabled=true \ … Specify chart values

@crichardson Test chart locally using Kind Kind container Kubernetes cluster $ kind create cluster Apache Kafka Customers Postgres DB Ingress NGINX Customer Service Port 80 $ helm install Authorization Service GET /customers Apache Zookeeper public class CustomerServiceHelmChartTest extends AbstractHelmChartTest { @Test public void testService() { var jwt = getJwt(); given() .when() .auth().preemptive().oauth2(jwt) .get("/customers") .then() .statusCode(200); } } $./gradlew helmChartTest

@crichardson Accelerating Helm chart tests Testing is a chart with Kind is heavyweight Pulling Kind image Starting Kind cluster Pulling application/infrastructure images into cluster THEREFORE: Use an Gradle incremental task to only run test when chart changes Pull application/infrastructure images from local registry

@crichardson Fast running Helm Template tests using Groovy + Spock class HelmChartSpec extends Specification { def helm = new HelmTemplater(true) def "defaultDataSourceUrl"() { given: def values = """ postgresEnabled: false """ when: def manifests = helm.template(values) then: def deployment = manifests.findDeployment() def value = (deployment.spec.template.spec.containers as List)[0].env. find { it.name == "SPRING_DATASOURCE_URL" }.value value == "jdbc:postgresql://app-customer-service-postgres/customer_service" } $ helm template … => YAML Inspired by https://github.com/robmoore-i/helm-chart-automated-tests $./gradlew helmTemplateTest

@crichardson Publishing images and Helm chart Container registry Container image Helm chart - name: Publish run: | docker buildx build —push … helm package helm-chart/ —version … helm push chart-1.0.0.tgz oci://…

@crichardson Agenda The deployment pipeline: automating the path to production The test pyramid for microservices Designing a deployment pipeline for Kubernetes Deploying and releasing changes

@crichardson Don’t deploy the chart manually Container registry Container image Helm chart Kubernetes Cluster ? Service Deployment … $ helm upgrade … X

Use GitOps-based deployment Git repository = desired state = reproducible git commit/push - to deploy a change git revert/push - rollback a change git log - what changed, when and by whom Audit log Production problem -> look for recent change https://www.gitops.tech/#what-is-gitops Git Repository … chart: customer-service repository:… version: 0.1.0-BUILD… values: … … … chart: customer-service repository:… version: 0.1.0-BUILD… values: … … Kubernetes Cluster De fi nes desired state

@crichardson Cluster Git repository apps/ Manifests Container registry GitOps with Flux Kubernetes Cluster Flux Kubernetes API Container image Helm chart Service deployment pipeline https:// fl uxcd.io/ Git commit/push Periodically: Git pull apply/delete clusters/dev/ Manifests dev prod 1 2 3

@crichardson Deploying a chart with a HelmRelease apiVersion: helm.toolkit.fluxcd.io/v2beta1 kind: HelmRelease metadata: name: customer—service namespace: default spec: chart: spec: sourceRef: kind: HelmRepository name: application chart: customer—service version: 0.1.0-BUILD.20… interval: 15s values: replicaCount: 2 … Chart name and version Chart values

@crichardson kind: HelmRelease metadata: name: customer-service spec: chart: spec: chart: customer-service version: 0.1.0-BUILD.20… # {"$imagepolicy": “default:customer-service-image-policy:tag"} Cluster Git repository apps/ Manifests Container registry Automatically deploy a new chart version Kubernetes Cluster Flux Kubernetes API Container image Helm chart Service deployment pipeline Poll apply/delete dev Publish Git commit /push 1 2 3 4

@crichardson Problem: how to support multiple environments? Kubernetes prod cluster Customer Service HelmRelease Kubernetes dev cluster Customer Service HelmRelease Dev values Prod values Dev version Prod version How to tailor to an environment? Service deployment pipeline How to propagate changes? Automatic updates

@crichardson Using Kustomize apps/base kustomization Customer Service HelmRelease apps/dev overlay kustomization kustomize.yaml kustomize.yaml Patch Manifest apps/prod overlay kustomization kustomize.yaml Patch Manifest Patch Manifest Patch Manifest Modi fi es Modi fi es https://kustomize.io/ kind: Kustomization metadata: name: apps spec: path: apps/dev kind: Kustomization metadata: name: apps spec: path: apps/prod clusters/dev/apps.yaml clusters/prod/apps.yaml e.g. automated version updates

@crichardson dev/patch-customer-service.yaml Patching HelmRelease manifests kind: HelmRelease metadata: name: customer-service spec: chart: spec: chart: customer-service sourceRef: kind: HelmRepository name: application … kind: HelmRelease metadata: name: customer-service spec: chart: spec: version: "0.1.0-BUILD…" # {"$imagepolicy": … } … kind: HelmRelease metadata: name: customer-service spec: chart: spec: version: "0.1.0-BUILD.20231101154007" base/CustomerServiceHelmRelease.yml prod/patch-customer-service.yaml dev overlay prod overlay

@crichardson Automating promotion to production Kubernetes dev cluster Customer Service HelmRelease GitHub Promotion Action Helm upgrade succeeded event Github repository kind: HelmRelease metadata: name: customer-service spec: chart: spec: version: … prod/patch-customer-service.yaml Kubernetes prod cluster Customer Service HelmRelease edit/commit/push upgrade 1 2 3 kind: Alert metadata: name: … spec: providerRef: name: github kind: Provider metadata: name: github spec: type: githubdispatch Run tests?

@crichardson Improving safety with canary releases Don’t immediately release change to all users Instead First, test it Then gradually shift traf fi c from old version to new version while monitoring its behavior

Order Service V1 Order Service V2 Canary releases with Flagger 1.Deploy V2 alongside V1 2.Test V2 3.Release V2 to a small % of production users 4.Monitor/test (latency, errors) - undo rollout if errors 5. Increase % of production traf fi c going to V2 6.Repeat until 100% of traf fi c going to V2 7.Eventually undeploy V1 Ingress or Service Mesh Customer Service V1 Customer Service V2 Monitoring system e.g. Prometheus https:// fl agger.app/ Flagger Queries Con fi gures

@crichardson Summary Fast automated deployment pipeline is essential Push tests down the test pyramid Avoid end-to-end testing Build, test and publish container image and Helm Chart Use GitOps-style deployment Use canary releases to safely update production Fast, automated deployment pipeline

@crichardson @crichardson [email protected] http://adopt.microservices.io Questions?