10.05.23 How to Build the Greatest and Best Pipeline in the World Richard Gross (he/him) Hypermedia- Designer Archaeologist Auditor richargh.de/ speakerdeck.com/richargh @arghrich

Slides by richargh.de and Which Company has the Greatest and Best Pipeline in the World? 2

Slides by richargh.de and Deploy all the things 3 Amazon (~2012)a1 • Multiple “Primitives” • Max deploys in one hour: 1,079 fl1 https://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr e1 https://www.infoq.com/news/2014/03/etsy-deploy-50-times-a-day/ fb1 https://engineering.fb.com/2012/08/03/uncategorized/ship-early-and-ship-twice-as-often/ fb2 https://engineering.fb.com/2017/08/31/web/rapid-release-at-massive-scale/ a1 Velocity Culture https://www.youtube.com/watch?v=dxk8b9rSKOo g1 https://cacm.acm.org/magazines/2016/7/204032-why-google-stores-billions-of-lines-of-code-in-a-single-repository/fulltext Facebook (~2012)fb1 • 1,5 TB Monolith • 2 Deploys per Day Flickr (~2007)fl1 • 10 deploys per Day • Where DevOps went public Etsy (~2014)fl1 • 50 deploys per Day • 60 million monthly visits Facebook (~2017)fb2 • TB Monolith • Quasi-Continuous Deploys Google (~2016)g1 • Millions of LoC in one Repo • Where MonoRepo went public • Quasi-Continuous Deploys Works with all architectures Also when you are heavily regulated

Slides by richargh.de and 4 Why How Who

Slides by richargh.de and 5 Why How Who

Slides by richargh.de and Big bang releases only lead to big bangs 6

Slides by richargh.de and (1 We are more often wrong then right about which features produce value 7 1 https://blackswanfarming.com/how-to-tilt-the-innovation-playing-field 1

Slides by richargh.de and (2) Big Releases are High-Risk • The effect of sth big is extremely hard to anticipate, way too many variables • If something goes wrong • You have a huge change to sift through • It’s hard to roll-back 8 2

Slides by richargh.de and (2) Big Releases are Expensive 9 3 Cost Batch Size Transaction Cost Holding Cost Total Cost Optimal Batch Size What if we could bring the transaction costs way down? 1 https://hbr.org/2012/05/six-myths-of-product-development

Slides by richargh.de and Low-risk releases are incremental 10

Slides by richargh.de and Delivery Deliver small experiments and gather feedback Idea Slice It Bet on it Code it Commit Build It Production Deploy It Release It Feedback

Slides by richargh.de and The Deployment Pipeline Make it economic to work in small batches 12

Slides by richargh.de and 13 Why How Who

Slides by richargh.de and “If we only just work harder we’ll catch up” That never works 14

Slides by richargh.de and Prefer to work Smarter, with a sharper saw1 15 1 Image by Henrik Kniberg

Slides by richargh.de and 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Legacy Intermediate Normal Team Work Innovation Sharpening the Saw Unplanned Work + Rework Sharpen the Saw to go faster 16 Features not on-time, on-budget, full of bugs and you can’t get to the interesting stuff. Improve Architecture, automated tests, pipeline and your process. Continuously sharpen, so you have time for features 1 Loosely based on Accelerate – State of DevOps 2018

Slides by richargh.de and The Story of the Hp Laserjet Firmware Team 17 2008 10% - code integration 20% - detailed planning 25% - porting code 25% - current product support 15% - manual testing ~5% - innovation 1 A Practical Approach to Large Scale Agile Development https://www.youtube.com/watch?v=Trqjj3d3lhQ 2 Why Scaling Agile Does not Work https://www.youtube.com/watch?v=2zYxWEZ0gYg 2011 2% - continuous integration 5% - agile planning 15% - one main branch 10% - one branch cpe 5% - most testing automated ~40% - innovation 23% managing automated tests

Slides by richargh.de and Remember though, you are not AWS or Google Or do you have hundreds or thousands of engineers working on improving the release? 18

Slides by richargh.de and Build your Greatest and Best Pipeline in the World Make it economic to work in small batches 19

Slides by richargh.de and 20 Why How Who Prepare Build Optimize

Slides by richargh.de and Make it economic to work in small batches • Good pipelines require good architecture • Uncouple your code • Test on multiple levels • Version everything à EverythingAsCode • Convince your Stakeholders 21

Slides by richargh.de and Uncouple your Code High Coupling Low Cohesion Low Coupling High Cohesion 22 Dependency Unknown Source

Slides by richargh.de and Write and test focused modules • Systems consist of modules • Modules • Provide a simple api • Hide their internals • Are deep, not shallow1 • Call api of other modules 23 1 A Philosophy of Software Design Dependency

Slides by richargh.de and Make it clear what your module does, what the api is and what is internal src/features/core/ ├── renting/ │ ├── internal/ │ │ ├── a.service.ts │ │ ├── a.repository.ts │ │ └── ... │ ├── exposed-types/ │ │ ├── a.type.ts │ │ └── ... │ ├── a.renting.facade.ts │ └── another.renting.facade.ts ├── paying/ │ ├── internal/ │ ├── exposed-types/ │ └── a.paying.facade.ts src/commons/── paying/ │ ├── internal/ 24 ✘ The focus of a module is declared via verb Internals are declared via your convention Facade and types define the module api Great place to test • Test is structure-insensitive • If the test is simple to write, the api is also simple Great place to test • If it’s a leaf • If the test is simple to write, the code simple Bad place to test • If it composes multiple other internals • The test now locks down your internal structure

Slides by richargh.de and Convince your Stakeholders 25

Slides by richargh.de and Speed and Stability are possible without tradeoffs1,2 (Research-confirmed since 2013) 26 1 Accelerate – State of DevOps Report 2018 2 The Data Behind DevOps https://www.youtube.com/watch?v=DgpsX5yLXQw Throughput Stability

Slides by richargh.de and Speed and Stability enable one another 27 1 Accelerate – State of DevOps Report 2018 Low Performer Between once per week and once per month Between one month and six months Between one week and one month 46-60% Elite Performer On demand (multiple deploys per day) Less than one hour Less than one hour 0-15% Throughput Deployment Frequency Lead Time for Changes Stability Time to Restore Service Change Failure Rate

Slides by richargh.de and 28 Why How Who Prepare Build Optimize

Slides by richargh.de and Automate, automate, automate Automate as much as possible but don’t overdo it1,2 Deployment should be at most one mouse click away Lots of automated Checks to guard quality 29 1 https://en.wikipedia.org/wiki/Ironies_of_Automation 2 How Complex Systems Fail

Slides by richargh.de and 30 Commit Stage Build ✓ Package & Store ✓ Static Code Analysis (Lint) ✓ Unit Tests ✓ Integration Tests ✓ 1 AWS DPRA https://pipelines.devops.aws.dev/ Self-Service Automatic Explore Stage Deploy & Smoke ✓ Man. exploratory tests ✓ 5min Explore Env Git Source Code Test Code Infra Code Data Migrations Configuration … Required ✓ Situational ✓ Start here, then iterate

Slides by richargh.de and 31 Commit Stage Beta Stage Build ✓ Package & Store ✓ Static Code Analysis (Lint) ✓ Unit Tests ✓ Prepare Stage Prod Stage Integration Tests ✓ 1 AWS DPRA https://pipelines.devops.aws.dev/ Fetch & Cache Dependencies ✓ Beta Env Prod Env Self-Service Automatic Deploy & Smoke ✓ Explore Stage Deploy & Smoke ✓ Man. exploratory tests ✓ System Tests ✓ 5min Deploy & Smoke ✓ Monitoring & Logging ✓ Explore Env ~30min Git Source Code Test Code Infra Code Data Migrations Configuration … Required ✓ Situational ✓

Slides by richargh.de and 32 Commit Stage Beta Stage Gamma Stage Build ✓ Package & Store ✓ Static Code Analysis (Lint) ✓ Unit Tests ✓ Prepare Stage Prod Stage Integration Tests ✓ Secrets Detection ✓ “Dependency” Analysis (SCA) ✓ Static Security Tests (SAST) ✓ 1 AWS DPRA https://pipelines.devops.aws.dev/ Fetch & Cache Dependencies ✓ Beta Env Gamma Env Prod Env Self-Service Automatic Deploy & Smoke ✓ Explore Stage Deploy & Smoke ✓ Man. exploratory tests ✓ System Tests ✓ Deploy & Smoke ✓ Resilience Tests ✓ Soak Tests ✓ Dynamic Security Tests (DAST) ✓ Monitoring & Logging ✓ Performance Tests ✓ 5min Deploy & Smoke ✓ Monitoring & Logging ✓ Explore Env ~30min ~4h Git Source Code Test Code Infra Code Data Migrations Configuration … Required ✓ Situational ✓

Slides by richargh.de and 33 Commit Stage Beta Stage Gamma Stage Prod Stage 1 AWS DPRA https://pipelines.devops.aws.dev/ Beta Env Gamma Env Prod Env Self-Service Automatic Explore Stage Explore Env Required ✓ Situational ✓ Does it work? Does it work in a real environment? Does it work in a production-like environment? Does it work for real? Does it work in unexpected situations? Does it work for me?

Slides by richargh.de and More infos? AWS has a guide for you J Deployment Pipeline Reference Architecture (DPRA) 34 1 AWS DPRA https://pipelines.devops.aws.dev/

Slides by richargh.de and Does it work? 35 Commit Stage Build ✓ Package & Store ✓ Static Code Analysis (Lint) ✓ Unit Tests ✓ Integration Tests ✓ Secrets Detection ✓ “Dependency” Analysis (SCA) ✓ Static Security Tests (SAST) ✓ Prepare Stage Fetch & Cache Dependencies ✓ Static Security Analysis Dynamic Feature Tests

Slides by richargh.de and 36 Prepare Stage Fetch & Cache Dependencies ✓ # in /.gitlab-ci.yaml 1. # image+env-vars for all jobs 2. variables: 3. NODE_IMAGE: /node:18.x-alpine3.y 4. image: $NODE_IMAGE 5. 6. .node:cache: 7. cache: 8. policy: pull 9. key: 10. files: 11. - package-lock.json 12. prefix: node 13. paths: 14. - node_modules/ 15. 16.node_update_dependencies: 17. stage: prepare 18. extends: 19. - .node:cache 20. cache: 21. policy: pull-push 22. script: 23. - npm ci Commit Stage Build ✓ # in /.gitlab-ci.yaml 1. build: 2. stage: commit 3. extends: 4. - .node:cache 5. script: 6. - npm run build:prod 7. artifacts: 8. paths: 9. - dist/ 10. expire_in: 8 hours 1 Gitlab Examples https://docs.gitlab.com/ee/ci/examples/ 2 Cache https://docs.gitlab.com/ee/ci/yaml/index.html#cache Update cache Use cache Share build result with other jobs

Slides by richargh.de and 37 Commit Stage 1. # circular analysis 2. - npx madge --extensions ts 3. --circular ./src/ 4. # architecture 5. - npx ts-arch 6. - npx depcruise --config .arch.js src 7. # for Java: use ArchUnit Static Analysis: Architecture ✓ 1. - npx eslint src/ tests/ 2. - hadolint “Dockerfile” 3. - ... Sonar, shellcheck Static Analysis: Common Issues ✓ 1. - gitleaks detect Static Analysis: Secrets Detection ✓ 1. - trivy fs --scanners vuln,secret 2. ,config myproject/ 3. - trivy image “${IMAGE}:” “Dependency” Analysis (SCA) ✓ 1. - sonar-scanner Static Security Tests (SAST) ✓

Slides by richargh.de and 38 Commit Stage Beta Stage Gamma Stage Prepare Stage 1 https://www.gocd.org/2016/06/08/Add-performance-testing-to-your-Continuous-Delivery-pipeline/ Gamma Env Self-Service Automatic Deploy & Smoke ✓ Resilience Tests ✓ Soak Tests ✓ Dynamic Security Tests (DAST) ✓ Monitoring & Logging ✓ Performance Tests1 ✓ ~4h Required ✓ Situational ✓ Does it work in a production-like environment? Load and Stress tests Continuous Load over hours Inject failures, kill services Try to exploit security holes (OWASP ZAP)

Slides by richargh.de and 39 Gamma Stage # In your perf-scenario.yaml 1. config: 2. target: “https:// 3. phases: 4. # how your virtual users arrive 5. - duration: 60 6. arrivalRate: 5 7. name: Warm up 8. # your service level aggreements 9. ensure: 10. thresholds: 11. - "'http.response_time.p95'": 200 12. 13.scenarios: 14. # what scenarios to run 15. - name: ”Create Orders" 16. flow: 17. - post: 18. name: ”Create order" 19. url: "/api/orders” 20. json: 21. name: "{{ name }}" Performance Tests ✓ 1 Artillery https://www.artillery.io/docs/guides/getting-started/writing-your-first-test

Slides by richargh.de and Monitor at least the RED1 Metrics 40 1 https://grafana.com/files/grafanacon_eu_2018/Tom_Wilkie_GrafanaCon_EU_2018.pdf 2 https://github.com/grafana/intro-to-mlt Rate Delay Errors

Slides by richargh.de and These are all Feedback Loops • Automated Unit/Int. Tests • Static Tests (Architecture, …) • Integration into Trunk • Automated Large Tests • Performance, etc. Tests • Exploratory Tests • Production Metrics • Customer Reactions 41 Idea …

Slides by richargh.de and 42 Why How Who Prepare Build Optimize

Slides by richargh.de and The pipeline consists of Everything we need to go from commit to releasable outcome 43

Slides by richargh.de and The Goal of a Pipeline Have we done everything we can to disprove that this is a releasable candidate? 44

Slides by richargh.de and Build on the shoulder of giants So much stuff you can use and most of it is free 45

Slides by richargh.de and Automate Dependency Updates 46 1 https://docs.renovatebot.com/configuration-options/ Renovate Stage # in /.gitlab-ci.yaml 1. renovate: 2. stage: renovate 3. image: $REOVATE_IMAGE 4. script: 5. - renovate $(cat repositories.txt | xargs) --token $TOKEN_GIT –log-file renovate.log 6. artifacts: 7. paths: 8. - renovate.log # in /repositories.txt 1. / In it’s own repository with a bi-hourly schedule # in /config.js 1. module.exports = { 2. “branchPrefix”: “chore/renovate-”, 3. ... 4. }

Slides by richargh.de and Parallize Whatever you can 47 1 https://continuousdelivery.com/2010/09/deployment-pipeline-anti-patterns/ Commit Stage Build ✓ Package & Store ✓ Static Code Analysis (Lint) ✓ Unit Tests ✓ Integration Tests ✓ Secrets Detection ✓ “Dependency” Analysis (SCA) ✓ Static Security Tests (SAST) ✓ Most of the stage can and should run in parallel. Run tests in parallel Run test in parallel with separate db schema per test set

Slides by richargh.de and Be Flexible - Stay Wide and Short1 48 1 https://continuousdelivery.com/2010/09/deployment-pipeline-anti-patterns/ Commit Stage Beta Stage Gamma Stage Prod Stage Beta Env Gamma Env Prod Env Explore Stage Explore Env Give choices to smart humans.

Slides by richargh.de and For daily deployments, consider Gamma ONE deployments1 49 1 Automating safe, hands-off deployments https://aws.amazon.com/builders-library/automating-safe-hands-off-deployments/ Commit Stage Beta Stage Gamma Stage Prod Stage Beta Env Gamma Env Prod Env Is the service backwards compatible? (1) Deploy ONE instance side-by with old version (2) Unleash normal test traffic and measure (3) If all ok, after 1 hour, deploy all instances (4) Run regular Gamma Stage tests

Slides by @arghrich It’s not just about the Pipeline 50 Pipeline Easy Hard Techniques Something you do Methods When you do it Mindset What you actually do

Slides by @arghrich Continuous Delivery Principles • Build quality in • Work in small batches • Computers Perform Repetitive Tasks, People Solve Problems • Relentlessly Pursue Continuous Improvement • Everyone is Responsible 51 CD Principles by Jez Humble Four Principles of Low-Risk Software Releases by Jez Humble

Slides by richargh.de and We are what we repeatedly do. Excellence then, is not an act, but a habit.1 52 1 The Story of Philosophy

Slides by @arghrich Thanks 53 Richard Gross (he/him) richargh.de/ speakerdeck.com/richargh @arghrich Works for maibornwolff.de/ People. Code. Commitment. DE TN ES Archaeologist Auditor Hypermedia- Designer

Slides by richargh.de and Backup

Slides by richargh.de and Gitlab-Ci Links • Create a Job: https://docs.gitlab.com/ee/ci/jobs/ • Caching&Artifacts https://docs.gitlab.com/ee/ci/caching/ • Predefined Gitlab Variables: https://docs.gitlab.com/ee/ci/variables/predefined_variables.html • Unit Test Reports: https://docs.gitlab.com/ee/ci/testing/unit_test_reports.html • Test Coverage: https://docs.gitlab.com/ee/ci/testing/test_coverage_visualization.html#exa mple-test-coverage-configurations • Metrics Reports: https://docs.gitlab.com/ee/ci/testing/metrics_reports.html • Gitlab-Ci Yaml reference: https://docs.gitlab.com/ee/ci/yaml/index.html • Gitlab-Ci Examples: https://docs.gitlab.com/ee/ci/examples/ 55

Slides by richargh.de and Flickr 2009 • 10 Deploys per Day 56 1 https://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr

Slides by richargh.de and Etsy 20141 • 50 Deploys/Day 57 1 https://www.infoq.com/news/2014/03/etsy-deploy-50-times-a-day/

Slides by richargh.de and Facebook 20xx1 • Over 1B users worldwide • About 700 developers committing against their frontend source code repo • Single frontend code binary about 1.5GB in size • Pushed out to many thousands of servers (the number is not public) • Changes can go from check-in to end users in as quickly as 40 minutes • Release process almost entirely invisible to the users • 2 Deploys per Day in 20122 • Quasi-Continuous Deploy in 20173 58 1 https://dzone.com/articles/pushing-twice-daily-our 2 https://engineering.fb.com/2012/08/03/uncategorized/ship-early-and-ship-twice-as-often/ 3 https://engineering.fb.com/2017/08/31/web/rapid-release-at-massive-scale/ 4 https://research.facebook.com/publications/continuous-deployment-at-facebook-and-oanda/

Slides by richargh.de and AWS Pipeline • Thousands of teams + Microservice architecture + Multiple environments + Continuous Delivery • = 50 million deployments a year (2015) 1 1. Source 2. Build (1st build, 2nd Unit test) 3. Beta (1st Deploy, 2nd UI test) 4. Gamma (1st Deploy, 2nd Load test) 5. Production (1st Deploy region1, 2nd Deploy region2, 3rd Deploy region3) 59 1 Transforming Software Development https://www.youtube.com/watch?v=YCrhemssYuI

Slides by richargh.de and Amazon Stats (around 2012)1 11.6 seconds Mean time between deplovments (weekday) 1,079 Max # of deployments in a single hour 10,000 Mean # of hosts simultaneously receiving a deployment 30,000 Max # of hosts simultaneously receiving a deployment 60 1 Velocity Culture https://www.youtube.com/watch?v=dxk8b9rSKOo

Slides by richargh.de and Google Repository (around 2016)1 • Billions of LoC in a single repository (35 million commits) • Including configuration and documentation • > 2.000 projects under active development • 40.000 commits per workday • Multiple Services with one version: HEAD • Simplified dependency management: static linking without diamond dependency problem • Atomic changes, large scale refactoring • Homegrown VCS (Piper -> Bazel is a subset) • Trunk-Based Development • More than 50 million test cases run per day • 25.000 Developers 61 1 https://cacm.acm.org/magazines/2016/7/204032-why-google-stores-billions-of-lines-of-code-in-a-single-repository/fulltext 2 https://www.infoq.com/presentations/Continuous-Testing-Build-Cloud/

Slides by richargh.de and Google Pipeline • Code is partitioned into packages, folders with a BUILD file • The BUILD file describes what other packages it depends on and what it provides 62 1 https://google-engtools.blogspot.com/2011/08/build-in-cloud-how-build-system-works.html 2 https://www.infoq.com/presentations/Continuous-Testing-Build-Cloud/

Slides by richargh.de and Netflix • Red/Black-Deployment • Quasi-Continuous Deploys 63 1 https://netflixtechblog.com/deploying-the-netflix-api-79b6176cc3f0?gi=386880e182fb

Slides by richargh.de and Stackoverflow Architecture • The main, retail, public site serves over a hundred million unique users and 6,000 page views per second. • 1.3 BILLION page views per month • 1 Monolith deployed to Own data center • 9 Web Servers • 4 SQL Servers • 2 Redis Servers • 3 Elastic Search Servers • 2 HAProxy Servers (Load Balancers) 64 1 https://stackexchange.com/performance 2 https://www.zdnet.com/article/stack-overflow-cto-from-bootstrapped-to-scaling-one-of-the-webs-biggest-properties/ 2022

Slides by richargh.de and Now Unit Test1 your Modules • By which you mean one test file per class? (no ✘) • By which you mean together with the database? (no ✘) • By which you mean … (probably also no ✘) 65 1 A Set of Unit Testing Rules https://www.artima.com/weblogs/viewpost.jsp?thread=126923

Slides by richargh.de and A Set of Unit Testing Rules A test is not a unit test if: • It talks to a database • It communicates across the network • It touches the file system • It can't run at the same time as any of your other unit tests • You have to do special things to your environment (such as editing config files) to run it. Tests that do these things aren't bad. Often they are worth writing, and they can be written in a unit test harness. However, it is important to be able to separate them from true unit tests so that we can keep a set of tests that we can run fast whenever we make our changes. 66 1 A Set of Unit Testing Rules https://www.artima.com/weblogs/viewpost.jsp?thread=126923

Slides by richargh.de and Test Sizes1, not bike sheds 67 1 Google Test Sizes https://testing.googleblog.com/2010/12/test-sizes.html Feature Small Medium Large Network access No Localhost only Yes Database No Yes Yes File system access No Yes Yes Use external system No Discouraged Yes Multiple threads No Yes Yes Sleep statements No Yes Yes System properties No Yes Yes Time limit (seconds) 60 300 900+ Test: modules, functions Test: (Repo) adapters Test: BE Api, FE&BE together

Slides by richargh.de and Larger Size Complicated Test Harness Imprecise Feedback Coarse assertions Slower à $$$ Smaller Size Simple Test Harness Specific Feedback Strict assertions Faster à $ Test Sizes Guide Dependency Stubbed-Dep. Small Code being tested Test Amount Medium DB L a r g e Exploratory

Slides by richargh.de and 69 Commit Stage # in /.gitlab-ci.yaml 1. small_tests: 2. stage: commit 3. extends: 4. - .node:cache 5. variables: 6. KUBERNETES_CPU_REQUEST: 1000m 7. KUBERNETES_MEMORY_REQUEST: 2500Mi 8. script: 9. - npx jest tests/small --ci --reporters=default --reporters=jest-junit 10. # optional artifacts but this way you’ll see nice reports in Gitlab 11. artifacts: 12. paths: 13. - junit.xml 14. expire_in: 5 days 15. reports: 16. junit: junit.xml Small Tests ✓ 1 Gitlab Unit Test Reports https://docs.gitlab.com/ee/ci/testing/unit_test_reports.html

Slides by richargh.de and 70 Commit Stage # in /.gitlab-ci.yaml 1. medium_tests: 2. stage: commit 3. extends: 4. - .node:cache 5. variables: 6. KUBERNETES_CPU_REQUEST: 1000m 7. KUBERNETES_MEMORY_REQUEST: 2500Mi 8. services: 9. # start a database 10. # remove for small tests they don’t need services 11. - name: 12. alias: mongo 13. script: 14. # create junit report because Gitlab can display that 15. - npx jest tests/medium --ci --reporters=default --reporters=jest-junit 16. # optional artifacts but this way you’ll see nice reports in Gitlab 17. artifacts: 18. paths: 19. - junit.xml 20. expire_in: 5 days 21. reports: 22. junit: junit.xml Medium Tests ✓ 1 Gitlab Unit Test Reports https://docs.gitlab.com/ee/ci/testing/unit_test_reports.html

Slides by richargh.de and 71 Commit Stage # in /.gitlab-ci.yaml 1. build: 2. stage: commit 3. extends: 4. - .node:cache 5. script: 6. - npm run build:prod 7. artifacts: 8. paths: 9. - dist/ 10. expire_in: 8 hours Commit Stage Build ✓ # in /.gitlab-ci.yaml 1. package: 2. stage: commit 3. needs: 4. - job: build 5. artifacts: true 6. image: 7. name: /kaniko-project/executor:xyz 8. entrypoint: [ "" ] 9. script: 10. - DEST=”" 11. - /kaniko/executor 12. --context "${CI_PROJECT_DIR}" 13. --dockerfile "${CI_PROJECT_DIR}/Dockerfile" 14. --destination “${DEST}:${CI_COMMIT_SHORT_SHA}” 15. --destination “${DEST}:latest” 16. --push-retry 3 1 Kaniko https://github.com/GoogleContainerTools/kaniko 2 Gitlab Kaniko https://docs.gitlab.com/ee/ci/docker/using_kaniko.html Package & Store ✓ Create artifacts Use artifacts

Slides by richargh.de and 72 Commit Stage Beta Stage Prepare Stage 1 AWS DPRA https://pipelines.devops.aws.dev/ Beta Env Self-Service Automatic Deploy & Smoke ✓ Large Tests ✓ ~30min Required ✓ Situational ✓ Does it work in a real environment?

Slides by richargh.de and 73 Beta Stage 1. deploy_beta: 2. stage: beta 3. image: $HELM_IMAGE 4. variables: 5. # do not clone source code 6. GIT_STRATEGY: none 7. # Gitlab can display latest deployment 8. environment: 9. name: beta 10. script: 11. # visualize deploy changes 12. - helm diff upgrade 13. --install --color --context 5 14. --detailed-exitcode <...> 15. # deploy 16. - helm upgrade 17. --install 18. --values .helm/values-beta.yaml 19. --namespace beta 20. -beta 21. .helm/ 22. # wait for deployment to finish 23. - kubectl rollout status 24. --request-timeout=5s 25. -n=beta 26. -beta Deploy ✓ Beta Stage 1. smoke_beta: 2. stage: commit 3. needs: 4. - job: deploy_beta 5. artifacts: false 6. script: 7. - npm run test:smoke -- -ns=beta Smoke ✓ 1 Gitlab Environments https://docs.gitlab.com/ee/ci/environments/ 2 Helm Diff Plugin https://github.com/databus23/helm-diff A smoke test Uses non-destructive requests That are invisible to the user Examples: GET /users but also PUT /orders PUT /orders

Slides by richargh.de and AWS DPRA1 74 1 https://pipelines.devops.aws.dev/application-pipeline/