Upgrade to Pro — share decks privately, control downloads, hide ads and more …

How to Build the Greatest and Best Pipeline in the World - JAX 2023

How to Build the Greatest and Best Pipeline in the World - JAX 2023

The Heart of any well-performing team is it‘s own hand-crafted Engineering process. But a heart would be useless without veins that pass along the blood and can take the pressure that the heart builds. In a similar manner a team needs it’s own hand-crafted pipeline. Most teams today have one but only few teams have a pipeline that can take the pressure and not collapse. This talk is how to build this Greatest and Best Pipeline in the world.

To be realistic though, it‘s more of a tribute to the greatest and best pipeline in the world. Not because we couldn’t remember but because we want to show you how you can build your greatest and best pipeline.

Since your process is different from ours we‘ll show you how to hand-craft your pipeline and not prescripe a solution that won't fit.

The examples will cover building, bundling, testing and securing your deployment artifact. They'll be written in Gitlab-Ci format and deploy to Kubernetes, but you can apply the same principles and patterns to any other pipeline and any other environment.

Richard

May 10, 2023
Tweet

More Decks by Richard

Other Decks in Technology

Transcript

  1. 10.05.23
    How to Build the Greatest
    and Best Pipeline in the
    World
    Richard Gross (he/him)
    Hypermedia-
    Designer
    Archaeologist Auditor
    richargh.de/
    speakerdeck.com/richargh
    @arghrich

    View full-size slide

  2. Slides by richargh.de and
    Which Company has
    the Greatest and Best
    Pipeline in the World?
    2

    View full-size slide

  3. Slides by richargh.de and
    Deploy all the things
    3
    Amazon (~2012)a1
    • Multiple “Primitives”
    • Max deploys in one hour:
    1,079
    fl1 https://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr
    e1 https://www.infoq.com/news/2014/03/etsy-deploy-50-times-a-day/
    fb1 https://engineering.fb.com/2012/08/03/uncategorized/ship-early-and-ship-twice-as-often/
    fb2 https://engineering.fb.com/2017/08/31/web/rapid-release-at-massive-scale/
    a1 Velocity Culture https://www.youtube.com/watch?v=dxk8b9rSKOo
    g1 https://cacm.acm.org/magazines/2016/7/204032-why-google-stores-billions-of-lines-of-code-in-a-single-repository/fulltext
    Facebook (~2012)fb1
    • 1,5 TB Monolith
    • 2 Deploys per Day
    Flickr (~2007)fl1
    • 10 deploys per Day
    • Where DevOps went public
    Etsy (~2014)fl1
    • 50 deploys per Day
    • 60 million monthly visits
    Facebook (~2017)fb2
    • TB Monolith
    • Quasi-Continuous Deploys
    Google (~2016)g1
    • Millions of LoC in one Repo
    • Where MonoRepo went public
    • Quasi-Continuous Deploys
    Works with all architectures
    Also when you are heavily regulated

    View full-size slide

  4. Slides by richargh.de and
    4
    Why How
    Who

    View full-size slide

  5. Slides by richargh.de and
    5
    Why How
    Who

    View full-size slide

  6. Slides by richargh.de and
    Big bang releases only lead to
    big bangs
    6

    View full-size slide

  7. Slides by richargh.de and
    (1 We are more often wrong then right
    about which features produce value
    7
    1 https://blackswanfarming.com/how-to-tilt-the-innovation-playing-field
    1

    View full-size slide

  8. Slides by richargh.de and
    (2) Big Releases are High-Risk
    • The effect of sth big is extremely hard to anticipate, way too
    many variables
    • If something goes wrong
    • You have a huge change to sift through
    • It’s hard to roll-back
    8
    2

    View full-size slide

  9. Slides by richargh.de and
    (2) Big Releases are Expensive
    9
    3
    Cost
    Batch Size
    Transaction Cost
    Holding Cost
    Total Cost
    Optimal
    Batch Size
    What if we could bring
    the transaction costs
    way down?
    1 https://hbr.org/2012/05/six-myths-of-product-development

    View full-size slide

  10. Slides by richargh.de and
    Low-risk releases are
    incremental
    10

    View full-size slide

  11. Slides by richargh.de and
    Delivery
    Deliver small experiments and
    gather feedback
    Idea Slice It
    Bet on it Code it
    Commit
    Build It
    Production
    Deploy It
    Release It
    Feedback

    View full-size slide

  12. Slides by richargh.de and
    The Deployment Pipeline
    Make it economic to work in small batches
    12

    View full-size slide

  13. Slides by richargh.de and
    13
    Why How
    Who

    View full-size slide

  14. Slides by richargh.de and
    “If we only just work harder
    we’ll catch up”
    That never works
    14

    View full-size slide

  15. Slides by richargh.de and
    Prefer to work Smarter, with a sharper
    saw1
    15
    1 Image by Henrik Kniberg

    View full-size slide

  16. Slides by richargh.de and
    0%
    10%
    20%
    30%
    40%
    50%
    60%
    70%
    80%
    90%
    100%
    Legacy Intermediate Normal
    Team Work
    Innovation Sharpening the Saw Unplanned Work + Rework
    Sharpen the Saw to go faster
    16
    Features not on-time,
    on-budget, full of bugs
    and you can’t get to the
    interesting stuff.
    Improve Architecture,
    automated tests,
    pipeline and your
    process.
    Continuously sharpen,
    so you have time for
    features
    1 Loosely based on Accelerate – State of DevOps 2018

    View full-size slide

  17. Slides by richargh.de and
    The Story of the
    Hp Laserjet Firmware Team
    17
    2008
    10% - code integration
    20% - detailed planning
    25% - porting code
    25% - current product support
    15% - manual testing
    ~5% - innovation
    1 A Practical Approach to Large Scale Agile Development https://www.youtube.com/watch?v=Trqjj3d3lhQ
    2 Why Scaling Agile Does not Work https://www.youtube.com/watch?v=2zYxWEZ0gYg
    2011
    2% - continuous integration
    5% - agile planning
    15% - one main branch
    10% - one branch cpe
    5% - most testing automated
    ~40% - innovation
    23% managing automated tests

    View full-size slide

  18. Slides by richargh.de and
    Remember though, you are not
    AWS or Google
    Or do you have hundreds or thousands of engineers working on improving the
    release?
    18

    View full-size slide

  19. Slides by richargh.de and
    Build your Greatest and Best
    Pipeline in the World
    Make it economic to work in small batches
    19

    View full-size slide

  20. Slides by richargh.de and
    20
    Why How
    Who
    Prepare
    Build
    Optimize

    View full-size slide

  21. Slides by richargh.de and
    Make it economic
    to work in small batches
    • Good pipelines require good architecture
    • Uncouple your code
    • Test on multiple levels
    • Version everything à EverythingAsCode
    • Convince your Stakeholders
    21

    View full-size slide

  22. Slides by richargh.de and
    Uncouple your Code
    High Coupling
    Low Cohesion
    Low Coupling
    High Cohesion
    22
    Dependency
    Unknown Source

    View full-size slide

  23. Slides by richargh.de and
    Write and test focused modules
    • Systems consist of modules
    • Modules
    • Provide a simple api
    • Hide their internals
    • Are deep, not shallow1
    • Call api of other modules
    23 1 A Philosophy of Software Design
    Dependency

    View full-size slide

  24. Slides by richargh.de and
    Make it clear what your module does,
    what the api is and what is internal
    src/features/core/
    ├── renting/
    │ ├── internal/
    │ │ ├── a.service.ts
    │ │ ├── a.repository.ts
    │ │ └── ...
    │ ├── exposed-types/
    │ │ ├── a.type.ts
    │ │ └── ...
    │ ├── a.renting.facade.ts
    │ └── another.renting.facade.ts
    ├── paying/
    │ ├── internal/
    │ ├── exposed-types/
    │ └── a.paying.facade.ts
    src/commons/── paying/
    │ ├── internal/
    24

    The focus of a module
    is declared via verb
    Internals are declared
    via your convention
    Facade and types
    define the module api
    Great place to test
    • Test is structure-insensitive
    • If the test is simple to write, the
    api is also simple
    Great place to test
    • If it’s a leaf
    • If the test is simple to write, the
    code simple
    Bad place to test
    • If it composes multiple other
    internals
    • The test now locks down your
    internal structure

    View full-size slide

  25. Slides by richargh.de and
    Convince your Stakeholders
    25

    View full-size slide

  26. Slides by richargh.de and
    Speed and Stability are possible without
    tradeoffs1,2 (Research-confirmed since 2013)
    26
    1 Accelerate – State of DevOps Report 2018
    2 The Data Behind DevOps https://www.youtube.com/watch?v=DgpsX5yLXQw
    Throughput Stability

    View full-size slide

  27. Slides by richargh.de and
    Speed and Stability enable one another
    27
    1 Accelerate – State of DevOps Report 2018
    Low Performer
    Between once per week
    and once per month
    Between one month
    and six months
    Between one week
    and one month
    46-60%
    Elite Performer
    On demand
    (multiple deploys per day)
    Less than
    one hour
    Less than
    one hour
    0-15%
    Throughput
    Deployment
    Frequency
    Lead Time
    for Changes
    Stability
    Time to
    Restore Service
    Change
    Failure Rate

    View full-size slide

  28. Slides by richargh.de and
    28
    Why How
    Who
    Prepare
    Build
    Optimize

    View full-size slide

  29. Slides by richargh.de and
    Automate, automate, automate
    Automate as much as possible but don’t overdo it1,2
    Deployment should be at most one mouse click away
    Lots of automated Checks to guard quality
    29
    1 https://en.wikipedia.org/wiki/Ironies_of_Automation
    2 How Complex Systems Fail

    View full-size slide

  30. Slides by richargh.de and
    30
    Commit Stage
    Build

    Package & Store

    Static Code
    Analysis (Lint)

    Unit Tests

    Integration Tests

    1 AWS DPRA https://pipelines.devops.aws.dev/
    Self-Service
    Automatic
    Explore Stage
    Deploy & Smoke

    Man. exploratory
    tests

    5min
    Explore Env
    Git
    Source Code
    Test Code
    Infra Code
    Data Migrations
    Configuration
    … Required

    Situational

    Start here, then iterate

    View full-size slide

  31. Slides by richargh.de and
    31
    Commit Stage Beta Stage
    Build

    Package & Store

    Static Code
    Analysis (Lint)

    Unit Tests

    Prepare Stage Prod Stage
    Integration Tests

    1 AWS DPRA https://pipelines.devops.aws.dev/
    Fetch & Cache
    Dependencies

    Beta Env Prod Env
    Self-Service
    Automatic
    Deploy & Smoke

    Explore Stage
    Deploy & Smoke

    Man. exploratory
    tests

    System Tests

    5min
    Deploy & Smoke

    Monitoring
    & Logging

    Explore Env
    ~30min
    Git
    Source Code
    Test Code
    Infra Code
    Data Migrations
    Configuration
    … Required

    Situational

    View full-size slide

  32. Slides by richargh.de and
    32
    Commit Stage Beta Stage Gamma Stage
    Build

    Package & Store

    Static Code
    Analysis (Lint)

    Unit Tests

    Prepare Stage Prod Stage
    Integration Tests

    Secrets Detection

    “Dependency”
    Analysis (SCA)

    Static Security
    Tests (SAST)
    ✓ 1 AWS DPRA https://pipelines.devops.aws.dev/
    Fetch & Cache
    Dependencies

    Beta Env Gamma Env Prod Env
    Self-Service
    Automatic
    Deploy & Smoke

    Explore Stage
    Deploy & Smoke

    Man. exploratory
    tests

    System Tests

    Deploy & Smoke

    Resilience Tests

    Soak Tests

    Dynamic Security
    Tests (DAST)

    Monitoring &
    Logging

    Performance
    Tests

    5min
    Deploy & Smoke

    Monitoring
    & Logging

    Explore Env
    ~30min ~4h
    Git
    Source Code
    Test Code
    Infra Code
    Data Migrations
    Configuration
    … Required

    Situational

    View full-size slide

  33. Slides by richargh.de and
    33
    Commit Stage Beta Stage Gamma Stage Prod Stage
    1 AWS DPRA https://pipelines.devops.aws.dev/
    Beta Env Gamma Env Prod Env
    Self-Service
    Automatic
    Explore Stage
    Explore Env
    Required

    Situational

    Does it work? Does it work
    in a real
    environment?
    Does it work
    in a production-like
    environment?
    Does it work for real?
    Does it work
    in unexpected situations?
    Does it work for me?

    View full-size slide

  34. Slides by richargh.de and
    More infos? AWS has a guide
    for you J
    Deployment Pipeline Reference Architecture (DPRA)
    34
    1 AWS DPRA https://pipelines.devops.aws.dev/

    View full-size slide

  35. Slides by richargh.de and
    Does it work?
    35
    Commit Stage
    Build

    Package & Store

    Static Code
    Analysis (Lint)

    Unit Tests

    Integration Tests

    Secrets Detection

    “Dependency”
    Analysis (SCA)

    Static Security
    Tests (SAST)

    Prepare Stage
    Fetch & Cache
    Dependencies

    Static Security Analysis
    Dynamic Feature Tests

    View full-size slide

  36. Slides by richargh.de and
    36
    Prepare Stage
    Fetch &
    Cache
    Dependencies

    # in /.gitlab-ci.yaml
    1. # image+env-vars for all jobs
    2. variables:
    3. NODE_IMAGE: /node:18.x-alpine3.y
    4. image: $NODE_IMAGE
    5.
    6. .node:cache:
    7. cache:
    8. policy: pull
    9. key:
    10. files:
    11. - package-lock.json
    12. prefix: node
    13. paths:
    14. - node_modules/
    15.
    16.node_update_dependencies:
    17. stage: prepare
    18. extends:
    19. - .node:cache
    20. cache:
    21. policy: pull-push
    22. script:
    23. - npm ci
    Commit Stage Build

    # in /.gitlab-ci.yaml
    1. build:
    2. stage: commit
    3. extends:
    4. - .node:cache
    5. script:
    6. - npm run build:prod
    7. artifacts:
    8. paths:
    9. - dist/
    10. expire_in: 8 hours
    1 Gitlab Examples https://docs.gitlab.com/ee/ci/examples/
    2 Cache https://docs.gitlab.com/ee/ci/yaml/index.html#cache
    Update cache
    Use cache
    Share build result with
    other jobs

    View full-size slide

  37. Slides by richargh.de and
    37
    Commit Stage
    1. # circular analysis
    2. - npx madge --extensions ts
    3. --circular ./src/
    4. # architecture
    5. - npx ts-arch
    6. - npx depcruise --config .arch.js src
    7. # for Java: use ArchUnit
    Static Analysis: Architecture

    1. - npx eslint src/ tests/
    2. - hadolint “Dockerfile”
    3. - ... Sonar, shellcheck
    Static Analysis: Common Issues

    1. - gitleaks detect
    Static Analysis: Secrets Detection

    1. - trivy fs --scanners vuln,secret
    2. ,config myproject/
    3. - trivy image “${IMAGE}:”
    “Dependency” Analysis (SCA)

    1. - sonar-scanner
    Static Security Tests (SAST)

    View full-size slide

  38. Slides by richargh.de and
    38
    Commit Stage Beta Stage Gamma Stage
    Prepare Stage
    1 https://www.gocd.org/2016/06/08/Add-performance-testing-to-your-Continuous-Delivery-pipeline/
    Gamma Env
    Self-Service
    Automatic
    Deploy & Smoke

    Resilience Tests

    Soak Tests

    Dynamic Security
    Tests (DAST)

    Monitoring &
    Logging

    Performance
    Tests1

    ~4h
    Required

    Situational

    Does it work in a production-like
    environment?
    Load and Stress tests
    Continuous Load over hours
    Inject failures, kill services
    Try to exploit security holes
    (OWASP ZAP)

    View full-size slide

  39. Slides by richargh.de and
    39
    Gamma Stage
    # In your perf-scenario.yaml
    1. config:
    2. target: “https://
    3. phases:
    4. # how your virtual users arrive
    5. - duration: 60
    6. arrivalRate: 5
    7. name: Warm up
    8. # your service level aggreements
    9. ensure:
    10. thresholds:
    11. - "'http.response_time.p95'": 200
    12.
    13.scenarios:
    14. # what scenarios to run
    15. - name: ”Create Orders"
    16. flow:
    17. - post:
    18. name: ”Create order"
    19. url: "/api/orders”
    20. json:
    21. name: "{{ name }}"
    Performance Tests

    1 Artillery https://www.artillery.io/docs/guides/getting-started/writing-your-first-test

    View full-size slide

  40. Slides by richargh.de and
    Monitor at least the RED1 Metrics
    40
    1 https://grafana.com/files/grafanacon_eu_2018/Tom_Wilkie_GrafanaCon_EU_2018.pdf
    2 https://github.com/grafana/intro-to-mlt
    Rate
    Delay
    Errors

    View full-size slide

  41. Slides by richargh.de and
    These are all Feedback Loops
    • Automated Unit/Int. Tests
    • Static Tests (Architecture, …)
    • Integration into Trunk
    • Automated Large Tests
    • Performance, etc. Tests
    • Exploratory Tests
    • Production Metrics
    • Customer Reactions
    41
    Idea

    View full-size slide

  42. Slides by richargh.de and
    42
    Why How
    Who
    Prepare
    Build
    Optimize

    View full-size slide

  43. Slides by richargh.de and
    The pipeline consists of
    Everything we need to go from commit to releasable outcome
    43

    View full-size slide

  44. Slides by richargh.de and
    The Goal of a Pipeline
    Have we done everything we can to disprove that this is a releasable
    candidate?
    44

    View full-size slide

  45. Slides by richargh.de and
    Build on the shoulder of giants
    So much stuff you can use and most of it is free
    45

    View full-size slide

  46. Slides by richargh.de and
    Automate Dependency Updates
    46
    1 https://docs.renovatebot.com/configuration-options/
    Renovate Stage
    # in /.gitlab-ci.yaml
    1. renovate:
    2. stage: renovate
    3. image: $REOVATE_IMAGE
    4. script:
    5. - renovate $(cat repositories.txt | xargs) --token $TOKEN_GIT –log-file renovate.log
    6. artifacts:
    7. paths:
    8. - renovate.log
    # in /repositories.txt
    1. /
    In it’s own repository with a
    bi-hourly schedule
    # in /config.js
    1. module.exports = {
    2. “branchPrefix”: “chore/renovate-”,
    3. ...
    4. }

    View full-size slide

  47. Slides by richargh.de and
    Parallize Whatever you can
    47
    1 https://continuousdelivery.com/2010/09/deployment-pipeline-anti-patterns/
    Commit Stage
    Build

    Package & Store

    Static Code
    Analysis (Lint)

    Unit Tests

    Integration Tests

    Secrets Detection

    “Dependency”
    Analysis (SCA)

    Static Security
    Tests (SAST)

    Most of the stage can and
    should run in parallel.
    Run tests in
    parallel
    Run test in parallel with
    separate db schema per test set

    View full-size slide

  48. Slides by richargh.de and
    Be Flexible - Stay Wide and Short1
    48
    1 https://continuousdelivery.com/2010/09/deployment-pipeline-anti-patterns/
    Commit Stage Beta Stage Gamma Stage Prod Stage
    Beta Env Gamma Env Prod Env
    Explore Stage
    Explore Env
    Give choices to smart humans.

    View full-size slide

  49. Slides by richargh.de and
    For daily deployments, consider Gamma
    ONE deployments1
    49
    1 Automating safe, hands-off deployments https://aws.amazon.com/builders-library/automating-safe-hands-off-deployments/
    Commit Stage Beta Stage Gamma Stage Prod Stage
    Beta Env Gamma Env Prod Env
    Is the service backwards compatible?
    (1) Deploy ONE instance side-by with old version
    (2) Unleash normal test traffic and measure
    (3) If all ok, after 1 hour, deploy all instances
    (4) Run regular Gamma Stage tests

    View full-size slide

  50. Slides by @arghrich
    It’s not just about the Pipeline
    50
    Pipeline
    Easy Hard
    Techniques
    Something you do
    Methods
    When you do it
    Mindset
    What you actually do

    View full-size slide

  51. Slides by @arghrich
    Continuous Delivery Principles
    • Build quality in
    • Work in small batches
    • Computers Perform Repetitive Tasks,
    People Solve Problems
    • Relentlessly Pursue Continuous Improvement
    • Everyone is Responsible
    51
    CD Principles by Jez Humble
    Four Principles of Low-Risk Software Releases by Jez Humble

    View full-size slide

  52. Slides by richargh.de and
    We are what we repeatedly do.
    Excellence then, is not an act,
    but a habit.1
    52
    1 The Story of Philosophy

    View full-size slide

  53. Slides by @arghrich
    Thanks
    53
    Richard Gross (he/him)
    richargh.de/
    speakerdeck.com/richargh
    @arghrich
    Works for maibornwolff.de/
    People. Code. Commitment.
    DE TN ES
    Archaeologist Auditor
    Hypermedia-
    Designer

    View full-size slide

  54. Slides by richargh.de and
    Backup

    View full-size slide

  55. Slides by richargh.de and
    Gitlab-Ci Links
    • Create a Job: https://docs.gitlab.com/ee/ci/jobs/
    • Caching&Artifacts https://docs.gitlab.com/ee/ci/caching/
    • Predefined Gitlab Variables:
    https://docs.gitlab.com/ee/ci/variables/predefined_variables.html
    • Unit Test Reports:
    https://docs.gitlab.com/ee/ci/testing/unit_test_reports.html
    • Test Coverage:
    https://docs.gitlab.com/ee/ci/testing/test_coverage_visualization.html#exa
    mple-test-coverage-configurations
    • Metrics Reports: https://docs.gitlab.com/ee/ci/testing/metrics_reports.html
    • Gitlab-Ci Yaml reference: https://docs.gitlab.com/ee/ci/yaml/index.html
    • Gitlab-Ci Examples: https://docs.gitlab.com/ee/ci/examples/
    55

    View full-size slide

  56. Slides by richargh.de and
    Flickr 2009
    • 10 Deploys per Day
    56
    1 https://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr

    View full-size slide

  57. Slides by richargh.de and
    Etsy 20141
    • 50 Deploys/Day
    57
    1 https://www.infoq.com/news/2014/03/etsy-deploy-50-times-a-day/

    View full-size slide

  58. Slides by richargh.de and
    Facebook 20xx1
    • Over 1B users worldwide
    • About 700 developers committing against their frontend source code
    repo
    • Single frontend code binary about 1.5GB in size
    • Pushed out to many thousands of servers (the number is not public)
    • Changes can go from check-in to end users in as quickly as 40
    minutes
    • Release process almost entirely invisible to the users
    • 2 Deploys per Day in 20122
    • Quasi-Continuous Deploy in 20173
    58
    1 https://dzone.com/articles/pushing-twice-daily-our
    2 https://engineering.fb.com/2012/08/03/uncategorized/ship-early-and-ship-twice-as-often/
    3 https://engineering.fb.com/2017/08/31/web/rapid-release-at-massive-scale/
    4 https://research.facebook.com/publications/continuous-deployment-at-facebook-and-oanda/

    View full-size slide

  59. Slides by richargh.de and
    AWS Pipeline
    • Thousands of teams + Microservice architecture + Multiple
    environments + Continuous Delivery
    • = 50 million deployments a year (2015) 1
    1. Source
    2. Build (1st build, 2nd Unit test)
    3. Beta (1st Deploy, 2nd UI test)
    4. Gamma (1st Deploy, 2nd Load test)
    5. Production (1st Deploy region1, 2nd Deploy region2, 3rd Deploy
    region3)
    59
    1 Transforming Software Development https://www.youtube.com/watch?v=YCrhemssYuI

    View full-size slide

  60. Slides by richargh.de and
    Amazon Stats (around 2012)1
    11.6 seconds
    Mean time between deplovments (weekday)
    1,079
    Max # of deployments in a single hour
    10,000
    Mean # of hosts simultaneously receiving a deployment
    30,000
    Max # of hosts simultaneously receiving a deployment
    60
    1 Velocity Culture https://www.youtube.com/watch?v=dxk8b9rSKOo

    View full-size slide

  61. Slides by richargh.de and
    Google Repository (around 2016)1
    • Billions of LoC in a single repository (35 million commits)
    • Including configuration and documentation
    • > 2.000 projects under active development
    • 40.000 commits per workday
    • Multiple Services with one version: HEAD
    • Simplified dependency management: static linking without diamond
    dependency problem
    • Atomic changes, large scale refactoring
    • Homegrown VCS (Piper -> Bazel is a subset)
    • Trunk-Based Development
    • More than 50 million test cases run per day
    • 25.000 Developers
    61
    1 https://cacm.acm.org/magazines/2016/7/204032-why-google-stores-billions-of-lines-of-code-in-a-single-repository/fulltext
    2 https://www.infoq.com/presentations/Continuous-Testing-Build-Cloud/

    View full-size slide

  62. Slides by richargh.de and
    Google Pipeline
    • Code is partitioned into packages, folders with a BUILD file
    • The BUILD file describes what other packages it depends on and
    what it provides
    62
    1 https://google-engtools.blogspot.com/2011/08/build-in-cloud-how-build-system-works.html
    2 https://www.infoq.com/presentations/Continuous-Testing-Build-Cloud/

    View full-size slide

  63. Slides by richargh.de and
    Netflix
    • Red/Black-Deployment
    • Quasi-Continuous Deploys
    63
    1 https://netflixtechblog.com/deploying-the-netflix-api-79b6176cc3f0?gi=386880e182fb

    View full-size slide

  64. Slides by richargh.de and
    Stackoverflow Architecture
    • The main, retail, public site serves over a hundred million unique
    users and 6,000 page views per second.
    • 1.3 BILLION page views per month
    • 1 Monolith deployed to Own data center
    • 9 Web Servers
    • 4 SQL Servers
    • 2 Redis Servers
    • 3 Elastic Search Servers
    • 2 HAProxy Servers (Load Balancers)
    64
    1 https://stackexchange.com/performance
    2 https://www.zdnet.com/article/stack-overflow-cto-from-bootstrapped-to-scaling-one-of-the-webs-biggest-properties/ 2022

    View full-size slide

  65. Slides by richargh.de and
    Now Unit Test1 your Modules
    • By which you mean one test file per class? (no ✘)
    • By which you mean together with the database? (no ✘)
    • By which you mean … (probably also no ✘)
    65
    1 A Set of Unit Testing Rules https://www.artima.com/weblogs/viewpost.jsp?thread=126923

    View full-size slide

  66. Slides by richargh.de and
    A Set of Unit Testing Rules
    A test is not a unit test if:
    • It talks to a database
    • It communicates across the network
    • It touches the file system
    • It can't run at the same time as any of your other unit tests
    • You have to do special things to your environment (such as editing
    config files) to run it.
    Tests that do these things aren't bad. Often they are worth writing, and
    they can be written in a unit test harness. However, it is important to be
    able to separate them from true unit tests so that we can keep a set of
    tests that we can run fast whenever we make our changes.
    66
    1 A Set of Unit Testing Rules https://www.artima.com/weblogs/viewpost.jsp?thread=126923

    View full-size slide

  67. Slides by richargh.de and
    Test Sizes1, not bike sheds
    67
    1 Google Test Sizes https://testing.googleblog.com/2010/12/test-sizes.html
    Feature Small Medium Large
    Network access No Localhost only Yes
    Database No Yes Yes
    File system access No Yes Yes
    Use external system No Discouraged Yes
    Multiple threads No Yes Yes
    Sleep statements No Yes Yes
    System properties No Yes Yes
    Time limit (seconds) 60 300 900+
    Test:
    modules, functions
    Test:
    (Repo) adapters
    Test:
    BE Api, FE&BE together

    View full-size slide

  68. Slides by richargh.de and
    Larger Size
    Complicated Test Harness
    Imprecise Feedback
    Coarse assertions
    Slower
    à $$$
    Smaller Size
    Simple Test Harness
    Specific Feedback
    Strict assertions
    Faster
    à $
    Test Sizes Guide
    Dependency
    Stubbed-Dep.
    Small
    Code being tested
    Test Amount
    Medium
    DB
    L
    a
    r
    g
    e
    Exploratory

    View full-size slide

  69. Slides by richargh.de and
    69
    Commit Stage
    # in /.gitlab-ci.yaml
    1. small_tests:
    2. stage: commit
    3. extends:
    4. - .node:cache
    5. variables:
    6. KUBERNETES_CPU_REQUEST: 1000m
    7. KUBERNETES_MEMORY_REQUEST: 2500Mi
    8. script:
    9. - npx jest tests/small --ci --reporters=default --reporters=jest-junit
    10. # optional artifacts but this way you’ll see nice reports in Gitlab
    11. artifacts:
    12. paths:
    13. - junit.xml
    14. expire_in: 5 days
    15. reports:
    16. junit: junit.xml
    Small Tests

    1 Gitlab Unit Test Reports https://docs.gitlab.com/ee/ci/testing/unit_test_reports.html

    View full-size slide

  70. Slides by richargh.de and
    70
    Commit Stage
    # in /.gitlab-ci.yaml
    1. medium_tests:
    2. stage: commit
    3. extends:
    4. - .node:cache
    5. variables:
    6. KUBERNETES_CPU_REQUEST: 1000m
    7. KUBERNETES_MEMORY_REQUEST: 2500Mi
    8. services:
    9. # start a database
    10. # remove for small tests they don’t need services
    11. - name:
    12. alias: mongo
    13. script:
    14. # create junit report because Gitlab can display that
    15. - npx jest tests/medium --ci --reporters=default --reporters=jest-junit
    16. # optional artifacts but this way you’ll see nice reports in Gitlab
    17. artifacts:
    18. paths:
    19. - junit.xml
    20. expire_in: 5 days
    21. reports:
    22. junit: junit.xml
    Medium Tests

    1 Gitlab Unit Test Reports https://docs.gitlab.com/ee/ci/testing/unit_test_reports.html

    View full-size slide

  71. Slides by richargh.de and
    71
    Commit Stage
    # in /.gitlab-ci.yaml
    1. build:
    2. stage: commit
    3. extends:
    4. - .node:cache
    5. script:
    6. - npm run build:prod
    7. artifacts:
    8. paths:
    9. - dist/
    10. expire_in: 8 hours
    Commit Stage
    Build

    # in /.gitlab-ci.yaml
    1. package:
    2. stage: commit
    3. needs:
    4. - job: build
    5. artifacts: true
    6. image:
    7. name: /kaniko-project/executor:xyz
    8. entrypoint: [ "" ]
    9. script:
    10. - DEST=”"
    11. - /kaniko/executor
    12. --context "${CI_PROJECT_DIR}"
    13. --dockerfile "${CI_PROJECT_DIR}/Dockerfile"
    14. --destination “${DEST}:${CI_COMMIT_SHORT_SHA}”
    15. --destination “${DEST}:latest”
    16. --push-retry 3
    1 Kaniko https://github.com/GoogleContainerTools/kaniko
    2 Gitlab Kaniko https://docs.gitlab.com/ee/ci/docker/using_kaniko.html
    Package & Store

    Create artifacts
    Use artifacts

    View full-size slide

  72. Slides by richargh.de and
    72
    Commit Stage Beta Stage
    Prepare Stage
    1 AWS DPRA https://pipelines.devops.aws.dev/
    Beta Env
    Self-Service
    Automatic
    Deploy & Smoke

    Large Tests

    ~30min
    Required

    Situational

    Does it work in a real environment?

    View full-size slide

  73. Slides by richargh.de and
    73
    Beta Stage
    1. deploy_beta:
    2. stage: beta
    3. image: $HELM_IMAGE
    4. variables:
    5. # do not clone source code
    6. GIT_STRATEGY: none
    7. # Gitlab can display latest deployment
    8. environment:
    9. name: beta
    10. script:
    11. # visualize deploy changes
    12. - helm diff upgrade
    13. --install --color --context 5
    14. --detailed-exitcode <...>
    15. # deploy
    16. - helm upgrade
    17. --install
    18. --values .helm/values-beta.yaml
    19. --namespace beta
    20. -beta
    21. .helm/
    22. # wait for deployment to finish
    23. - kubectl rollout status
    24. --request-timeout=5s
    25. -n=beta
    26. -beta
    Deploy
    ✓ Beta Stage
    1. smoke_beta:
    2. stage: commit
    3. needs:
    4. - job: deploy_beta
    5. artifacts: false
    6. script:
    7. - npm run test:smoke -- -ns=beta
    Smoke

    1 Gitlab Environments https://docs.gitlab.com/ee/ci/environments/
    2 Helm Diff Plugin https://github.com/databus23/helm-diff
    A smoke test
    Uses non-destructive requests
    That are invisible to the user
    Examples:
    GET /users but also
    PUT /orders
    PUT /orders

    View full-size slide

  74. Slides by richargh.de and
    AWS DPRA1
    74
    1 https://pipelines.devops.aws.dev/application-pipeline/

    View full-size slide