How to Build the Greatest and Best Pipeline in the World - JAX 2023

The Heart of any well-performing team is it‘s own hand-crafted Engineering process. But a heart would be useless without veins that pass along the blood and can take the pressure that the heart builds. In a similar manner a team needs it’s own hand-crafted pipeline. Most teams today have one but only few teams have a pipeline that can take the pressure and not collapse. This talk is how to build this Greatest and Best Pipeline in the world.

To be realistic though, it‘s more of a tribute to the greatest and best pipeline in the world. Not because we couldn’t remember but because we want to show you how you can build your greatest and best pipeline.

Since your process is different from ours we‘ll show you how to hand-craft your pipeline and not prescripe a solution that won't fit.

The examples will cover building, bundling, testing and securing your deployment artifact. They'll be written in Gitlab-Ci format and deploy to Kubernetes, but you can apply the same principles and patterns to any other pipeline and any other environment.

Transcript

1. 10.05.23
   How to Build the Greatest
   and Best Pipeline in the
   World
   Richard Gross (he/him)
   Hypermedia-
   Designer
   Archaeologist Auditor
   richargh.de/
   speakerdeck.com/richargh
   @arghrich
   

   View Slide

2. Slides by richargh.de and
   Which Company has
   the Greatest and Best
   Pipeline in the World?
   2
   

   View Slide

3. Slides by richargh.de and
   Deploy all the things
   3
   Amazon (~2012)a1
   • Multiple “Primitives”
   • Max deploys in one hour:
   1,079
   fl1 https://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr
   e1 https://www.infoq.com/news/2014/03/etsy-deploy-50-times-a-day/
   fb1 https://engineering.fb.com/2012/08/03/uncategorized/ship-early-and-ship-twice-as-often/
   fb2 https://engineering.fb.com/2017/08/31/web/rapid-release-at-massive-scale/
   a1 Velocity Culture https://www.youtube.com/watch?v=dxk8b9rSKOo
   g1 https://cacm.acm.org/magazines/2016/7/204032-why-google-stores-billions-of-lines-of-code-in-a-single-repository/fulltext
   Facebook (~2012)fb1
   • 1,5 TB Monolith
   • 2 Deploys per Day
   Flickr (~2007)fl1
   • 10 deploys per Day
   • Where DevOps went public
   Etsy (~2014)fl1
   • 50 deploys per Day
   • 60 million monthly visits
   Facebook (~2017)fb2
   • TB Monolith
   • Quasi-Continuous Deploys
   Google (~2016)g1
   • Millions of LoC in one Repo
   • Where MonoRepo went public
   • Quasi-Continuous Deploys
   Works with all architectures
   Also when you are heavily regulated
   

   View Slide

4. Slides by richargh.de and
   4
   Why How
   Who
   

   View Slide

5. Slides by richargh.de and
   5
   Why How
   Who
   

   View Slide

6. Slides by richargh.de and
   Big bang releases only lead to
   big bangs
   6
   

   View Slide

7. Slides by richargh.de and
   (1 We are more often wrong then right
   about which features produce value
   7
   1 https://blackswanfarming.com/how-to-tilt-the-innovation-playing-field
   1
   

   View Slide

8. Slides by richargh.de and
   (2) Big Releases are High-Risk
   • The effect of sth big is extremely hard to anticipate, way too
   many variables
   • If something goes wrong
   • You have a huge change to sift through
   • It’s hard to roll-back
   8
   2
   

   View Slide

9. Slides by richargh.de and
   (2) Big Releases are Expensive
   9
   3
   Cost
   Batch Size
   Transaction Cost
   Holding Cost
   Total Cost
   Optimal
   Batch Size
   What if we could bring
   the transaction costs
   way down?
   1 https://hbr.org/2012/05/six-myths-of-product-development
   

   View Slide

10. Slides by richargh.de and
    Low-risk releases are
    incremental
    10
    

    View Slide

11. Slides by richargh.de and
    Delivery
    Deliver small experiments and
    gather feedback
    Idea Slice It
    Bet on it Code it
    Commit
    Build It
    Production
    Deploy It
    Release It
    Feedback
    

    View Slide

12. Slides by richargh.de and
    The Deployment Pipeline
    Make it economic to work in small batches
    12
    

    View Slide

13. Slides by richargh.de and
    13
    Why How
    Who
    

    View Slide

14. Slides by richargh.de and
    “If we only just work harder
    we’ll catch up”
    That never works
    14
    

    View Slide

15. Slides by richargh.de and
    Prefer to work Smarter, with a sharper
    saw1
    15
    1 Image by Henrik Kniberg
    

    View Slide

16. Slides by richargh.de and
    0%
    10%
    20%
    30%
    40%
    50%
    60%
    70%
    80%
    90%
    100%
    Legacy Intermediate Normal
    Team Work
    Innovation Sharpening the Saw Unplanned Work + Rework
    Sharpen the Saw to go faster
    16
    Features not on-time,
    on-budget, full of bugs
    and you can’t get to the
    interesting stuff.
    Improve Architecture,
    automated tests,
    pipeline and your
    process.
    Continuously sharpen,
    so you have time for
    features
    1 Loosely based on Accelerate – State of DevOps 2018
    

    View Slide

17. Slides by richargh.de and
    The Story of the
    Hp Laserjet Firmware Team
    17
    2008
    10% - code integration
    20% - detailed planning
    25% - porting code
    25% - current product support
    15% - manual testing
    ~5% - innovation
    1 A Practical Approach to Large Scale Agile Development https://www.youtube.com/watch?v=Trqjj3d3lhQ
    2 Why Scaling Agile Does not Work https://www.youtube.com/watch?v=2zYxWEZ0gYg
    2011
    2% - continuous integration
    5% - agile planning
    15% - one main branch
    10% - one branch cpe
    5% - most testing automated
    ~40% - innovation
    23% managing automated tests
    

    View Slide

18. Slides by richargh.de and
    Remember though, you are not
    AWS or Google
    Or do you have hundreds or thousands of engineers working on improving the
    release?
    18
    

    View Slide

19. Slides by richargh.de and
    Build your Greatest and Best
    Pipeline in the World
    Make it economic to work in small batches
    19
    

    View Slide

20. Slides by richargh.de and
    20
    Why How
    Who
    Prepare
    Build
    Optimize
    

    View Slide

21. Slides by richargh.de and
    Make it economic
    to work in small batches
    • Good pipelines require good architecture
    • Uncouple your code
    • Test on multiple levels
    • Version everything à EverythingAsCode
    • Convince your Stakeholders
    21
    

    View Slide

22. Slides by richargh.de and
    Uncouple your Code
    High Coupling
    Low Cohesion
    Low Coupling
    High Cohesion
    22
    Dependency
    Unknown Source
    

    View Slide

23. Slides by richargh.de and
    Write and test focused modules
    • Systems consist of modules
    • Modules
    • Provide a simple api
    • Hide their internals
    • Are deep, not shallow1
    • Call api of other modules
    23 1 A Philosophy of Software Design
    Dependency
    

    View Slide

24. Slides by richargh.de and
    Make it clear what your module does,
    what the api is and what is internal
    src/features/core/
    ├── renting/
    │ ├── internal/
    │ │ ├── a.service.ts
    │ │ ├── a.repository.ts
    │ │ └── ...
    │ ├── exposed-types/
    │ │ ├── a.type.ts
    │ │ └── ...
    │ ├── a.renting.facade.ts
    │ └── another.renting.facade.ts
    ├── paying/
    │ ├── internal/
    │ ├── exposed-types/
    │ └── a.paying.facade.ts
    src/commons/── paying/
    │ ├── internal/
    24
    ✘
    The focus of a module
    is declared via verb
    Internals are declared
    via your convention
    Facade and types
    define the module api
    Great place to test
    • Test is structure-insensitive
    • If the test is simple to write, the
    api is also simple
    Great place to test
    • If it’s a leaf
    • If the test is simple to write, the
    code simple
    Bad place to test
    • If it composes multiple other
    internals
    • The test now locks down your
    internal structure
    

    View Slide

25. Slides by richargh.de and
    Convince your Stakeholders
    25
    

    View Slide

26. Slides by richargh.de and
    Speed and Stability are possible without
    tradeoffs1,2 (Research-confirmed since 2013)
    26
    1 Accelerate – State of DevOps Report 2018
    2 The Data Behind DevOps https://www.youtube.com/watch?v=DgpsX5yLXQw
    Throughput Stability
    

    View Slide

27. Slides by richargh.de and
    Speed and Stability enable one another
    27
    1 Accelerate – State of DevOps Report 2018
    Low Performer
    Between once per week
    and once per month
    Between one month
    and six months
    Between one week
    and one month
    46-60%
    Elite Performer
    On demand
    (multiple deploys per day)
    Less than
    one hour
    Less than
    one hour
    0-15%
    Throughput
    Deployment
    Frequency
    Lead Time
    for Changes
    Stability
    Time to
    Restore Service
    Change
    Failure Rate
    

    View Slide

28. Slides by richargh.de and
    28
    Why How
    Who
    Prepare
    Build
    Optimize
    

    View Slide

29. Slides by richargh.de and
    Automate, automate, automate
    Automate as much as possible but don’t overdo it1,2
    Deployment should be at most one mouse click away
    Lots of automated Checks to guard quality
    29
    1 https://en.wikipedia.org/wiki/Ironies_of_Automation
    2 How Complex Systems Fail
    

    View Slide

30. Slides by richargh.de and
    30
    Commit Stage
    Build
    ✓
    Package & Store
    ✓
    Static Code
    Analysis (Lint)
    ✓
    Unit Tests
    ✓
    Integration Tests
    ✓
    1 AWS DPRA https://pipelines.devops.aws.dev/
    Self-Service
    Automatic
    Explore Stage
    Deploy & Smoke
    ✓
    Man. exploratory
    tests
    ✓
    5min
    Explore Env
    Git
    Source Code
    Test Code
    Infra Code
    Data Migrations
    Configuration
    … Required
    ✓
    Situational
    ✓
    Start here, then iterate
    

    View Slide

31. Slides by richargh.de and
    31
    Commit Stage Beta Stage
    Build
    ✓
    Package & Store
    ✓
    Static Code
    Analysis (Lint)
    ✓
    Unit Tests
    ✓
    Prepare Stage Prod Stage
    Integration Tests
    ✓
    1 AWS DPRA https://pipelines.devops.aws.dev/
    Fetch & Cache
    Dependencies
    ✓
    Beta Env Prod Env
    Self-Service
    Automatic
    Deploy & Smoke
    ✓
    Explore Stage
    Deploy & Smoke
    ✓
    Man. exploratory
    tests
    ✓
    System Tests
    ✓
    5min
    Deploy & Smoke
    ✓
    Monitoring
    & Logging
    ✓
    Explore Env
    ~30min
    Git
    Source Code
    Test Code
    Infra Code
    Data Migrations
    Configuration
    … Required
    ✓
    Situational
    ✓
    

    View Slide

32. Slides by richargh.de and
    32
    Commit Stage Beta Stage Gamma Stage
    Build
    ✓
    Package & Store
    ✓
    Static Code
    Analysis (Lint)
    ✓
    Unit Tests
    ✓
    Prepare Stage Prod Stage
    Integration Tests
    ✓
    Secrets Detection
    ✓
    “Dependency”
    Analysis (SCA)
    ✓
    Static Security
    Tests (SAST)
    ✓ 1 AWS DPRA https://pipelines.devops.aws.dev/
    Fetch & Cache
    Dependencies
    ✓
    Beta Env Gamma Env Prod Env
    Self-Service
    Automatic
    Deploy & Smoke
    ✓
    Explore Stage
    Deploy & Smoke
    ✓
    Man. exploratory
    tests
    ✓
    System Tests
    ✓
    Deploy & Smoke
    ✓
    Resilience Tests
    ✓
    Soak Tests
    ✓
    Dynamic Security
    Tests (DAST)
    ✓
    Monitoring &
    Logging
    ✓
    Performance
    Tests
    ✓
    5min
    Deploy & Smoke
    ✓
    Monitoring
    & Logging
    ✓
    Explore Env
    ~30min ~4h
    Git
    Source Code
    Test Code
    Infra Code
    Data Migrations
    Configuration
    … Required
    ✓
    Situational
    ✓
    

    View Slide

33. Slides by richargh.de and
    33
    Commit Stage Beta Stage Gamma Stage Prod Stage
    1 AWS DPRA https://pipelines.devops.aws.dev/
    Beta Env Gamma Env Prod Env
    Self-Service
    Automatic
    Explore Stage
    Explore Env
    Required
    ✓
    Situational
    ✓
    Does it work? Does it work
    in a real
    environment?
    Does it work
    in a production-like
    environment?
    Does it work for real?
    Does it work
    in unexpected situations?
    Does it work for me?
    

    View Slide

34. Slides by richargh.de and
    More infos? AWS has a guide
    for you J
    Deployment Pipeline Reference Architecture (DPRA)
    34
    1 AWS DPRA https://pipelines.devops.aws.dev/
    

    View Slide

35. Slides by richargh.de and
    Does it work?
    35
    Commit Stage
    Build
    ✓
    Package & Store
    ✓
    Static Code
    Analysis (Lint)
    ✓
    Unit Tests
    ✓
    Integration Tests
    ✓
    Secrets Detection
    ✓
    “Dependency”
    Analysis (SCA)
    ✓
    Static Security
    Tests (SAST)
    ✓
    Prepare Stage
    Fetch & Cache
    Dependencies
    ✓
    Static Security Analysis
    Dynamic Feature Tests
    

    View Slide

36. Slides by richargh.de and
    36
    Prepare Stage
    Fetch &
    Cache
    Dependencies
    ✓
    # in /.gitlab-ci.yaml
    1. # image+env-vars for all jobs
    2. variables:
    3. NODE_IMAGE: /node:18.x-alpine3.y
    4. image: $NODE_IMAGE
    5.
    6. .node:cache:
    7. cache:
    8. policy: pull
    9. key:
    10. files:
    11. - package-lock.json
    12. prefix: node
    13. paths:
    14. - node_modules/
    15.
    16.node_update_dependencies:
    17. stage: prepare
    18. extends:
    19. - .node:cache
    20. cache:
    21. policy: pull-push
    22. script:
    23. - npm ci
    Commit Stage Build
    ✓
    # in /.gitlab-ci.yaml
    1. build:
    2. stage: commit
    3. extends:
    4. - .node:cache
    5. script:
    6. - npm run build:prod
    7. artifacts:
    8. paths:
    9. - dist/
    10. expire_in: 8 hours
    1 Gitlab Examples https://docs.gitlab.com/ee/ci/examples/
    2 Cache https://docs.gitlab.com/ee/ci/yaml/index.html#cache
    Update cache
    Use cache
    Share build result with
    other jobs
    

    View Slide

37. Slides by richargh.de and
    37
    Commit Stage
    1. # circular analysis
    2. - npx madge --extensions ts
    3. --circular ./src/
    4. # architecture
    5. - npx ts-arch
    6. - npx depcruise --config .arch.js src
    7. # for Java: use ArchUnit
    Static Analysis: Architecture
    ✓
    1. - npx eslint src/ tests/
    2. - hadolint “Dockerfile”
    3. - ... Sonar, shellcheck
    Static Analysis: Common Issues
    ✓
    1. - gitleaks detect
    Static Analysis: Secrets Detection
    ✓
    1. - trivy fs --scanners vuln,secret
    2. ,config myproject/
    3. - trivy image “${IMAGE}:”
    “Dependency” Analysis (SCA)
    ✓
    1. - sonar-scanner
    Static Security Tests (SAST)
    ✓
    

    View Slide

38. Slides by richargh.de and
    38
    Commit Stage Beta Stage Gamma Stage
    Prepare Stage
    1 https://www.gocd.org/2016/06/08/Add-performance-testing-to-your-Continuous-Delivery-pipeline/
    Gamma Env
    Self-Service
    Automatic
    Deploy & Smoke
    ✓
    Resilience Tests
    ✓
    Soak Tests
    ✓
    Dynamic Security
    Tests (DAST)
    ✓
    Monitoring &
    Logging
    ✓
    Performance
    Tests1
    ✓
    ~4h
    Required
    ✓
    Situational
    ✓
    Does it work in a production-like
    environment?
    Load and Stress tests
    Continuous Load over hours
    Inject failures, kill services
    Try to exploit security holes
    (OWASP ZAP)
    

    View Slide

39. Slides by richargh.de and
    39
    Gamma Stage
    # In your perf-scenario.yaml
    1. config:
    2. target: “https://
    3. phases:
    4. # how your virtual users arrive
    5. - duration: 60
    6. arrivalRate: 5
    7. name: Warm up
    8. # your service level aggreements
    9. ensure:
    10. thresholds:
    11. - "'http.response_time.p95'": 200
    12.
    13.scenarios:
    14. # what scenarios to run
    15. - name: ”Create Orders"
    16. flow:
    17. - post:
    18. name: ”Create order"
    19. url: "/api/orders”
    20. json:
    21. name: "{{ name }}"
    Performance Tests
    ✓
    1 Artillery https://www.artillery.io/docs/guides/getting-started/writing-your-first-test
    

    View Slide

40. Slides by richargh.de and
    Monitor at least the RED1 Metrics
    40
    1 https://grafana.com/files/grafanacon_eu_2018/Tom_Wilkie_GrafanaCon_EU_2018.pdf
    2 https://github.com/grafana/intro-to-mlt
    Rate
    Delay
    Errors
    

    View Slide

41. Slides by richargh.de and
    These are all Feedback Loops
    • Automated Unit/Int. Tests
    • Static Tests (Architecture, …)
    • Integration into Trunk
    • Automated Large Tests
    • Performance, etc. Tests
    • Exploratory Tests
    • Production Metrics
    • Customer Reactions
    41
    Idea
    …
    

    View Slide

42. Slides by richargh.de and
    42
    Why How
    Who
    Prepare
    Build
    Optimize
    

    View Slide

43. Slides by richargh.de and
    The pipeline consists of
    Everything we need to go from commit to releasable outcome
    43
    

    View Slide

44. Slides by richargh.de and
    The Goal of a Pipeline
    Have we done everything we can to disprove that this is a releasable
    candidate?
    44
    

    View Slide

45. Slides by richargh.de and
    Build on the shoulder of giants
    So much stuff you can use and most of it is free
    45
    

    View Slide

46. Slides by richargh.de and
    Automate Dependency Updates
    46
    1 https://docs.renovatebot.com/configuration-options/
    Renovate Stage
    # in /.gitlab-ci.yaml
    1. renovate:
    2. stage: renovate
    3. image: $REOVATE_IMAGE
    4. script:
    5. - renovate $(cat repositories.txt | xargs) --token $TOKEN_GIT –log-file renovate.log
    6. artifacts:
    7. paths:
    8. - renovate.log
    # in /repositories.txt
    1. /
    In it’s own repository with a
    bi-hourly schedule
    # in /config.js
    1. module.exports = {
    2. “branchPrefix”: “chore/renovate-”,
    3. ...
    4. }
    

    View Slide

47. Slides by richargh.de and
    Parallize Whatever you can
    47
    1 https://continuousdelivery.com/2010/09/deployment-pipeline-anti-patterns/
    Commit Stage
    Build
    ✓
    Package & Store
    ✓
    Static Code
    Analysis (Lint)
    ✓
    Unit Tests
    ✓
    Integration Tests
    ✓
    Secrets Detection
    ✓
    “Dependency”
    Analysis (SCA)
    ✓
    Static Security
    Tests (SAST)
    ✓
    Most of the stage can and
    should run in parallel.
    Run tests in
    parallel
    Run test in parallel with
    separate db schema per test set
    

    View Slide

48. Slides by richargh.de and
    Be Flexible - Stay Wide and Short1
    48
    1 https://continuousdelivery.com/2010/09/deployment-pipeline-anti-patterns/
    Commit Stage Beta Stage Gamma Stage Prod Stage
    Beta Env Gamma Env Prod Env
    Explore Stage
    Explore Env
    Give choices to smart humans.
    

    View Slide

49. Slides by richargh.de and
    For daily deployments, consider Gamma
    ONE deployments1
    49
    1 Automating safe, hands-off deployments https://aws.amazon.com/builders-library/automating-safe-hands-off-deployments/
    Commit Stage Beta Stage Gamma Stage Prod Stage
    Beta Env Gamma Env Prod Env
    Is the service backwards compatible?
    (1) Deploy ONE instance side-by with old version
    (2) Unleash normal test traffic and measure
    (3) If all ok, after 1 hour, deploy all instances
    (4) Run regular Gamma Stage tests
    

    View Slide

50. Slides by @arghrich
    It’s not just about the Pipeline
    50
    Pipeline
    Easy Hard
    Techniques
    Something you do
    Methods
    When you do it
    Mindset
    What you actually do
    

    View Slide

51. Slides by @arghrich
    Continuous Delivery Principles
    • Build quality in
    • Work in small batches
    • Computers Perform Repetitive Tasks,
    People Solve Problems
    • Relentlessly Pursue Continuous Improvement
    • Everyone is Responsible
    51
    CD Principles by Jez Humble
    Four Principles of Low-Risk Software Releases by Jez Humble
    

    View Slide

52. Slides by richargh.de and
    We are what we repeatedly do.
    Excellence then, is not an act,
    but a habit.1
    52
    1 The Story of Philosophy
    

    View Slide

53. Slides by @arghrich
    Thanks
    53
    Richard Gross (he/him)
    richargh.de/
    speakerdeck.com/richargh
    @arghrich
    Works for maibornwolff.de/
    People. Code. Commitment.
    DE TN ES
    Archaeologist Auditor
    Hypermedia-
    Designer
    

    View Slide

54. Slides by richargh.de and
    Backup
    

    View Slide

55. Slides by richargh.de and
    Gitlab-Ci Links
    • Create a Job: https://docs.gitlab.com/ee/ci/jobs/
    • Caching&Artifacts https://docs.gitlab.com/ee/ci/caching/
    • Predefined Gitlab Variables:
    https://docs.gitlab.com/ee/ci/variables/predefined_variables.html
    • Unit Test Reports:
    https://docs.gitlab.com/ee/ci/testing/unit_test_reports.html
    • Test Coverage:
    https://docs.gitlab.com/ee/ci/testing/test_coverage_visualization.html#exa
    mple-test-coverage-configurations
    • Metrics Reports: https://docs.gitlab.com/ee/ci/testing/metrics_reports.html
    • Gitlab-Ci Yaml reference: https://docs.gitlab.com/ee/ci/yaml/index.html
    • Gitlab-Ci Examples: https://docs.gitlab.com/ee/ci/examples/
    55
    

    View Slide

56. Slides by richargh.de and
    Flickr 2009
    • 10 Deploys per Day
    56
    1 https://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr
    

    View Slide

57. Slides by richargh.de and
    Etsy 20141
    • 50 Deploys/Day
    57
    1 https://www.infoq.com/news/2014/03/etsy-deploy-50-times-a-day/
    

    View Slide

58. Slides by richargh.de and
    Facebook 20xx1
    • Over 1B users worldwide
    • About 700 developers committing against their frontend source code
    repo
    • Single frontend code binary about 1.5GB in size
    • Pushed out to many thousands of servers (the number is not public)
    • Changes can go from check-in to end users in as quickly as 40
    minutes
    • Release process almost entirely invisible to the users
    • 2 Deploys per Day in 20122
    • Quasi-Continuous Deploy in 20173
    58
    1 https://dzone.com/articles/pushing-twice-daily-our
    2 https://engineering.fb.com/2012/08/03/uncategorized/ship-early-and-ship-twice-as-often/
    3 https://engineering.fb.com/2017/08/31/web/rapid-release-at-massive-scale/
    4 https://research.facebook.com/publications/continuous-deployment-at-facebook-and-oanda/
    

    View Slide

59. Slides by richargh.de and
    AWS Pipeline
    • Thousands of teams + Microservice architecture + Multiple
    environments + Continuous Delivery
    • = 50 million deployments a year (2015) 1
    1. Source
    2. Build (1st build, 2nd Unit test)
    3. Beta (1st Deploy, 2nd UI test)
    4. Gamma (1st Deploy, 2nd Load test)
    5. Production (1st Deploy region1, 2nd Deploy region2, 3rd Deploy
    region3)
    59
    1 Transforming Software Development https://www.youtube.com/watch?v=YCrhemssYuI
    

    View Slide

60. Slides by richargh.de and
    Amazon Stats (around 2012)1
    11.6 seconds
    Mean time between deplovments (weekday)
    1,079
    Max # of deployments in a single hour
    10,000
    Mean # of hosts simultaneously receiving a deployment
    30,000
    Max # of hosts simultaneously receiving a deployment
    60
    1 Velocity Culture https://www.youtube.com/watch?v=dxk8b9rSKOo
    

    View Slide

61. Slides by richargh.de and
    Google Repository (around 2016)1
    • Billions of LoC in a single repository (35 million commits)
    • Including configuration and documentation
    • > 2.000 projects under active development
    • 40.000 commits per workday
    • Multiple Services with one version: HEAD
    • Simplified dependency management: static linking without diamond
    dependency problem
    • Atomic changes, large scale refactoring
    • Homegrown VCS (Piper -> Bazel is a subset)
    • Trunk-Based Development
    • More than 50 million test cases run per day
    • 25.000 Developers
    61
    1 https://cacm.acm.org/magazines/2016/7/204032-why-google-stores-billions-of-lines-of-code-in-a-single-repository/fulltext
    2 https://www.infoq.com/presentations/Continuous-Testing-Build-Cloud/
    

    View Slide

62. Slides by richargh.de and
    Google Pipeline
    • Code is partitioned into packages, folders with a BUILD file
    • The BUILD file describes what other packages it depends on and
    what it provides
    62
    1 https://google-engtools.blogspot.com/2011/08/build-in-cloud-how-build-system-works.html
    2 https://www.infoq.com/presentations/Continuous-Testing-Build-Cloud/
    

    View Slide

63. Slides by richargh.de and
    Netflix
    • Red/Black-Deployment
    • Quasi-Continuous Deploys
    63
    1 https://netflixtechblog.com/deploying-the-netflix-api-79b6176cc3f0?gi=386880e182fb
    

    View Slide

64. Slides by richargh.de and
    Stackoverflow Architecture
    • The main, retail, public site serves over a hundred million unique
    users and 6,000 page views per second.
    • 1.3 BILLION page views per month
    • 1 Monolith deployed to Own data center
    • 9 Web Servers
    • 4 SQL Servers
    • 2 Redis Servers
    • 3 Elastic Search Servers
    • 2 HAProxy Servers (Load Balancers)
    64
    1 https://stackexchange.com/performance
    2 https://www.zdnet.com/article/stack-overflow-cto-from-bootstrapped-to-scaling-one-of-the-webs-biggest-properties/ 2022
    

    View Slide

65. Slides by richargh.de and
    Now Unit Test1 your Modules
    • By which you mean one test file per class? (no ✘)
    • By which you mean together with the database? (no ✘)
    • By which you mean … (probably also no ✘)
    65
    1 A Set of Unit Testing Rules https://www.artima.com/weblogs/viewpost.jsp?thread=126923
    

    View Slide

66. Slides by richargh.de and
    A Set of Unit Testing Rules
    A test is not a unit test if:
    • It talks to a database
    • It communicates across the network
    • It touches the file system
    • It can't run at the same time as any of your other unit tests
    • You have to do special things to your environment (such as editing
    config files) to run it.
    Tests that do these things aren't bad. Often they are worth writing, and
    they can be written in a unit test harness. However, it is important to be
    able to separate them from true unit tests so that we can keep a set of
    tests that we can run fast whenever we make our changes.
    66
    1 A Set of Unit Testing Rules https://www.artima.com/weblogs/viewpost.jsp?thread=126923
    

    View Slide

67. Slides by richargh.de and
    Test Sizes1, not bike sheds
    67
    1 Google Test Sizes https://testing.googleblog.com/2010/12/test-sizes.html
    Feature Small Medium Large
    Network access No Localhost only Yes
    Database No Yes Yes
    File system access No Yes Yes
    Use external system No Discouraged Yes
    Multiple threads No Yes Yes
    Sleep statements No Yes Yes
    System properties No Yes Yes
    Time limit (seconds) 60 300 900+
    Test:
    modules, functions
    Test:
    (Repo) adapters
    Test:
    BE Api, FE&BE together
    

    View Slide

68. Slides by richargh.de and
    Larger Size
    Complicated Test Harness
    Imprecise Feedback
    Coarse assertions
    Slower
    à $$$
    Smaller Size
    Simple Test Harness
    Specific Feedback
    Strict assertions
    Faster
    à $
    Test Sizes Guide
    Dependency
    Stubbed-Dep.
    Small
    Code being tested
    Test Amount
    Medium
    DB
    L
    a
    r
    g
    e
    Exploratory
    

    View Slide

69. Slides by richargh.de and
    69
    Commit Stage
    # in /.gitlab-ci.yaml
    1. small_tests:
    2. stage: commit
    3. extends:
    4. - .node:cache
    5. variables:
    6. KUBERNETES_CPU_REQUEST: 1000m
    7. KUBERNETES_MEMORY_REQUEST: 2500Mi
    8. script:
    9. - npx jest tests/small --ci --reporters=default --reporters=jest-junit
    10. # optional artifacts but this way you’ll see nice reports in Gitlab
    11. artifacts:
    12. paths:
    13. - junit.xml
    14. expire_in: 5 days
    15. reports:
    16. junit: junit.xml
    Small Tests
    ✓
    1 Gitlab Unit Test Reports https://docs.gitlab.com/ee/ci/testing/unit_test_reports.html
    

    View Slide

70. Slides by richargh.de and
    70
    Commit Stage
    # in /.gitlab-ci.yaml
    1. medium_tests:
    2. stage: commit
    3. extends:
    4. - .node:cache
    5. variables:
    6. KUBERNETES_CPU_REQUEST: 1000m
    7. KUBERNETES_MEMORY_REQUEST: 2500Mi
    8. services:
    9. # start a database
    10. # remove for small tests they don’t need services
    11. - name:
    12. alias: mongo
    13. script:
    14. # create junit report because Gitlab can display that
    15. - npx jest tests/medium --ci --reporters=default --reporters=jest-junit
    16. # optional artifacts but this way you’ll see nice reports in Gitlab
    17. artifacts:
    18. paths:
    19. - junit.xml
    20. expire_in: 5 days
    21. reports:
    22. junit: junit.xml
    Medium Tests
    ✓
    1 Gitlab Unit Test Reports https://docs.gitlab.com/ee/ci/testing/unit_test_reports.html
    

    View Slide

71. Slides by richargh.de and
    71
    Commit Stage
    # in /.gitlab-ci.yaml
    1. build:
    2. stage: commit
    3. extends:
    4. - .node:cache
    5. script:
    6. - npm run build:prod
    7. artifacts:
    8. paths:
    9. - dist/
    10. expire_in: 8 hours
    Commit Stage
    Build
    ✓
    # in /.gitlab-ci.yaml
    1. package:
    2. stage: commit
    3. needs:
    4. - job: build
    5. artifacts: true
    6. image:
    7. name: /kaniko-project/executor:xyz
    8. entrypoint: [ "" ]
    9. script:
    10. - DEST=”"
    11. - /kaniko/executor
    12. --context "${CI_PROJECT_DIR}"
    13. --dockerfile "${CI_PROJECT_DIR}/Dockerfile"
    14. --destination “${DEST}:${CI_COMMIT_SHORT_SHA}”
    15. --destination “${DEST}:latest”
    16. --push-retry 3
    1 Kaniko https://github.com/GoogleContainerTools/kaniko
    2 Gitlab Kaniko https://docs.gitlab.com/ee/ci/docker/using_kaniko.html
    Package & Store
    ✓
    Create artifacts
    Use artifacts
    

    View Slide

72. Slides by richargh.de and
    72
    Commit Stage Beta Stage
    Prepare Stage
    1 AWS DPRA https://pipelines.devops.aws.dev/
    Beta Env
    Self-Service
    Automatic
    Deploy & Smoke
    ✓
    Large Tests
    ✓
    ~30min
    Required
    ✓
    Situational
    ✓
    Does it work in a real environment?
    

    View Slide

73. Slides by richargh.de and
    73
    Beta Stage
    1. deploy_beta:
    2. stage: beta
    3. image: $HELM_IMAGE
    4. variables:
    5. # do not clone source code
    6. GIT_STRATEGY: none
    7. # Gitlab can display latest deployment
    8. environment:
    9. name: beta
    10. script:
    11. # visualize deploy changes
    12. - helm diff upgrade
    13. --install --color --context 5
    14. --detailed-exitcode <...>
    15. # deploy
    16. - helm upgrade
    17. --install
    18. --values .helm/values-beta.yaml
    19. --namespace beta
    20. -beta
    21. .helm/
    22. # wait for deployment to finish
    23. - kubectl rollout status
    24. --request-timeout=5s
    25. -n=beta
    26. -beta
    Deploy
    ✓ Beta Stage
    1. smoke_beta:
    2. stage: commit
    3. needs:
    4. - job: deploy_beta
    5. artifacts: false
    6. script:
    7. - npm run test:smoke -- -ns=beta
    Smoke
    ✓
    1 Gitlab Environments https://docs.gitlab.com/ee/ci/environments/
    2 Helm Diff Plugin https://github.com/databus23/helm-diff
    A smoke test
    Uses non-destructive requests
    That are invisible to the user
    Examples:
    GET /users but also
    PUT /orders
    PUT /orders
    

    View Slide

74. Slides by richargh.de and
    AWS DPRA1
    74
    1 https://pipelines.devops.aws.dev/application-pipeline/
    

    View Slide