Tweet
Tweet

Slide 1

Slide 1 text

REAL WORLD CD LEARN, ADAPT & IMPROVE MICHIEL ROOK @MICHIELTCS

Slide 2

Slide 2 text

LARGE SEMI-GOV IN NL

Slide 3

Slide 3 text

@michieltcs MYSQL ELASTIC INTERNET

Slide 4

Slide 4 text

@michieltcs TOPIC TEAMS

Slide 5

Slide 5 text

@michieltcs FIRE TEAM

Slide 6

Slide 6 text

@michieltcs PACKAGE/DEP UPDATES

Slide 7

Slide 7 text

@michieltcs BUG TRIAGE

Slide 8

Slide 8 text

@michieltcs RELEASE ROLLOUT

Slide 9

Slide 9 text

@michieltcs A TYPICAL RELEASE...

Slide 10

Slide 10 text

@michieltcs RELEASE CHECKLIST

Slide 11

Slide 11 text

@michieltcs @michieltcs Create release Test release Deploy release every 2 weeks Thursday (sprint end) Monday (sprint start) Tuesday

Slide 12

Slide 12 text

@michieltcs 1. BRANCH RC 2. DEPLOY TO ACC 3. SIGNAL TESTER(S) 4. SIGNAL PO 5. FIX ON BRANCH 6. GET APPROVAL 7. DEPLOY TO PROD 8. VERIFY PROD

Slide 13

Slide 13 text

@michieltcs 2-3 DAYS MANUAL WORK

Slide 14

Slide 14 text

@michieltcs ... HOT FIXES?

Slide 15

Slide 15 text

@michieltcs GOALS

Slide 16

Slide 16 text

@michieltcs REDUCE COST

Slide 17

Slide 17 text

@michieltcs FAST FEEDBACK

Slide 18

Slide 18 text

@michieltcs REDUCE TOIL

Slide 19

Slide 19 text

@michieltcs REDUCE TOIL the kind of work tied to running a production service that tends to be manual, repetitive, automatable, tactical, devoid of enduring value, and that scales linearly as a service grows

Slide 20

Slide 20 text

@michieltcs PUT PO/BIZ BACK IN DRIVER'S SEAT

Slide 21

Slide 21 text

@michieltcs WHY CD THEN?

Slide 22

Slide 22 text

@michieltcs MAKE THINGS SMALL

Slide 23

Slide 23 text

@michieltcs BIG STEPS

Slide 24

Slide 24 text

@michieltcs FAIL BIG

Slide 25

Slide 25 text

@michieltcs SMALL STEPS

Slide 26

Slide 26 text

@michieltcs FAIL SMALL

Slide 27

Slide 27 text

@michieltcs IF IT HURTS DO IT MORE OFTEN

Slide 28

Slide 28 text

@michieltcs $ = REALIZED VALUE CREDITS TO @FGOULDING

Slide 29

Slide 29 text

@michieltcs @michieltcs HIGH VS. LOW PERFORMING TEAMS Source: Accelerate - The Science Of DevOps

Slide 30

Slide 30 text

APPROACH

Slide 31

Slide 31 text

@michieltcs 3 PHASES

Slide 32

Slide 32 text

@michieltcs DEFER SPLITTING THE MONOLITH

Slide 33

Slide 33 text

@michieltcs INCREASE TEMPO & AUTOMATION

Slide 34

Slide 34 text

@michieltcs SINGLE PIPELINE

Slide 35

Slide 35 text

@michieltcs IMPROVEMENTS & TESTING

Slide 36

Slide 36 text

PHASE 1: INCREASING RELEASE CADENCE

Slide 37

Slide 37 text

@michieltcs WEEKLY RELEASE

Slide 38

Slide 38 text

@michieltcs AUTOMATION

Slide 39

Slide 39 text

@michieltcs @michieltcs Build / Package Automated Tests Acceptance Environment Production Environment

Slide 40

Slide 40 text

@michieltcs @michieltcs Build / Package Automated Tests Acceptance Environment Production Environment

Slide 41

Slide 41 text

@michieltcs pipeline {
 stages {
 stage('Run tests') {
 sh "gradle check"
 }
 
 stage('Build docker image') {
 sh "docker build -t jobservice:${env.BUILD_NUMBER} ."
 sh "docker push jobservice:${env.BUILD_NUMBER}"
 }
 
 stage('Deploy staging') {
 sh "ansible-playbook -e BUILD=${env.BUILD_NUMBER}
 -i staging deploy.yml"
 }
 
 stage('Deploy production') {
 sh "ansible-playbook -e BUILD=${env.BUILD_NUMBER}
 -i prod deploy.yml"
 }
 }
 PIPELINE AS CODE image: registry.local/runner:latest
 
 stages:
 - test
 - images
 - deploy
 
 phpunit:
 stage: test
 script:
 - bin/phpunit
 
 images:
 stage: images
 script:
 - make images publish
 only:
 - master
 
 deploy:
 stage: deploy
 script:
 - make deploy
 only:
 - master

Slide 42

Slide 42 text

No content

Slide 43

Slide 43 text

No content

Slide 44

Slide 44 text

@michieltcs WAY OF WORKING

Slide 45

Slide 45 text

@michieltcs PAIR PROGRAMMING

Slide 46

Slide 46 text

BOY SCOUT RULE

Slide 47

Slide 47 text

@michieltcs NO MORE RELEASE BRANCHES

Slide 48

Slide 48 text

@michieltcs @michieltcs

Slide 49

Slide 49 text

@michieltcs @michieltcs

Slide 50

Slide 50 text

@michieltcs @michieltcs

Slide 51

Slide 51 text

PHASE 2: SINGLE PIPELINE

Slide 52

Slide 52 text

@michieltcs @michieltcs Build / Package Automated Tests Acceptance Environment Production Environment

Slide 53

Slide 53 text

@michieltcs CONTINUOUS DEPLOYMENT

Slide 54

Slide 54 text

@michieltcs ZERO DOWNTIME DEPLOYS

Slide 55

Slide 55 text

@michieltcs ROLLING DEPLOYS

Slide 56

Slide 56 text

@michieltcs @michieltcs LOAD BALANCER ROLLING UPDATE SERVICE V1.0 SERVICE V1.0 SERVICE V1.0 @michieltcs

Slide 57

Slide 57 text

@michieltcs @michieltcs LOAD BALANCER ROLLING UPDATE SERVICE V1.0 SERVICE V1.0 SERVICE V1.0 SERVICE V1.1 @michieltcs

Slide 58

Slide 58 text

@michieltcs @michieltcs LOAD BALANCER ROLLING UPDATE SERVICE V1.0 SERVICE V1.1 SERVICE V1.0 SERVICE V1.0 @michieltcs

Slide 59

Slide 59 text

@michieltcs @michieltcs LOAD BALANCER ROLLING UPDATE SERVICE V1.0 SERVICE V1.1 SERVICE V1.0 SERVICE V1.1 @michieltcs

Slide 60

Slide 60 text

@michieltcs @michieltcs LOAD BALANCER ROLLING UPDATE SERVICE V1.0 SERVICE V1.1 SERVICE V1.1 SERVICE V1.0 @michieltcs

Slide 61

Slide 61 text

@michieltcs @michieltcs LOAD BALANCER ROLLING UPDATE SERVICE V1.0 SERVICE V1.1 SERVICE V1.1 SERVICE V1.1 @michieltcs

Slide 62

Slide 62 text

@michieltcs @michieltcs LOAD BALANCER ROLLING UPDATE SERVICE V1.1 SERVICE V1.1 SERVICE V1.1 SERVICE V1.0 @michieltcs

Slide 63

Slide 63 text

@michieltcs @michieltcs LOAD BALANCER ROLLING UPDATE SERVICE V1.1 SERVICE V1.1 SERVICE V1.1 @michieltcs

Slide 64

Slide 64 text

@michieltcs PIPELINE FAILURES

Slide 65

Slide 65 text

@michieltcs FLAKY TESTS

Slide 66

Slide 66 text

@michieltcs TIMEOUTS

Slide 67

Slide 67 text

@michieltcs NETWORK STABILITY ISSUES

Slide 68

Slide 68 text

@michieltcs EXTERNAL DEPS IN TESTS

Slide 69

Slide 69 text

@michieltcs BRITTLE INFRASTRUCTURE

Slide 70

Slide 70 text

@michieltcs IF YOU CAN'T TRUST IT - REMOVE IT

Slide 71

Slide 71 text

STOP THE LINE

Slide 72

Slide 72 text

@michieltcs EXTREME FEEDBACK

Slide 73

Slide 73 text

@michieltcs @michieltcs

Slide 74

Slide 74 text

@michieltcs RELEASE TIMING?

Slide 75

Slide 75 text

FEATURE TOGGLES

Slide 76

Slide 76 text

@michieltcs DECOUPLE DEPLOYMENTS FROM RELEASES

Slide 77

Slide 77 text

@michieltcs @michieltcs

Slide 78

Slide 78 text

@michieltcs @michieltcs

Slide 79

Slide 79 text

@michieltcs CAUTION: FEATURE TOGGLE DEBT

Slide 80

Slide 80 text

PHASE 3: IMPROVEMENTS & TESTING

Slide 81

Slide 81 text

@michieltcs PIPELINE SPEED

Slide 82

Slide 82 text

@michieltcs BUY A BIGGER BOX

Slide 83

Slide 83 text

@michieltcs CLEVER PIPELINE IMPROVEMENTS: PARALLELIZE

Slide 84

Slide 84 text

@michieltcs TESTING IMPROVEMENTS

Slide 85

Slide 85 text

@michieltcs @michieltcs UNIT TESTS E2E / VISUAL TESTS INTEGRATION TESTS LOTS OF MANUAL TESTING E2E TESTS

Slide 86

Slide 86 text

@michieltcs QUALITY ENGINEERING

Slide 87

Slide 87 text

@michieltcs AUTOMATION

Slide 88

Slide 88 text

@michieltcs @michieltcs CONTRACT TESTING

Slide 89

Slide 89 text

@michieltcs @michieltcs UI TESTING

Slide 90

Slide 90 text

@michieltcs @michieltcs VISUAL TESTING

Slide 91

Slide 91 text

@michieltcs @michieltcs DEPENDENCY SCANNING

Slide 92

Slide 92 text

@michieltcs @michieltcs UNIT TESTS UI / VISUAL TESTS Exploratory testing & user feedback Monitoring & alerting INTEGRATION / CONTRACT TESTS COST SPEED

Slide 93

Slide 93 text

RESULTS & TAKEAWAYS

Slide 94

Slide 94 text

@michieltcs LEARNINGS

Slide 95

Slide 95 text

@michieltcs PIPELINE SPEED & STABILITY

Slide 96

Slide 96 text

@michieltcs TEST THE PIPELINE

Slide 97

Slide 97 text

@michieltcs WAY OF WORKING & MINDSET

Slide 98

Slide 98 text

@michieltcs VISUAL TESTING TOOLS

Slide 99

Slide 99 text

@michieltcs SOME STATS

Slide 100

Slide 100 text

@michieltcs ~4000 TESTS PER DEPLOYMENT

Slide 101

Slide 101 text

@michieltcs ~10 MINUTES PER PIPELINE RUN

Slide 102

Slide 102 text

@michieltcs ~1.5 HOURS FROM DEV TO PROD

Slide 103

Slide 103 text

@michieltcs IN 6 MONTHS: MORE THAN 5000 DEPLOYMENTS

Slide 104

Slide 104 text

@michieltcs @michieltcs THANK YOU FOR LISTENING! @michieltcs / michiel@michielrook.nl www.michielrook.nl