$30 off During Our Annual Pro Sale. View Details »

DevOpsPorto Meetup18: Continuous Delivery Patterns for Boring Releases by Manuel Pais

DevOpsPorto Meetup18: Continuous Delivery Patterns for Boring Releases by Manuel Pais

Talk delivered by Manuel Pais

DevOpsPorto

July 12, 2018
Tweet

More Decks by DevOpsPorto

Other Decks in Technology

Transcript

  1. Continuous Delivery Patterns
    for Boring Releases

    View Slide

  2. About me
    Manuel Pais
    MS Software Eng
    @manupaisable
    manuelpais.net
    [email protected]
    DevOps and Delivery Consultant
    Focused on teams and flow
    2
    @manupaisable | manuelpais.net

    View Slide

  3. @manupaisable | manuelpais.net 3
    Today
    1. Intro to boring releases
    2. Patterns for safer releases
    3. Patterns for faster releases
    4. Patterns for sustainable delivery

    View Slide

  4. @manupaisable | manuelpais.net 4
    Today
    1. Intro to boring releases
    2. Patterns for safer releases
    3. Patterns for faster releases
    4. Patterns for sustainable delivery

    View Slide

  5. 5
    @manupaisable | manuelpais.net

    View Slide

  6. 6
    App down,
    refuses to restart
    @manupaisable | manuelpais.net

    View Slide

  7. 7
    App down,
    refuses to restart
    Data migration
    completed… partially
    @manupaisable | manuelpais.net

    View Slide

  8. 8
    App down,
    refuses to restart…
    Missing
    libraries…
    Data migration
    completed… partially
    @manupaisable | manuelpais.net

    View Slide

  9. 9
    App down,
    refuses to restart…
    The password master…
    Data migration
    completed… partially
    Missing
    libraries…
    @manupaisable | manuelpais.net

    View Slide

  10. 10
    @manupaisable | manuelpais.net

    View Slide

  11. 11
    @manupaisable | manuelpais.net

    View Slide

  12. @manupaisable | manuelpais.net 12

    View Slide

  13. @manupaisable | manuelpais.net 13

    View Slide

  14. @manupaisable | manuelpais.net 14
    Build Test Deploy

    View Slide

  15. @manupaisable | manuelpais.net 15
    Build Test Deploy
    Build Test Deploy

    View Slide

  16. @manupaisable | manuelpais.net 16
    Build Test Deploy
    Build Test Deploy
    Build Test Deploy

    View Slide

  17. @manupaisable | manuelpais.net 17
    Build Test Deploy
    Build Test Deploy
    Build Test Deploy Monitor

    View Slide

  18. 18
    Outcomes
    @manupaisable | manuelpais.net

    View Slide

  19. “ability to get changes of all types,
    into production, or into the hands of
    users, safely and quickly in a
    sustainable way”
    –Dave Farley & Jez Humble
    continuousdelivery.com
    @manupaisable | manuelpais.net 19

    View Slide

  20. @manupaisable | manuelpais.net 20
    Today
    1. Intro to boring releases
    2. Patterns for safer releases
    3. Patterns for faster releases
    4. Patterns for sustainable delivery

    View Slide

  21. “ability to get changes of all types,
    into production, or into the hands of
    users, safely and quickly in a
    sustainable way”
    –Dave Farley & Jez Humble
    continuousdelivery.com
    @manupaisable | manuelpais.net 21

    View Slide

  22. STEP ONE
    map all release activities in a
    deployment pipeline
    22
    @manupaisable | manuelpais.net

    View Slide

  23. 23
    @manupaisable | manuelpais.net

    View Slide

  24. Gains from
    improving visibility to everyone,
    and increasing repeatability and
    traceability are enormous
    24
    @manupaisable | manuelpais.net

    View Slide

  25. Value Stream Mapping
    @manupaisable | manuelpais.net 25

    View Slide

  26. Value Stream Mapping
    is a pen and paper tool that
    highlights bottlenecks in a matter of
    hours and raises awareness of what
    difficulties other teams face
    @manupaisable | manuelpais.net 26

    View Slide

  27. STEP TWO
    measure key metrics on:
    -speed (e.g. cycle time)
    -quality (e.g. defect rate)
    -operability (e.g. MTTR)
    27
    @manupaisable | manuelpais.net

    View Slide

  28. 28
    @manupaisable | manuelpais.net

    View Slide

  29. STEP THREE
    PUT IN THE WORK !!!
    29
    @manupaisable | manuelpais.net

    View Slide

  30. Which work?
    Automated builds
    in clean environments
    30
    @manupaisable | manuelpais.net

    View Slide

  31. Which work?
    You can provision a
    mini-replica of production
    environnment
    31
    @manupaisable | manuelpais.net

    View Slide

  32. Which work?
    You have application health checks
    and fast smoke tests
    for sanity checking
    32
    @manupaisable | manuelpais.net

    View Slide

  33. Which work?
    You have acceptance tests
    with reasonable coverage,
    including failure scenarios
    33
    @manupaisable | manuelpais.net

    View Slide

  34. Which work?
    Single source of truth.
    Single binary.
    Single path to production.
    34
    @manupaisable | manuelpais.net

    View Slide

  35. STEP THREE
    PUT IN THE WORK !!!
    (but in iterative fashion…
    walking skeletons FTW!)
    35
    @manupaisable | manuelpais.net

    View Slide

  36. 36
    @manupaisable | manuelpais.net

    View Slide

  37. 37
    practices first,
    tools second
    @manupaisable | manuelpais.net

    View Slide

  38. Practices for boring releases
    •Automated build (and unit tests)
    •Provision prod replica in pipeline
    •Automated acceptance tests (BDD)
    •No “invisible” activities
    •One source of truth
    •One path to production
    @manupaisable | manuelpais.net 38

    View Slide

  39. @manupaisable | manuelpais.net 39
    Today
    1. Intro to boring releases
    2. Patterns for safer releases
    3. Patterns for faster releases
    4. Patterns for sustainable delivery

    View Slide

  40. “ability to get changes of all types,
    into production, or into the hands of
    users, safely and quickly in a
    sustainable way”
    –Dave Farley & Jez Humble
    continuousdelivery.com
    @manupaisable | manuelpais.net 40

    View Slide

  41. @manupaisable | manuelpais.net 41
    can’t auto-
    scale people
    How to cope with
    ever increasing
    cognitive load on
    teams to build and
    run applications?

    View Slide

  42. @manupaisable | manuelpais.net 42
    CI
    Peer
    review
    Infra Security Comply

    View Slide

  43. @manupaisable | manuelpais.net 43
    CI
    Peer
    review
    Infra Security Comply
    Database Accept UX Deploy

    View Slide

  44. @manupaisable | manuelpais.net 44
    Fix
    Monitor
    Run
    CI
    Peer
    review
    Infra Security Comply
    Database Accept UX Deploy

    View Slide

  45. Multiple approaches needed
    1. Smarter pipelines
    2. Team structures
    3. Self-service platforms
    4. Resilient delivery system
    @manupaisable | manuelpais.net 45

    View Slide

  46. Multiple approaches needed
    1. Smarter pipelines
    2. Well-thought team structures
    3. Self-service platforms
    4. Resilient delivery system
    @manupaisable | manuelpais.net 46

    View Slide

  47. Multiple approaches needed
    1. Smarter pipelines
    2. Well-thought team structures
    3. Self-service platforms
    4. Resilient delivery system
    @manupaisable | manuelpais.net 47

    View Slide

  48. Multiple approaches needed
    1. Smarter pipelines
    2. Well-thought team structures
    3. Self-service platforms
    @manupaisable | manuelpais.net 49

    View Slide

  49. Reduce wait times
    Minimum path to production
    Risk-based activities
    Continuous pruning
    50
    @manupaisable | manuelpais.net
    1. Smarter pipelines

    View Slide

  50. Waiting for pipeline
    @manupaisable | manuelpais.net 51

    View Slide

  51. Waiting for pipeline
    @manupaisable | manuelpais.net 52

    View Slide

  52. Reduce wait for pipeline
    @manupaisable | manuelpais.net 53

    View Slide

  53. Scalable CI/CD Infrastructure
    •Solved problem in cloud systems
    •Starts with infrastructure-as-code
    •Agent farm (auto scaling if possible)
    •Pipelines need to evolve as number of
    teams grows
    @manupaisable | manuelpais.net 54

    View Slide

  54. Waiting for dependencies
    @manupaisable | manuelpais.net 55

    View Slide

  55. Waiting for dependencies
    @manupaisable | manuelpais.net 56

    View Slide

  56. Flow efficiency =
    “teams who aren’t paying attention to this concept generally
    have flow efficiencies around the 15% mark - that means that
    work normally spends 85% of its lifecycle waiting on something.”
    http://leankanban.com/flow-efficiency-a-great-metric-you-probably-arent-using
    @manupaisable | manuelpais.net 57

    View Slide

  57. issue is not how long it takes to do
    something, it's how long we're
    waiting for it to get done
    @manupaisable | manuelpais.net 58

    View Slide

  58. @manupaisable | manuelpais.net 59

    View Slide

  59. Initial state: serial pipeline
    @manupaisable | manuelpais.net 60

    View Slide

  60. Intermediate state: CAB in path to prod
    @manupaisable | manuelpais.net 61

    View Slide

  61. End state: short and wide pipeline
    @manupaisable | manuelpais.net 62

    View Slide

  62. Reduce wait times
    Minimum path to production
    Risk-based activities
    Continuous pruning
    63
    @manupaisable | manuelpais.net
    1. Smarter pipelines

    View Slide

  63. Reduce wait times
    Minimum path to production
    Risk-based activities
    Continuous pruning
    64
    @manupaisable | manuelpais.net
    1. Smarter pipelines

    View Slide

  64. Reduce wait times
    Minimum path to production
    Risk-based activities
    Continuous pruning
    65
    @manupaisable | manuelpais.net
    1. Smarter pipelines

    View Slide

  65. Reduce wait times
    Minimum path to production
    Risk-based activities
    Continuous pruning
    66
    @manupaisable | manuelpais.net
    1. Smarter pipelines

    View Slide

  66. @manupaisable | manuelpais.net 70
    Today
    1. Intro to boring releases
    2. Patterns for safer releases
    3. Patterns for faster releases
    4. Patterns for sustainable delivery

    View Slide

  67. “ability to get changes of all types,
    into production, or into the hands of
    users, safely and quickly in a
    sustainable way”
    –Dave Farley & Jez Humble
    continuousdelivery.com
    @manupaisable | manuelpais.net 71

    View Slide

  68. @manupaisable | manuelpais.net 72

    View Slide

  69. Delivery system itself should be boring, not just the pipeline.
    73
    @manupaisable | manuelpais.net

    View Slide

  70. Resilient delivery system
    that enables
    fast(er) feedback loops
    74
    @manupaisable | manuelpais.net

    View Slide

  71. Delivery system
    •CI tool
    •Pipeline orchestration tool
    •Orchestration plugins / 3rd party tools
    •Pipeline definitions
    •Source repos
    •CI + CD infrastructure
    •And ?
    @manupaisable | manuelpais.net 75

    View Slide

  72. Delivery system
    @manupaisable | manuelpais.net 76

    View Slide

  73. Delivery system
    CI/CD Toolchain
    @manupaisable | manuelpais.net 77

    View Slide

  74. Delivery system
    CI/CD Toolchain
    App 1
    App 2
    @manupaisable | manuelpais.net 78

    View Slide

  75. Delivery system
    CI/CD Toolchain
    Pipelines
    App 1
    App 2
    @manupaisable | manuelpais.net 79

    View Slide

  76. Delivery system
    CI/CD Toolchain
    Pipelines
    Infra
    App 1
    App 2
    @manupaisable | manuelpais.net 80

    View Slide

  77. How resilient delivery looks like
    •Tooling and configuration changes do
    not impact regular delivery
    •Gracefully handles peak load of
    pipeline runs
    •Issues with underlying infra/tools
    handled swiftly, rollback if needed
    •Disaster recovery at a click of a button
    @manupaisable | manuelpais.net 81

    View Slide

  78. How resilient delivery looks like
    •Changes (plugins, configuration, jobs,
    etc) do not impact regular delivery
    •Gracefully handles peak load of
    pipeline runs
    •Issues with underlying infra/tools
    handled swiftly, rollback if needed
    •Disaster recovery at a click of a button
    @manupaisable | manuelpais.net 82

    View Slide

  79. How resilient delivery looks like
    •Changes (plugins, configuration, jobs,
    etc) do not impact regular delivery
    •Gracefully handles peak load of
    pipeline runs
    •Issues with underlying infra/tools
    handled swiftly, roll backed if needed
    •Disaster recovery at a click of a button
    @manupaisable | manuelpais.net 83

    View Slide

  80. How resilient delivery looks like
    •Changes (plugins, configuration, jobs,
    etc) do not impact regular delivery
    •Gracefully handles peak load of
    pipeline runs
    •Issues with underlying infra/tools
    handled swiftly, rollback if needed
    •Disaster recovery at a click of a button
    (almost)
    @manupaisable | manuelpais.net 84

    View Slide

  81. Practices for resiliency (1/3)
    •Pipeline infrastructure-as-code
    •Pipeline configuration-as-code
    •Build and release from zero to live
    @manupaisable | manuelpais.net 85

    View Slide

  82. Practices for resiliency (1/3)
    •Pipeline infrastructure-as-code
    •Pipeline configuration-as-code
    •Build and release from zero to live
    @manupaisable | manuelpais.net 86

    View Slide

  83. @manupaisable | manuelpais.net 87

    View Slide

  84. Bonus Points
    pipeline-as-code allows
    designing future states of the
    value stream and draw evolution
    @manupaisable | manuelpais.net 88

    View Slide

  85. Practices for resiliency (1/3)
    •Pipeline infrastructure-as-code
    •Pipeline configuration-as-code
    •Build and release from zero to live
    @manupaisable | manuelpais.net 89

    View Slide

  86. Team A Team B
    Team C Team D
    @manupaisable | manuelpais.net 90

    View Slide

  87. The glitch is believed to have been caused by a power supply issue and there is
    no evidence of a cyber-attack, the airline said.
    @manupaisable | manuelpais.net 91

    View Slide

  88. Team A Team B
    Team C Team D
    @manupaisable | manuelpais.net 92

    View Slide

  89. Team D
    Team B
    Team A
    Team C
    @manupaisable | manuelpais.net 93

    View Slide

  90. Team A
    @manupaisable | manuelpais.net 94
    Team D
    Team B
    Team C

    View Slide

  91. Team A
    Team C
    @manupaisable | manuelpais.net 95
    Team D
    Team B

    View Slide

  92. Team A Team B
    Team C
    @manupaisable | manuelpais.net 96
    Team D

    View Slide

  93. Team A Team B
    Team C Team D
    @manupaisable | manuelpais.net 97

    View Slide

  94. Borat’s Law / Caveat
    @manupaisable | manuelpais.net 98

    View Slide

  95. Practices for resiliency (2/3)
    •Pipeline immutable infrastructure
    •Blue-green pipeline deployments
    @manupaisable | manuelpais.net 99

    View Slide

  96. Practices for resiliency (2/3)
    •Pipeline immutable infrastructure
    •Blue-green pipeline deployments
    @manupaisable | manuelpais.net 100

    View Slide

  97. @manupaisable | manuelpais.net 101

    View Slide

  98. @manupaisable | manuelpais.net 102

    View Slide

  99. Practices for resiliency (3/3)
    •Aggregated logging
    •Monitoring & alerting
    •Auto-scaling
    @manupaisable | manuelpais.net 103

    View Slide

  100. Books
    104
    @manupaisable | manuelpais.net

    View Slide

  101. releasabilitybook.com
    Book sample out now!
    Team Guide to
    Software Releasability
    by Chris O’Dell & Manuel Pais
    105
    @manupaisable | manuelpais.net

    View Slide

  102. @manupaisable | manuelpais.net 106
    manuelpais.net

    View Slide

  103. thank you
    Manuel Pais
    MS Software Eng
    @manupaisable
    manuelpais.net
    [email protected]
    DevOps and Delivery Consultant
    Focused on teams and flow
    107
    @manupaisable | manuelpais.net

    View Slide