Evolving Continuous Delivery

This is a story of guiding a team's CD adoption and maturity, from a chaotic start, to being awesome.

This is a journey from a troublesome, struggling start of chaotic manual deployments, merge hell, regular production roll backs and lost code, to deliver a single commit to master automatically and reliably, under an hour, many times a day.

We’ll look at a few simple practices that worked, to introduce CD practices such as monitoring, build automation and test automation gradually, instead of going all in at first.

You’ll start to see that it isn’t daunting at all, and start delivering with joy.

Related post: http://www.hibri.net//2016/06/18/continuous-delivery-rags-to-riches/

Transcript

1. Evolving
   Continuous
   Delivery
   From Rags to Riches
   Hibri Marzook
   @hibri
   

   View Slide

2. @hibri
   http://www.hibri.net
   

   View Slide

3. Hibri Marzook, February 2017
   @hibri
   Fintech
   Post startup phase
   Successful product in the market
   ~ 30 in engineering
   ~ 200 employees
   Context
   

   View Slide

4. Hibri Marzook, February 2017
   @hibri
   First impressions
   Public facing API, running in the Cloud
   No automation
   Inconsistent environments
   Context switching. Not finishing work in progress
   Old, stale branches merged into active branch
   Manual deploys
   

   View Slide

5. Hibri Marzook, February 2017
   @hibri
   

   View Slide

6. Hibri Marzook, February 2017
   @hibri
   What to fix first?
   Branching strategy?
   Lack of automation?
   Lack of technical practices? TDD?
   Lack of CI?
   Process?
   Scrum, Kanban ceremonies?
   

   View Slide

7. Hibri Marzook, February 2017
   @hibri
   A focus on delivery is
   needed for continuous
   improvement
   

   View Slide

8. Hibri Marzook, February 2017
   @hibri
   Gain situational awareness
   Photo Credit - Hugh Trainer via Flickr
   

   View Slide

9. Hibri Marzook, February 2017
   @hibri
   Gaining situational
   awareness
   Install an application monitoring tool. New Relic
   Zero friction install
   Alerting out of the box
   PagerDuty integration
   

   View Slide

10. Hibri Marzook, February 2017
    @hibri
    We could be the first responders
    
    How much traffic?
    
    How many users are affected?
    
    Usage trends?
    
    Error rates?
    
    Can this scale?
    
    

    View Slide

11. Hibri Marzook, February 2017
    @hibri
    Shared understanding of the
    system
    Draw environment map
    Draw components of the system
    Ensure that all team members have the same
    understanding
    Keep maps visible
    

    View Slide

12. Hibri Marzook, February 2017
    @hibri
    Better situational awareness
    Photo Credit - Steve Jurveston via Flickr
    

    View Slide

13. Hibri Marzook, February 2017
    @hibri
    

    View Slide

14. Hibri Marzook, February 2017
    @hibri
    Expose the realities of the
    system instead of relying
    on assumptions
    

    View Slide

15. Hibri Marzook, February 2017
    @hibri
    In a legacy distributed application, there is a
    disconnect between how the system behaves in
    production and how the creators think it “should”
    behave.
    

    View Slide

16. Hibri Marzook, February 2017
    @hibri
    Create visibility
    

    View Slide

17. Hibri Marzook, February 2017
    @hibri
    Agile ≠ JIRA
    

    View Slide

18. Hibri Marzook, February 2017
    @hibri
    Creating visibility
    Made work visible with a simple card wall
    Pull from the right attitude
    Use JIRA only for tracking
    Created transparency about the work we are doing
    

    View Slide

19. Hibri Marzook, February 2017
    @hibri
    Limit work in progress
    Did one thing at a time
    Got into the habit of finishing one thing
    Whole team focussed on delivery
    Only then did we do more
    

    View Slide

20. Hibri Marzook, February 2017
    @hibri
    When you don’t have
    anything else, put
    everyone together
    

    View Slide

21. Hibri Marzook, February 2017
    @hibri
    Put everyone together
    Everyone we worked with sat in the same area
    Introduced Slack
    Devs are always around to fix an issue
    Practice remote working early
    

    View Slide

22. Hibri Marzook, February 2017
    @hibri
    

    View Slide

23. Hibri Marzook, February 2017
    @hibri
    Use slack time to learn
    and improve
    

    View Slide

24. Hibri Marzook, February 2017
    @hibri
    How did we create slack?
    WIP Limits
    Production issues are quantified, don’t need to jump
    on every issue
    Have data to push back on management pressure
    Can ignore some problems
    

    View Slide

25. Hibri Marzook, February 2017
    @hibri
    Long term change
    happens during slack
    time
    

    View Slide

26. Hibri Marzook, February 2017
    @hibri
    Before CI
    Couldn’t reliably build the system. “Works on my
    machine”
    Key man dependencies to do a release.
    Build time dependencies have to be fixed manually.
    

    View Slide

27. Hibri Marzook, February 2017
    @hibri
    

    View Slide

28. Hibri Marzook, February 2017
    @hibri
    When the team spends more time
    talking about branching and
    merging, you have the wrong
    strategy
    

    View Slide

29. Hibri Marzook, February 2017
    @hibri
    Introducing CI
    Built the system reliably and repeatedly on a CI
    server
    Removed the key man dependency
    Used this build as the source for manual deploys
    No ATDD yet
    Manual regression testing
    

    View Slide

30. Hibri Marzook, February 2017
    @hibri
    Getting branching under
    control
    

    View Slide

31. Hibri Marzook, February 2017
    @hibri
    

    View Slide

32. Hibri Marzook, February 2017
    @hibri
    Amplify the good
    things
    

    View Slide

33. Hibri Marzook, February 2017
    @hibri
    Adding automated testing
    Made the tests work in a local dev environment
    Made the same tests run in other environments
    Kept adding tests for all new work and fixes
    Refactored tests continuously
    

    View Slide

34. Hibri Marzook, February 2017
    @hibri
    Adding automated deploys
    Deployed to test environments on every commit
    Used Azure’s “slots” feature for blue/green
    TeamCity + Octopus Deploy
    Smoke tests after every deploy
    

    View Slide

35. Hibri Marzook, February 2017
    @hibri
    

    View Slide

36. Hibri Marzook, February 2017
    @hibri
    The ‘A’ word
    

    View Slide

37. Hibri Marzook, February 2017
    @hibri
    Talk and discuss the values
    Focus on the impact
    Explain cause and effect
    Introduce and repeat the principles
    Agile manifesto. Left over right
    

    View Slide

38. Hibri Marzook, February 2017
    @hibri
    https://landing.google.com/sre/book/chapters/part3.html#fig_part-practices_reliability-hierarchy
    

    View Slide

39. Hibri Marzook, February 2017
    @hibri
    Evolutionary approach
    Deal with current constraints
    Sometimes it’s OK to do things manually
    Every change should make sense and help the team a little bit
    more
    Team should not have to wait for all the changes to gain value
    

    View Slide

40. Hibri Marzook, February 2017
    @hibri
    Have a primal push to get
    your software into the
    hands of users
    

    View Slide

41. Hibri Marzook, February 2017
    @hibri
    Thanks!
    Questions?
    @hibri
    hibri@hibri.net
    

    View Slide