$30 off During Our Annual Pro Sale. View Details »

Evolving Continuous Delivery

Hibri Marzook
February 27, 2017

Evolving Continuous Delivery

This is a story of guiding a team's CD adoption and maturity, from a chaotic start, to being awesome.

This is a journey from a troublesome, struggling start of chaotic manual deployments, merge hell, regular production roll backs and lost code, to deliver a single commit to master automatically and reliably, under an hour, many times a day.

We’ll look at a few simple practices that worked, to introduce CD practices such as monitoring, build automation and test automation gradually, instead of going all in at first.

You’ll start to see that it isn’t daunting at all, and start delivering with joy.

Related post: http://www.hibri.net//2016/06/18/continuous-delivery-rags-to-riches/

Hibri Marzook

February 27, 2017
Tweet

More Decks by Hibri Marzook

Other Decks in Technology

Transcript

  1. Evolving
    Continuous
    Delivery
    From Rags to Riches
    Hibri Marzook
    @hibri

    View Slide

  2. @hibri
    http://www.hibri.net

    View Slide

  3. Hibri Marzook, February 2017
    @hibri
    Fintech
    Post startup phase
    Successful product in the market
    ~ 30 in engineering
    ~ 200 employees
    Context

    View Slide

  4. Hibri Marzook, February 2017
    @hibri
    First impressions
    Public facing API, running in the Cloud
    No automation
    Inconsistent environments
    Context switching. Not finishing work in progress
    Old, stale branches merged into active branch
    Manual deploys

    View Slide

  5. Hibri Marzook, February 2017
    @hibri

    View Slide

  6. Hibri Marzook, February 2017
    @hibri
    What to fix first?
    Branching strategy?
    Lack of automation?
    Lack of technical practices? TDD?
    Lack of CI?
    Process?
    Scrum, Kanban ceremonies?

    View Slide

  7. Hibri Marzook, February 2017
    @hibri
    A focus on delivery is
    needed for continuous
    improvement

    View Slide

  8. Hibri Marzook, February 2017
    @hibri
    Gain situational awareness
    Photo Credit - Hugh Trainer via Flickr

    View Slide

  9. Hibri Marzook, February 2017
    @hibri
    Gaining situational
    awareness
    Install an application monitoring tool. New Relic
    Zero friction install
    Alerting out of the box
    PagerDuty integration

    View Slide

  10. Hibri Marzook, February 2017
    @hibri
    We could be the first responders

    How much traffic?

    How many users are affected?

    Usage trends?

    Error rates?

    Can this scale?

    View Slide

  11. Hibri Marzook, February 2017
    @hibri
    Shared understanding of the
    system
    Draw environment map
    Draw components of the system
    Ensure that all team members have the same
    understanding
    Keep maps visible

    View Slide

  12. Hibri Marzook, February 2017
    @hibri
    Better situational awareness
    Photo Credit - Steve Jurveston via Flickr

    View Slide

  13. Hibri Marzook, February 2017
    @hibri

    View Slide

  14. Hibri Marzook, February 2017
    @hibri
    Expose the realities of the
    system instead of relying
    on assumptions

    View Slide

  15. Hibri Marzook, February 2017
    @hibri
    In a legacy distributed application, there is a
    disconnect between how the system behaves in
    production and how the creators think it “should”
    behave.

    View Slide

  16. Hibri Marzook, February 2017
    @hibri
    Create visibility

    View Slide

  17. Hibri Marzook, February 2017
    @hibri
    Agile ≠ JIRA

    View Slide

  18. Hibri Marzook, February 2017
    @hibri
    Creating visibility
    Made work visible with a simple card wall
    Pull from the right attitude
    Use JIRA only for tracking
    Created transparency about the work we are doing

    View Slide

  19. Hibri Marzook, February 2017
    @hibri
    Limit work in progress
    Did one thing at a time
    Got into the habit of finishing one thing
    Whole team focussed on delivery
    Only then did we do more

    View Slide

  20. Hibri Marzook, February 2017
    @hibri
    When you don’t have
    anything else, put
    everyone together

    View Slide

  21. Hibri Marzook, February 2017
    @hibri
    Put everyone together
    Everyone we worked with sat in the same area
    Introduced Slack
    Devs are always around to fix an issue
    Practice remote working early

    View Slide

  22. Hibri Marzook, February 2017
    @hibri

    View Slide

  23. Hibri Marzook, February 2017
    @hibri
    Use slack time to learn
    and improve

    View Slide

  24. Hibri Marzook, February 2017
    @hibri
    How did we create slack?
    WIP Limits
    Production issues are quantified, don’t need to jump
    on every issue
    Have data to push back on management pressure
    Can ignore some problems

    View Slide

  25. Hibri Marzook, February 2017
    @hibri
    Long term change
    happens during slack
    time

    View Slide

  26. Hibri Marzook, February 2017
    @hibri
    Before CI
    Couldn’t reliably build the system. “Works on my
    machine”
    Key man dependencies to do a release.
    Build time dependencies have to be fixed manually.

    View Slide

  27. Hibri Marzook, February 2017
    @hibri

    View Slide

  28. Hibri Marzook, February 2017
    @hibri
    When the team spends more time
    talking about branching and
    merging, you have the wrong
    strategy

    View Slide

  29. Hibri Marzook, February 2017
    @hibri
    Introducing CI
    Built the system reliably and repeatedly on a CI
    server
    Removed the key man dependency
    Used this build as the source for manual deploys
    No ATDD yet
    Manual regression testing

    View Slide

  30. Hibri Marzook, February 2017
    @hibri
    Getting branching under
    control

    View Slide

  31. Hibri Marzook, February 2017
    @hibri

    View Slide

  32. Hibri Marzook, February 2017
    @hibri
    Amplify the good
    things

    View Slide

  33. Hibri Marzook, February 2017
    @hibri
    Adding automated testing
    Made the tests work in a local dev environment
    Made the same tests run in other environments
    Kept adding tests for all new work and fixes
    Refactored tests continuously

    View Slide

  34. Hibri Marzook, February 2017
    @hibri
    Adding automated deploys
    Deployed to test environments on every commit
    Used Azure’s “slots” feature for blue/green
    TeamCity + Octopus Deploy
    Smoke tests after every deploy

    View Slide

  35. Hibri Marzook, February 2017
    @hibri

    View Slide

  36. Hibri Marzook, February 2017
    @hibri
    The ‘A’ word

    View Slide

  37. Hibri Marzook, February 2017
    @hibri
    Talk and discuss the values
    Focus on the impact
    Explain cause and effect
    Introduce and repeat the principles
    Agile manifesto. Left over right

    View Slide

  38. Hibri Marzook, February 2017
    @hibri
    https://landing.google.com/sre/book/chapters/part3.html#fig_part-practices_reliability-hierarchy

    View Slide

  39. Hibri Marzook, February 2017
    @hibri
    Evolutionary approach
    Deal with current constraints
    Sometimes it’s OK to do things manually
    Every change should make sense and help the team a little bit
    more
    Team should not have to wait for all the changes to gain value

    View Slide

  40. Hibri Marzook, February 2017
    @hibri
    Have a primal push to get
    your software into the
    hands of users

    View Slide

  41. Hibri Marzook, February 2017
    @hibri
    Thanks!
    Questions?
    @hibri
    [email protected]

    View Slide