Upgrade to Pro — share decks privately, control downloads, hide ads and more …

CI/CD For the Enterprise

CI/CD For the Enterprise

As organizations adopt continuous integration/continuous delivery (CI/CD) into their software practives, it often stops at building artifacts and running test suites. In the typical enterprise, there are many mory processes surrounding a software release such as change management and product management sign-off. Integrating these buisness processes into CI/CD pipelines allows software teams to spend more time delivering value to customers and less time filling out paperwork.

In this session, you will follow a software development team in Red Hat IT as they journeyed to fully integrate business processes into their CI/CD pipeline.

You will learn how to:
* Integrate code quality checks into your pipeline
* Automate project management processes
* Adapt change management processes to more frequent software releases
* Use a single pipeline to follow code from initial commit to production

Patrick Easters

May 01, 2019
Tweet

More Decks by Patrick Easters

Other Decks in Technology

Transcript

  1. CI/CD FOR THE ENTERPRISE:
    TEACHING AN OLD DOG NEW TRICKS
    Open Infrastructure Summit 2019
    Patrick Easters
    Sr. Software Applications Engineer
    May 1, 2019

    View full-size slide

  2. 2
    THE PIPE DREAM
    CODE
    QUALITY
    SECURITY
    SCANNING
    CHANGE
    MGMT
    PORTFOLIO
    MGMT
    BUILD TEST
    AUTOMATICALLY
    DEPLOY TO
    PRODUCTION
    CONTINUOUS
    INTEGRATION
    CONTINUOUS
    DELIVERY

    View full-size slide

  3. 3
    THE SQUAD
    WHO WE ARE
    ● 22 people (managers, developers,
    SREs, quality engineers, BAs)
    ● Distributed across 3 states in the
    US and 3 continents
    WHAT WE DO
    ● Build and maintain applications to
    help Red Hat customers manage
    their subscriptions and access
    content

    View full-size slide

  4. 4
    WHERE WE BEGIN
    ● Early stages of adopting DevOps practices, but still very siloed
    ● IT Operations group performs majority of releases
    ● Good use of Configuration Management, though it was complex
    ● Averaged 2 releases/mo for our flagship application

    View full-size slide

  5. 5
    THE TERRIBLE, HORRIBLE, NO GOOD,
    VERY BAD RELEASE PROCESS
    QA
    Ops
    Dev
    BA
    Manager
    Change
    Manager
    Dev
    QA
    Stage
    Prod
    Product
    Owner
    PHASE

    View full-size slide

  6. 6
    THE TERRIBLE, HORRIBLE, NO GOOD,
    VERY BAD RELEASE PROCESS

    View full-size slide

  7. 7
    OPPORTUNITIES FOR IMPROVEMENT
    ● Releases were a significant source of toil
    ● Infrequent releases led to long cycle times
    ● Confusing process

    View full-size slide

  8. 8
    PAAS TO THE RESCUE?
    ● Red Hat IT launched new PaaS offering based on OpenShift 3
    ● Application teams given access to their apps in production
    ● Automated build and release jobs “for free”

    View full-size slide

  9. 11
    SCALING OUT
    ● So far we’ve only migrated a few low-touch apps
    ● Migrated our flagship app: access.redhat.com/management
    ● Started a greenfield app: api.access.redhat.com/management

    View full-size slide

  10. SOLVING CONSTRAINTS

    View full-size slide

  11. 13
    DEV-ON-DEMAND
    ● A single dev environment was no longer sufficient
    ● Moved to short-lived instances following Merge Requests
    ● Shifted left: code quality scans, unit tests
    ● Quick feedback for QEs and BAs

    View full-size slide

  12. 14
    OPERATION “GO GREEN”
    ● Many automated tests flickered and became unstable over time
    ● Manual triage required by Quality Engineers
    ● Conscious choice to allocate time for fixing tests instead of
    developing new ones

    View full-size slide

  13. 15
    MODERNIZING CHANGE MANAGEMENT
    ● Better risk categorization
    ● Removed arbitrary lead time requirement and change freezes
    ● Used “Standard Change” type that is pre-approved
    ● Used ServiceNow API to open/close changes

    View full-size slide

  14. 16
    USER STORIES, NOT FAKE NEWS
    ● Up-to-date Kanban board is important
    ○ WIP Limits
    ○ Current Status
    ● Integrated pipeline with Rally APIs to move stories and create
    milestones

    View full-size slide

  15. 17
    IT’S BEGINNING TO LOOK A LOT LIKE CI/CD

    View full-size slide

  16. MEASURING UP

    View full-size slide

  17. 19
    DELIVERY PERFORMANCE
    ELITE HIGH MEDIUM LOW
    Deployment
    Frequency
    on-demand hourly-daily weekly-monthly weekly-monthly
    Change Lead
    Time
    <1h 1d - 1w 1w - 1m 1m - 6m
    MTTR <1h <1d <1d 1w - 1m
    Change Failure
    Rate
    0-15% 0-15% 0-15% 46-60%
    From DORA’s 2018 State of DevOps report

    View full-size slide

  18. 20
    RHSM WEB CHANGE RATE
    Change Requests per Month

    View full-size slide

  19. 22
    THE HUMAN FACTOR
    “Our release process used to feel more like filing taxes, but now it’s
    more like sending money with Venmo”

    View full-size slide

  20. 23
    THE HUMAN FACTOR
    “My favorite part of our CI/CD adoption has been the freedom to
    experiment: having disposable merge instances for giving early
    feedback, feature flags that let us try things with a targeted audience
    and fine-tune before turning on new functionality for the entire
    customer base, and even the experimentation that comes with every
    member of the team feeling empowered to try automating some piece
    of our process.”

    View full-size slide

  21. 24
    THE HE-MAN FACTOR
    “I am basically He-Man. By the power of CI/CD, I
    have the power*!”
    *To push changes to production myself when an appropriate
    threshold of quality and organizational preparedness, much of
    which is built into our pipelines, is met.

    View full-size slide

  22. THANK YOU
    Email: [email protected]
    Twitter: @patrickeasters

    View full-size slide