Upgrade to Pro — share decks privately, control downloads, hide ads and more …

CI/CD For the Enterprise

CI/CD For the Enterprise

As organizations adopt continuous integration/continuous delivery (CI/CD) into their software practives, it often stops at building artifacts and running test suites. In the typical enterprise, there are many mory processes surrounding a software release such as change management and product management sign-off. Integrating these buisness processes into CI/CD pipelines allows software teams to spend more time delivering value to customers and less time filling out paperwork.

In this session, you will follow a software development team in Red Hat IT as they journeyed to fully integrate business processes into their CI/CD pipeline.

You will learn how to:
* Integrate code quality checks into your pipeline
* Automate project management processes
* Adapt change management processes to more frequent software releases
* Use a single pipeline to follow code from initial commit to production

98780d04a49b212b2f4dc5d65ed30add?s=128

Patrick Easters

May 01, 2019
Tweet

More Decks by Patrick Easters

Other Decks in Technology

Transcript

  1. CI/CD FOR THE ENTERPRISE: TEACHING AN OLD DOG NEW TRICKS

    Open Infrastructure Summit 2019 Patrick Easters Sr. Software Applications Engineer May 1, 2019
  2. 2 THE PIPE DREAM CODE QUALITY SECURITY SCANNING CHANGE MGMT

    PORTFOLIO MGMT BUILD TEST AUTOMATICALLY DEPLOY TO PRODUCTION CONTINUOUS INTEGRATION CONTINUOUS DELIVERY
  3. 3 THE SQUAD WHO WE ARE • 22 people (managers,

    developers, SREs, quality engineers, BAs) • Distributed across 3 states in the US and 3 continents WHAT WE DO • Build and maintain applications to help Red Hat customers manage their subscriptions and access content
  4. 4 WHERE WE BEGIN • Early stages of adopting DevOps

    practices, but still very siloed • IT Operations group performs majority of releases • Good use of Configuration Management, though it was complex • Averaged 2 releases/mo for our flagship application
  5. 5 THE TERRIBLE, HORRIBLE, NO GOOD, VERY BAD RELEASE PROCESS

    QA Ops Dev BA Manager Change Manager Dev QA Stage Prod Product Owner PHASE
  6. 6 THE TERRIBLE, HORRIBLE, NO GOOD, VERY BAD RELEASE PROCESS

  7. 7 OPPORTUNITIES FOR IMPROVEMENT • Releases were a significant source

    of toil • Infrequent releases led to long cycle times • Confusing process
  8. 8 PAAS TO THE RESCUE? • Red Hat IT launched

    new PaaS offering based on OpenShift 3 • Application teams given access to their apps in production • Automated build and release jobs “for free”
  9. 9

  10. 10 CI/CD

  11. 11 SCALING OUT • So far we’ve only migrated a

    few low-touch apps • Migrated our flagship app: access.redhat.com/management • Started a greenfield app: api.access.redhat.com/management
  12. SOLVING CONSTRAINTS

  13. 13 DEV-ON-DEMAND • A single dev environment was no longer

    sufficient • Moved to short-lived instances following Merge Requests • Shifted left: code quality scans, unit tests • Quick feedback for QEs and BAs
  14. 14 OPERATION “GO GREEN” • Many automated tests flickered and

    became unstable over time • Manual triage required by Quality Engineers • Conscious choice to allocate time for fixing tests instead of developing new ones
  15. 15 MODERNIZING CHANGE MANAGEMENT • Better risk categorization • Removed

    arbitrary lead time requirement and change freezes • Used “Standard Change” type that is pre-approved • Used ServiceNow API to open/close changes
  16. 16 USER STORIES, NOT FAKE NEWS • Up-to-date Kanban board

    is important ◦ WIP Limits ◦ Current Status • Integrated pipeline with Rally APIs to move stories and create milestones
  17. 17 IT’S BEGINNING TO LOOK A LOT LIKE CI/CD

  18. MEASURING UP

  19. 19 DELIVERY PERFORMANCE ELITE HIGH MEDIUM LOW Deployment Frequency on-demand

    hourly-daily weekly-monthly weekly-monthly Change Lead Time <1h 1d - 1w 1w - 1m 1m - 6m MTTR <1h <1d <1d 1w - 1m Change Failure Rate 0-15% 0-15% 0-15% 46-60% From DORA’s 2018 State of DevOps report
  20. 20 RHSM WEB CHANGE RATE Change Requests per Month

  21. 21 LEAD TIME

  22. 22 THE HUMAN FACTOR “Our release process used to feel

    more like filing taxes, but now it’s more like sending money with Venmo”
  23. 23 THE HUMAN FACTOR “My favorite part of our CI/CD

    adoption has been the freedom to experiment: having disposable merge instances for giving early feedback, feature flags that let us try things with a targeted audience and fine-tune before turning on new functionality for the entire customer base, and even the experimentation that comes with every member of the team feeling empowered to try automating some piece of our process.”
  24. 24 THE HE-MAN FACTOR “I am basically He-Man. By the

    power of CI/CD, I have the power*!” *To push changes to production myself when an appropriate threshold of quality and organizational preparedness, much of which is built into our pipelines, is met.
  25. Q+A

  26. THANK YOU Email: peasters@redhat.com Twitter: @patrickeasters