DevOps and Continuous Development

With the drive towards Continuous Delivery, how we develop and deliver quality products is changing at a rapid pace. Our once slow delivery cycles worked well for on-premise deployments but now we are challenged to deliver in ever shorter cycles to support our services in the cloud while ensuring quality for all deployment targets. The diversity of what we need to support is ever increasing while the pressure to shorten delivery cycles pushes us to find ever more efficiency in our pipeline and processes.

This talk will share our journey as we adopted DevOps and Continuous Delivery: from the cultural changes we experienced, the lessons we learned, and, finally, how Cognitive Software Delivery and Testing has enabled those changes and driven greater quality and efficiency in our lifecycle.

Transcript

1. The drive to continuous delivery A quality focused delivery lifecycle Elastic Build Cloud Moving testing into the cloud at scale Cognitive Software Testing Driving quality through analytics WAS Liberty Overview Elastic Build and Test Ecosystem Cognitive Software Delivery Demo / Examples Open Liberty Scalable - Cross reference data from many sources to develop greater insights. Higher quality through focused attention on key issues Fast and adaptive - create new views of data in minutes, not days Real-time insights into the delivery pipeline Efficient - remove manual effort and optimise infrastructure and delivery pipeline Cognitive Software Delivery Kevin Smith - Cognitive Software Delivery Architect e: smithk6@uk.ibm.com © 2019 IBM Corporation A highly instrumented, scalable, cloud based elastic build and test ecosystem analysed in realtime to provide insights into product quality and process optimizations.
2. WAS Liberty © 2019 IBM Corporation Liberty - A brief history 2010 Project Inception A few scrum teams working in closely together using AGILE methodologies 2012 First release of WebSphere Liberty WebSphere Liberty released in June 2012 The drive to continuous delivery A quality focused delivery lifecycle Elastic Build Cloud Moving testing into the cloud at scale Cognitive Software Testing Driving quality through analytics Overview Elastic Build and Test Ecosystem Cognitive Software Delivery Demo / Examples Open Liberty Kevin Smith - Cognitive Software Delivery Architect e: smithk6@uk.ibm.com
3. WAS Liberty © 2019 IBM Corporation Liberty - A brief history 2010 Project Inception A few scrum teams working in closely together using AGILE methodologies 2012 First release of WebSphere Liberty WebSphere Liberty released in June 2012 2013 Big Bang restructure for continuous delivery Large scale re-org of work item management as well as source code repository to enable continuous delivery. 2016 Migration from RTC to GitHub Enterprise Large scale re-org of work item management as well as source code repository to enable continuous delivery. 2015 IBM starts pushing DevOps and shared tooling Internal development supported by CIO office as we start seeing external tools being available internally starting with Slack The drive to continuous delivery A quality focused delivery lifecycle Elastic Build Cloud Moving testing into the cloud at scale Cognitive Software Testing Driving quality through analytics Overview Elastic Build and Test Ecosystem Cognitive Software Delivery Demo / Examples Open Liberty Kevin Smith - Cognitive Software Delivery Architect e: smithk6@uk.ibm.com water fall SCRUM Product Vision Requirements Analysis Specification Design Approval QA Testing Acceptance SCRUM Code Test Documentation Service Build/Test triage
4. WAS Liberty © 2019 IBM Corporation Liberty - A brief history 2010 Project Inception A few scrum teams working in closely together using AGILE methodologies 2012 First release of WebSphere Liberty WebSphere Liberty released in June 2012 2013 Big Bang restructure for continuous delivery Large scale re-org of work item management as well as source code repository to enable continuous delivery. 2016 Migration from RTC to GitHub Enterprise Large scale re-org of work item management as well as source code repository to enable continuous delivery. 2015 IBM starts pushing DevOps and shared tooling Internal development supported by CIO office as we start seeing external tools being available internally starting with Slack The drive to continuous delivery A quality focused delivery lifecycle Elastic Build Cloud Moving testing into the cloud at scale Cognitive Software Testing Driving quality through analytics Overview Elastic Build and Test Ecosystem Cognitive Software Delivery Demo / Examples Open Liberty Kevin Smith - Cognitive Software Delivery Architect e: smithk6@uk.ibm.com water fall SCRUM Squad Product Vision Requirements Analysis Specification Design Approval QA Testing Acceptance Squad Squad Squad Autonomous Squads, Tightly Aligned SCRUM Code Test Documentation Service Build/Test triage
5. WAS Liberty © 2019 IBM Corporation Liberty - A brief history 2010 Project Inception A few scrum teams working in closely together using AGILE methodologies 2012 First release of WebSphere Liberty WebSphere Liberty released in June 2012 2013 Big Bang restructure for continuous delivery Large scale re-org of work item management as well as source code repository to enable continuous delivery. 2016 Migration from RTC to GitHub Enterprise Large scale re-org of work item management as well as source code repository to enable continuous delivery. 2015 IBM starts pushing DevOps and shared tooling Internal development supported by CIO office as we start seeing external tools being available internally starting with Slack 2017 Opening Up - OpenLiberty is born Large parts of Liberty code base moved to GitHub.com and made available to the Open Source Community. 2019 Change to monthly releases Move from a 3 monthly release cycle to monthly. The drive to continuous delivery A quality focused delivery lifecycle Elastic Build Cloud Moving testing into the cloud at scale Cognitive Software Testing Driving quality through analytics Overview Elastic Build and Test Ecosystem Cognitive Software Delivery Demo / Examples Open Liberty Kevin Smith - Cognitive Software Delivery Architect e: smithk6@uk.ibm.com water fall SCRUM Squad Product Vision Requirements Analysis Specification Design Approval QA Testing Acceptance Squad Squad Squad Autonomous Squads, Tightly Aligned SCRUM Code Test Documentation Service Build/Test triage
6. 
 
 
 
 
 
 
 Stabilisation Phase Time to known quality Tolerance of regressions Test run frequency Team would generally start to make software ready for release when all functionality has been developed. Team should get their software ready for release throughout development at periodic intervals. Team keeps software ready for release at all times during development. Months Weeks Days / Hours N/A Days Hours High Low Zero Weekly Daily Every Change WAS Liberty © 2019 IBM Corporation Waterfall Agile Continuous Delivery The drive to continuous delivery A quality focused delivery lifecycle Elastic Build Cloud Moving testing into the cloud at scale Cognitive Software Testing Driving quality through analytics Overview Elastic Build and Test Ecosystem Cognitive Software Delivery Demo / Examples Open Liberty Kevin Smith - Cognitive Software Delivery Architect e: smithk6@uk.ibm.com
7. © 2019 IBM Corporation Personal 
 Build Continuous
 Build Release
 Build Re-Iterate Re-Iterate Exit Gates: To progress the code either all tests must pass or only known defects be encountered. • Unit Test: 
 Verifies smallest testable pieces of code. • Build Verification Test: 
 Small set of tests used to initially validate the build. • Functional Test (Lite): 
 Verifies correct behaviour of functions. The “Lite” version is around 70% of the entire corpus, focusing on golden path testing and common error paths. • Functional Test (Full Corpus): 
 Entire corpus of functional tests. A superset of the “Lite” functional tests adding in uncommon code paths, long running tests and rare error paths.
 • Supported Operating Environment Test
 Runs the Entire corpus of functional tests against a large matrix of OS/JDK combinations supported by the product. • Installation Verification:
 Verify the product has been correctly packaged. • Persona (SVT):
 Large scale testing including customer like scenarios, load, scalability, recoverability tests. Majority are run manually to include some exploratory testing. • Continuous Persona Test (NEST):
 24/7 test environment running customer like scenarios constantly, updated using DevOps principles. • Performance Test
 Designed to identify performance issues and bottlenecks as well as verify the throughput of the product. • Compliance/Compatibility Test (CTS):
 A set of tests and tools to confirm the product performs to an industry standard specification (J2EE). Functional Test (Full Corpus) Supported Operating Environment Test (SOE) Installation Verification Test Persona (SVT) - Regression plus exploratory testing Continuous Persona Test (NEST) Performance Test Compliance / Compatability Test (CTS) The drive to continuous delivery A quality focused delivery lifecycle Elastic Build Cloud Moving testing into the cloud at scale Cognitive Software Testing Driving quality through analytics Overview Elastic Build and Test Ecosystem Cognitive Software Delivery Demo / Examples Code Change Exit Gate Exit Gate Release Build Unit Test Build Verification Test Functional Test (Lite) Unit Test Build Verification Test Functional Test (Lite) Unit Test Build Verification Test Functional Test (Lite) Open Liberty Kevin Smith - Cognitive Software Delivery Architect e: smithk6@uk.ibm.com
8. WAS Liberty © 2019 IBM Corporation The Build Environment • Liberty has a fully automated build and test system. 
 • Infrastructure as code: Definition of machines, software stack, build scripts are all defined as code.
 • Our build/test infrastructure is backed by our “Elastic Build Ecosystem”.
 • Dynamically provisioned build/test machines created and destroyed for each job.
 • Use multiple clouds for high availability.
 • Up to 20000 virtual machines provisioned in a single week.
 • Each Machine individually configured with the stack required for the specific job.
 • Ability to spin up multiple machine topologies in support of a single job.
 • Highly instrumented build feeding details in real time to the cognitive solution.
 • Dependant Infrastructure monitored and managed by Consul. • Supported Operating Environment testing runs 120 hours worth of tests on each of 190 platform/JDK combinations. • Running between one and two years worth of testing (machine time) every single day. The drive to continuous delivery Elastic Build Cloud Moving testing into the cloud at scale Cognitive Software Testing Driving quality through analytics Overview Elastic Build and Test Ecosystem Cognitive Software Delivery Demo / Examples Open Liberty Kevin Smith - Cognitive Software Delivery Architect e: smithk6@uk.ibm.com
9. WAS Liberty © 2019 IBM Corporation 2 4 6 1 3 5 Detect Build Monitor build system and spot when a new request is made. Create Plan Identify all possible solutions then select the best one considering available data including load, capacity and reliability. Create and Configure Create/Identify the base machine or container then install and configure any required components Verify Verify that the created environment and components are working as expected. Run Build Execute the build as requested and monitor its progress. Cleanup Completely clean up all the requested resources. Kafka Elastic Build Ecosystem - Workflow The drive to continuous delivery Elastic Build Cloud Moving testing into the cloud at scale Cognitive Software Testing Driving quality through analytics Overview Elastic Build and Test Ecosystem Cognitive Software Delivery Demo / Examples Open Liberty Kevin Smith - Cognitive Software Delivery Architect e: smithk6@uk.ibm.com
10. WAS Liberty © 2019 IBM Corporation The drive to continuous delivery Elastic Build Cloud Moving testing into the cloud at scale Cognitive Software Testing Driving quality through analytics Overview Elastic Build and Test Ecosystem Cognitive Software Delivery Demo / Examples Open Liberty Kevin Smith - Cognitive Software Delivery Architect e: smithk6@uk.ibm.com
11. WAS Liberty © 2019 IBM Corporation The drive to continuous delivery Elastic Build Cloud Moving testing into the cloud at scale Cognitive Software Testing Driving quality through analytics Overview Elastic Build and Test Ecosystem Cognitive Software Delivery Demo / Examples Open Liberty Kevin Smith - Cognitive Software Delivery Architect e: smithk6@uk.ibm.com
12. WAS Liberty © 2019 IBM Corporation The Original Build The drive to continuous delivery Elastic Build Cloud Moving testing into the cloud at scale Cognitive Software Testing Driving quality through analytics Overview Elastic Build and Test Ecosystem Cognitive Software Delivery Demo / Examples Open Liberty Kevin Smith - Cognitive Software Delivery Architect e: smithk6@uk.ibm.com
13. WAS Liberty © 2019 IBM Corporation Cognitive Software Delivery Application of analytics to provide real-time insights into product quality and the delivery pipeline to driving greater efficiency and customer satisfaction. The drive to continuous delivery Cognitive Software Testing Driving quality through analytics Overview Elastic Build and Test Ecosystem Cognitive Software Delivery Demo / Examples Open Liberty Kevin Smith - Cognitive Software Delivery Architect e: smithk6@uk.ibm.com
14. WAS Liberty We can be smarter in what we build, how we build it and what we test Manual analysis of the data is very expensive Vast amounts of data hundreds of code changes / day 200+ personal builds / day >1 year of testing / day Fast feedback of regressions to delivery teams Data is complex 5 1 2 3 4 Why we need cognitive software delivery © 2019 IBM Corporation The drive to continuous delivery Cognitive Software Testing Driving quality through analytics Overview Elastic Build and Test Ecosystem Cognitive Software Delivery Demo / Examples Open Liberty Kevin Smith - Cognitive Software Delivery Architect e: smithk6@uk.ibm.com
15. Cognitive Software Delivery WAS Liberty © 2019 IBM Corporation The drive to continuous delivery Cognitive Software Testing Driving quality through analytics Overview Elastic Build and Test Ecosystem Cognitive Software Delivery Demo / Examples Open Liberty Kevin Smith - Cognitive Software Delivery Architect e: smithk6@uk.ibm.com Elastic Build Ecosystem Virtual Machines Virtual Machines Docker/Kubernetees Static Machines Ecosystem Orchestrator Data Collection Source Control Defect Management Visualization Engine Analytics/AI Engine Web Frontend Scheduler Kafka HTTP Rest
16. WAS Liberty © 2019 IBM Corporation No longer assume we have to keep everything. As we move to continuous delivery test results are in fact transient and the defects become the system of record. Assert: Test results are transient No longer develop a solution for one product. Instead always consider how other products can use the solution and seek to be generic. 
 MQ, CICS Liberty, I2 and I2 counter fraud are adopting the solution. Solution must be generic System must be cheap for others to adopt, both in terms of infrastructure requirements and the learning curve. Low adoption cost If we can drive for a generic database schema that works for many products, the investment in Analytics on top of that data become sharable and re-usable. Standard data. Although initially set up to capture functional test data, the system should be easily expanded to cover new areas. We are already capturing information on our builds, change sets etc to drive further capabilities. Extensible Guiding
 Principles The drive to continuous delivery Cognitive Software Testing Driving quality through analytics Overview Elastic Build and Test Ecosystem Cognitive Software Delivery Demo / Examples Open Liberty Kevin Smith - Cognitive Software Delivery Architect e: smithk6@uk.ibm.com
17. Advanced interrogation of our data: Analytics has allowed us to quickly and easily interrogate our data by creating new streams. A real world example: Our builds were taking longer than normal to run, causing build queues, intermittent failures and slowing development. We used analytics to identify a relationship between build times and the virtualized hardware running the build. This entire process had a turn-around time of 8 hours. 25% reduction in execution time. 150% increase in “green” builds. WAS Liberty © 2019 IBM Corporation Demo / Examples The drive to continuous delivery Overview Elastic Build and Test Ecosystem Cognitive Software Delivery Open Liberty Kevin Smith - Cognitive Software Delivery Architect e: smithk6@uk.ibm.com
18. Demo / Examples WAS Liberty © 2019 IBM Corporation Live Real Time Dashboards The drive to continuous delivery Overview Elastic Build and Test Ecosystem Cognitive Software Delivery Open Liberty Kevin Smith - Cognitive Software Delivery Architect e: smithk6@uk.ibm.com
19. WAS Liberty © 2019 IBM Corporation Introducing the dot plot Sorted Platform Test Suite sorted to group common elements together Note: A dot represents one or more failures in a test suite Demo / Examples The drive to continuous delivery Overview Elastic Build and Test Ecosystem Cognitive Software Delivery Open Liberty Kevin Smith - Cognitive Software Delivery Architect e: smithk6@uk.ibm.com
20. WAS Liberty © 2019 IBM Corporation Introducing the dot plot Sorted Platform Test Suite sorted to group common elements together Note: A dot represents one or more failures in a test suite Demo / Examples The drive to continuous delivery Overview Elastic Build and Test Ecosystem Cognitive Software Delivery Open Liberty Kevin Smith - Cognitive Software Delivery Architect e: smithk6@uk.ibm.com
21. WAS Liberty © 2019 IBM Corporation What it actually looks like (All platforms for one build) Demo / Examples The drive to continuous delivery Overview Elastic Build and Test Ecosystem Cognitive Software Delivery Open Liberty Kevin Smith - Cognitive Software Delivery Architect e: smithk6@uk.ibm.com
22. WAS Liberty © 2019 IBM Corporation More Recent Older Runs An alternative view: Single platform over time Demo / Examples The drive to continuous delivery Overview Elastic Build and Test Ecosystem Cognitive Software Delivery Open Liberty Kevin Smith - Cognitive Software Delivery Architect e: smithk6@uk.ibm.com
23. WAS Liberty © 2019 IBM Corporation Semi-Auto Triage Demo / Examples The drive to continuous delivery Overview Elastic Build and Test Ecosystem Cognitive Software Delivery Open Liberty Kevin Smith - Cognitive Software Delivery Architect e: smithk6@uk.ibm.com
24. WAS Liberty © 2019 IBM Corporation Semi-Auto Triage Demo / Examples The drive to continuous delivery Overview Elastic Build and Test Ecosystem Cognitive Software Delivery Open Liberty Kevin Smith - Cognitive Software Delivery Architect e: smithk6@uk.ibm.com
25. WAS Liberty © 2019 IBM Corporation Semi-Auto Triage Demo / Examples The drive to continuous delivery Overview Elastic Build and Test Ecosystem Cognitive Software Delivery Open Liberty Kevin Smith - Cognitive Software Delivery Architect e: smithk6@uk.ibm.com