Automating test runs on hardware with Jenkins and Pipeline-as-Code

Slide 1

Slide 1 text

© 2016 CloudBees, Inc. All Rights Reserved 1 © 2016 CloudBees, Inc. All Rights Reserved Running tests on Hardware with Jenkins and Pipeline as Code Oleg Nenashev (@oleg_nenashev), Jenkins Developer CloudBees, Inc. Böblingen, April 26, 2016

Slide 2

Slide 2 text

© 2016 CloudBees, Inc. All Rights Reserved 2 • Jenkins Developer at CloudBees – CloudBees Jenkins Platform (Enterprise Edition) o CloudBees Jenkins Operations Center o CloudBees Jenkins Analytics – Customer support and consulting • Jenkins project – Core team member – Maintainer of plugins (Ownership, Custom tools, Promoted builds, etc.) – Google Summer of Code 2016 org admin > whoami -now @oleg_nenashev oleg-nenashev onenashev

Slide 3

Slide 3 text

© 2016 CloudBees, Inc. All Rights Reserved 3 • PhD, Hardware engineering • Background: R&D and Automation engineer – HW and Embedded Area – ~8 years o Intel, Sitronics, Synopsys – Development of embedded systems – Automation for IPs, SoCs and toolchains – Large-scale automation infrastructures > whoami -background @oleg_nenashev oleg-nenashev onenashev

Slide 4

Slide 4 text

Slide 5

Slide 5 text

Slide 6

Slide 6 text

© 2016 CloudBees, Inc. All Rights Reserved 6 What is an Embedded Product? ` Hardware External IP Firmware / Embedded SW External SW Development tools / EDAs Demo software Demo hardware Developer kits “Main product” Documentation

Slide 7

Slide 7 text

© 2016 CloudBees, Inc. All Rights Reserved 7 And then… integrate components ? Dependencies of products ? Tons of reports ? Who has broken the build ? Is it an infra issue ? What did we actually test Component 1 Component 2 Component 3 Component 4 Product Boom!

Slide 8

Slide 8 text

© 2016 CloudBees, Inc. All Rights Reserved 8 Embedded area. Welcome to the automation hell! • Almost everything needs to be automated • Multiple technologies and tools • Unstable hardware & networks • Legacy stuff is everywhere • HW engineers are not SW experts • Limited resources

Slide 9

Slide 9 text

© 2016 CloudBees, Inc. All Rights Reserved 9 1. Avoid spending much resources on it 2. Provide feedback to developers as soon as possible – Classic approaches – N days/weeks – Now – N minutes/hours 3. Ensure the transparency of the project status 4. Frequently ship development versions of components to other teams for integration testing – “Continuous delivery” What do we want from Automation?

Slide 10

Slide 10 text

© 2016 CloudBees, Inc. All Rights Reserved 10 Why Jenkins? • It’s the most popular automation server in the world • It’s open-source: – You can customize Jenkins plugins – Plenty of available materials and examples • It’s a framework => you can do everything • Commercial support is available for self-hosted installations

Slide 11

Slide 11 text

© 2016 CloudBees, Inc. All Rights Reserved 11 Jenkins Is Powerful Over 1000 Jenkins Plugins Integration with hundreds of tools Orchestration of the Tool- chains End-to-End CI/CD Pipelines Deploy Release Scan & Test Code & Commit Build & Config

Slide 12

Slide 12 text

Slide 13

Slide 13 text

Slide 14

Slide 14 text

© 2016 CloudBees, Inc. All Rights Reserved 14 Abstract component build flow Get sources SCMs Binary repositories Build artifacts Setup environment “Build” “Test” “Analyze” Publish reports Deploy artifacts Tool-chains SW Dev. Tools EDA tools Integration stuff Multiple parallel tasks Manual Timers SCM events Notify stakeholders

Slide 15

Slide 15 text

Slide 16

Slide 16 text

© 2016 CloudBees, Inc. All Rights Reserved 16 • PROS – Good diagnosability and reproducibility – No need in expensive hardware peripherals • CONS – Models may contain their own bugs – Real-time execution is commonly non-achievable – Long execution time (Cycle Accurate & HW simulations) o Slow spot-checks (on-commit changes, pull request builds) o Need in computing grids for massive runs (random tests) Simulators “We run hello-world in 2 hours”

Slide 17

Slide 17 text

© 2016 CloudBees, Inc. All Rights Reserved 17 Suggested Verification Flow Analysis of build results and reports Stage II Hardware Stage I Pre-flight checks Stage III Simulator FAILED Rerun on another board in the case of infra failure PASSED Verify failure, provide reproducible reports to developers Parallelize on-demand

Slide 18

Slide 18 text

© 2016 CloudBees, Inc. All Rights Reserved 18 Hardware stuff for testing • S - EZ1CUSB @ http://www.easyfpga.com • M - http://www.xilinx.com/univ/xupv5-lx110t.htm • L - HAPS-64, www.synopsys.com S - x00$ M – x000$ L –x0000$ FPGA accelerators & prototyping boards • Various prototypes, developer kits, FPGA boards, etc.

Slide 19

Slide 19 text

© 2016 CloudBees, Inc. All Rights Reserved 19 And XXL… http://blogs.synopsys.com/breakingthethreelaws/ 2014/04/the-secret-ninja-fu-for-higher- performance-prototype-operation/ HAPS S96 Guess the price… Your device prototype is here

Slide 20

Slide 20 text

© 2016 CloudBees, Inc. All Rights Reserved 20 We want to… – … minimize the amount of hardware we use – … utilize the hardware as much as possible – … share hardware between projects Nice to have: – ... reuse the existing hardware (from developer computers, manual QA labs, etc.) Issue #1. Hardware is expensive!

Slide 21

Slide 21 text

© 2016 CloudBees, Inc. All Rights Reserved 21 Example. Jenkins node with an attached board Power Reset when it hangs “Deploy bit-files” Blasting cables Debugging cables, firmware upload We don’t change them, huh? Jenkins node drivers tools peripherals Attached peripherals runtime VM or bare- metal

Slide 22

Slide 22 text

© 2016 CloudBees, Inc. All Rights Reserved 22 Mapping automation infra to existing HW Nodes @ Site A Nodes @ Site C Nodes @ Site B “We borrowed FPGAs here, somebody takes them randomly” “This stuff has Windows drivers only” “No infra team members here” “We run Jenkins nodes on developer laptops” Frankenboards “We built them from the garbage we had” L L L L L

Slide 23

Slide 23 text

© 2016 CloudBees, Inc. All Rights Reserved 23 • Any run may fail due to infrastructure issues • Testing cycle is long, rerun should be automatic – Helps with board/network failures – Master failure is fatal • Hardware tests are barely reproducible => need to confirm failures with simulations • Clean environment issue Issue #2. Hardware is unstable

Slide 24

Slide 24 text

© 2016 CloudBees, Inc. All Rights Reserved 24 • Access should be dispatched by Jenkins • Builds utilize the hardware during a single phase only • Scheduled availability of nodes • Teams fight for resources • Clean environment (again) Issue #3. Hardware is shared

Slide 25

Slide 25 text

© 2016 CloudBees, Inc. All Rights Reserved 25 Approach: Use Free-style and Matrix: – https://jenkins.io/solutions/embedded/ Classic approaches JUC2015/London More info: Keywords: • Script steps to run the logic • Custom Tools Plugin to manage the tools • Throttle Concurrent Builds and Lockable Resources to dispatch the access to the hardware • Naginator Plugin to restart the builds on- demand • Parameterized Trigger / Copy artifacts to parallelize tests and to collect results

Slide 26

Slide 26 text

© 2016 CloudBees, Inc. All Rights Reserved 26 Classic Approaches Spaghetti Automation • Many job dependencies • Difficult build flow tracking • Complex configurations • Duplication • Expensive maintenance

Slide 27

Slide 27 text

© 2016 CloudBees, Inc. All Rights Reserved 27 Classic Approaches. Partial solutions Ø Script-based frameworks Ø Templates for common stuff – CloudBees Templates Plugin in the CJP – Template Project & Inheritance Plugins in open-source Ø Fingerprinting of artifacts – Allows to track the builds at least

Slide 28

Slide 28 text

Slide 29

Slide 29 text

© 2016 CloudBees, Inc. All Rights Reserved 29 • Now it’s an CI automation server • Focus on user experience • Minimal compatibility changes, all 1.x plugins work well • UX improvements: – Web UI improvements (configuration pages, etc.) – New installation wizard – New website with docs • Security is enabled by default • Pipeline as Code Jenkins 2.0. What’s inside? http://bit.ly/jenkins20

Slide 30

Slide 30 text

Slide 31

Slide 31 text

© 2016 CloudBees, Inc. All Rights Reserved 31 • Groovy DSL for build flow description • Job type in Jenkins (Pipeline Job) • Ecosystem for automation-as-code – ~15 dedicated plugins – Integration in many other plugins Pipeline is a…

Slide 32

Slide 32 text

© 2016 CloudBees, Inc. All Rights Reserved 32 Pipeline. Example #1 32 • Syntax: Groovy-based DSL • Job descriptions can be stored in SCM • E.g. Jenkinsfile in the repo root • http://bit.ly/pipeline-tutorial // Run on a node having the “my_fpga” label node("linux && ml509") { git url:"http://github.com/oleg-nenashev/pipeline_hw_samples" sh "make all" }

Slide 33

Slide 33 text

© 2016 CloudBees, Inc. All Rights Reserved 33 • Robustness against restarts of Jenkins master • Robustness against network disconnects – sh() steps are based on the Durable Task plugin – Jenkins continues the execution flow once the node reconnects • Parallelization across multiple nodes: – It can be achieved via combination of parallel() and node() steps – No need to use the trigger/copy chain (but you can do it) • Ability to store the shared logic in standalone Pipeline libraries Pipeline. Useful features

Slide 34

Slide 34 text

Slide 35

Slide 35 text

© 2016 CloudBees, Inc. All Rights Reserved 35 Example 2. Running on two nodes in parallel … stage ‘test' parallel ‘board1' : { node(‘linux && board1') { unstash 'source' sh "./bin/runTest --target=board1" step([$class:'JUnitResultArchiver', testResults:'**/test-results/*.xml']) } }, ’board2': { node(‘linux && board2') ) { unstash 'source' sh "./bin/runTest --target=board2" step([$class:'JUnitResultArchiver', testResults:'**/test-results/*.xml']) } }

Slide 36

Slide 36 text

© 2016 CloudBees, Inc. All Rights Reserved 36 • Lockable Resources plugin is integrated with Pipeline • Throttle Concurrent Builds plugin is not integrated, license limitations may be a problem Resource management echo 'Starting' node('board1') { stage(’Compile') // Build project // ... stage(’Run tests') lock(env.NODE_NAME + "-board1") { // Run something on the board // ... } }

Slide 37

Slide 37 text

© 2016 CloudBees, Inc. All Rights Reserved 37 Node failover • Pipeline allows re-running unreliable stages on other nodes. System failure Node #1 Node #2 . . . . . . for (def board : boards) { echo "trying board " + board; try { node(board) { checkout scm sh ‘./bin/run.sh’ // Call passed => DONE break; } } catch (Exception ex) { if (ex.message.contains ("exit code 255")) { // Fatal error fail("Test run failed") } } } Try all possible boards:

Slide 38

Slide 38 text

© 2016 CloudBees, Inc. All Rights Reserved 38 Pipeline features. Interactive step // Build the project, blast it to the boards // Prepare the run in the test management system // Send the notification to testers. Then wait for the input echo ’Automatic tests passed’ hipchatSend('@QATeam The build ' + buildVersion + ' has been staged: ' + stagingBoard + '. Please Test it and report the result back to ' + build.url + '. TestLink run is here: ' + testLinkRunURL) input message: 'Have manual spot-‐checks passed? Ready to release?' // Proceed on the successful path mail to: "[email protected]", subject: "Dream Product -‐ build is released", body: "Hi there, The new build of the product has been released. Link:" + artifactoryLink ~~~~~~ Code ~~~~~~ ~~~~~~ Code ~~~~~~ Pipeline can interact with users in real-time via input steps

Slide 39

Slide 39 text

Slide 40

Slide 40 text

Slide 41

Slide 41 text

Slide 42

Slide 42 text

© 2016 CloudBees, Inc. All Rights Reserved 42 CJP Features. Templates! • OSS Plugins have a limited functionality: – Template Project, Inheritance Plugin, Job Generator, etc. • Parameterized trigger causes UI overheads – Multi-page configuration in UI: “Trigger with several parameters => wait => copy artifacts => analyze” • Templates allow to save much time on the implementations • Templates support pipeline

Slide 43

Slide 43 text

© 2016 CloudBees, Inc. All Rights Reserved 43 CJP Features. Shared slaves for rare hardware • Use-case: multiple Jenkins instances in a single org • Instances may require a specific resource • Out-of-the-box throttling of CJOC shared slaves

Slide 44

Slide 44 text

© 2016 CloudBees, Inc. All Rights Reserved 44 • It is a flexible monitoring and reporting engine • Use-cases: – Node utilization analysis. When do we need to expand the infra? – Monitoring of distributed environments CJP Features. Analytics Example:

Slide 45

Slide 45 text

© 2016 CloudBees, Inc. All Rights Reserved 45 • Allows to restart Pipeline from a particular step • E.g. “Retry deployment” to save time – Needs an actual workspace CJP Features. Pipeline checkpoints node { sh './build-and-test' } checkpoint 'Completed tests' node { sh './deploy' }

Slide 46

Slide 46 text

© 2016 CloudBees, Inc. All Rights Reserved 46 Summary • Jenkins can be used to run tests on hardware in embedded and automotive projects • Jenkins Pipeline is helpful for test runs • CloudBees Jenkins Platform helps to decrease the maintenance cost and to improve user experience

Slide 47

Slide 47 text

© 2016 CloudBees, Inc. All Rights Reserved 47 • Management of resources. – Integration with Throttle Concurrent Builds plugin – Use-case: limiting the license utilization in automation infrastructures (JENKINS-31801) • Better support of tools. – Common build flow – dozens of tools – Integration with Custom Tools Plugin seems to be the best option, especially in the case of multiple tool versions (JENKINS-30680) • Pipeline package manager (JENKINS-34186) – Lack of plugins for EDA tools in Jenkins, code duplications – A common approach is to keep the shared "functions" in libraries – Pipeline Global Library and Pipeline Remote Loader can be used Pipeline improvement areas*

Slide 48

Slide 48 text

© 2016 CloudBees, Inc. All Rights Reserved 48 More info • There are many articles about Jenkins in the Embedded area – Most of them describe the “classic approaches” based on Freestyle jobs • Embedded solutions page: – https://jenkins.io/solutions/embedded/ Embedded Solutions Page …

Slide 49

Slide 49 text

© 2016 CloudBees, Inc. All Rights Reserved 49 Useful links • Jenkins project – Website: https://jenkins.io – Embedded Solutions page: https://jenkins.io/solutions/embedded/ • Pipeline as Code tutorial: – https://github.com/jenkinsci/workflow- plugin/blob/master/TUTORIAL.md • CloudBees website – Website: https://www.cloudbees.com – CloudBees Jenkins Platform: https://www.cloudbees.com/products/cloudbees-jenkins-platform

Slide 50

Slide 50 text

Slide 51

Slide 51 text

© 2016 CloudBees, Inc. All Rights Reserved 51 // Synthesize on any node def imageId="" node("linux") { stage "Prepare environment" git url:"http://github.com/oleg-nenashev/pipeline_hw_samples" // Construct the bitfile image ID from commit ID sh 'git rev-parse HEAD > GIT_COMMIT' imageId= "myprj-${fpgaId}-" + readFile('GIT_COMMIT').take(6) stage "Synthesize project" sh "make FPGA_TYPE=$fpgaId synthesize_for_fpga" /* We archive the bitfile before running the test, so it won't be lost it if something happens with the FPGA run stage. */ archive "target/image_${fpgaId}.bit" stash includes: "target/image_${fpgaId}.bit", name: 'bitfile’ } …. Example 3. Optimize the node usage [1/2]

Slide 52

Slide 52 text

© 2016 CloudBees, Inc. All Rights Reserved 52 … /* Run on a node with 'my_fpga' label. In this example it means that the Jenkins node contains the attacked FPGA of such type.*/ node ("linux && $fpgaId") { stage "Blast bitfile" git url:"http://github.com/oleg-nenashev/pipeline_hw_samples" def artifact='target/image_'+fpgaId+'.bit' unstash 'bitfile' sh "make FPGA_TYPE=$fpgaId impact" /* Run automatic tests. Then report test results from the generated Junit */ stage "Auto Tests" sh "make FPGA_TYPE=$fpgaId tests" sh "perl scripts/convertToJunit.pl --from=target/test-results/*” + "--to=target/report_${fpgaId}.xml --classPrefix=\"myprj-${fpgaId}.\"" step([$class:"JUnitResultArchiver", testResults:"target/report_${fpgaId}.xml"]) } Example 3. Optimize the node usage [2/2]