Automating test runs on hardware with Jenkins and Pipeline-as-Code

In automotive projects it's frequently required to run tests on special hardware peripherals: development boards, prototypes, etc. It may be required for both software and hardware areas, and especially for products involving both worlds.
CI and CD methodologies require continuous integration and system testing, and Jenkins comes to help here.

Jenkins is an automation framework, which can be adjusted to reliably work with hardware attached to its nodes. My presentation addresses common use-cases like peripheral connection, development tools usage, resource sharing between teams, and infrastructure monitoring. I will present potential implementations for classic Jenkins freestyle jobs and Pipeline-as-Code. We will also talk about the added value being provided by CloudBees Jenkins Platform.

Transcript

1. © 2016 CloudBees, Inc. All Rights Reserved 1 © 2016 CloudBees, Inc. All Rights Reserved Running tests on Hardware with Jenkins and Pipeline as Code Oleg Nenashev (@oleg_nenashev), Jenkins Developer CloudBees, Inc. Böblingen,  April  26,  2016
2. © 2016 CloudBees, Inc. All Rights Reserved 2 • Jenkins Developer at CloudBees – CloudBees Jenkins Platform (Enterprise Edition) o CloudBees Jenkins Operations Center o CloudBees Jenkins Analytics – Customer support and consulting • Jenkins project – Core team member – Maintainer of plugins (Ownership, Custom tools, Promoted builds, etc.) – Google Summer of Code 2016 org admin > whoami -now @oleg_nenashev oleg-­nenashev onenashev
3. © 2016 CloudBees, Inc. All Rights Reserved 3 • PhD, Hardware engineering • Background: R&D and Automation engineer – HW and Embedded Area – ~8 years o Intel, Sitronics, Synopsys – Development of embedded systems – Automation for IPs, SoCs and toolchains – Large-scale automation infrastructures > whoami -background @oleg_nenashev oleg-­nenashev onenashev
4. © 2016 CloudBees, Inc. All Rights Reserved 4 • https://www.cloudbees.com/jenkins/juc- 2015/abstracts/europe/02-01-1130-nenashev Prior work JUC2015/London
5. © 2016 CloudBees, Inc. All Rights Reserved 5 Ø Automation in the Embedded world. Why Jenkins? Ø Using hardware peripherals Ø Pipeline as Code Ø Added value of CloudBees Jenkins Platform Agenda
6. © 2016 CloudBees, Inc. All Rights Reserved 6 What is an Embedded Product? ` Hardware External IP Firmware / Embedded SW External SW Development tools / EDAs Demo software Demo hardware Developer kits “Main  product” Documentation
7. © 2016 CloudBees, Inc. All Rights Reserved 7 And then… integrate components ?  Dependencies  of  products ?  Tons  of  reports ?  Who  has  broken  the  build ?  Is  it  an  infra  issue ?  What  did  we  actually  test Component 1 Component 2 Component 3 Component 4 Product Boom!
8. © 2016 CloudBees, Inc. All Rights Reserved 8 Embedded area. Welcome to the automation hell! • Almost everything needs to be automated • Multiple technologies and tools • Unstable hardware & networks • Legacy stuff is everywhere • HW engineers are not SW experts • Limited resources
9. © 2016 CloudBees, Inc. All Rights Reserved 9 1. Avoid spending much resources on it 2. Provide feedback to developers as soon as possible – Classic approaches – N days/weeks – Now – N minutes/hours 3. Ensure the transparency of the project status 4. Frequently ship development versions of components to other teams for integration testing – “Continuous delivery” What do we want from Automation?
10. © 2016 CloudBees, Inc. All Rights Reserved 10 Why Jenkins? • It’s the most popular automation server in the world • It’s open-source: – You can customize Jenkins plugins – Plenty of available materials and examples • It’s a framework => you can do everything • Commercial support is available for self-hosted installations
11. © 2016 CloudBees, Inc. All Rights Reserved 11 Jenkins Is Powerful Over 1000 Jenkins Plugins Integration with hundreds of tools Orchestration of the Tool- chains End-to-End CI/CD Pipelines Deploy Release Scan  &  Test Code  &   Commit Build  &  Config
12. © 2016 CloudBees, Inc. All Rights Reserved 12 Система Docs Plugins Configs Infra-­ structure Samples Demos Jenkins-based Automation System Infra-­ structure Infra-­ structure
13. © 2016 CloudBees, Inc. All Rights Reserved 13 Jenkins users in the Embedded World • The most of Embedded companies have Jenkins instances • Examples from a 5-minute search in public materials
14. © 2016 CloudBees, Inc. All Rights Reserved 14 Abstract component build flow Get sources SCMs Binary  repositories Build  artifacts Setup environment “Build” “Test” “Analyze” Publish reports Deploy artifacts Tool-­chains SW  Dev.  Tools EDA  tools Integration  stuff Multiple   parallel   tasks Manual Timers SCM  events Notify stakeholders
15. © 2016 CloudBees, Inc. All Rights Reserved 15 Embedded components. Testing approaches Software Hardware Simulators, EDA tools FPGAs,  ASIC  prototypes HW  Accelerators,  etc. x
16. © 2016 CloudBees, Inc. All Rights Reserved 16 • PROS – Good diagnosability and reproducibility – No need in expensive hardware peripherals • CONS – Models may contain their own bugs – Real-time execution is commonly non-achievable – Long execution time (Cycle Accurate & HW simulations) o Slow spot-checks (on-commit changes, pull request builds) o Need in computing grids for massive runs (random tests) Simulators “We  run  hello-­world   in  2  hours”
17. © 2016 CloudBees, Inc. All Rights Reserved 17 Suggested Verification Flow Analysis of build results and reports Stage II Hardware Stage I Pre-flight checks Stage III Simulator FAILED Rerun on another board in the case of infra failure PASSED Verify failure, provide reproducible reports to developers Parallelize on-demand
18. © 2016 CloudBees, Inc. All Rights Reserved 18 Hardware stuff for testing • S  -­ EZ1CUSB  @  http://www.easyfpga.com • M  -­ http://www.xilinx.com/univ/xupv5-­lx110t.htm • L    -­ HAPS-­64,  www.synopsys.com S  -­ x00$ M  – x000$ L  –x0000$ FPGA  accelerators   &  prototyping   boards • Various  prototypes,  developer  kits,  FPGA  boards,  etc.  
19. © 2016 CloudBees, Inc. All Rights Reserved 19 And XXL… http://blogs.synopsys.com/breakingthethreelaws/ 2014/04/the-­secret-­ninja-­fu-­for-­higher-­ performance-­prototype-­operation/ HAPS  S96 Guess  the  price… Your  device prototype  is   here
20. © 2016 CloudBees, Inc. All Rights Reserved 20 We want to… – … minimize the amount of hardware we use – … utilize the hardware as much as possible – … share hardware between projects Nice to have: – ... reuse the existing hardware (from developer computers, manual QA labs, etc.) Issue #1. Hardware is expensive!
21. © 2016 CloudBees, Inc. All Rights Reserved 21 Example. Jenkins node with an attached board Power Reset   when  it  hangs “Deploy  bit-­files” Blasting   cables Debugging   cables, firmware  upload We  don’t   change   them,  huh? Jenkins  node drivers tools peripherals Attached peripherals runtime VM or bare- metal
22. © 2016 CloudBees, Inc. All Rights Reserved 22 Mapping automation infra to existing HW Nodes  @  Site  A Nodes  @  Site  C Nodes  @  Site  B “We  borrowed  FPGAs   here,  somebody  takes   them  randomly” “This  stuff  has   Windows  drivers  only” “No  infra  team   members  here” “We  run  Jenkins   nodes  on  developer   laptops” Frankenboards “We  built  them  from   the  garbage  we  had” L L L L L
23. © 2016 CloudBees, Inc. All Rights Reserved 23 • Any run may fail due to infrastructure issues • Testing cycle is long, rerun should be automatic – Helps with board/network failures – Master failure is fatal • Hardware tests are barely reproducible => need to confirm failures with simulations • Clean environment issue Issue #2. Hardware is unstable
24. © 2016 CloudBees, Inc. All Rights Reserved 24 • Access should be dispatched by Jenkins • Builds utilize the hardware during a single phase only • Scheduled availability of nodes • Teams fight for resources • Clean environment (again) Issue #3. Hardware is shared
25. © 2016 CloudBees, Inc. All Rights Reserved 25 Approach: Use Free-style and Matrix: – https://jenkins.io/solutions/embedded/ Classic approaches JUC2015/London More info: Keywords: • Script  steps  to  run  the  logic • Custom  Tools  Plugin   to  manage  the  tools • Throttle  Concurrent  Builds   and  Lockable   Resources   to  dispatch  the  access  to  the   hardware • Naginator Plugin  to  restart  the  builds  on-­ demand • Parameterized   Trigger  /  Copy  artifacts  to   parallelize   tests  and  to  collect  results
26. © 2016 CloudBees, Inc. All Rights Reserved 26 Classic Approaches Spaghetti Automation • Many job dependencies • Difficult build flow tracking • Complex configurations • Duplication • Expensive maintenance
27. © 2016 CloudBees, Inc. All Rights Reserved 27 Classic Approaches. Partial solutions Ø Script-based frameworks Ø Templates for common stuff – CloudBees Templates Plugin in the CJP – Template Project & Inheritance Plugins in open-source Ø Fingerprinting of artifacts – Allows to track the builds at least
28. © 2016 CloudBees, Inc. All Rights Reserved 28 Need a better solution? Describe the build flow as a code!
29. © 2016 CloudBees, Inc. All Rights Reserved 29 • Now it’s an CI automation server • Focus on user experience • Minimal compatibility changes, all 1.x plugins work well • UX improvements: – Web UI improvements (configuration pages, etc.) – New installation wizard – New website with docs • Security is enabled by default • Pipeline as Code Jenkins 2.0. What’s inside? http://bit.ly/jenkins20
30. © 2016 CloudBees, Inc. All Rights Reserved 30 Automation as Code in Jenkins Job DSL Job Builder Plugin Pipeline (fka Workflow) Pipeline  is  offered  by  default  in  Jenkins  2.0 … Groovy YAML Groovy
31. © 2016 CloudBees, Inc. All Rights Reserved 31 • Groovy DSL for build flow description • Job type in Jenkins (Pipeline Job) • Ecosystem for automation-as-code – ~15 dedicated plugins – Integration in many other plugins Pipeline is a…
32. © 2016 CloudBees, Inc. All Rights Reserved 32 Pipeline. Example #1 32 • Syntax:  Groovy-­based  DSL • Job  descriptions  can  be  stored  in  SCM • E.g.  Jenkinsfile in  the  repo  root • http://bit.ly/pipeline-­tutorial //  Run  on  a  node  having  the  “my_fpga”  label node("linux &&  ml509")  {   git url:"http://github.com/oleg-­nenashev/pipeline_hw_samples" sh "make  all" }
33. © 2016 CloudBees, Inc. All Rights Reserved 33 • Robustness against restarts of Jenkins master • Robustness against network disconnects – sh() steps are based on the Durable Task plugin – Jenkins continues the execution flow once the node reconnects • Parallelization across multiple nodes: – It can be achieved via combination of parallel() and node() steps – No need to use the trigger/copy chain (but you can do it) • Ability to store the shared logic in standalone Pipeline libraries Pipeline. Useful features
34. © 2016 CloudBees, Inc. All Rights Reserved 34 Stage view
35. © 2016 CloudBees, Inc. All Rights Reserved 35 Example 2. Running on two nodes in parallel … stage ‘test' parallel ‘board1' : { node(‘linux &&  board1')  { unstash 'source' sh "./bin/runTest -­-­target=board1" step([$class:'JUnitResultArchiver',   testResults:'**/test-­results/*.xml']) } },  ’board2': { node(‘linux &&  board2')  )  { unstash 'source' sh "./bin/runTest -­-­target=board2" step([$class:'JUnitResultArchiver',   testResults:'**/test-­results/*.xml']) } }  
36. © 2016 CloudBees, Inc. All Rights Reserved 36 • Lockable Resources plugin is integrated with Pipeline • Throttle Concurrent Builds plugin is not integrated, license limitations may be a problem Resource management echo  'Starting' node('board1') { stage(’Compile') //  Build  project //  ... stage(’Run  tests') lock(env.NODE_NAME + "-­board1") { //  Run  something  on  the  board //  ... } }
37. © 2016 CloudBees, Inc. All Rights Reserved 37 Node failover • Pipeline allows re-running unreliable stages on other nodes. System  failure Node  #1 Node  #2 .  .  . .  .  . for (def board : boards) { echo "trying board " + board; try { node(board) { checkout scm sh ‘./bin/run.sh’ // Call passed => DONE break; } } catch (Exception ex) { if (ex.message.contains ("exit code 255")) { // Fatal error fail("Test run failed") } } } Try all possible boards:
38. © 2016 CloudBees, Inc. All Rights Reserved 38 Pipeline features. Interactive step // Build  the  project,  blast  it  to  the  boards //  Prepare  the  run  in  the  test  management  system //  Send  the  notification   to  testers.  Then  wait  for  the  input echo  ’Automatic  tests  passed’   hipchatSend('@QATeam The  build  ' + buildVersion + '  has  been  staged:  ' + stagingBoard + '.  Please  Test  it  and  report  the  result  back  to  ' + build.url + '.  TestLink run  is  here:  ' + testLinkRunURL) input  message:  'Have  manual  spot-­‐checks  passed?  Ready  to  release?' //  Proceed  on  the  successful  path mail  to:  "dream_team@cloudbees.com",   subject:  "Dream  Product  -­‐ build  is  released",   body:  "Hi  there,  The  new  build  of  the  product   has  been  released.  Link:" +  artifactoryLink ~~~~~~  Code  ~~~~~~ ~~~~~~  Code  ~~~~~~ Pipeline can interact with users in real-time via input steps
39. © 2016 CloudBees, Inc. All Rights Reserved 39 More examples: https://github.com/jenkinsci/pipeline- examples
40. © 2016 CloudBees, Inc. All Rights Reserved 40 © 2016 CloudBees, Inc. All Rights Reserved 40 What’s about CloudBees Jenkins Platform?
41. © 2016 CloudBees, Inc. All Rights Reserved 41 Common features All  common  features  are   applicable,  out  of  the  scope See  CloudBees  Docs
42. © 2016 CloudBees, Inc. All Rights Reserved 42 CJP Features. Templates! • OSS Plugins have a limited functionality: – Template Project, Inheritance Plugin, Job Generator, etc. • Parameterized trigger causes UI overheads – Multi-page configuration in UI: “Trigger with several parameters => wait => copy artifacts => analyze” • Templates allow to save much time on the implementations • Templates support pipeline
43. © 2016 CloudBees, Inc. All Rights Reserved 43 CJP Features. Shared slaves for rare hardware • Use-case: multiple Jenkins instances in a single org • Instances may require a specific resource • Out-of-the-box throttling of CJOC shared slaves
44. © 2016 CloudBees, Inc. All Rights Reserved 44 • It is a flexible monitoring and reporting engine • Use-cases: – Node utilization analysis. When do we need to expand the infra? – Monitoring of distributed environments CJP Features. Analytics Example:
45. © 2016 CloudBees, Inc. All Rights Reserved 45 • Allows to restart Pipeline from a particular step • E.g. “Retry deployment” to save time – Needs an actual workspace CJP Features. Pipeline checkpoints node  {   sh './build-­and-­test'  }   checkpoint   'Completed   tests'   node  {   sh './deploy'   }
46. © 2016 CloudBees, Inc. All Rights Reserved 46 Summary • Jenkins can be used to run tests on hardware in embedded and automotive projects • Jenkins Pipeline is helpful for test runs • CloudBees Jenkins Platform helps to decrease the maintenance cost and to improve user experience
47. © 2016 CloudBees, Inc. All Rights Reserved 47 • Management of resources. – Integration with Throttle Concurrent Builds plugin – Use-case: limiting the license utilization in automation infrastructures (JENKINS-31801) • Better support of tools. – Common build flow – dozens of tools – Integration with Custom Tools Plugin seems to be the best option, especially in the case of multiple tool versions (JENKINS-30680) • Pipeline package manager (JENKINS-34186) – Lack of plugins for EDA tools in Jenkins, code duplications – A common approach is to keep the shared "functions" in libraries – Pipeline Global Library and Pipeline Remote Loader can be used Pipeline improvement areas*
48. © 2016 CloudBees, Inc. All Rights Reserved 48 More info • There are many articles about Jenkins in the Embedded area – Most of them describe the “classic approaches” based on Freestyle jobs • Embedded solutions page: – https://jenkins.io/solutions/embedded/ Embedded Solutions Page …
49. © 2016 CloudBees, Inc. All Rights Reserved 49 Useful links • Jenkins project – Website: https://jenkins.io – Embedded Solutions page: https://jenkins.io/solutions/embedded/ • Pipeline as Code tutorial: – https://github.com/jenkinsci/workflow- plugin/blob/master/TUTORIAL.md • CloudBees website – Website: https://www.cloudbees.com – CloudBees Jenkins Platform: https://www.cloudbees.com/products/cloudbees-jenkins-platform
50. © 2016 CloudBees, Inc. All Rights Reserved 50 © 2016 CloudBees, Inc. All Rights Reserved Contacts: E-­mail:  onenashev@cloudbees.com GitUb/Gitter: Twitter:  @oleg_nenashev oleg-­nenashev
51. © 2016 CloudBees, Inc. All Rights Reserved 51 // Synthesize on any node def imageId="" node("linux") { stage "Prepare environment" git url:"http://github.com/oleg-nenashev/pipeline_hw_samples" // Construct the bitfile image ID from commit ID sh 'git rev-parse HEAD > GIT_COMMIT' imageId= "myprj-${fpgaId}-" + readFile('GIT_COMMIT').take(6) stage "Synthesize project" sh "make FPGA_TYPE=$fpgaId synthesize_for_fpga" /* We archive the bitfile before running the test, so it won't be lost it if something happens with the FPGA run stage. */ archive "target/image_${fpgaId}.bit" stash includes: "target/image_${fpgaId}.bit", name: 'bitfile’ } …. Example 3. Optimize the node usage [1/2]
52. © 2016 CloudBees, Inc. All Rights Reserved 52 … /* Run on a node with 'my_fpga' label. In this example it means that the Jenkins node contains the attacked FPGA of such type.*/ node ("linux && $fpgaId") { stage "Blast bitfile" git url:"http://github.com/oleg-nenashev/pipeline_hw_samples" def artifact='target/image_'+fpgaId+'.bit' unstash 'bitfile' sh "make FPGA_TYPE=$fpgaId impact" /* Run automatic tests. Then report test results from the generated Junit */ stage "Auto Tests" sh "make FPGA_TYPE=$fpgaId tests" sh "perl scripts/convertToJunit.pl --from=target/test-results/*” + "--to=target/report_${fpgaId}.xml --classPrefix=\"myprj-${fpgaId}.\"" step([$class:"JUnitResultArchiver", testResults:"target/report_${fpgaId}.xml"]) } Example 3. Optimize the node usage [2/2]