Gothenburg Jenkins Meetup. Testing on Hardware with Jenkins Pipeline

Running tests on hardware with Jenkins Pipeline Oleg Nenashev (@oleg_nenashev)
CloudBees, Inc. Days of Jenkins Göteborg, May 29, 2017

© 2017 CloudBees, Inc. All Rights Reserved. 2 #DayOfJenkins Göteborg
2017 About me @oleg_nenashev oleg-nenashev LibreCores project St. Petersburg Polytechnic University Jenkins meetups

2017 Oleg’s “Hall of Shame”(c) • Jenkins Core • Remoting • Windows Service Wrapper • Plugins

2017 About you…

2017 ØJenkins Pipeline in Embedded • Managing workspaces • Managing shared resources • Tools Ø[TBD] - My Adopter experience Agenda

2017 • Pipeline 101 • Pipeline Generic Features • Blue Ocean • Declarative Pipeline • Automation Infrastructure • Agents, Computing Grids, Virtualization • Added-value Pipeline features in CloudBees Jenkins Enterprise What is NOT in the talk? Robert’s talk Q&A Session Q&A

2017 What is a typical Embedded Product? ` Hardware External IP Firmware / Embedded SW External SW Development tools / EDAs Demo software Demo hardware Developer kits “Main product” Documentation

2017 Automotive… Source: http://www.popsautoelectric.com/auto-electric/ • System of systems • High level of integration • 3rd-party components • Many standards • High quality requirements

2017 Embedded area. Welcome to the automation hell! • Continuous integration is must-have • Multiple technologies and tools • Legacy is everywhere • Unstable hardware & networks • …

2017 1. The most popular CI/CD automation server in the world 2. It’s a framework => flexible 3. It’s open source => customizable 4. Commercial support is available 5. Useful plugins Why Jenkins?

2017 Система Docs Plugins Configs Infra- structure Samples Demos How do we build CI Systems? Infra- structure Infra- structure • Agents (FKA ”slaves”) • Tools • Hardware peripherals • Computing grids

Testing of Embedded projects…

2017 Where do build/tests run? Software Hardware Simulators, EDA tools FPGAs, ASIC prototypes HW Accelerators, etc. x

2017 Tests are reproducible Easy to debug issues No need in expensive hardware (+) Single-shot VMs • No environment pollution, etc. Simulators. PROS “We run Hello World in two hours.”

2017 Models are just… models • Not accurate for all cases • May contain bugs Long execution time • Slow spot-checks • Computing grids for massive runs Simulators. CONS “We run Hello World in two hours.”

2017 And hardware…

2017 Example. Jenkins Node with an Attached Board 17 VM or bare-metal Power Reset when it hangs Deploy bit-files, Blasting cables Debugging cables, firmware upload We don’t change them, huh? Jenkins agent Drivers Tools Peripherals Attached peripherals Runtime

2017 Minimize amount of hardware tools Utilize hardware as much as possible Share hardware between projects • Also use it outside Jenkins... Reuse the existing hardware 18 Issue #1. Hardware is Expensive! M – x000$ L –x0000$ M - http://www.xilinx.com/univ/xupv5-lx110t.htm L - HAPS-64, www.synopsys.com

2017 System monitoring • Discover hanging runs • Discover deadlocks Automatic recovery from Infra issues • Check if it is an infra issues • Rerun tests if possible • Common use-case: confirm HW failures with simulations Environment management • Environment cleanup • Workspace cleanup 19 Issue #2. Hardware is Unstable

What do we need?

2017 What do we need? Resource management • Shared resources: hardware, licenses • Access management: Permissions, etc. Tools • Integration with tools • Shared tool infrastructures, preinstalled tools • Tool versioning Reusable solutions • Templates • Pipeline libraries

2017 Approach: Using “classic” Job types: • Freestyle, Matrix, JobDSL, … Classic Approaches Keywords: • Script steps to run the logic • Custom Tools Plugin to manage tools • Throttle Concurrent Builds and Lockable Resources to dispatch access to hardware • Naginator Plugin to restart builds on-demand • Parameterized Trigger / Copy artifacts to parallelize tests and to collect results JUC2015/London https://jenkins.io/solutions/embedded/

2017 Classic Approaches Spaghetti Automation • Barely traceable dependencies • Complex build flow • Complex configurations • Duplication • Expensive maintenance

2017 ü Store build definitions with the project ü Track changes together with the project ü Tests together with the project Configuration as Code

2017 Configuration as Code in Jenkins Job Configuration Jenkins system configuration

2017 TODO: Oil Pipelines Image Pipelines?

2017 Jenkins Pipeline is a… 1. Groovy DSL for automatic flow description - defined in SCM (Jenkinsfile) - defined in Job configuration // Run on a node having the “my_fpga” label node("linux && ml509") { git url:"http://github.com/myorg/myproject.git" sh "make all" } http://bit.ly/pipeline-tutorial

2017 Main features Configuration-as-Code Independent from node/workspace Parallelization Robustness

2017 Jenkins Pipeline is a… 1. Groovy DSL for automatic flow description 2. Item type Pipeline Job Multi-Branch Pipeline Organization Folder • GitHub • BitBucket* • …

2017 Jenkins Pipeline is a… 1. Groovy DSL for automatic flow description 2. Item type 3. Pipeline as Code Ecosystem

2017 Branch- Source, BlueOcean, ... 89 - explicit support 42 - Pipeline- specific Documentation Samples Development Tools Jenkins Pipeline Ecosystem Plugins:

Jenkins Pipeline in Embedded 2016

2017 2016 http://bit.ly/nenashev-automotive-day-2016

2017 • Management of resources. • Integration with Throttle Concurrent Builds plugin • Use-case: limiting the license utilization in automation infrastructures (JENKINS-31801) • Better support of tools. • Common build flow – dozens of tools • Integration with Custom Tools Plugin seems to be the best option, especially in the case of multiple tool versions (JENKINS-30680) • Pipeline package manager (JENKINS-34186) • Lack of plugins for EDA tools in Jenkins, code duplications • A common approach is to keep the shared "functions" in libraries • Pipeline Global Library and Pipeline Remote Loader can be used 2016 – Pipeline improvement areas From my talk at CloudBees | Jenkins Automotive and Embedded Day 2016

2017 Resource Management • Lockable Resources Plugin • Since 2016 https://plugins.jenkins.io/lockable-resources

2017 Resource Management • Throttle Concurrent Builds • Since 2.0, May 2017 • Supports only categories • Throttle inside run • Throttle across many runs https://plugins.jenkins.io/throttle-concurrents

2017 Throttle parallel tasks inside Pipeline

2017 Management of resources • Lockable Resources • Throttle Concurrent Builds • Pipeline Milestones – milestone() step • Priority Sorter Plugin • Job Fan-In plugin • … http://bit.ly/pipeline-compatibility

2017 Tools Support • withEnv() for environment variables • tool() step in Pipeline was improved • Tool reference in the Job • Section in Declarative Pipeline

2017 Tools Support • EnvInject Plugin – partially integrated • Only basic use-cases • No ETA • Custom Tools Plugin – partially integrated • Follow JENKINS-30680 • ETA – 2017

2017 Pipeline Libs • Early 2016 • Global Pipeline Library • Pipeline Remote Loader Plugin • Late 2016 • Pipeline Shared Libraries • Support of @Grab – Pipeline libs can pull common Groovy libs Library Structure:

2017 /me as a Pipeline Adopter

2017 Pipeline Early adoption It is working! * Image: http://devopsreactions.tumblr.com/post/151281670953/trying-the-first-prototype-good-enough

Lessons Learnt. Obstacles? • Missing integrations • Pipeline DSL !=
Groovy • Error handling • Development Tools

2017 Missing integration ØTools/plugins •http://bit.ly/pipeline-compatibility •Trusted pipeline libraries should help much ØBuild Promotion (Promoted Builds, Release, …)

2017 For a reason… Pipeline DSL is not Groovy

2017 Reason 1. Context retention – CPS/NonCPS • No way to propagate any data • Solution - @NonCPS steps within the library steps

2017 Reason 2. Script Security Script approval Groovy Sandbox Mode • Mandatory for Jenkinsfile from SCM • Mandatory for untrusted libs Limited access to Java API • Script approvals by admins • Debug fun “Run & Fail & Approve & Rerun”

2017 Error Handling • IMHO: Pipeline is designed for Success Path scenarios When it “Just works” When it does not

2017 Error Handling. Top-level Scripted Pipeline Declarative Pipeline • Global try/catch() When it does not…

2017 Error Handling When it “Just works” When it does not The interesting part… Failover, repeat

2017 Error Handling. Hardware failover • Pipeline allows re-running unreliable stages on other nodes. System failure (script returns 253) Node #1 Node #2 . . . . . . for (def board : boards) { echo "trying board " + board; try { node(board) { checkout scm sh ‘./bin/run.sh’ // Call passed => DONE break; } } catch (Exception ex) { if (ex.message.contains ("exit code 253")) { // Fatal error fail("Test run failed") } } } Try all possible boards:

2017 Error handling. Hardware failover for (def board : boards) { echo "trying board " + board; try { node(board) { checkout scm sh ‘./bin/run.sh’ // Call passed => DONE break; } } catch (Exception ex) { if (ex.message.contains ("exit code 255")) { // Fatal error fail("Test run failed") } } } ??? • Hard to extract error cause • Special handling is required for every case • Cannot just use existing parsers • Build Failure Analyzer • Log Parser • Warnings • Try-catch blocks pollute the code • Hard to test

2017 Error handling. Hardware failover Declarative Pipeline Encapsulation via Libraries Simple use-case Other • Exception try/catch • timeout() wrapper • Proper failover • Logic in @NonCPS

2017 Pipeline. Dev Tools IDE Integration Unit Test Framework Static Analysis Library manager Debug tools Documen- tation Easy script deployment Freestyle => Pipeline Converter Good Not enough Missing

2017 How do I work? Intellij IDEA Filesystem SCM Plugin (with patch) Disclaimer: There may be better ways • Syntax file • Built-in documentation Jenkins Test Instance Source Code (local .git repos) • Production System Snapshot • Custom Pipeline Library Settings • Repositories for libraries • Jenkinsfile repos

2017 Getting the Syntax File

2017 Filesystem SCM Plugin • Pending pull request: • https://github.com/jenkinsci/filesystem_scm-plugin/pull/2 Library definition in the test instance !!!

2017 Pipeline Testing • No standard test framework offered by the Jenkins project • External project: https://github.com/lesfurets/JenkinsPipelineUnit “mvn verify” has been invoked… And the execution passed… For the source Jenkinsfile…

Lessons learnt • No area-specific issues so far • High
barrier to entry • It worth investment

2017 What I need? 2017 • Development Tools for Pipeline • IDE, Debugger, Test Tools • Community Pipeline Libraries • Common Embedded cases without plugins • Promotion engine for Pipeline • Ideally: Merge of standard Promoted Builds and Release plugins • Plugins for area-specific tools

2017 Takeaways • Jenkins Pipeline is a powerful engine for highly-customized Jenkins instances • Jenkins Pipeline can be effectively used in Embedded/Automotive use-cases • Barrier to entry is still high • Just try it…

2017 Thank you! Contacts: E-mail: [email protected] GitHub: oleg-nenashev Twitter: @oleg_nenashev

2017 More info • Embedded solutions page: • https://jenkins.io/solutions/embedded/ • Links to the articles and videos • Some categorization • To be updated soon… Embedded Solutions Page …

2017 • Jenkins project • Website: https://jenkins.io • Meetup: https://www.meetup.com/Jenkins-online-meetup/ • Embedded Solutions page: https://jenkins.io/solutions/embedded/ • Jenkins Pipeline • Tutorial: http://bit.ly/pipeline-tutorial • Compatibility: http://bit.ly/pipeline-compatibility • Examples: https://github.com/jenkinsci/pipeline-examples • CloudBees • Community portal: https://go.cloudbees.com/ Useful Links

2017 Thank you! Contacts: E-mail: [email protected] GitHub: oleg-nenashev Twitter: @oleg_nenashev

Software at the speed of ideas THANK YOU! www.cloudbees.com

Gothenburg Jenkins Meetup. Testing on Hardware ...

Gothenburg Jenkins Meetup. Testing on Hardware with Jenkins Pipeline

More Decks by Oleg Nenashev

Other Decks in Programming

Featured

Transcript