Automating hardware projects with Jenkins Bologna, October 8th, 2016 Oleg Nenashev, Jenkins project @oleg_nenashev 

@oleg_nenashev, #jenkinsci, #orconf2016 October, 2016 • CloudBees • Area-specific consulting to customers • Intel Labs, ex. SoC Architecture Group • Research, Embedded Processor Core • Branch Prediction, Binary Code Compression • Part-time project automation • Synopsys, ARC Processors department • Large-scale automation infrastructure lead • PhD • Automated hardware reengineering • Some prototypes were based on OpenRISC 1200 Who am I? HW Background @oleg_nenashev oleg-nenashev 2 

@oleg_nenashev, #jenkinsci, #orconf2016 October, 2016 Who am I? Now • PhD, Hardware Engineering • St. Petersburg Polytechnic University • Jenkins project, Core Team Member • Jenkins core and community • Meetup organizer, GSoC org admin • Plugin Maintainer • LibreCores, Newbie Contributor • Working on LibreCores CI • CloudBees, Engineer • CB is one of the top Jenkins contributors • CloudBees Jenkins Platform @oleg_nenashev oleg-nenashev 3 

@oleg_nenashev, #jenkinsci, #orconf2016 October, 2016 • Automation in the area • Who is Mr. Jenkins? • Why Jenkins? Why NOT Jenkins? • Common cases • Tool Management • Runs on FPGAs and peripherals • Simplifying automation with Jenkins Pipeline Agenda 4 

@oleg_nenashev, #jenkinsci, #orconf2016 October, 2016 Abstract component build flow Get sources SCMs Binary repositories Build artifacts Setup environment “Build” “Test” “Analyze” Publish reports Deploy artifacts Tool-chains SW Dev. Tools EDA tools Integration stuff Multiple parallel tasks Manual Timers SCM events Notify stakeholders 5 

@oleg_nenashev, #jenkinsci, #orconf2016 October, 2016 Abstract Hardware project ` Hardware External IP Firmware / embedded software External software Development tools / EDAs Demo software Demo hardware Developer kits “Main deliverable” Documentation Integ- ration Boom! 6 

@oleg_nenashev, #jenkinsci, #orconf2016 October, 2016 • Almost everything needs to be automated • Legacy stuff is everywhere • Multiple technologies and tools • Unstable hardware and networks • Limited resources • Hardware engineers are not software experts 7 Welcome to Automation Hell! Technical Non-technical 

@oleg_nenashev, #jenkinsci, #orconf2016 October, 2016 üImprove the quality of projects • Discover issues on early stages üImprove contributor experience • Pull request builders • Validated merge üTransparency of the project status (in enterprise) üFrequently ship DEV versions to other teams/users • Integration Testing • “Continuous delivery” in hardware What do we want from Automation? 8 

@oleg_nenashev, #jenkinsci, #orconf2016 October, 2016 Who is Mr. Jenkins? 9 1. The most popular automation server in the world • After CRON ... 2. It’s a framework • Swiss Army Knife in automation 3. It’s open source • You can customize it 4. It’s scalable • Hundreds of build agents • Thousands of jobs 5. Many users in the hardware area 

@oleg_nenashev, #jenkinsci, #orconf2016 October, 2016 Jenkins is Popular http://stats.jenkins-ci.org/jenkins-stats/ • >150k installations • ~300 active committers (6 months) • ~3 new plugins every week • ~50 plugin releases every week Hudson Jenkins 10 

@oleg_nenashev, #jenkinsci, #orconf2016 October, 2016 Jenkins is Powerful Deploy Release Scan & Test Code & Commit Build & Config 11 

@oleg_nenashev, #jenkinsci, #orconf2016 October, 2016 •LTS release lines •Backward compatibility •Jenkins 1.x was compatible for 8+ years •Jenkins 2.x is almost compatible with Jenkins 1.x •It’s also conservative about continuous improvement Jenkins is Conservative 12 

@oleg_nenashev, #jenkinsci, #orconf2016 October, 2016 Jenkins in the Hardware Area After 5 minutes of googling… 13 

@oleg_nenashev, #jenkinsci, #orconf2016 October, 2016 • Lack of [documentation/experience sharing] in the hardware area • Lack of area-specific plugins • Configuration and maintenance are no so trivial (it’s a framework!) • General obstacles: • Complex and outdated UI • Plugin inter-compatibility • Updates require significant efforts • Community is not always responsive Why NOT Jenkins? 14 

@oleg_nenashev, #jenkinsci, #orconf2016 October, 2016 Jenkins-based Automation System Computing grids Docs HW peripherals Plugins Configs Build agents Jobs Jenkins-based Automation System 15 

@oleg_nenashev, #jenkinsci, #orconf2016 October, 2016 • There is not so much plugins for EDA tools • BUT: we can use Script build steps (sh, batch, etc.) • Issues: • We need dozens of EDA tools • We also need SW development tools (compilers, etc.) • Sometimes we need multiple versions of tools Working with tools in Jenkins 16 

@oleg_nenashev, #jenkinsci, #orconf2016 October, 2016 1. Local tool installation on a node 2. External Tool Managers 3. Virtualization 4. Jenkins-based solutions • Via plugins (if available) • Generic approaches: Environment setup, Custom Tools Tool usage approaches 17 

@oleg_nenashev, #jenkinsci, #orconf2016 October, 2016 • Advantages: • Straightforward • Can attach existing preconfigured machines • Disadvantages: • Not scalable • Difficult administration • Risk of the INFRA consistency loss • Build agent environment pollution by tools Local tool installation & Tool Managers 18 

@oleg_nenashev, #jenkinsci, #orconf2016 October, 2016 • Advantages: • Quick setup if you have ready infrastructure • Jenkins provides plugins • AWS, EC2, vSphere, Mesos, Docker, … • Sharing physical hardware: vCenter, Docker Swarm • Works well with configuration as code (e.g. Puppet) • Always-clean environment • Disadvantages: • Poor scalability when you have many tool combinations • Complicated for BIG tools (common case for EDAs) Virtualization of nodes and tools 19 

@oleg_nenashev, #jenkinsci, #orconf2016 October, 2016 • Created for LibreCores CI • https://github.com/oleg- nenashev/fusesoc-docker (To Be Moved) • Publicly available on DockerHub: • https://hub.docker.com/r/onenashev/fus esoc-icarus-quartus/ • Usage: • docker run onenashev/fusesoc- icarus-quartus fusesoc sim wb_sdram_ctrl • Jenkins can automagically create nodes for Docker images Example: “FuseSoC-Icarus-Quartus” image 20 

@oleg_nenashev, #jenkinsci, #orconf2016 October, 2016 Jenkins-based solutions •Specific plugins: • Many plugins for SW development tools • No so many plugins for EDA tools •Exceptions: Questa VRM, Cadence vManager •Useful generic plugins: • Custom Tools Plugin • EnvInject Plugin • Tool Environment Plugin • … 21 

@oleg_nenashev, #jenkinsci, #orconf2016 October, 2016 •Use virtualization when it is possible •Otherwise - features provided by Jenkins •Special Plugins •Custom Tools Plugin •Backend infrastructure •Use local tool installations as a a last resort •Drivers for nodes with FPGAs •… Recommendations 22 

@oleg_nenashev, #jenkinsci, #orconf2016 October, 2016 Running tests in Jenkins SW HW Simulators EDA tools FPGAs ASIC prototypes Hardware Accelerators. x 23 

@oleg_nenashev, #jenkinsci, #orconf2016 October, 2016 • PROS • Good diagnosability and reproducibility • No need for expensive hardware peripherals • Single-shot VMs => no environment pollution • Tests can be parallelized on computing grids • CONS • Models may contain their own bugs • Real-time execution is commonly non-achievable • Long execution time (cycle accurate & hardware simulations) • Slow spot-checks (on-commit changes, pull request builds) • Need in computing grids for massive runs (random tests) Simulators 24 

@oleg_nenashev, #jenkinsci, #orconf2016 October, 2016 Computing Grids. Usage in Jenkins Manage tasks from CLI (script steps) “Transparent” – launch Jenkins nodes on the grid • Use-case: Legacy scripts • Difficult management (job abort, status polling, etc.) • Manual launch • “Start by script on Master” • LSF and SGE Cloud Plugins • No FOSS plugins for other grids L “Launch task” build & pipeline steps • No available open-source plugins L • Script steps instead 25 

@oleg_nenashev, #jenkinsci, #orconf2016 October, 2016 Simulators “We start up Linux in 6 hours.” In Cloud: • $ per test run • $$ for daily runs • $$$ for PR builders 26 Let’s try hardware? 

@oleg_nenashev, #jenkinsci, #orconf2016 October, 2016 We want to… • minimize the amount of hardware we use • utilize the hardware as much as possible • share hardware between projects Nice to have: • reuse the existing hardware •Developer computers (nightly builds) •Manual QA labs •… Issue #1. Hardware is Expensive! M – x000$ L –x0000$ M - http://www.xilinx.com/univ/xupv5-lx110t.htm L - HAPS-64, www.synopsys.com 27 

@oleg_nenashev, #jenkinsci, #orconf2016 October, 2016 Power Reset when it hangs Deploy bit-files, Blasting cables Debugging cables, firmware upload We don’t change them, huh? Jenkins node Drivers Tools Peripherals Attached peripherals Runtime Example. Jenkins Node with FPGA VM or bare- metal 28 

@oleg_nenashev, #jenkinsci, #orconf2016 October, 2016 Mapping Infra to Existing Hardware Nodes @ Site A Nodes @ Site C Nodes @ Site B “We borrowed FPGAs here, somebody takes them randomly.” “This stuff has Windows drivers only.” “No infra team members here.” “We run Jenkins nodes on developer laptops.” Frankenboards “We built them from the garbage we had.” L L L L L 29 

@oleg_nenashev, #jenkinsci, #orconf2016 October, 2016 •Any run may fail due to infrastructure issues •Test rerun should be automatic • Partian solution - Naginator plugin • Helps with board/network failures • Master failure is fatal •Hardware tests failures are barely reproducible • Need to re-confirm failures with simulations • Two-level test run •Clean environment issue Issue #2. Hardware is Unstable 30 

@oleg_nenashev, #jenkinsci, #orconf2016 October, 2016 • Access should be dispatched by Jenkins • Builds utilize hardware during a single phase only • One job should be running on multiple nodes • Scheduled availability of nodes • OOTB feature in the Jenkins core • Teams fight for resources • Job prioritization (e.g. Pipeline sorter) • Clean environment (again) • Workspace Cleanup Plugin Issue #3. Hardware is Shared 31 

@oleg_nenashev, #jenkinsci, #orconf2016 October, 2016 Classic Approaches JUC2015/London More info: Keywords: • Script steps to run the logic • Custom Tools plugin to manage the tools • Throttle Concurrent Builds and Lockable Resources to dispatch the access to the hardware • Naginator plugin to restart the builds on-demand • Parameterized Trigger / Copy Artifact plugins to parallelize tests and to collect results Approach: Use Free-style and Matrix: https://jenkins.io/solutions/embedded/ 32 

@oleg_nenashev, #jenkinsci, #orconf2016 October, 2016 Classic Approaches Spaghetti Automation • Many job dependencies • Difficult build flow tracking • Complex configurations • Duplication • Maintenance is expensive 33 

@oleg_nenashev, #jenkinsci, #orconf2016 October, 2016 Need a Better Solution? Jenkins 2 •NEW WEBSITE •NOW – CI AUTOMATION SERVER •FOCUS ON USER EXPERIENCE •UI IMPROVEMENTS •SECURITY IS ENABLED BY DEFAULT •PIPELINE AS CODE http://bit.ly/jenkins20 Stats: • Released on April 2016 • LTS line – July 2016 • 50% adoption over six months 34 

@oleg_nenashev, #jenkinsci, #orconf2016 October, 2016 Ø Groovy DSL for automatic flow description • Alternatives – Job DSL, Jenkins Job Builder Ø Job type in Jenkins (Pipeline Job) • Job definitions can be stored in SCM • Automatic job creation from repositories and organizations Ø Ecosystem for automation as code • ~20 dedicated plugins • Integration in many other plugins • Suggested by default in Jenkins 2 Jenkins Pipeline is a… // Run on a node having the “my_fpga” label node("linux && ml509") { git url:"http://github.com/myorg/myproject.git" sh "make all" } http://bit.ly/pipeline-tutorial 35 

@oleg_nenashev, #jenkinsci, #orconf2016 October, 2016 • Robustness against restarts of Jenkins master • Robustness against network disconnects • sh() steps are based on the Durable Task plugin • Jenkins continues execution once the node reconnects • Parallelization across multiple nodes: • It can be achieved via combination of parallel() and node() steps • No need to use the trigger/copy chain (but you can do it) • Ability to store the shared logic in Pipeline libraries • Visualization: Pipeline Stage View, Blue Ocean Pipeline. Useful Features 36 

@oleg_nenashev, #jenkinsci, #orconf2016 October, 2016 Pipeline Stage View Plugin New UI: Blue Ocean is coming soon! 37 

Example 2: Running tests in Parallel 38 

@oleg_nenashev, #jenkinsci, #orconf2016 October, 2016 Resource Management •Dispatching parallel access: • Lockable Resources plugin is integrated with Pipeline • Throttle Concurrent Builds plugin is not integrated, license limitations may be a problem •Tool Management: • Environment management via withEnv() step • Custom Tools Plugin is partially integrated with Pipeline • Restricting access: • Job Restrictions Plugin (for nodes) • Access schedule in node configs echo 'Starting' node('board1') { stage('Build') git url: 'http://github.com/myorg/myprj.git' withEnv(["PATH+GCC=${tool ’gcc4'}/bin"]) { sh 'make clean build' } } stage(’Run tests') lock(env.NODE_NAME + "-board1") { // Run something on the board // ... } } 39 

@oleg_nenashev, #jenkinsci, #orconf2016 October, 2016 Node Failover Pipeline allows re-running unreliable stages on other nodes. System failure Node #1 Node #2 . . . . . . for (def board : boards) { echo ”Try board " + board; try { node(board) { checkout scm sh ‘./bin/run.sh’ // Call passed => DONE break; } } catch (Exception ex) { if (ex.message.contains ("exit code 255")) { // Fatal error fail("Test run failed") } } } Try all possible boards: 40 

@oleg_nenashev, #jenkinsci, #orconf2016 October, 2016 Example: Pipeline with FuseSoC (LibreCores CI) 41 

@oleg_nenashev, #jenkinsci, #orconf2016 October, 2016 Summary. Hardware projects & Jenkins 1. Jenkins can be effectively used to run automation in hardware projects 2. Jenkins lacks some EDA integration plugins, but it’s not a blocker 3. It is possible to address common use- cases in the area 4. Jenkins Pipeline is helpful for test runs on hardware. It’s easy! 42 

@oleg_nenashev, #jenkinsci, #orconf2016 October, 2016 • Many articles • 90% - about “classic approaches” based on Freestyle jobs • Jenkins website aggregates articles • Embedded solutions page: • https://jenkins.io/solutions/embedded/ • TODO: Hardware solutions page: • https://jenkins.io/solutions/hardware/ • To be created soon • Blogposts: • https://jenkins.io/node/ More Info Embedded Solutions Page … 43 

@oleg_nenashev, #jenkinsci, #orconf2016 October, 2016 • Jenkins project: • Website: https://jenkins.io • Embedded Solutions page: https://jenkins.io/solutions/embedded/ • Hardware Solutions page (TODO): https://jenkins.io/solutions/hardware/ • Pipeline: • Tutorial: https://github.com/jenkinsci/workflow-plugin/blob/master/TUTORIAL.md • Compatibility: https://github.com/jenkinsci/pipeline- plugin/blob/master/COMPATIBILITY.md • Examples: https://github.com/jenkinsci/pipeline-examples • LibreCores: • Website: http://librecores.org/ • CI instance (work in progress): https://ci.librecores.org/ Useful Links 44 

@oleg_nenashev, #jenkinsci, #orconf2016 October, 2016 Q & A Contacts: • E-mail: [email protected] • Gitter: /oleg-nenashev • Twitter: @oleg_nenashev 45