Managing Embedded System test infrastructures with Docker and Jenkins

Embed

Start on current slide

Slide 1

Slide 1 text

Managing Embedded Systems test infrastructures with Docker and Jenkins Oleg Nenashev, CloudBees CloudBees | Jenkins Automotive and Embedded Day Böblingen, April 11, 2018

Slide 2

Slide 2 text

© 2017 CloudBees, Inc. All Rights Reserved. 2 About me @oleg_nenashev oleg-nenashev Contributor, LibreCores PhD, St. Petersburg Polytechnic University Core Team, Jenkins Engineering, Architecture & Platform Reliability

Slide 3

Slide 3 text

Slide 4

Slide 4 text

Slide 5

Slide 5 text

Slide 6

Slide 6 text

Slide 7

Slide 7 text

Slide 8

Slide 8 text

Slide 9

Slide 9 text

Slide 10

Slide 10 text

Slide 11

Slide 11 text

Slide 12

Slide 12 text

Slide 13

Slide 13 text

Slide 14

Slide 14 text

Slide 15

Slide 15 text

Slide 16

Slide 16 text

© 2017 CloudBees, Inc. All Rights Reserved. 16 Docker • Production proven container engine • Developer-friendly • Linux, Windows, Mac + amd64/ARM • Huge ecosystem • Public/private image repos (DockerHub) • Multi-container apps - Docker Compose • Clustering - K8s, Docker Swarm, etc. https://www.docker.com/

Slide 17

Slide 17 text

Slide 18

Slide 18 text

Slide 19

Slide 19 text

Slide 20

Slide 20 text

Slide 21

Slide 21 text

Slide 22

Slide 22 text

© 2017 CloudBees, Inc. All Rights Reserved. 22 1. It’s popular, big community 2. It’s an automation framework 3. Pipeline as Code • Scripted/Declarative, Blue Ocean Jenkins in Embedded. Why? // Run on a node having the “my_fpga” label node("linux && ml509") { git url:"http://github.com/myorg/myproject.git" sh "make all" } http://bit.ly/pipeline-tutorial

Slide 23

Slide 23 text

Slide 24

Slide 24 text

© 2017 CloudBees, Inc. All Rights Reserved. 24 1. It’s popular, big community 2. It’s an automation framework 3. Pipeline as Code 4. Many users in the embedded area 5. Commercial support is available • CloudBees: go.cloudbees.com Jenkins in Embedded. Why?

Slide 25

Slide 25 text

Slide 26

Slide 26 text

© 2017 CloudBees, Inc. All Rights Reserved. 26 Jenkins & Docker. Plugins • Docker Plugin • Docker Pipeline Plugin • Docker Custom Build Environment Plugin • … • Yet Another Docker Plugin • Kubernetes Plugin • Kubernetes Pipeline Plugin • …

Slide 27

Slide 27 text

© 2017 CloudBees, Inc. All Rights Reserved. 27 Example: Docker Pipeline Plugin https://plugins.jenkins.io/docker-workflow docker.image('onenashev/gcc-riscv:6.4-rc').inside { checkout scm sh "make clean test" }

Slide 28

Slide 28 text

So, how do we use containers?

Slide 29

Slide 29 text

Slide 30

Slide 30 text

Slide 31

Slide 31 text

© 2017 CloudBees, Inc. All Rights Reserved. 31 PROS •Good diagnosability and reproducibility •No need for expensive peripherals •Single-shot containers •Container-friendly CONS •Real-time execution is commonly non-achievable •Long execution time (esp. time-accurate simulations) •Models may contain their own defects Simulators “We run Hello World in two hours.”

Slide 32

Slide 32 text

Slide 33

Slide 33 text

© 2017 CloudBees, Inc. All Rights Reserved. 33 Docker for Simulations • Minimize using physical hardware when possible • You get: Low overhead and startup time • You get: Resource sharing and quota management •CPU/RAM •Licenses •Priorities

Slide 34

Slide 34 text

© 2017 CloudBees, Inc. All Rights Reserved. 34 Example: Kubernetes Plugin def label = "gcc-riscv-${UUID.randomUUID().toString()}" podTemplate(name: 'test', label: label, containers: [ containerTemplate(name: 'gcc-riscv', image: 'onenashev/gcc-riscv:6.4-rc’, resourceRequestCpu: ‘2’, resourceLimitMemory: ‘8Gi' ttyEnabled: true, command: 'cat')]) { node(label) { checkout scm sh "make clean test" step([$class: 'TapPublisher', testResults: 'output/test/report.tap', ...]) } } https://plugins.jenkins.io/kubernetes

Slide 35

Slide 35 text

© 2017 CloudBees, Inc. All Rights Reserved. 35 Docker for Simulations • Minimize using physical hardware when possible • You get: low overhead and startup time • You get: Resource sharing and quota management • You get: Multi-container tests •Kubernetes PODs, Docker Compose

Slide 36

Slide 36 text

Slide 37

Slide 37 text

© 2017 CloudBees, Inc. All Rights Reserved. 37 Testing with hardware VM, container or bare-metal Power Reset when it hangs Deploy bit-files, Blasting cables Debugging cables, firmware upload We don’t change them, huh? Jenkins node Drivers Tools Peripherals Attached peripherals Runtime

Slide 38

Slide 38 text

Slide 39

Slide 39 text

© 2017 CloudBees, Inc. All Rights Reserved. 39 Hardware tests with containers? • It works on a success path • BUT: software errors, timeouts, access conflicts, device hanging when switching containers They say Reality • “It works” • “You can run containers in privileged mode” • “You can use drivers from host OS”

Slide 40

Slide 40 text

© 2017 CloudBees, Inc. All Rights Reserved. 40 Hardware tests with containers? • It works on a success path • BUT: software errors, timeouts, access conflicts, device hanging when switching containers • BUT: you still need special test equipment and drivers They say Reality • “It works” • “You can run containers in privileged mode” • “You can use drivers from host OS”

Slide 41

Slide 41 text

© 2017 CloudBees, Inc. All Rights Reserved. 41 Hardware tests with containers? • It works on a success path • BUT: software errors, timeouts, access conflicts, device hanging when switching containers • BUT: you still need special test equipment and drivers • BUT: it’s still not flexible They say Reality • “It works” • “You can run containers in privileged mode” • “You can use drivers from host OS”

Slide 42

Slide 42 text

© 2017 CloudBees, Inc. All Rights Reserved. 42 Hardware tests with containers? • It works on a success path • BUT: software errors, timeouts, access conflicts, device hanging when switching containers • BUT: you still need special test equipment and drivers • BUT: it’s still not flexible They say Reality => Not always helpful for testing on hardware • “It works” • “You can run containers in privileged mode” • “You can use drivers from host OS”

Slide 43

Slide 43 text

Example. LibreCores CI • LibreCores – hub for open-source hardware projects • https://www.librecores.org/ • Work in Progress: CI as a service

Slide 44

Slide 44 text

© 2017 CloudBees, Inc. All Rights Reserved. 44 Example. LibreCores CI • LibreCores – hub for FOSS HW projects • Work in Progress: CI as a service for projects • Docker as an engine • Powered by Jenkins and Jenkins Pipeline Jenkins Docker (Swarm) Other Integrated Components Physical nodes FPGAs and other HW peripherals LibreCores CI Infrastructure Project-specific Infrastructure On-demand build agents SSH, JNLP over VPN Instance in Docker: https://github.com/librecores/librecores-ci

Slide 45

Slide 45 text

© 2017 CloudBees, Inc. All Rights Reserved. 45 Example. LibreCores CI We provide a framework to users: incapsulation, etc. Pipeline libraries + minimal Jenkinsfile @Library(“fusesoc”) import org.librecores.fusesoc.FuseSoCBuilder; def builder = new FuseSoCBuilder() builder.defaultCoreCIBuild(this, repo: 'wb_sdram_ctrl', organization: 'oleg-nenashev', nodeLabels: 'docker-fusesoc-icarus' extraArgs: '--transactions 10’) Example: Run build and test https://github.com/librecores/librecores-pipeline-lib

Slide 46

Slide 46 text

© 2017 CloudBees, Inc. All Rights Reserved. 46 Example. LibreCores CI We provide a framework to users: incapsulation, etc. Pipeline libraries + minimal Jenkinsfile Standard containers, bundled tools @Library(“fusesoc”) import org.librecores.fusesoc.FuseSoCBuilder; def builder = new FuseSoCBuilder() builder.defaultCoreCIBuild(this, repo: 'wb_sdram_ctrl', organization: 'oleg-nenashev', nodeLabels: 'docker-fusesoc-icarus' extraArgs: '--transactions 10’) Example: Run build and test https://github.com/librecores/librecores-pipeline-lib

Slide 47

Slide 47 text

© 2017 CloudBees, Inc. All Rights Reserved. 47 Example. LibreCores CI We provide a framework to users: incapsulation, etc. Pipeline libraries + minimal Jenkinsfile Standard containers, bundled tools Yet Another Docker Plugin @Library(‘lib‘) import org.librecores.ci.Modules def lcci = new Modules(steps) node('librecores-ci-modules') { lcci.load(["eda/verilator/3.902"]) stage('Simulation Build') { lcci.sh 'verilator -f verilate.vc' } } Example: Use container-external tools https://github.com/librecores/librecores-pipeline-lib

Slide 48

Slide 48 text

© 2017 CloudBees, Inc. All Rights Reserved. 48 Example. LibreCores CI We provide a framework to users: incapsulation, etc. Pipeline libraries + minimal Jenkinsfile Standard containers, bundled tools Yet Another Docker Plugin Modules Tool Environment @Library(‘lib‘) import org.librecores.ci.Modules def lcci = new Modules(steps) node('librecores-ci-modules') { lcci.load(["eda/verilator/3.902"]) stage('Simulation Build') { lcci.sh 'verilator -f verilate.vc' } } Example: Use container-external tools https://github.com/librecores/librecores-pipeline-lib

Slide 49

Slide 49 text

So, how to migrate?

Slide 50

Slide 50 text

© 2017 CloudBees, Inc. All Rights Reserved. 50 Abstract Project Build modules Run spot-ch ecks Test on simu-la tor Tests on hard-w are Static Analysis • Jenkins Pipeline or Freestyle • Migrate to Pipeline at same time if needed

Slide 51

Slide 51 text

So, how to migrate? 1. Virtualize 2. Use single-shot containers 3. Split to isolated stages 4. Parallelize

Slide 52

Slide 52 text

© 2017 CloudBees, Inc. All Rights Reserved. 52 Step 1. Virtualize • Agent is provisioned once for the system • Agent can be suspended when not needed so save resources • Environment cleanup should be done by runs Tool 1 Tool 2 Tool 3 Multiple CPUs, much RAM Build 2 Build 1 Your CI tool

Slide 53

Slide 53 text

© 2017 CloudBees, Inc. All Rights Reserved. 53 Step 2. Use single-shot containers • Big blob containers are not effective • Lightweight containers are fast • We do not need to stick the entire build to a single machine Tool 1 Tool 2 Tool 3 Multiple CPUs, much RAM Build Agent Your CI tool

Slide 54

Slide 54 text

© 2017 CloudBees, Inc. All Rights Reserved. 54 Step 3. Split to stages Build module 1 Build module 2 Run spot-ch ecks Test on simu-la tor Tests on hard-w are Image 1, Tool 1 Image 1, Tool 1 Image 2, Tools 1&2 Image 3, Tool 3 Host with FPGA Your CI tool • Lightweight images • No combinatorial explosion

Slide 55

Slide 55 text

© 2017 CloudBees, Inc. All Rights Reserved. 55 Step 4. Parallelize Build module 1 Run spot-checks Simulator tests #1 Tests on hardware Host with FPGA Simulator tests #2 Simulator tests #3 Build module 2 Your CI tool • Better resource utilization • Quick response (CI) • Simplified failover Static analysis Security scans

Slide 56

Slide 56 text

© 2017 CloudBees, Inc. All Rights Reserved. 56 Lessons Learnt. Docker Containers • Docker simplifies test infra maintenance • Docker images should be small • Packaging of multiple tools is not effective • Test flows should be staged •Workspaces can be shared between stages, e.g. External Workspace Manager plugin • One Docker host is not enough, cluster it with Kubernetes & Co…

Slide 57

Slide 57 text

Slide 58

Slide 58 text

© 2017 CloudBees, Inc. All Rights Reserved. 58 Takeaways • Use: Test environment for simulations •Clean environments •Parallelization •Resource sharing • Proceed with caution: on-platform and hardware-in-the-loop testing •Connection of hardware

Slide 59

Slide 59 text

© 2017 CloudBees, Inc. All Rights Reserved. 59 Takeaways • Use: Test environment for simulations •Clean environments •Parallelization •Resource sharing • Proceed with caution: on-platform and hardware-in-the-loop testing •Connection of hardware • Recommended: Step-by-step migration

Slide 60

Slide 60 text

© 2017 CloudBees, Inc. All Rights Reserved. 60 • Docker •Website: https://www.docker.com/ • Jenkins project: •Website: https://jenkins.io •Embedded Solutions page: https://jenkins.io/solutions/embedded/ • CloudBees website: •Website: https://www.cloudbees.com •Information portal: https://go.cloudbees.com/ •These slides: https://pages.cloudbees.com/ ECS2017/Oleg_Nenashev Useful Links Embedded Solutions Page