Managing Embedded System test infrastructures with Docker and Jenkins

Virtualization and Containerization are one of the hottest areas in the test automation of software systems.
Such approaches simplify management of test environments, reduce costs and improve reproducibility of tests.
Docker is one of the popular container platforms in the area.

Are such approaches applicable to the Embedded Systems area?
Container technologies have limitations, especially when it comes to tests involving hardware-in-the-loop (HWIL) in the loop or drivers.
In test automation flows it is important to know Docker limitations to properly combine it with other approaches.

This talk focuses on Docker usage specifics in simulation-based and HWIL test automation with Docker and automation servers like Jenkins.
The talk includes overview of Docker advantages and disadvantages, and provides examples from real-world projects.
Main takeaways from the talk are the recommendations when and how to use Docker in Embedded Systems testing.
And how to migrate the automation flows to such new approaches.

Transcript

1. Managing Embedded Systems test infrastructures with Docker and Jenkins Oleg Nenashev, CloudBees CloudBees | Jenkins Automotive and Embedded Day Böblingen, April 11, 2018
2. © 2017 CloudBees, Inc. All Rights Reserved. 2 About me @oleg_nenashev oleg-nenashev Contributor, LibreCores PhD, St. Petersburg Polytechnic University Core Team, Jenkins Engineering, Architecture & Platform Reliability
3. © 2017 CloudBees, Inc. All Rights Reserved. 3 Previous talks
4. © 2017 CloudBees, Inc. All Rights Reserved. 4 Today…
5. © 2017 CloudBees, Inc. All Rights Reserved. 5 About you • Do you use virtual machines?
6. © 2017 CloudBees, Inc. All Rights Reserved. 6 About you • Do you use virtual machines? • Do you use containers?
7. © 2017 CloudBees, Inc. All Rights Reserved. 7 About you • Do you use virtual machines? • Do you use containers? • Do you use Docker?
8. © 2017 CloudBees, Inc. All Rights Reserved. 8 • Containers in Embedded. Why? • Docker and Jenkins • When to use them? • Examples • How to migrate? Today’s agenda
9. © 2017 CloudBees, Inc. All Rights Reserved. 9 Why containers? Configuration Management
10. © 2017 CloudBees, Inc. All Rights Reserved. 10 Why containers? Configuration Management Fast provisioning
11. © 2017 CloudBees, Inc. All Rights Reserved. 11 Why containers? Configuration Management Fast provisioning Clean environments
12. © 2017 CloudBees, Inc. All Rights Reserved. 12 Why containers? Configuration Management Fast provisioning Clean environments Disposability
13. © 2017 CloudBees, Inc. All Rights Reserved. 13 Why containers? Configuration Management Fast provisioning Clean environments Disposability Clustering
14. © 2017 CloudBees, Inc. All Rights Reserved. 14 Docker • Production proven container engine • Developer-friendly • Linux, Windows, Mac + amd64/ARM • Huge ecosystem https://www.docker.com/
15. © 2017 CloudBees, Inc. All Rights Reserved. 15 http://slidedeck.io/dpdornseifer/reveal_docker
16. © 2017 CloudBees, Inc. All Rights Reserved. 16 Docker • Production proven container engine • Developer-friendly • Linux, Windows, Mac + amd64/ARM • Huge ecosystem • Public/private image repos (DockerHub) • Multi-container apps - Docker Compose • Clustering - K8s, Docker Swarm, etc. https://www.docker.com/
17. © 2017 CloudBees, Inc. All Rights Reserved. 17 Docker in Embedded?
18. © 2017 CloudBees, Inc. All Rights Reserved. 18 Docker in Embedded Prototypes, Dev Tools Docker Swarm Cluster on Raspberry Pi
19. © 2017 CloudBees, Inc. All Rights Reserved. 19 Docker in Embedded Prototypes, Dev Tools Test Infrastructures, CI/CD Docker Swarm Cluster on Raspberry Pi
20. © 2017 CloudBees, Inc. All Rights Reserved. 20 1. It’s popular, big community Why Jenkins?
21. © 2017 CloudBees, Inc. All Rights Reserved. 21 1. It’s popular, big community 2. It’s an automation framework Jenkins in Embedded. Why?
22. © 2017 CloudBees, Inc. All Rights Reserved. 22 1. It’s popular, big community 2. It’s an automation framework 3. Pipeline as Code • Scripted/Declarative, Blue Ocean Jenkins in Embedded. Why? // Run on a node having the “my_fpga” label node("linux && ml509") { git url:"http://github.com/myorg/myproject.git" sh "make all" } http://bit.ly/pipeline-tutorial
23. © 2017 CloudBees, Inc. All Rights Reserved. 23 1. It’s popular, big community 2. It’s an automation framework 3. Pipeline as Code 4. Many users in the embedded area Jenkins in Embedded. Why?
24. © 2017 CloudBees, Inc. All Rights Reserved. 24 1. It’s popular, big community 2. It’s an automation framework 3. Pipeline as Code 4. Many users in the embedded area 5. Commercial support is available • CloudBees: go.cloudbees.com Jenkins in Embedded. Why?
25. © 2017 CloudBees, Inc. All Rights Reserved. 25 Jenkins & Docker. Packaging https://hub.docker.com/r/jenkins/ AND https://hub.docker.com/r/jenkinsci
26. © 2017 CloudBees, Inc. All Rights Reserved. 26 Jenkins & Docker. Plugins • Docker Plugin • Docker Pipeline Plugin • Docker Custom Build Environment Plugin • … • Yet Another Docker Plugin • Kubernetes Plugin • Kubernetes Pipeline Plugin • …
27. © 2017 CloudBees, Inc. All Rights Reserved. 27 Example: Docker Pipeline Plugin https://plugins.jenkins.io/docker-workflow docker.image('onenashev/gcc-riscv:6.4-rc').inside { checkout scm sh "make clean test" }
28. So, how do we use containers?
29. © 2017 CloudBees, Inc. All Rights Reserved. 29 Embedded Components Testing Approaches Software Hardware Simulators EDA tools FPGAs ASIC prototypes Hardware Accelerators, etc. x
30. © 2017 CloudBees, Inc. All Rights Reserved. 30 PROS •Good diagnosability and reproducibility •No need for expensive peripherals •Single-shot containers •Container-friendly Simulators
31. © 2017 CloudBees, Inc. All Rights Reserved. 31 PROS •Good diagnosability and reproducibility •No need for expensive peripherals •Single-shot containers •Container-friendly CONS •Real-time execution is commonly non-achievable •Long execution time (esp. time-accurate simulations) •Models may contain their own defects Simulators “We run Hello World in two hours.”
32. © 2017 CloudBees, Inc. All Rights Reserved. 32 Docker for Simulations • Minimize using physical hardware when possible • You get: low overhead and startup time
33. © 2017 CloudBees, Inc. All Rights Reserved. 33 Docker for Simulations • Minimize using physical hardware when possible • You get: Low overhead and startup time • You get: Resource sharing and quota management •CPU/RAM •Licenses •Priorities
34. © 2017 CloudBees, Inc. All Rights Reserved. 34 Example: Kubernetes Plugin def label = "gcc-riscv-${UUID.randomUUID().toString()}" podTemplate(name: 'test', label: label, containers: [ containerTemplate(name: 'gcc-riscv', image: 'onenashev/gcc-riscv:6.4-rc’, resourceRequestCpu: ‘2’, resourceLimitMemory: ‘8Gi' ttyEnabled: true, command: 'cat')]) { node(label) { checkout scm sh "make clean test" step([$class: 'TapPublisher', testResults: 'output/test/report.tap', ...]) } } https://plugins.jenkins.io/kubernetes
35. © 2017 CloudBees, Inc. All Rights Reserved. 35 Docker for Simulations • Minimize using physical hardware when possible • You get: low overhead and startup time • You get: Resource sharing and quota management • You get: Multi-container tests •Kubernetes PODs, Docker Compose
36. © 2017 CloudBees, Inc. All Rights Reserved. 36 OK, and what’s about Hardware?
37. © 2017 CloudBees, Inc. All Rights Reserved. 37 Testing with hardware VM, container or bare-metal Power Reset when it hangs Deploy bit-files, Blasting cables Debugging cables, firmware upload We don’t change them, huh? Jenkins node Drivers Tools Peripherals Attached peripherals Runtime
38. © 2017 CloudBees, Inc. All Rights Reserved. 38 Hardware tests with containers? • “It works” • “You can run containers in privileged mode” • “You can use drivers from host OS” They say
39. © 2017 CloudBees, Inc. All Rights Reserved. 39 Hardware tests with containers? • It works on a success path • BUT: software errors, timeouts, access conflicts, device hanging when switching containers They say Reality • “It works” • “You can run containers in privileged mode” • “You can use drivers from host OS”
40. © 2017 CloudBees, Inc. All Rights Reserved. 40 Hardware tests with containers? • It works on a success path • BUT: software errors, timeouts, access conflicts, device hanging when switching containers • BUT: you still need special test equipment and drivers They say Reality • “It works” • “You can run containers in privileged mode” • “You can use drivers from host OS”
41. © 2017 CloudBees, Inc. All Rights Reserved. 41 Hardware tests with containers? • It works on a success path • BUT: software errors, timeouts, access conflicts, device hanging when switching containers • BUT: you still need special test equipment and drivers • BUT: it’s still not flexible They say Reality • “It works” • “You can run containers in privileged mode” • “You can use drivers from host OS”
42. © 2017 CloudBees, Inc. All Rights Reserved. 42 Hardware tests with containers? • It works on a success path • BUT: software errors, timeouts, access conflicts, device hanging when switching containers • BUT: you still need special test equipment and drivers • BUT: it’s still not flexible They say Reality => Not always helpful for testing on hardware • “It works” • “You can run containers in privileged mode” • “You can use drivers from host OS”
43. Example. LibreCores CI • LibreCores – hub for open-source hardware projects • https://www.librecores.org/ • Work in Progress: CI as a service
44. © 2017 CloudBees, Inc. All Rights Reserved. 44 Example. LibreCores CI • LibreCores – hub for FOSS HW projects • Work in Progress: CI as a service for projects • Docker as an engine • Powered by Jenkins and Jenkins Pipeline Jenkins Docker (Swarm) Other Integrated Components Physical nodes FPGAs and other HW peripherals LibreCores CI Infrastructure Project-specific Infrastructure On-demand build agents SSH, JNLP over VPN Instance in Docker: https://github.com/librecores/librecores-ci
45. © 2017 CloudBees, Inc. All Rights Reserved. 45 Example. LibreCores CI We provide a framework to users: incapsulation, etc. Pipeline libraries + minimal Jenkinsfile @Library(“fusesoc”) import org.librecores.fusesoc.FuseSoCBuilder; def builder = new FuseSoCBuilder() builder.defaultCoreCIBuild(this, repo: 'wb_sdram_ctrl', organization: 'oleg-nenashev', nodeLabels: 'docker-fusesoc-icarus' extraArgs: '--transactions 10’) Example: Run build and test https://github.com/librecores/librecores-pipeline-lib
46. © 2017 CloudBees, Inc. All Rights Reserved. 46 Example. LibreCores CI We provide a framework to users: incapsulation, etc. Pipeline libraries + minimal Jenkinsfile Standard containers, bundled tools @Library(“fusesoc”) import org.librecores.fusesoc.FuseSoCBuilder; def builder = new FuseSoCBuilder() builder.defaultCoreCIBuild(this, repo: 'wb_sdram_ctrl', organization: 'oleg-nenashev', nodeLabels: 'docker-fusesoc-icarus' extraArgs: '--transactions 10’) Example: Run build and test https://github.com/librecores/librecores-pipeline-lib
47. © 2017 CloudBees, Inc. All Rights Reserved. 47 Example. LibreCores CI We provide a framework to users: incapsulation, etc. Pipeline libraries + minimal Jenkinsfile Standard containers, bundled tools Yet Another Docker Plugin @Library(‘lib‘) import org.librecores.ci.Modules def lcci = new Modules(steps) node('librecores-ci-modules') { lcci.load(["eda/verilator/3.902"]) stage('Simulation Build') { lcci.sh 'verilator -f verilate.vc' } } Example: Use container-external tools https://github.com/librecores/librecores-pipeline-lib
48. © 2017 CloudBees, Inc. All Rights Reserved. 48 Example. LibreCores CI We provide a framework to users: incapsulation, etc. Pipeline libraries + minimal Jenkinsfile Standard containers, bundled tools Yet Another Docker Plugin Modules Tool Environment @Library(‘lib‘) import org.librecores.ci.Modules def lcci = new Modules(steps) node('librecores-ci-modules') { lcci.load(["eda/verilator/3.902"]) stage('Simulation Build') { lcci.sh 'verilator -f verilate.vc' } } Example: Use container-external tools https://github.com/librecores/librecores-pipeline-lib
49. So, how to migrate?
50. © 2017 CloudBees, Inc. All Rights Reserved. 50 Abstract Project Build modules Run spot-ch ecks Test on simu-la tor Tests on hard-w are Static Analysis • Jenkins Pipeline or Freestyle • Migrate to Pipeline at same time if needed
51. So, how to migrate? 1. Virtualize 2. Use single-shot containers 3. Split to isolated stages 4. Parallelize
52. © 2017 CloudBees, Inc. All Rights Reserved. 52 Step 1. Virtualize • Agent is provisioned once for the system • Agent can be suspended when not needed so save resources • Environment cleanup should be done by runs Tool 1 Tool 2 Tool 3 Multiple CPUs, much RAM Build 2 Build 1 Your CI tool
53. © 2017 CloudBees, Inc. All Rights Reserved. 53 Step 2. Use single-shot containers • Big blob containers are not effective • Lightweight containers are fast • We do not need to stick the entire build to a single machine Tool 1 Tool 2 Tool 3 Multiple CPUs, much RAM Build Agent Your CI tool
54. © 2017 CloudBees, Inc. All Rights Reserved. 54 Step 3. Split to stages Build module 1 Build module 2 Run spot-ch ecks Test on simu-la tor Tests on hard-w are Image 1, Tool 1 Image 1, Tool 1 Image 2, Tools 1&2 Image 3, Tool 3 Host with FPGA Your CI tool • Lightweight images • No combinatorial explosion
55. © 2017 CloudBees, Inc. All Rights Reserved. 55 Step 4. Parallelize Build module 1 Run spot-checks Simulator tests #1 Tests on hardware Host with FPGA Simulator tests #2 Simulator tests #3 Build module 2 Your CI tool • Better resource utilization • Quick response (CI) • Simplified failover Static analysis Security scans
56. © 2017 CloudBees, Inc. All Rights Reserved. 56 Lessons Learnt. Docker Containers • Docker simplifies test infra maintenance • Docker images should be small • Packaging of multiple tools is not effective • Test flows should be staged •Workspaces can be shared between stages, e.g. External Workspace Manager plugin • One Docker host is not enough, cluster it with Kubernetes & Co…
57. © 2017 CloudBees, Inc. All Rights Reserved. 57 Takeaways • Use: Test environment for simulations •Clean environments •Parallelization •Resource sharing
58. © 2017 CloudBees, Inc. All Rights Reserved. 58 Takeaways • Use: Test environment for simulations •Clean environments •Parallelization •Resource sharing • Proceed with caution: on-platform and hardware-in-the-loop testing •Connection of hardware
59. © 2017 CloudBees, Inc. All Rights Reserved. 59 Takeaways • Use: Test environment for simulations •Clean environments •Parallelization •Resource sharing • Proceed with caution: on-platform and hardware-in-the-loop testing •Connection of hardware • Recommended: Step-by-step migration
60. © 2017 CloudBees, Inc. All Rights Reserved. 60 • Docker •Website: https://www.docker.com/ • Jenkins project: •Website: https://jenkins.io •Embedded Solutions page: https://jenkins.io/solutions/embedded/ • CloudBees website: •Website: https://www.cloudbees.com •Information portal: https://go.cloudbees.com/ •These slides: https://pages.cloudbees.com/ ECS2017/Oleg_Nenashev Useful Links Embedded Solutions Page
61. © 2017 CloudBees, Inc. All Rights Reserved. 61 Contacts: E-mail: onenashev@cloudbees.com GitHub: oleg-nenashev Twitter: @oleg_nenashev https://go.cloudbees.com/
62. THANK YOU! https://go.cloudbees.com/