ORCONF 2017. State of LibreCores CI

At ORCONF 2016 we announced the work on LibreCores CI - a continuous integration service for open-source Hardware projects. During the last year we made some progress on this front, and this presentation reflects the current progress on infrastructure and development tools.

Transcript

1. #librecores, #ORConf Sep 09, 2017 LibreCores Free and Open Digital Hardware State of LibreCores CI Oleg Nenashev Stefan Wallentowitz 2017
2. #librecores, #ORConf Sep 09, 2017 About us 2 @oleg_nenashev @wallento • FOSSi, Director • OpTiMSoC • Jenkins Project • CloudBees • LibreCores
3. #librecores, #ORConf Sep 09, 2017 Agenda • LibreCores CI • State of the infrastructure* • Example: OpTiMSoC • Pitfalls we hit • Current state & future work • Demo Disclaimer: the project is in the prototyping stage 3 Oleg Stefan Oleg
4. #librecores, #ORConf Sep 09, 2017 What is LibreCores CI? Continuous Integration service for open-source HW projects 4 LibreCores Web
5. #librecores, #ORConf Sep 09, 2017 LibreCore pre-history Started in 2016 Announced at ORCONF2016 5 http://bit.ly/orconf2016-librecores-ci
6. #librecores, #ORConf Sep 09, 2017 Vision 2016 LibreCores CI GitHub SSO User management LibreCores Web Top-level Web UI Project repositories GitHub BitBucket SVN GitHub is a main target system 6 CI Status Gitlab
7. #librecores, #ORConf Sep 09, 2017 Vision 2016. Infrastructure Jenkins Docker (Swarm) Other Integrated Components Physical nodes FPGAs and other HW peripherals LibreCores CI Infrastructure Project-specific Infrastructure On-demand build agents SSH, JNLP over VPN 7
8. #librecores, #ORConf Sep 09, 2017 •Dozens of build machines •Computing grids for simulations •FPGA grids for hardware testing •Licenses for all EDA tools LibreCores does NOT* provide… *According to the current plan/resources 8
9. #librecores, #ORConf Sep 09, 2017 •Dozens of build machines •Computing grids for simulations •FPGA grids for hardware testing •Licenses for all EDA tools LibreCores does NOT* provide… 9 BRING YOUR OWN INFRASTRUCTURE! *According to the current plan/resources
10. #librecores, #ORConf Sep 09, 2017 Librecores Components Master • PoC master: ci.librecores.org • New reusable Docker image Agents 10 • Build images with packaged tools • Examples: • Librecores CI • Librecores CI with Modules • FuseSoC + Icarus • FuseSoC + Icarus + Quartus • FuseSoC + RISCV toolchain • To be replaced by Modules Tools • LCCI CLI • Docker images for Building Tools • Shared Pipeline libraries
11. #librecores, #ORConf Sep 09, 2017 Example. FuseSoC Core 11 #Jenkinsfile import org.librecores.fusesoc.FuseSoCBuilder; OrgFuseSoCBuilder.defaultCoreCIBuild(this, repo: 'wb_sdram_ctrl', organization: 'oleg-nenashev', nodeLabels: 'docker-fusesoc-icarus' extraArgs: '--transactions 10’) https://github.com/oleg-nenashev/wb_sdram_ctrl Build Definition:
12. #librecores, #ORConf Sep 09, 2017 Usage of LibreCores CI As a service Deploy on your own Use tools • EDA tool Images • Shared Libraries for Jenkins Pipeline • Docker packages • Configuration scripts • ci.librecores.org • State: PoC It is not just a service! 12
13. #librecores, #ORConf Sep 09, 2017 •Modules •LCCI Tools •OpTiMSoC Example Under the hood. Tool Infrastructure 13 LibreCores Free and Open Digital Hardware
14. #librecores, #ORConf Sep 09, 2017 Module-based Environment • Problem: Diverse tool requirements • Many different tools are needed • Different versions of the same tool may be used by different projects • Environment Modules: Set environment • Configured by „modulefiles“ • http://modules.sourceforge.net/ $> module load verilator/3.900 PATH=/tools/verilator/3.900/bin/:… VERILATOR_ROOT=/tools/verilator/3.900 $> module load verilator/3.902 PATH=/tools/verilator/3.902/bin/:… VERILATOR_ROOT=/tools/verilator/3.902 14
15. #librecores, #ORConf Sep 09, 2017 Module-based LibreCores CI • Library in Jenkinsfile • Load required tools and versions • Execute shell commands in environment • https://github.com/libre cores/librecores- pipeline-lib @Library('librecoresci') import org.librecores.ci.Modules def lcci = new Modules(steps) node('librecores-ci-modules') { lcci.load(["eda/verilator/3.902"]) stage('Simulation Build') { lcci.sh 'verilator -f verilate.vc' } } 15
16. #librecores, #ORConf Sep 09, 2017 Module-based Tool Management • Standard Docker images for LibreCores CI • Do not include tools in docker images (only basic) • Mount /tools volume from host • Tools volume • Tools in different versions • Module infrastructure • LibreCores CI provides simple Docker images to install tools • Re-producible installations • In the same OS environment (Ubuntu 16.04) as agents 16
17. #librecores, #ORConf Sep 09, 2017 LCCI Command Line • Quick tool installation with simple command line tool • https://github.com/librecores/lcci-python • Example: pip install lcci • Simple YML configuration file • Simply install a tool • Example: lcci tools install verilator:3.902 • Install a whole „standard tool package“ • Example: lcci tools install lcci-2017.1 main: github-api-token: <TOKEN> volumes: tools: /home/lcci-tools 17
18. #librecores, #ORConf Sep 09, 2017 LCCI CLI. (FPGA) Agents • Command: „lcci start agent fpga-nexys4ddr“ • Add agents to lcci.yaml • Each agent is registered with LCCI • Agents run the docker image • The tools volume is mapped • Connect to jenkins and authenticate • Map extra devices (for FPGA agents) agents: fpga-nexys4ddr: type: modules agent-id: <LCCI-AGENT-ID> agent-secret: <LCCI-AGENT-SECRET> devices: - /dev/bus/usb/001/003 - /dev/ttyUSB1 18
19. #librecores, #ORConf Sep 09, 2017 Example: OpTiMSoC @Library('librecoresci') import org.librecores.ci.Modules def lcci = new Modules(steps) node('fpga-nexys4ddr') { lcci.load(["eda/xilinx/vivado/2016.4", "eda/fusesoc/1.6.1", "eda/verilator/3.902", "compilers/openrisc/gcc/5.3.0-newlib2.3.0-1", "base/python/3.5"]) stage("Checkout") { checkout scm } stage('Simulation Build') { lcci.sh 'make BUILD_DOCS=no BUILD_EXAMPLES_FPGA=no' } 19 https://github.com/optimsoc/optimsoc/blob/master/Jenkinsfile
20. #librecores, #ORConf Sep 09, 2017 Example: OpTiMSoC stage('Simulation Tests') { lcci.sh 'source objdir/dist/optimsoc-environment.sh; pytest -s -v test/systemtest/test_tutorial.py::TestTutorial' } stage('FPGA Build') { lcci.sh 'make BUILD_DOCS=no' } stage('FPGA Tests') { sh "echo 'boards:' > localconf.yaml" sh "echo ' nexys4ddr:' >> localconf.yaml" sh "echo ' device: /dev/ttyUSB1' >> localconf.yaml" lcci.sh 'source objdir/dist/optimsoc-environment.sh; export OPTIMSOC_TEST_LOCALCONF=$PWD/localconf.yaml; cat $OPTIMSOC_TEST_LOCALCONF; pytest -s -v test/systemtest/test_tutorial.py::TestTutorialFpga' } } 20 https://github.com/optimsoc/optimsoc/blob/master/Jenkinsfile
21. #librecores, #ORConf Sep 09, 2017 •[Demo] •Current state •Future plans What’s next? 21 LibreCores Free and Open Digital Hardware
22. #librecores, #ORConf Sep 09, 2017 Plans - 2016 1. Setup integration with LibreCores Web POC 2. Build definition libraries for FuseSoC YES 3. Setup flows for projects • OpTiMSoC • Other… SOME 4. Start defining usage conditions for projects TODO 5. Ensure the infrastructure can scale enough SOME 6. Start beta-testing TODO 22
23. #librecores, #ORConf Sep 09, 2017 Current state • Many plans, not so much time • Very first prototype • https://ci.librecores.org • To be replaced by the new version • Dockerized infrastructure • https://github.com/librecores/librecores-ci • Jenkins Master • Agents 23
24. #librecores, #ORConf Sep 09, 2017 https://hub.docker.com/r/ librecores/librecores-ci-dev/ 24
25. #librecores, #ORConf Sep 09, 2017 Future work 25 1. Deploy pending changes to https://ci.librecores.org 2. Finalize reference FPGA flows • OpTiMSoC 3. Start beta-testing • LibreCores CI Instance • Standalone setups (Docker) 4. Expand Tooling Support
26. #librecores, #ORConf Sep 09, 2017 Contributing • Try it • https://hub.docker.com/r/librecores/librecores-ci-dev/ • Discussions: • Mailing list: dev@lists.librecores.org • Gitter: https://gitter.im/librecores/librecores-ci • GitHub: • Code repository and task tracker • https://github.com/librecores/librecores-ci 26
27. #librecores, #ORConf Sep 09, 2017 Q & A Contacts: • Mailing list: dev@lists.librecores.org • Gitter: https://gitter.im/librecores/Lobby • Twitter: @oleg_nenashev, @wallento 27