Upgrade to Pro — share decks privately, control downloads, hide ads and more …

ORCONF 2017. State of LibreCores CI

Oleg Nenashev
September 09, 2017

ORCONF 2017. State of LibreCores CI

At ORCONF 2016 we announced the work on LibreCores CI - a continuous integration service for open-source Hardware projects. During the last year we made some progress on this front, and this presentation reflects the current progress on infrastructure and development tools.

Oleg Nenashev

September 09, 2017
Tweet

More Decks by Oleg Nenashev

Other Decks in Technology

Transcript

  1. #librecores, #ORConf Sep 09, 2017 LibreCores Free and Open Digital

    Hardware State of LibreCores CI Oleg Nenashev Stefan Wallentowitz 2017
  2. #librecores, #ORConf Sep 09, 2017 About us 2 @oleg_nenashev @wallento

    • FOSSi, Director • OpTiMSoC • Jenkins Project • CloudBees • LibreCores
  3. #librecores, #ORConf Sep 09, 2017 Agenda • LibreCores CI •

    State of the infrastructure* • Example: OpTiMSoC • Pitfalls we hit • Current state & future work • Demo Disclaimer: the project is in the prototyping stage 3 Oleg Stefan Oleg
  4. #librecores, #ORConf Sep 09, 2017 What is LibreCores CI? Continuous

    Integration service for open-source HW projects 4 LibreCores Web
  5. #librecores, #ORConf Sep 09, 2017 LibreCore pre-history Started in 2016

    Announced at ORCONF2016 5 http://bit.ly/orconf2016-librecores-ci
  6. #librecores, #ORConf Sep 09, 2017 Vision 2016 LibreCores CI GitHub

    SSO User management LibreCores Web Top-level Web UI Project repositories GitHub BitBucket SVN GitHub is a main target system 6 CI Status Gitlab
  7. #librecores, #ORConf Sep 09, 2017 Vision 2016. Infrastructure Jenkins Docker

    (Swarm) Other Integrated Components Physical nodes FPGAs and other HW peripherals LibreCores CI Infrastructure Project-specific Infrastructure On-demand build agents SSH, JNLP over VPN 7
  8. #librecores, #ORConf Sep 09, 2017 •Dozens of build machines •Computing

    grids for simulations •FPGA grids for hardware testing •Licenses for all EDA tools LibreCores does NOT* provide… *According to the current plan/resources 8
  9. #librecores, #ORConf Sep 09, 2017 •Dozens of build machines •Computing

    grids for simulations •FPGA grids for hardware testing •Licenses for all EDA tools LibreCores does NOT* provide… 9 BRING YOUR OWN INFRASTRUCTURE! *According to the current plan/resources
  10. #librecores, #ORConf Sep 09, 2017 Librecores Components Master • PoC

    master: ci.librecores.org • New reusable Docker image Agents 10 • Build images with packaged tools • Examples: • Librecores CI • Librecores CI with Modules • FuseSoC + Icarus • FuseSoC + Icarus + Quartus • FuseSoC + RISCV toolchain • To be replaced by Modules Tools • LCCI CLI • Docker images for Building Tools • Shared Pipeline libraries
  11. #librecores, #ORConf Sep 09, 2017 Example. FuseSoC Core 11 #Jenkinsfile

    import org.librecores.fusesoc.FuseSoCBuilder; OrgFuseSoCBuilder.defaultCoreCIBuild(this, repo: 'wb_sdram_ctrl', organization: 'oleg-nenashev', nodeLabels: 'docker-fusesoc-icarus' extraArgs: '--transactions 10’) https://github.com/oleg-nenashev/wb_sdram_ctrl Build Definition:
  12. #librecores, #ORConf Sep 09, 2017 Usage of LibreCores CI As

    a service Deploy on your own Use tools • EDA tool Images • Shared Libraries for Jenkins Pipeline • Docker packages • Configuration scripts • ci.librecores.org • State: PoC It is not just a service! 12
  13. #librecores, #ORConf Sep 09, 2017 •Modules •LCCI Tools •OpTiMSoC Example

    Under the hood. Tool Infrastructure 13 LibreCores Free and Open Digital Hardware
  14. #librecores, #ORConf Sep 09, 2017 Module-based Environment • Problem: Diverse

    tool requirements • Many different tools are needed • Different versions of the same tool may be used by different projects • Environment Modules: Set environment • Configured by „modulefiles“ • http://modules.sourceforge.net/ $> module load verilator/3.900 PATH=/tools/verilator/3.900/bin/:… VERILATOR_ROOT=/tools/verilator/3.900 $> module load verilator/3.902 PATH=/tools/verilator/3.902/bin/:… VERILATOR_ROOT=/tools/verilator/3.902 14
  15. #librecores, #ORConf Sep 09, 2017 Module-based LibreCores CI • Library

    in Jenkinsfile • Load required tools and versions • Execute shell commands in environment • https://github.com/libre cores/librecores- pipeline-lib @Library('librecoresci') import org.librecores.ci.Modules def lcci = new Modules(steps) node('librecores-ci-modules') { lcci.load(["eda/verilator/3.902"]) stage('Simulation Build') { lcci.sh 'verilator -f verilate.vc' } } 15
  16. #librecores, #ORConf Sep 09, 2017 Module-based Tool Management • Standard

    Docker images for LibreCores CI • Do not include tools in docker images (only basic) • Mount /tools volume from host • Tools volume • Tools in different versions • Module infrastructure • LibreCores CI provides simple Docker images to install tools • Re-producible installations • In the same OS environment (Ubuntu 16.04) as agents 16
  17. #librecores, #ORConf Sep 09, 2017 LCCI Command Line • Quick

    tool installation with simple command line tool • https://github.com/librecores/lcci-python • Example: pip install lcci • Simple YML configuration file • Simply install a tool • Example: lcci tools install verilator:3.902 • Install a whole „standard tool package“ • Example: lcci tools install lcci-2017.1 main: github-api-token: <TOKEN> volumes: tools: /home/lcci-tools 17
  18. #librecores, #ORConf Sep 09, 2017 LCCI CLI. (FPGA) Agents •

    Command: „lcci start agent fpga-nexys4ddr“ • Add agents to lcci.yaml • Each agent is registered with LCCI • Agents run the docker image • The tools volume is mapped • Connect to jenkins and authenticate • Map extra devices (for FPGA agents) agents: fpga-nexys4ddr: type: modules agent-id: <LCCI-AGENT-ID> agent-secret: <LCCI-AGENT-SECRET> devices: - /dev/bus/usb/001/003 - /dev/ttyUSB1 18
  19. #librecores, #ORConf Sep 09, 2017 Example: OpTiMSoC @Library('librecoresci') import org.librecores.ci.Modules

    def lcci = new Modules(steps) node('fpga-nexys4ddr') { lcci.load(["eda/xilinx/vivado/2016.4", "eda/fusesoc/1.6.1", "eda/verilator/3.902", "compilers/openrisc/gcc/5.3.0-newlib2.3.0-1", "base/python/3.5"]) stage("Checkout") { checkout scm } stage('Simulation Build') { lcci.sh 'make BUILD_DOCS=no BUILD_EXAMPLES_FPGA=no' } 19 https://github.com/optimsoc/optimsoc/blob/master/Jenkinsfile
  20. #librecores, #ORConf Sep 09, 2017 Example: OpTiMSoC stage('Simulation Tests') {

    lcci.sh 'source objdir/dist/optimsoc-environment.sh; pytest -s -v test/systemtest/test_tutorial.py::TestTutorial' } stage('FPGA Build') { lcci.sh 'make BUILD_DOCS=no' } stage('FPGA Tests') { sh "echo 'boards:' > localconf.yaml" sh "echo ' nexys4ddr:' >> localconf.yaml" sh "echo ' device: /dev/ttyUSB1' >> localconf.yaml" lcci.sh 'source objdir/dist/optimsoc-environment.sh; export OPTIMSOC_TEST_LOCALCONF=$PWD/localconf.yaml; cat $OPTIMSOC_TEST_LOCALCONF; pytest -s -v test/systemtest/test_tutorial.py::TestTutorialFpga' } } 20 https://github.com/optimsoc/optimsoc/blob/master/Jenkinsfile
  21. #librecores, #ORConf Sep 09, 2017 •[Demo] •Current state •Future plans

    What’s next? 21 LibreCores Free and Open Digital Hardware
  22. #librecores, #ORConf Sep 09, 2017 Plans - 2016 1. Setup

    integration with LibreCores Web POC 2. Build definition libraries for FuseSoC YES 3. Setup flows for projects • OpTiMSoC • Other… SOME 4. Start defining usage conditions for projects TODO 5. Ensure the infrastructure can scale enough SOME 6. Start beta-testing TODO 22
  23. #librecores, #ORConf Sep 09, 2017 Current state • Many plans,

    not so much time • Very first prototype • https://ci.librecores.org • To be replaced by the new version • Dockerized infrastructure • https://github.com/librecores/librecores-ci • Jenkins Master • Agents 23
  24. #librecores, #ORConf Sep 09, 2017 Future work 25 1. Deploy

    pending changes to https://ci.librecores.org 2. Finalize reference FPGA flows • OpTiMSoC 3. Start beta-testing • LibreCores CI Instance • Standalone setups (Docker) 4. Expand Tooling Support
  25. #librecores, #ORConf Sep 09, 2017 Contributing • Try it •

    https://hub.docker.com/r/librecores/librecores-ci-dev/ • Discussions: • Mailing list: [email protected] • Gitter: https://gitter.im/librecores/librecores-ci • GitHub: • Code repository and task tracker • https://github.com/librecores/librecores-ci 26
  26. #librecores, #ORConf Sep 09, 2017 Q & A Contacts: •

    Mailing list: [email protected] • Gitter: https://gitter.im/librecores/Lobby • Twitter: @oleg_nenashev, @wallento 27