Slide 1

Slide 1 text

Jenkins World Tour 2015 London, UK 23-24 June

Slide 2

Slide 2 text

Footer #jenkinsconf Jenkins-based CI for heterogeneous Hardware/Software projects Oleg Nenashev Jenkins Developer @ CloudBees 2

Slide 3

Slide 3 text

Footer #jenkinsconf > whoami • •  Background: ~10 years of… –  R&D: HW and Embedded systems –  CI in HW, Embedded and EDA areas –  Department-wide CI frameworks •  Current: –  Jenkins CI contributor –  Jenkins Developer at CloudBees –  PhD student, EDA Lab @ SPBPU 3

Slide 4

Slide 4 text

Footer #jenkinsconf > whoami --list jenkins •  >6 years of Hudson/Jenkins experience •  3 years of open-source contribution to Jenkins •  Maintenance of large-scale installations •  I maintain about 10 open-source plugins •  Contributions to other Jenkins stuff and INFRA •  Ownership •  Job Restrictions •  Role Strategy (legacy) •  Custom Tools (legacy) •  Extra Tool Installers •  Cygwin Process Killer Tool belts Ownership-based security

Slide 5

Slide 5 text

Footer #jenkinsconf Outline •  CI in HW/Embedded world. Why Jenkins? •  Integrating Jenkins with the common stuff –  EDA tools, FPGA farms, Computing Grids •  Several examples and case studies •  Getting added value from CloudBees Jenkins Platform 5

Slide 6

Slide 6 text

Footer #jenkinsconf Jenkins users from HW world •  The most of HW companies have Jenkins instances •  Examples from a 5min search in public materials 6

Slide 7

Slide 7 text

Footer #jenkinsconf Related talks / articles •  Robert Martin, “Multi-Stage-CI with Jenkins in an Embedded World” @ JUC2014/Berlin •  Martin d’Anjou, “FPGA Continuous Integration with Jenkins” @SNUG2013/Canada •  Alan Fitch, “Continuous Integration for FPGA Design and Verification” @ Verification Futures, 2015 •  JL Gray, Gordon McGregor, “A 30 Minute Project Makeover Using Continuous Integration” @ DVCon, 2012 •  … •  Sarah Woodall, “The Famous Cows of Cambridge: A Non- Standard Use Case for Jenkins” @JUC2015/London

Slide 8

Slide 8 text

Footer #jenkinsconf INTRODUCTION CI in HW/Embedded world 8

Slide 9

Slide 9 text

Footer #jenkinsconf What is a hardware product? . . . Not so obvious… •  IP Libraries – reusable blocks •  Prototypes •  Development kits •  …

Slide 10

Slide 10 text

Footer #jenkinsconf HW Development Flow 10 Synthesis Place & Route RTL Design Specification = Compilation = Linking Design = Coding VHDL Verilog / SystemVerilog SystemC … Verification Packaging

Slide 11

Slide 11 text

Footer #jenkinsconf HW CI Example 11 Get sources SCMs Binary repositories Build artifacts Setup environment “Build” = S&P&R “Test” “Analyze” Publish reports Deploy artifacts Static analysis Formal verification Timing analysis Specialized tools Unit tests Integration tests EDA Tools Electronic Design Automation Test results Coverages System resources utilization Much parallel tasks in flows

Slide 12

Slide 12 text

Footer #jenkinsconf Embedded SW build process 12 Other software Embedded Software •  Different tools, but the approaches are similar •  Verification is a real challenge

Slide 13

Slide 13 text

Footer #jenkinsconf And what about an HW IP Product? Hardware part External IP Firmware / Embedded SW External SW Development tools / EDAs Demo SW Demo hardware Developer kits Yes, we need to automate everything “Main product”

Slide 14

Slide 14 text

Footer #jenkinsconf Welcome to CI hell! •  Almost everything needs to be automated •  Multiple technologies and tools •  Unstable hardware & networks •  Legacy stuff is everywhere •  HW engineers are not SW experts •  … •  Limited resources •  Limited OSS usage and contribution options

Slide 15

Slide 15 text

Footer #jenkinsconf Why Jenkins? •  It’s a framework => you can do everything •  It’s open-source: –  You can customize Jenkins core and plugins –  Plenty of available materials and examples •  “Stable” LTS versions •  Commercial support for self-hosted installations

Slide 16

Slide 16 text

Footer #jenkinsconf AREA SPECIFICS •  EDA Software •  Computing grids •  FPGAs 16

Slide 17

Slide 17 text

Footer #jenkinsconf EDA Software •  Almost no OSS plugins –  Exception - Cadence vManager Plugin •  Specifics: –  Dozens of tools in a common design flow –  Different platform requirements –  Multiple tool versions are required –  Big size of tools – up to 100 Gigabytes –  Limited licenses being shared with externals 17

Slide 18

Slide 18 text

Footer #jenkinsconf EDA Software. Possible solution •  Store tools on shared folders –  On Windows services kohsuke/winsw/pull/71 may be used –  Extra Tool Installers Plugin provide a stub installer •  Access tools: Labels + Custom Tools Plugin •  Idea of the plugin: –  Configurable installers –  Setup of a custom environment (PATH, env. vars) –  Built-in versioning based on Extended Choice Parameter 18

Slide 19

Slide 19 text

Footer #jenkinsconf Custom Tools. Global Configuration 19 Can be stored remotely => automatic tool deployment Node and global env. vars are supported

Slide 20

Slide 20 text

Footer #jenkinsconf Custom Tools. Job Configuration 20 Allow versioning Automatic installation

Slide 21

Slide 21 text

Footer #jenkinsconf Custom Tools. Usage 21 In build steps: In build launch dialog:

Slide 22

Slide 22 text

Footer #jenkinsconf EDA Software. Publishers •  Jenkins has publishers for many SW dev. tools –  Unit test results (xUnit et all.) –  Performance test results –  Coverages •  No solutions for hardware, but you can use converters •  Example for VCS coverages: 1.  Get an HTML or raw XML report 2.  Use XSLT to convert it to Cobertura format 3.  Publish data using a standard plugin 22

Slide 23

Slide 23 text

Footer #jenkinsconf EDA Software. Licenses •  FLEX License Manager is everywhere in EDA •  Main problem: shared usage of licenses –  No sporadic prioritization in FLEX => long waiting –  Dedicated licenses are not cost-effective –  Limited availability of tools => custom accounts •  Minimal solution – Throttling, custom “wait for a license” scripts •  Best solution - FLEX LM plugin, no OSS ones 23

Slide 24

Slide 24 text

Footer #jenkinsconf Hardware testing •  Simulators – specialized EDA tools •  Hardware – FPGAs, ASIC prototypes •  Combined solutions (HW Accelerators, etc.) 24

Slide 25

Slide 25 text

Footer #jenkinsconf Cloudless skies •  Simulation tests require… •  Public/private clouds are not cost-effective •  HW Companies mostly use computing grids 25 Thousands CPU hours Terabytes of TMP files and reports All RAM you have

Slide 26

Slide 26 text

Footer #jenkinsconf Computing grids Examples: •  Sun Grid Engine (SGE, UGE) •  Platform Load Sharing Facility (LSF) •  NetBatch @ Intel •  … 26 •  Datacenter-wide grids •  Sharing across departments “We run hello-world in 2 hours”

Slide 27

Slide 27 text

Footer #jenkinsconf Computing Grids Usage 27 Manage tasks from CLI (script steps) “Transparent” – launch Jenkins nodes on the grid •  Use-case: Legacy scripts •  Difficult management (job abort, status polling, etc.) •  Manual launch •  “Start by script on Master” •  LSF Cloud Plugin •  No FOSS plugins for other grids L “Launch task” build & workflow steps •  No FOSS plugins !

Slide 28

Slide 28 text

Footer #jenkinsconf Simulations are slow, so… 28 •  S - EZ1CUSB @ •  M - •  L - HAPS-64, S - x00$ M – x000$ L –x0000$ FPGA accelerators & prototyping boards

Slide 29

Slide 29 text

Footer #jenkinsconf 29 And XXL… • breakingthethreelaws/2014/04/ the-secret-ninja-fu-for-higher- performance-prototype- operation/ HAPS S96 Guess the price… Could CI have its own stuff? Your CPU prototype is here

Slide 30

Slide 30 text

Footer #jenkinsconf Mapping CI to existing HW 30 Nodes @ Site A Nodes @ Site C Nodes @ Site B “We borrowed FPGAs here, somebody takes them randomly” “This stuff has Windows drivers only” “No infra team members here” “We run nodes on dev. laptops” Frankenboards “We built it from the garbage we had” Infrastructure after the “budget optimization” L L L L L

Slide 31

Slide 31 text

Footer #jenkinsconf How it works 31 Power When it hangs “Deploy bit-files” Blasting cables Debugging cables, firmware upload We don’t change them, huh? Jenkins node drivers tools peripherals Attached hardware runtime

Slide 32

Slide 32 text

Footer #jenkinsconf How do I implement jobs? •  Throttling of builds •  Shared functionality is decoupled to external project (Parameterized Trigger or Template Project) –  FPGA blasting –  Firmware Upload –  Run of a test case (binary upload) using a debugger provided by the toolchain (gdb, etc.) –  etc. •  Failover and restart on infra issues => Naginator 32

Slide 33

Slide 33 text

Footer #jenkinsconf Throttling of FPGA jobs •  Throttle Concurrent Builds Plugin –  Jobs can be throttled by a category –  Performance issues (esp. till 1.609.1), can be partially solved by Label Throttle plugin (CloudBees Jenkins Enterprise) –  Job developers must follow rules, no enforcement Global configuration: 33

Slide 34

Slide 34 text

Footer #jenkinsconf Throttling of FPGA jobs Job configuration: 34

Slide 35

Slide 35 text

Footer #jenkinsconf FPGAs monitoring •  Impossible to use an external system – shared resources •  Solution in Jenkins: 1.  Use periodic Multi-configuration Matrix Job for a monitoring 2.  Use Elastic Axis to generate label axes by an expression 3.  Run a spot-check on FPGA with blasting and a firmware test 4.  If it fails, disable the node using System Groovy Script 5.  Send notifications to INFRA admins 35

Slide 36

Slide 36 text

Footer #jenkinsconf ENTERPRISE FEATURES Getting added value from CloudBees Jenkins Platform 36 CJE – CloudBees Jenkins Enterprise CJOC – CloudBees Jenkins Operations Center

Slide 37

Slide 37 text

Footer #jenkinsconf Common features All common features are applicable, out of the scope See CloudBees Docs

Slide 38

Slide 38 text

Footer #jenkinsconf Templates! •  OSS Plugins have a limited functionality: –  Template Project, Inheritance Plugin, Job Generator, etc. •  Parameterized trigger causes UI overheads –  Multi-page configuration in UI: “Trigger with several parameters => wait => copy artifacts => analyze” •  Templates allow to save much time on the implementations •  Workflow + Scriptler Plugin are tasty as well 38

Slide 39

Slide 39 text

Footer #jenkinsconf Easy failover implementation •  Workflow allows to transparently re-run unreliable stages on other nodes. for (def board : boards) { try { // blasting … // run test … // test finished break; } catch (FPGAError error) { // try another board continue; } } System failure Node #1 Node #2 . . . . . .

Slide 40

Slide 40 text

Footer #jenkinsconf Shared slaves for rare hardware •  Use-case: multiple Jenkins instances in a single org •  Instances may require a specific resource •  No cross-Jenkins throttling features •  Out-of-the-box throttling in CJOC shared slaves 40

Slide 41

Slide 41 text

Footer #jenkinsconf Durable tasks •  Endurance tests may take weeks •  Jenkins slaves may be disconnected, masters can be restarted for maintenance purposes •  No stable solutions in Jenkins OSS if you don’t use computing grids •  Durable tasks allow to overcome the issue in many cases –  Shell scripts are running independently –  Jenkins polls for a completion asynchronously –  Serious limitation: cross-site installations 41

Slide 42

Slide 42 text

Footer #jenkinsconf Summary •  Jenkins is not only about software •  Jenkins can be used to run CI in hardware projects •  You can use Jenkins even without domain-specific plugins •  CJE/CJOC features greatly help to decrease the maintenance cost 42

Slide 43

Slide 43 text

Footer #jenkinsconf My HW-related wish-list •  Sun Grid Engine Cloud plugin •  FLEX LM plugin •  Flexible Windows Services •  Plugins for common EDA tools and report formats –  Build steps are OK => just use CLI utilities –  Publishing requires much efforts –  Logs parsing would be a killer feature 43

Slide 44

Slide 44 text

Footer #jenkinsconf Q&A