Pimp your Continuous Delivery Pipeline with Jenkins workflow (W-JAX 14)

Continuous delivery pipelines are, by definition, workflows with parallel job executions, join points, retries of jobs (Selenium tests are fragile) and manual steps (validation by a QA team). Come and discover how the new workflow engine of Jenkins CI and its Groovy-based DSL will give another dimension to your continuous delivery pipelines and greatly simplify your life.

Sample workflow groovy script used in this presentation: https://gist.github.com/cyrille-leclerc/796085e19d9cec4a71ef

Transcript

1. ©2014 CloudBees, Inc. All Rights Reserved Pimp your Continuous Delivery Pipeline with Jenkins workflow Cyrille Le Clerc, CloudBees
2. ©2014 CloudBees, Inc. All Rights Reserved About Me Cyrille Le Clerc   @cyrilleleclerc   Open  Source   CTO   DevOps,  Infra  as  Code,   Con9nuous  Delivery   Product  Management  
3. ©2014 CloudBees, Inc. All Rights Reserved About you Jenkins? CI? CD? Code Jenkins plugin?
4. ©2014 CloudBees, Inc. All Rights Reserved Agenda •  Con9nuous  Delivery   •  Jenkins  &  Con9nuous  Delivery   •  Workflows  in  Jenkins  today   •  The  new  Jenkins  Workflow  Engine   •  Jenkins  Workflow  Syntax  Card   •  Possible  future  
5. ©2014 CloudBees, Inc. All Rights Reserved Continuous Delivery
6. ©2014 CloudBees, Inc. All Rights Reserved Continuous Delivery •  “Reliable  So+ware  Releases   through  Build,  Test,  and   Deployment  Automa=on”   •  DONE  =  Shippable  into  Produc9on  
7. ©2014 CloudBees, Inc. All Rights Reserved Continuous Integration Steps Con$nuous  Integra$on   Source Code Quality Analysis Compile & Unit Tests Binaries
8. ©2014 CloudBees, Inc. All Rights Reserved Continuous Delivery Steps Con$nuous  Delivery   DEV DEV OPS QA Source Code Quality Analysis Compile & Unit Tests Binaries Perfs Environment Func Test Environment UAT Environment Production Environment GO LIVE! Release! Staging Environment
9. ©2014 CloudBees, Inc. All Rights Reserved Continuous Delivery Pipeline Continuous Delivery Pipeline Compile & Unit Tests Perfs Tests Selenium Tests Deploy UAT Sonar Analysis Deploy Staging Deploy Production
10. ©2014 CloudBees, Inc. All Rights Reserved Jenkins: Hub of Continuous Delivery •  Connect  the  dots   •  It  is  the  process   Source Code Perfs Environment Func Test Environment UAT Environment Binaries Quality Analysis Compile & Unit Tests
11. ©2014 CloudBees, Inc. All Rights Reserved CD Pipeline è Workflow Continuous Delivery Pipeline Compile & Unit Tests Perfs Tests Selenium Tests Deploy UAT Sonar Analysis Deploy Staging Deploy Production
12. ©2014 CloudBees, Inc. All Rights Reserved What we need •  Complex  pipelines  involving  mul9ple  stages   •  Non-­‐sequen$al  logic  such  as  loops,  forks  …   •  Long-­‐running  builds  must  survive  outages   •  Interac$ons  with  humans  including  pauses,  input   •  Restartable  builds  in  case  of  transient  errors   •  Reusable  defini$ons  to  avoid  duplica9on   •  Comprehensible  scripts  with  one  clear  defini9on  
13. ©2014 CloudBees, Inc. All Rights Reserved Workflow until today •  Job  chaining   •  Build  Flow  Plugin   •  Build  Pipeline  Plugin   •  Build  Delivery  Pipeline  
14. ©2014 CloudBees, Inc. All Rights Reserved Workflow until today •  Many  atomic  jobs   •  Hard  to  share  variables/state  between  jobs   •  Limited  logic   •  Mix  build  triggers,  parameterized  build  …    
15. ©2014 CloudBees, Inc. All Rights Reserved Build Flow Plugin •  Did  have  scriptability  and  extensibility   •  Did  not  address  configura9on  “sprawl”   •  Disjointed  view  of  what  really  ran   •  No  ability  to  survive  restarts   •  Almost  good  enough  but  could  not  go  further  
16. ©2014 CloudBees, Inc. All Rights Reserved Jenkins Workflow Engine
17. ©2014 CloudBees, Inc. All Rights Reserved Simplifying our sample workflow Compile & Unit Tests Perfs Tests Selenium Tests Deploy UAT Sonar Analysis Deploy Staging Deploy Production
18. ©2014 CloudBees, Inc. All Rights Reserved Simplifying our sample workflow Compile & Unit Tests Perfs Tests Selenium Tests Deploy UAT Sonar Analysis Deploy Staging Deploy Production
19. ©2014 CloudBees, Inc. All Rights Reserved What we need •  Complex  pipelines  involving  mul9ple  stages   •  Non-­‐sequen$al  logic  such  as  loops,  forks  …   •  Long-­‐running  builds  must  survive  outages   •  Interac$ons  with  humans  including  pauses,  input   •  Restartable  builds  in  case  of  transient  errors   •  Reusable  defini$ons  to  avoid  duplica9on   •  Comprehensible  scripts  with  one  clear  defini9on  
20. ©2014 CloudBees, Inc. All Rights Reserved New Workflow Syntax
21. ©2014 CloudBees, Inc. All Rights Reserved Key features •  En9re  flow  is  one  concise  Groovy  script   –  For  loops,  try-­‐finally,  fork-­‐join  …   •  Can  restart  Jenkins  while  flow  is  running   •  Allocate  slave  nodes  and  workspaces   –  As  many  as  you  want,  when  you  want   •  Stages  throYle  concurrency  of  builds   •  Human  input/approval  integrated  into  flow   •  Standard  project  concepts:  SCM,  ar9facts,  plugins  
22. ©2014 CloudBees, Inc. All Rights Reserved Groovy DSL vs. Graphical Workflow •  Familiar  control  flow  construc9on   •  Familiar  “tools”  for  building  abstrac9ons   Func9ons,  classes,  variables,  …   •  Workflow  in  version  control   •  As  opposed  to:   Graphical  workflow  designer  
23. ©2014 CloudBees, Inc. All Rights Reserved Project setup •  One  workflow  is  defined  as  a  job   •  Single  script  for  all  steps   •  Build  triggers  &  parameters  like  regular  projects   •  Each  workflow  execu9on  is  a  regular  Jenkins   build  displayed  in  regular  Jenkins  views   •  Graphical  visualiza9on  of  actual  build  possible   Not  of  visualiza9on  job  defini9on  but  of  build  execu9on  
24. ©2014 CloudBees, Inc. All Rights Reserved Non-sequential logic try {} catch {}
25. ©2014 CloudBees, Inc. All Rights Reserved Interaction with humans
26. ©2014 CloudBees, Inc. All Rights Reserved Restartable build / checkpoint
27. ©2014 CloudBees, Inc. All Rights Reserved Visualization
28. ©2014 CloudBees, Inc. All Rights Reserved Stages •  Semaphore   •  Visualiza9on  
29. ©2014 CloudBees, Inc. All Rights Reserved Comprehensible scripts
30. ©2014 CloudBees, Inc. All Rights Reserved Demo time!
31. ©2014 CloudBees, Inc. All Rights Reserved Workflow Syntax Card
32. ©2014 CloudBees, Inc. All Rights Reserved Workflow Syntax Card •  stage:  Enter  a  new  stage   •  node:  Allocate  node   •  ws:  Allocate  workspace   •  parallel:  Execute  sub-­‐workflow  in  parallel   •  retry:  Retry  the  body  up  to  N  9mes   •  catchError:  Catch  error  and  con9nue   •  input:  Input  /  human  interac9on   •  load:  Evaluate  a  Groovy  source  file  into  the  workflow  script  
33. ©2014 CloudBees, Inc. All Rights Reserved Workflow Syntax Card •  step:  General  build  step   •  sh:  Shell  script   •  bat:  Windows  batch  script  
34. ©2014 CloudBees, Inc. All Rights Reserved Workflow Syntax Card •  archive:  Archive  ar9facts   •  unarchive:  Copy  archived  ar9fact  into  the  workspace   •  echo:  Print  message   •  dir:  Change  directory   •  readFile:  Read  file  from  workspace   •  writeFile:  Write  file  to  workspace  
35. ©2014 CloudBees, Inc. All Rights Reserved Workflow Syntax Card •  git:  Git   •  svn:  Subversion   •  scm:  General  SCM   •  tool:  install  a  tool   •  build:  Build  a  job  
36. ©2014 CloudBees, Inc. All Rights Reserved Workflow Syntax Card •  checkpoint:  capture  the  state  of  the  execu9on  so   that  it  can  be  restarted  later  
37. ©2014 CloudBees, Inc. All Rights Reserved Possible Future
38. ©2014 CloudBees, Inc. All Rights Reserved Possible Future: probably OSS •  More  plugin  interoperability   •  Mul9-­‐branch  project  (&  SCM  API)  integra9on   •  More  advanced  SCM     •  In-­‐IDE  edi9ng  support   •  Concise  syntax  
39. ©2014 CloudBees, Inc. All Rights Reserved Possible Future: probably Jenkins Enterprise •  More  visualiza9ons   •  Validated  merge  integra9on   •  Deployment,  incl.  blue/green  with  rollback   •  Freestyle  and/or  Build  Flow  import   •  Templates  integra9on  
40. ©2014 CloudBees, Inc. All Rights Reserved Get the code •  hYps://gist.github.com/cyrille-­‐leclerc/ 796085e19d9cec4a71ef   – workflow.groovy  
41. ©2014 CloudBees, Inc. All Rights Reserved