Linuxwochen: Jenkins 2.0. What's inside?

The talk addresses major changes in Jenkins 2.0:

* Jenkins positioning shift: it goes beyond Continuous Integration (in particular, to Continuous Delivery and DevOps areas)
* User documentation and website improvements
* User experience improvements: Installation experience, configuration pages, etc.
* Pipeline-as-code: Ecosystem, which allows describing jobs in Groovy DSL located in SCMs

Event: https://cfp.linuxwochen.at/de/LWW16/public/events/382
Meetup page: http://www.meetup.com/jenkinsmeetup/events/230391829/

Transcript

1. Jenkins 2.0.  What’s  inside? Vienna,  April  29th, 2016 Oleg  Nenashev  (@oleg_nenashev) CloudBees,  Inc.
2. @oleg_nenashev,  @linuxwochen 2.0 April  29,  2016 >  whoami -­‐ Oleg  Nenashev •Jenkins  OSS  contributor •Core  &  Security teams •Plugin  Developer  and  Maintainer •Developer  at CloudBees •CloudBees  Jenkins  Platform •Background: •R&D, HW/Embedded  SW •Intel,  Sitronics,  Synopsys •Large-­‐scale  automation  infrastructure  lead @oleg_nenashev oleg-­‐nenashev 2
3. @oleg_nenashev,  @linuxwochen 2.0 April  29,  2016 •Hudson/Jenkins  – 8  years •Open-­‐source  contribution  – 4  years •My  plugins:  Ownership,  Job  Restrictions,… •Legacy:  Role  Strategy,  Custom  Tools,  Promoted  Builds •Core  &  Security  teams  in Jenkins •Meetup  organizer •Org  Admin  in Google  Summer  of  Code  2016 >  whoami –v  jenkins 3
4. @oleg_nenashev,  @linuxwochen 2.0 April  29,  2016 •CloudBees  – Jenkins  contributor  and  sponsor •Most  of  the  development  -­‐ open-­‐source •We  help  to  build  enterprise  automation   systems  based  on  Jenkins •CloudBees  Jenkins  Platform •24/7 support  of  enterprise  and  open-­‐source  stuff •Trainings,  professional  services About CloudBees 4
5. @oleg_nenashev,  @linuxwochen 2.0 April  29,  2016 Today’s  topic Released  on  April  20th https://jenkins-­‐ci.org/2.0/ 5
6. @oleg_nenashev,  @linuxwochen 2.0 April  29,  2016 Disclaimer •This  presentation  is  not  a  training  material •The  talk  represents  the  speaker’s  personal  opinion •This  opinion  may  differ  from  the  official  position  of   CloudBees  or  Jenkins  community 6
7. @oleg_nenashev,  @linuxwochen 2.0 April  29,  2016 Who  is  Mr.  Jenkins? Jenkins – Continuous  Integration System Automation Server  /  Framework 7
8. @oleg_nenashev,  @linuxwochen 2.0 April  29,  2016 Why  “automation”?   Continuous  Integration Continuous   Integration Continuous   Delivery DevOps Accounting   automation Home   Automation CRON  + WebUI 8
9. @oleg_nenashev,  @linuxwochen 2.0 April  29,  2016 Jenkins  101 >1000   plugind Integration   with  hundreds   of  tools Automation  flow   control Automation  of   any  project Code  &  Commit Build  &  Config Scan  &  Test Release Deploy 9
10. @oleg_nenashev,  @linuxwochen 2.0 April  29,  2016 Jenkins 101 • 2005  – Hudson  (Sun) • 2008  – It  becomes  popular,  Duke's  Choice  Award • 2011  – Conflict  with  Oracle,  renaming  to  Jenkins • 2012  – Jenkins  is  the  most  popelar tool • 2016  – Jenkins  2.0 • License:  MIT  (core) • https://github.com/jenkinsci/ 10
11. @oleg_nenashev,  @linuxwochen 2.0 April  29,  2016 Jenkins  is  popular http://stats.jenkins-­‐ci.org/jenkins-­‐stats/ • 1000k   installations • ~300  active   committers  (6   months) • ~3  new  plugins   every  week • ~30  plugin   releases  every   week Hudson Jenkins 11
12. @oleg_nenashev,  @linuxwochen 2.0 April  29,  2016 Why  “framework”? System Flexible  and  Extensible Minimal  OOTB  functionality Requires  customization Jenkins  is… 12
13. @oleg_nenashev,  @linuxwochen 2.0 April  29,  2016 Jenkins-­‐based  Automation  System Infra-­‐ structure Docs Infra-­‐ structure Plugins Configs Infra-­‐ structure Templates   Demos Jenkins-­‐based   Automation   System 13
14. 2.0 Why  Jenkins  2.0?
15. @oleg_nenashev,  @linuxwochen 2.0 April  29,  2016 Automation  Systems.  Evolvement  Trends •SW  area  shifts  focus  from  CI: •Continuous  Delivery •Continuous  Deployment •DevOps •Everything  as  Code •Centralization  of  automation  systems •High  requirements  and  strict  SLAs 15
16. @oleg_nenashev,  @linuxwochen 2.0 April  29,  2016 Jenkins  weaknesses.  User  experience Where  are  the  docs? How  to  find  a  plugin? Is  this  plugin  alive? How  to  configure  THIS? Why  do  they  ignore  my  issues? Where  is  the  maintainer? Why  the  UI  is  so  complicated? How  to  implement  it? 16
17. @oleg_nenashev,  @linuxwochen 2.0 April  29,  2016 Jenkins  Learning  Curve Basic  scripts,   CRON  with  UI Automation   flows  based   on  plugins Advanced  features, Custom  Plugins 17
18. @oleg_nenashev,  @linuxwochen 2.0 April  29,  2016 Jenkins.  Compatibility  costs • More  than  5  years  of  backward  compatibility • Hudson  2  plugins  still  work  on  Jenkins! • BUT: • Architecture  design  issues,  complex  APIs • Outdated  libs • Outdated  UI,  users  are  not  so  happy • Plugin  integration  issues 18
19. 2.0 Jenkins  2.0.  Prehistory
20. @oleg_nenashev,  @linuxwochen 2.0 April  29,  2016 1  April,  2010  – Hudson  2.0  Announcement Hudson 2.0 is expected to follow in the intrepid footsteps of other major revamps such as Netscape 5 and Perl 6 in its ambitiousness and innovation. Implementation of core in portable C for greater speed and code readability Revamped storage architecture around the Oracle 11g Database to provide more optimal synergy over flat XML files. https://jenkins-­‐ci.org/blog/2010/04/01/announcing-­‐the-­‐hudson-­‐2-­‐0-­‐roadmap/ Perl  6,  2016 Hudson  2.0,  ? 20
21. @oleg_nenashev,  @linuxwochen 2.0 April  29,  2016 •…  do  not  want  to  spend  4  years  on  2.0 •… want  to  retain  the  compatibility •…  want  to  support  major  architecture   changes  in  the  nearest  future •…  focus  on  UX  and CD In Jenkins  2.0 we… 21
22. @oleg_nenashev,  @linuxwochen 2.0 April  29,  2016 •2015  – Discussions  at  Jenkins  Office  Hours   •October  2015  -­‐ Jenkins  2.0  scope  proposals   • https://wiki.jenkins-­‐ci.org/display/JENKINS/Jenkins+2.0 •SCALEx16  and  FOSDEM2016  (Jan  30-­‐31) •First  2.0  Demos,  Face-­‐to-­‐face  discussions  with  users •Jenkins  2.0  -­‐ Contributor  Summit (Feb  1) • >20  Jenkins  Core  developers  and  contributors •Defined  the  release  scope •Notes:  http://bit.ly/jenkins2-­‐summit-­‐notes Jenkins  2.0.  Key  Events 22
23. @oleg_nenashev,  @linuxwochen 2.0 April  29,  2016 •Feb  26  – joined  Google  Summer  of  Code •http://bit.ly/jenkins-­‐gsoc-­‐wiki •9  project  ideas,  15  mentors,  dozens  of  applications •5  applications  have  been  accepted • Jenkins  web  UI  improvements • Support  Core  Plugin  improvement  (better  support  bundles,   submit  issue  from  Jenkins  UI,  etc.) • External  workspace  manager  plugin • Automatic  plugin  documentation  publishing  (Docs as  code) • Big  Data.  Plugin/Jenkins  usage  stats Key  Events 23
24. @oleg_nenashev,  @linuxwochen 2.0 April  29,  2016 Jenkins  roadmap Jenkins  1.x “Jenkins  2.x” Jenkins  2.0 April  20th End  of  2016  (?) • New  website • Focus  on  UX  and  project  infra • Minimal compatibility  changes • Major  changes  in  the  core   architecture • Further  UI  enhancements • At  least  1  LTS  line  in  2016 24
25. @oleg_nenashev,  @linuxwochen 2.0 April  29,  2016 •February  29:  public  alpha  version   •March  20:   •"feature  complete” •beta  version •wide  testing •April  6:   •Release  candidate •jenkins.io  launch   •April  20:  Release Release  timeline 25
26. @oleg_nenashev,  @linuxwochen 2.0 April  29,  2016 •Repackaging  of  plugins:  No  bundling  anymore •UX  improvements: •New  installation  wizard •WebUIimprovements  (configuration  pages,  etc.) •WebUI library  updates •Security  is  enabled  by  default •CSRF,  Slave-­‐2-­‐master  security •Jenkins  starts  up  with  Matrix-­‐Based  Security •Pipeline  as  Code Jenkins  2.0. What’s  inside  the  core? http://bit.ly/jenkins20 26
27. 2.0 Jenkins  2.0  website
28. @oleg_nenashev,  @linuxwochen 2.0 April  29,  2016 •New  domain:  http://jenkins.io •Static  web-­‐site,  sources  are  on  GitHub
29. @oleg_nenashev,  @linuxwochen 2.0 April  29,  2016 New  website  (http://jenkins.io) 29
30. Editing  in  GitHub:   Markdown,  Asciidoc,   HAML Sidepanels (Twitter,   Calendar,  etc.) Disclaimer:   legacy  layout (fits  the  slide   layout)
31. Solution  pages.  New  content! Disclaimer:   legacy  layout (fits  the  slide   layout)
32. @oleg_nenashev,  @linuxwochen 2.0 April  29,  2016 Wishlist •More  content •Solution  pages •CloudBees  moves  Jenkins  Handbook  to OSS •Centralized  documentation  for  plugins •«Plugin  gallery»   •Community  ratings •Easy  changelogaccess •Downgrade  notices 32
33. @oleg_nenashev,  @linuxwochen 2.0 April  29,  2016 •Focus  -­‐ Out  of  the  Box  Experience •Installation  wizard,  plugin   management,  configuration  pages Jenkins  2.0  UI 33
34. @oleg_nenashev,  @linuxwochen 2.0 April  29,  2016 Installation  Wizard.  Options 34
35. @oleg_nenashev,  @linuxwochen 2.0 April  29,  2016 Installation  Wizard.  Plugin  selection On  hover  – popup  with   the  dependency  list 35
36. @oleg_nenashev,  @linuxwochen 2.0 April  29,  2016 Job  configuration  pages.  Tabs Tabs Restore  the   old  view 36
37. 2.0 Pipeline  as  code DESCRIBING   JENKINS JOBS   VIA  CODE
38. @oleg_nenashev,  @linuxwochen 2.0 April  29,  2016 •Storage  along  with  the  project •Configuration  change  tracking Why  would  we  benefit  from  it? 38
39. @oleg_nenashev,  @linuxwochen 2.0 April  29,  2016 Automation  as  Code Job  DSL  Plugin Job  Builder  Plugin Pipeline  (fka Workflow) Pipeline  is  offered  by  default  in  Jenkins  2.0 .  .  . Groovy YAML Groovy 39
40. @oleg_nenashev,  @linuxwochen 2.0 April  29,  2016 •Groovy  DSL  for  build  flow  description •Job  type  in  Jenkins (Pipeline  Job) •Ecosystem  for  automation-­‐as-­‐code •~15  dedicated  plugins •Integration  in  many  other  plugins What  is  Pipeline? 40
41. @oleg_nenashev,  @linuxwochen 2.0 April  29,  2016 Pipeline.  Example  #1 • Syntax:  Groovy-­‐based  DSL • Job  descriptions  can  be  stored  in  SCM • E.g.  Jenkinsfile in  the  repo  root • http://bit.ly/pipeline-­‐tutorial 41
42. @oleg_nenashev,  @linuxwochen 2.0 April  29,  2016 Pipeline.  Available  steps • Build  steps  and   publishers  from  plugins • Core  steps  (file   operations,  etc.) • Extra  utility  steps 42
43. @oleg_nenashev,  @linuxwochen 2.0 April  29,  2016 •Snippet  generator  UI •User  sets  up  settings  in  UI •Jenkins  generates  Pipeline  command  for  copy&paste Pipeline  features 43
44. @oleg_nenashev,  @linuxwochen 2.0 April  29,  2016 •Parallelization •parallel()  – running  of  subscripts  in  parallel •No  Freestyle  job  triggering  spaghetti  anymore •Multiple  node()  operations  in  a  job •Parallel  runs  on  different  nodes  are  supported •Multiple  scm()  usages  in  a  single  job •Interactive  steps  (user  input  via  for) •You  can  involve  manual  QA  into  your  CD! Pipeline  features 44
45. @oleg_nenashev,  @linuxwochen 2.0 April  29,  2016 Example.  Parallel  tests … stage 'test' parallel 'unit' : { node  { unstash 'source' sh "./gradlew :test" stash includes: 'build/jacoco/*.exec',   name: 'unitCodeCoverage' step([$class:'JUnitResultArchiver',   testResults:'**/build/test-­‐results/*.xml']) } },  'integration': { node { unstash 'source' sh "./gradlew -­‐PhappyPath :integration-­‐test:test" step([$class:'JUnitResultArchiver',   testResults:'**/build/test-­‐results/*.xml']) } }   45
46. @oleg_nenashev,  @linuxwochen 2.0 April  29,  2016 Example.  Interactive  steps // Build,  perform  automatic  tests,  initialize  TMS //  Notify  manual  testers echo  ’Automatic  tests  passed’   hipchatSend('@QATeam The  build  ' + buildVersion + '  has  been  staged:  ' + stagingURL + '.  Please  Test  it  and  report  back  to  ' + build.url + '.  TestLink run  is  here:  ' + testLinkRunURL) //  Wait  till  QA  team  confirms  the  release input  message:  'Have  manual  spot-­‐checks  passed?  Ready  to  release?' //  Release  and  notifications mail  to:  "dream_team@cloudbees.com",   subject:  "Dream  Product  -­‐ build  is  released",   body:  "Hi  there,  the  new  build  has  been  released.  Link:  " +  artifactoryLink ~~~~~~  Code  ~~~~~~ ~~~~~~  Code  ~~~~~~ 46
47. @oleg_nenashev,  @linuxwochen 2.0 April  29,  2016 •Robustness  against  master  restarts •Pipeline  continues  jobs  after  the  restart •Robustness  against  network  issues •We  use  Durable  Task Plugin •sh()  call  won’t  fail  if  network  fails •Security: •Groovy  Sandbox  (Script  Security  Plugin) •Every  call  from  Pipeline  is  validated  (whitelists) Pipeline  features  – for  admins 47
48. Pipeline.  Example  #2.  Jenkins  IRC  bot  CD def imageName =  'jenkinsciinfra/ircbot’ node('docker')   { checkout scm //  Some  magic  for  getting   Docker image  tag sh 'git rev-­‐parse HEAD  >  GIT_COMMIT' shortCommit =  readFile('GIT_COMMIT').take(6) def imageTag =  "build${shortCommit}" stage 'Build ircbot' withMavenEnv (["BUILD_NUMBER=${env.BUILD_NUMBER}:${shortCommit}"])   { sh 'make bot'  //  Make  calls  Maven,  which  calls  Ant } stage 'Build container' def whale =  docker.build("${imageName}:${imageTag}”) stage 'Deploy container' whale.push() } https://github.com/jenkins-­‐infra/ircbot Temporary  solution
49. @oleg_nenashev,  @linuxwochen 2.0 April  29,  2016 Pipeline:  More  examples https://github.com/jenkinsci/ pipeline-­‐examples 49
50. @oleg_nenashev,  @linuxwochen 2.0 April  29,  2016 •Visualization  of  a  sequential  Pipeline •The  plugin  has  been  moved  to  OSS  by  CloudBees Pipeline  Stage  View • stage() • Independent   logs  for   each  stage • Interactive  steps  are   integrated  into  UI 50
51. @oleg_nenashev,  @linuxwochen 2.0 April  29,  2016 •Multi-­‐Branch  Pipeline  Plugin •Job  configuration  only  contains  reference  to  the   SCM (Git/GitHub,  BitBucket,  …) •Jenkins  takes  Pipeline  from  Jenkinsfilein  SCM •Jenkins  creates  separate  subtasks  for  branches •Builds  and  reports  results  separately •[COMING  SOON] – Pull-­‐request  builds •(+)  Same  thing  for  GitHub and BitBucket orgs •1  single  spec  in  Jenkins  for  the  many  projects Pipeline  Features.  Multi-­‐Branch 51
52. @oleg_nenashev,  @linuxwochen 2.0 April  29,  2016 Multi-­‐Branch  Pipeline.  Example Jobs  for  branches 52
53. @oleg_nenashev,  @linuxwochen 2.0 April  29,  2016 Multi-­‐Branch  Pipeline.  Example 53
54. @oleg_nenashev,  @linuxwochen 2.0 April  29,  2016 •load()  – loading  of  Groovy  code  from  a  file •CPS  Global  library •Comes  as  a  part  of  Pipeline  plugin  set •Creates  Git repo  within  Jenkins •Defines  global  variables  and  classes •Pipeline  Remote  Loader • https://github.com/jenkinsci/workflow-­‐remote-­‐loader-­‐plugin •Loads  Pipeline  from  Git/GitHub •The  plugin  logic  is  implemented  as  Pipeline Code  sharing  between  Pipelines 54
55. @oleg_nenashev,  @linuxwochen 2.0 April  29,  2016 Pipeline  Remote  Loader.  Example stage  'Load  files  from  GitHub' def environment,  helloworld fileLoader.withGit ('https://github.com/jenkinsci/workflow-­‐remote-­‐loader-­‐plugin.git',   'master',  null,  '')   {   helloworld = fileLoader.load('examples/fileLoader/helloworld');   environment  = fileLoader.load('examples/fileLoader/environment');   }   stage  'Run  methods  from  the  loaded  content' helloworld.printHello()   environment.dumpEnvVars() 55
56. @oleg_nenashev,  @linuxwochen 2.0 April  29,  2016 •More  plugin  integrations •Pipeline  Stage  view •Pipeline  syntax  highlighting  and  docs  in IDE •Graphical  editor  of  Jenkins  pipelines Coming  soon… 56
57. @oleg_nenashev,  @linuxwochen 2.0 April  29,  2016 •«Pretty»  integration  for  many  plugins •Compatibility  list:   •http://bit.ly/pipeline-­‐compatibility •If  you  need  other  steps… Compatibility Snippet  generator  is  your  friend 57
58. @oleg_nenashev,  @linuxwochen 2.0 April  29,  2016 •Syntax  highlighting  and  code  completion •IntellijIDEA  – GDSL  -­‐ works •Eclipse  DSLD  – PoC – almost  works •Built-­‐in  documentation  -­‐ TODO •Context  help  – TODO IDE  Integration 58
59. @oleg_nenashev,  @linuxwochen 2.0 April  29,  2016 •Representation  of  parallel  tasks •Representation  of  stages •Fine-­‐grained  step  layout Pipeline  Viewer  Plugin 59
60. @oleg_nenashev,  @linuxwochen 2.0 April  29,  2016 Pipeline  Editor  Plugin https://github.com/jenkinsci/pipeline-­‐editor-­‐plugin 60
61. @oleg_nenashev,  @linuxwochen 2.0 April  29,  2016 •Scalability  and  Availability •Database  storage  option •User  experience •Continuous  delivery.  More  stuff •Extension  of  Pipeline  Ecosystem Beyound 2.0,  Q4  2016.  Sketch  plans 61
62. @oleg_nenashev,  @linuxwochen 2.0 April  29,  2016 •Jenkins  2.0  goes  beyond  CI •Jenkins  2.0  provides  new  features  and   better  UX  for  existing  Jenkins  users   •Minimal  compatibility  changes,  you  can   update  your  non-­‐LTS  instances  now •Major  architecture  changes  are  ahead Summary 62
63. @oleg_nenashev,  @linuxwochen 2.0 April  29,  2016 •Core  &  Plugins •jenkins-­‐ci.org •Documentation •Solution  pages •Blogs •Conferences  and  Jenkins  Area  Meetups •Infrastructure Contribute 63
64. @oleg_nenashev,  @linuxwochen 2.0 April  29,  2016 •Jenkins  website:  http://jenkins.io •Jenkins  2.0:  https://jenkins-­‐ci.org/2.0/ •Pipeline  tutorial: •https://github.com/jenkinsci/workflow-­‐ plugin/blob/master/TUTORIAL.md •Need  help? •#jenkinson  Freenode Links 64
65. @oleg_nenashev,  @linuxwochen 2.0 April  29,  2016 Q  &  A Contacts: • E-­‐mail:  o.v.nenashev@gmail.com • Gitter:  /oleg-­‐nenashev • Twitter:  @oleg_nenashev