St.  Petersburg   Jenkins  Meetup @jenkins_spb Jenkins 2.0  and Pipeline-­‐as-­‐Code.   What’s  going  on? St.  Petersburg,  March  10th,2016 Oleg  Nenashev  (@oleg_nenashev) CloudBees,  Inc.

@oleg_nenashev,  @jenkins_spb 2.0 March  10,  2016 >  whoami – Oleg  Nenashev •Jenkins  OSS  contributor •Core  &  Security teams  member •Plugin  Developer  and  Maintainer •Developer  at CloudBees •CloudBees  Jenkins  Platform •Background: •R&D, HW/Embedded  SW •Intel,  Sitronics,  Synopsys •Large-­‐scale  automation  infrastructure  lead @oleg_nenashev oleg-­‐nenashev 2

@oleg_nenashev,  @jenkins_spb 2.0 March  10,  2016 •Hudson/Jenkins   – 8  years •Open-­‐source  contribution  – 4  years • My  plugins:  Ownership,  Job  Restrictions, … • Legacy:  Role  Strategy,  Custom  Tools,  Promoted  Builds • Core  &  Security  teams  in Jenkins •Meetup  organizer: • http://www.meetup.com/St-­‐Petersburg-­‐Jenkins-­‐Meetup/ •Jenkins  Org  Admin  in Google  Summer  of  Code  2016 >  whoami –v  jenkins –m  oss 3

@oleg_nenashev,  @jenkins_spb 2.0 March  10,  2016 Disclaimer • This  presentation  is  not  a  training  material • The  talk  represents  the  speaker’s   personal  opinion • This  opinion  may  differ  from  the  official  position  of   CloudBees  or  Jenkins  community 4

2.0 Previously… WHY  JENKINS  2.0? https://events.yandex.ru/lib/talks/3302/ “Who  is  mr. Jenkins?”

@oleg_nenashev,  @jenkins_spb 2.0 March  10,  2016 Automation  Systems  Evolvement  Trends •SW  area  shifts  focus  from  CI: •Continuous  Delivery •DevOps •Many  integrations  =>  communities  and  OSS •Centralization  of  automation  systems •High  requirements  and  strict  SLAs •Commercial  solutions 6

@oleg_nenashev,  @jenkins_spb 2.0 March  10,  2016 User  experience  issues Where  are  the  docs? How  to  find  a  plugin? Is  this  plugin  alive? How  to  configure  THIS? Why  do  they  ignore  my  issues? Where  is  the  maintainer? Why  the  UI  is  so  complicated? How  to  implement  it? 7

@oleg_nenashev,  @jenkins_spb 2.0 March  10,  2016 Jenkins  Learning  Curve Basic  scripts,   CRON  with  UI Automation   flows  based   on  plugins Advanced  features, Custom  Plugins 8

@oleg_nenashev,  @jenkins_spb 2.0 March  10,  2016 Jenkins  weaknesses.  Limited  scalability Node  1 Node  2 Node  N .  .  . • Master  handles  the  most  of  the  build  logic • Jobs  on  nodes  ALWAYS load  the  master • Input/Output on  master  is  also  a  bottleneck Multi-­‐ tenancy ??? Gearman Plugin  (OSS) CloudBees  Jenkins  Platform 9

@oleg_nenashev,  @jenkins_spb 2.0 March  10,  2016 Jenkins  weaknesses.  Compatibility  costs • More  than  5  years  of  backward  compatibility • Hudson  2  plugins  still  work  on  Jenkins! • BUT: • Architecture  design  issues,  complex  APIs • Outdated  libs • Outdated  UI • Plugin  integration  issues 10

2.0 Jenkins  2.0.  Prehistory

@oleg_nenashev,  @jenkins_spb 2.0 March  10,  2016 1  April,  2010  – Hudson  2.0  Announcement Hudson 2.0 is expected to follow in the intrepid footsteps of other major revamps such as Netscape 5 and Perl 6 in its ambitiousness and innovation. Implementation of core in portable C for greater speed and code readability Revamped storage architecture around the Oracle 11g Database to provide more optimal synergy over flat XML files. https://jenkins-­‐ci.org/blog/2010/04/01/announcing-­‐the-­‐hudson-­‐2-­‐0-­‐roadmap/ Perl  6,  2016 Hudson  2.0,  ? 12

@oleg_nenashev,  @jenkins_spb 2.0 March  10,  2016 •…  don’t  want  to  spend  4  years  on  2.0 •… want  to  keep  the  backward  compatibility •…  want  to  support  major  architecture   changes  in  the  nearest  future •…  focus  on  UX  and CD In Jenkins  2.0 we… 13

@oleg_nenashev,  @jenkins_spb 2.0 March  10,  2016 •2015  – Discussions  at  Jenkins  Office  Hours   •October  2015  -­‐ Jenkins  2.0  scope  proposals   •https://wiki.jenkins-­‐ ci.org/display/JENKINS/Jenkins+2.0 Jenkins  2.0.  Key  Events 14

@oleg_nenashev,  @jenkins_spb 2.0 March  10,  2016 •SCALEx16  and  FOSDEM2016  (Jan  30-­‐31) •First  2.0  Demos •Face-­‐to-­‐face  discussions  with  Jenkins  users •Jenkins  2.0  Contributor  Summit (1  февраля) • >20  Jenkins  Core  developers •Meetup:  http://bit.ly/jenkins2-­‐summit •Notes:  http://bit.ly/jenkins2-­‐summit-­‐notes •New  Plans •Review  of  Jenkins  2.x  improvements Jenkins  2.0.  Key  Events 15

@oleg_nenashev,  @jenkins_spb 2.0 March  10,  2016 •Jenkins  is  accepted  to  Google  Summer  of  Code •http://bit.ly/jenkins-­‐gsoc-­‐wiki •Main  Focus  – Jenkins  2.0 and UX •9  project  ideas,  15  mentors Key  Events @Students:  Student  applications  should  be   submitted  till  March 24th! Contact  the   community: http://bit.ly/jenkins-­‐gsoc-­‐apply 16

@oleg_nenashev,  @jenkins_spb 2.0 March  10,  2016 Initial  plans,  2015 Jenkins  1.x “Jenkins  N.x” Jenkins  2.x Middle  of  2016 End  of  2016  (?) • Website • Focus  on  UX  and  project  infra • Minimal compatibility  changes • Major  architectural   changes • At  least  1  LTS  line  in  2016 17

@oleg_nenashev,  @jenkins_spb 2.0 March  10,  2016 •In Jenkins  2.0  we  deliver  the  minimal  required   set  of  changes •We  do  not  break the  binary  compatibility  in   Jenkins  2.0 •All  Jenkins  1.x  plugins  should  work •We  continue  working  on  features  in  Jenkins  2.x New  plan,  Feb  2016 18

@oleg_nenashev,  @jenkins_spb 2.0 March  10,  2016 •UX  improvements: •“Plugin  gallery” •New  installation  wizard •WebUI improvements  (config pages,  etc.) •WebUI library  updates •Security  will  be  enabled  by  default •Pipeline-­‐as-­‐Code •API  deprecation  engine  (postponed  a  bit) Jenkins  2.0. What’s  inside  the  core? http://bit.ly/jenkins20 19

@oleg_nenashev,  @jenkins_spb 2.0 March  10,  2016 •February  29:  alpha  version  =>  DONE •March  16:   •"feature  complete” •beta  version •wide  testing •March  23:   •Release  candidate •jenkins.io  launch   •March  30:  (or  later)  – Release Current  plans 20

@oleg_nenashev,  @jenkins_spb 2.0 March  10,  2016 •Branches  are  being  integrated  now,  stay  tuned! •Regular  alpha  version  releases •https://jenkins-­‐ci.org/2.0/ Jenkins  2.0.  Alpha  version Overview  and  download  page 21

2.0 Jenkins  2.0  website

@oleg_nenashev,  @jenkins_spb 2.0 March  10,  2016 •New  domain:  http://jenkins.io •Static  web-­‐site,  sources  are  on  GitHub ⇒Easy  content  management  (CD!) ⇒Everybody  can  contribute  a  pull-­‐request •Content  rework •Structured  info •MOAR  information •Website  goes  beyond  Continuous  Integration Jenkins  2.0.  Website http://bit.ly/jenkins20 23

@oleg_nenashev,  @jenkins_spb 2.0 March  10,  2016 •Static  website  – DONE •Domain  and  new  layout  – DONE •Localization  -­‐ TODO •Content  rework  – Work-­‐in-­‐progress Status 24

@oleg_nenashev,  @jenkins_spb 2.0 March  10,  2016 New  website  (http://jenkins.io) 25

Editing  in  GitHub:   Markdown,  Asciidoc,   HAML Various  sidebar   panels  for   better   navigation

Solution  pages

@oleg_nenashev,  @jenkins_spb 2.0 March  10,  2016 Wishlist •More  content •Solution  pages •CloudBees  moves  Jenkins  Handbook  to OSS •Centralized  documentation  for  plugins •«Plugin  gallery»   28

@oleg_nenashev,  @jenkins_spb 2.0 March  10,  2016 •Focus  -­‐ Out  of  the  Box  Experience •Installation  wizard,  plugin   management,  configuration  pages Jenkins  2.0  UI 29

@oleg_nenashev,  @jenkins_spb 2.0 March  10,  2016 Setup  Wizard 30

@oleg_nenashev,  @jenkins_spb 2.0 March  10,  2016 Plugin  selection  in  alpha-­‐2 On  hover  – popup  with   the  dependency  list 31

@oleg_nenashev,  @jenkins_spb 2.0 March  10,  2016 Update  Manager  2.0.  Concept 32

@oleg_nenashev,  @jenkins_spb 2.0 March  10,  2016 Job  configuration  pages.  Tabs Tabs Restore  the   old  view 33

2.0 Pipeline  as  code DESCRIBING   JENKINS JOBS  VIA  CODE

@oleg_nenashev,  @jenkins_spb 2.0 March  10,  2016 •Storage  along  with  the  project •Configuration  change  tracking Why  would  we  benefit  from  it? 35

@oleg_nenashev,  @jenkins_spb 2.0 March  10,  2016 Automation  as  Code Job  DSL DotCI Pipeline  (fka Workflow) Pipeline  will  be  offered  by  default  in  Jenkins  2,0 36

@oleg_nenashev,  @jenkins_spb 2.0 March  10,  2016 •Groovy  DSL  for  build  flow  description •Job  type  in  Jenkins (Pipeline  Job) •Ecosystem  for  automation-­‐as-­‐code •~15  dedicated  plugins •Integration  in  many  other  plugins What  is  Pipeline? 37

@oleg_nenashev,  @jenkins_spb 2.0 March  10,  2016 Pipeline.  Example  #1 • Syntax:  Groovy-­‐based  DSL • Job  descriptions  can  be  stored  in  SCM • E.g.  Jenkinsfile in  the  repo  root • http://bit.ly/pipeline-­‐tutorial 38

@oleg_nenashev,  @jenkins_spb 2.0 March  10,  2016 Jenkinsfile example 39

@oleg_nenashev,  @jenkins_spb 2.0 March  10,  2016 Pipeline.  Available  steps • Build  steps  and   publishers  from  plugins • Core  steps  (file   operations,  etc.) • Extra  utility  steps 40

@oleg_nenashev,  @jenkins_spb 2.0 March  10,  2016 •Snippet  generator  UI •User  sets  up  settings  in  UI •Jenkins  generates  Pipeline  command  for  copy&paste Pipeline  features 41

@oleg_nenashev,  @jenkins_spb 2.0 March  10,  2016 •Parallelization •parallel()  – running  of  subscripts  in  parallel •No  Freestyle  job  triggering  spaghetti  anymore •Multiple  node()  operations  in  a  job •Parallel  runs  on  different  nodes  are  supported •Multiple  scm()  usages  in  a  single  job •Interactive  steps  (user  input  via  for) •You  can  involve  manual  QA  into  your  CD! Pipeline  features 42

@oleg_nenashev,  @jenkins_spb 2.0 March  10,  2016 •Robustness  against  master  restarts •Pipeline  stores  context  and  stores  jobs  after  the   restart •Robustness  against  network  issues •We  use  Durable  Task Plugin •sh()  call  won’t  fail  if  network  fails •Security: •Groovy  Sandbox  (Script  Security  Plugin) •Every  call  from  Pipeline  is  validated  (whitelists) Pipeline  features 43

Pipeline.  Example  #2.  Jenkins  IRC  bot  CD #!groovy def imageName =  'jenkinsciinfra/ircbot’ node('docker')   { checkout scm //  Some  magic  for  getting  Docker image  tag sh 'git rev-­‐parse HEAD  >  GIT_COMMIT' shortCommit =  readFile('GIT_COMMIT').take(6) def imageTag =  "build${shortCommit}" stage 'Build ircbot' withMavenEnv (["BUILD_NUMBER=${env.BUILD_NUMBER}:${shortCommit}"])   { sh 'make bot'  //  Make  calls  Maven,  which  calls  Ant } stage 'Build container' def whale =  docker.build("${imageName}:${imageTag}”) stage 'Deploy container' whale.push() } https://github.com/jenkins-­‐infra/ircbot Temporary  solution

@oleg_nenashev,  @jenkins_spb 2.0 March  10,  2016 Pipeline:  More  examples https://github.com/jenkinsci/ pipeline-­‐examples 45

@oleg_nenashev,  @jenkins_spb 2.0 March  10,  2016 •Multi-­‐Branch  Pipeline  Plugin •Job  configuration  only  contains  reference  to  the   SCM (Git/GitHub,  BitBucket,  …) •Jenkins  takes  Pipeline  from  Jenkinsfile in  SCM •Jenkins  creates  separate  subtasks  for  branches,  builds   and  reports  results  separately •[COMING  SOON] – Pull-­‐request  builds •(+)  Same  thing  for  GitHub and BitBucket orgs •1  single  spec  in  Jenkins  for  the  many  projects Pipeline  Features.  Multi-­‐Branch 46

@oleg_nenashev,  @jenkins_spb 2.0 March  10,  2016 Multibranch Pipeline.   Example 47

@oleg_nenashev,  @jenkins_spb 2.0 March  10,  2016 •load()  – loading  of  Groovy  code  from  a  file •CPS  Global  library •Comes  as  a  part  of  Pipeline  plugin  set •Creates  Git repo  within  Jenkins •Defines  global  variables  and  classes •Pipeline  Remote  Loader • https://github.com/jenkinsci/workflow-­‐remote-­‐loader-­‐plugin •Loads  Pipeline  from  Git/GitHub •The  plugin  logic  is  implemented  as  Pipeline Code  sharing  between  Pipelines 48

@oleg_nenashev,  @jenkins_spb 2.0 March  10,  2016 Pipeline  Remote  Loader.  Example stage  'Load  files  from  GitHub' def environment,  helloworld fileLoader.withGit ('https://github.com/jenkinsci/workflow-­‐remote-­‐loader-­‐plugin.git',   'master',  null,  '')   {   helloworld = fileLoader.load('examples/fileLoader/helloworld');   environment  = fileLoader.load('examples/fileLoader/environment');   }   stage  'Run  methods  from  the  loaded  content' helloworld.printHello()   environment.dumpEnvVars() 49

@oleg_nenashev,  @jenkins_spb 2.0 March  10,  2016 •More  plugin  integrations •Pipeline  Stage  view •Pipeline  syntax  highlighting  and  docs  in IDE •Graphical  editor  of  Jenkins  pipelines Coming  soon… 50

@oleg_nenashev,  @jenkins_spb 2.0 March  10,  2016 •«Pretty»  integration  for  many  plugins •Compatibility  list:   •http://bit.ly/pipeline-­‐compatibility •If  you  need  other  steps… Compatibility Snippet  generator  is  your  friend 51

@oleg_nenashev,  @jenkins_spb 2.0 March  10,  2016 • CloudBees  moves  the  project  to  OSS • Beta  version  is  available  in  the  experimental  update  center Pipeline   Stage  View • Stage() • Separate  logs  for   stages • Good  UI  for   interactive  steps 52

@oleg_nenashev,  @jenkins_spb 2.0 March  10,  2016 •Syntax  highlighting  and  code  completion •Intellij IDEA  – GDSL  -­‐ works •Eclipse  DSLD  – PoC – almost  works •Built-­‐in  documentation  -­‐ TODO •Context  help  – TODO IDE  Integration 53

@oleg_nenashev,  @jenkins_spb 2.0 March  10,  2016 Graphical  Pipeline  Editor https://github.com/jenkinsci/pipeline-­‐editor-­‐plugin 54

@oleg_nenashev,  @jenkins_spb 2.0 March  10,  2016 •Pipeline  Debugger •Multi-­‐configuration  pipelines Long-­‐term  plans… 55

@oleg_nenashev,  @jenkins_spb 2.0 March  10,  2016 •Jenkins  website:  https://jenkins-­‐ci.org •New  website  prototype:  http://jenkins.io •Jenkins  2.0:  https://jenkins-­‐ci.org/2.0/ •Russian-­‐speaking  community  resources: •Gitter:  https://gitter.im/jenkinsci-­‐ru/public •Twitter  – @jenkins_ru   •Pipeline-­‐as-­‐Code  tutorial: •https://github.com/jenkinsci/workflow-­‐ plugin/blob/master/TUTORIAL.md Links 56

@oleg_nenashev,  @jenkins_spb 2.0 March  10,  2016 Q  &  A Contacts: • E-­‐mail:  [email protected] • Gitter:  /oleg-­‐nenashev • Twitter:  @oleg_nenashev