Pipeline As Code – Kris Buytaert

Pipeline As Code – Kris Buytaert

Infrastructure as Code has been a native open source innovation happening over the past decade, with languages such as Puppet and Chef growing fast ops folks have been adopting development practices to run their infrastructures. In the mean time the operations people together with the developers ran into the limitations of traditional CI platforms where a lot of “Dirty Clicking” was the norm , creating a pipeline for a stack, or new project was a boring manual experience.

This is exactly what we tried to get rid of of ther past couple of years from an infrastructure point of view.

So allow me to introduce the solution to this, Pipeline as Code.

027edc76bf9f9c030820807f87c5dbdc?s=128

DevOpsDays Zurich

May 02, 2018
Tweet

Transcript

  1. 2.

    Kris Buytaert Kris Buytaert • I used to be a

    Dev, I used to be a Dev, • Then Became an Op Then Became an Op • CTO and Open Source Consultant CTO and Open Source Consultant @ @inuits.eu inuits.eu • Everything is a freaking DNS Everything is a freaking DNS Problem Problem • Evangelizing devops Evangelizing devops • Organiser of #devopsdays, Organiser of #devopsdays, #cfgmgmtcamp, #loadays, …. #cfgmgmtcamp, #loadays, ….
  2. 3.

    cAms cAms • Build Automation Build Automation • Test Automation

    Test Automation • Infrastructure as Code Infrastructure as Code • Monitoring as Code Monitoring as Code • Security as Code Security as Code • Pipeline as Code Pipeline as Code
  3. 4.

    Jenkins as Example Jenkins as Example • Open Source tool

    Open Source tool • Widely adopted in the Enterprise Widely adopted in the Enterprise • Plenty of plugins available Plenty of plugins available • UI Focused UI Focused
  4. 5.

    Pipelines Pipelines • Lots of them Lots of them •

    Similar ones, but not identical ones Similar ones, but not identical ones • One project = different deployment One project = different deployment targets targets • People move teams expect same People move teams expect same patterns patterns • Mostly unmaintaned Mostly unmaintaned
  5. 8.

    Scaling Pipelines Scaling Pipelines • Create a Pipeline, Create a

    Pipeline, • For job in Pipeline For job in Pipeline • Create new Job Based on OldJob Create new Job Based on OldJob • Update One Job Update One Job • Never refactor the rest Never refactor the rest
  6. 9.

    Generating Jenkins Pipelines Generating Jenkins Pipelines • Template the XML

    Template the XML • Put it in Puppet Put it in Puppet • Worked for stable pipelines Worked for stable pipelines • Kinda Kinda • XML gets rewritten on the fly XML gets rewritten on the fly
  7. 10.

    JenkinsJobBuilder JenkinsJobBuilder • First Usable Attempt First Usable Attempt •

    Python Based Python Based • Openstack Community Openstack Community • Limited Functionality Limited Functionality • Little Adoption / Openstack Little Adoption / Openstack Abandonned it Abandonned it
  8. 11.

    PipelineDSL PipelineDSL • Jenkinsfile Jenkinsfile • Inside a repo Inside

    a repo • Groovy Groovy • Limited functionality Limited functionality • Initially buggy Initially buggy • Popular for Easy tasks Popular for Easy tasks • (straight, no splits etc) (straight, no splits etc)
  9. 12.

    Jenkins Job DSL Jenkins Job DSL • Groovy Groovy •

    Flexible Flexible • Well Supported Well Supported • Suitable for more complex Pipelines Suitable for more complex Pipelines
  10. 13.

    Seedjobs Seedjobs • Groovy Groovy • Git Git • Rebuild

    jobs on Rebuild jobs on commit commit • Projects in Projects in folders folders
  11. 15.

    Example dsl project Example dsl project job("${foldername}/dashing-dashboard-build") job("${foldername}/dashing-dashboard-build") job("${foldername}/dashing-dashboard-test") job("${foldername}/dashing-dashboard-test")

    job("${foldername}/dashing-upload2repo") job("${foldername}/dashing-upload2repo") job("${foldername}/dashing-dashboard- job("${foldername}/dashing-dashboard- deploy") deploy") job("${foldername}/dashing-dashboard- job("${foldername}/dashing-dashboard- deploy-prd") deploy-prd") job("${foldername}/dashing-build") job("${foldername}/dashing-build")
  12. 16.

    Larger CI Stacks Larger CI Stacks • Generate Pipelines /

    Jobs based on Generate Pipelines / Jobs based on config files , config files , • Build libraries Build libraries • CheckoutJob CheckoutJob • DeployJob DeployJob • PackageJob PackageJob • Use Groovy / JobDSL to generate Use Groovy / JobDSL to generate PipelineDSL PipelineDSL
  13. 18.

    Problems solved Problems solved • One job per task, no

    reuse of jobs with One job per task, no reuse of jobs with different parameters different parameters • All Jobs are in sync, update one = All Jobs are in sync, update one = update all update all • Centrally managed jobs (git) Centrally managed jobs (git)
  14. 20.

    Contact Contact Kris Buytaert Kris Buytaert Kris.Buytaert@inuits.be Kris.Buytaert@inuits.be Further Reading

    Further Reading @krisbuytaert @krisbuytaert http://www.krisbuytaert.be/blog/ http://www.krisbuytaert.be/blog/ http://www.inuits.be/ http://www.inuits.be/ Inuits Inuits Essensteenweg Essensteenweg 31 31 Brasschaat Brasschaat Belgium Belgium 891.514.231 891.514.231