Pipeline As Code – Kris Buytaert

Infrastructure as Code has been a native open source innovation happening over the past decade, with languages such as Puppet and Chef growing fast ops folks have been adopting development practices to run their infrastructures. In the mean time the operations people together with the developers ran into the limitations of traditional CI platforms where a lot of “Dirty Clicking” was the norm , creating a pipeline for a stack, or new project was a boring manual experience.

This is exactly what we tried to get rid of of ther past couple of years from an infrastructure point of view.

So allow me to introduce the solution to this, Pipeline as Code.

Transcript

1. Pipeline as Code Pipeline as Code Kris Buytaert @krisbuytaert
2. Kris Buytaert Kris Buytaert • I used to be a Dev, I used to be a Dev, • Then Became an Op Then Became an Op • CTO and Open Source Consultant CTO and Open Source Consultant @ @inuits.eu inuits.eu • Everything is a freaking DNS Everything is a freaking DNS Problem Problem • Evangelizing devops Evangelizing devops • Organiser of #devopsdays, Organiser of #devopsdays, #cfgmgmtcamp, #loadays, …. #cfgmgmtcamp, #loadays, ….
3. cAms cAms • Build Automation Build Automation • Test Automation Test Automation • Infrastructure as Code Infrastructure as Code • Monitoring as Code Monitoring as Code • Security as Code Security as Code • Pipeline as Code Pipeline as Code
4. Jenkins as Example Jenkins as Example • Open Source tool Open Source tool • Widely adopted in the Enterprise Widely adopted in the Enterprise • Plenty of plugins available Plenty of plugins available • UI Focused UI Focused
5. Pipelines Pipelines • Lots of them Lots of them • Similar ones, but not identical ones Similar ones, but not identical ones • One project = different deployment One project = different deployment targets targets • People move teams expect same People move teams expect same patterns patterns • Mostly unmaintaned Mostly unmaintaned
6. Dirty Clickers Dirty Clickers
7. Complex Pipelines Complex Pipelines
8. Scaling Pipelines Scaling Pipelines • Create a Pipeline, Create a Pipeline, • For job in Pipeline For job in Pipeline • Create new Job Based on OldJob Create new Job Based on OldJob • Update One Job Update One Job • Never refactor the rest Never refactor the rest
9. Generating Jenkins Pipelines Generating Jenkins Pipelines • Template the XML Template the XML • Put it in Puppet Put it in Puppet • Worked for stable pipelines Worked for stable pipelines • Kinda Kinda • XML gets rewritten on the fly XML gets rewritten on the fly
10. JenkinsJobBuilder JenkinsJobBuilder • First Usable Attempt First Usable Attempt • Python Based Python Based • Openstack Community Openstack Community • Limited Functionality Limited Functionality • Little Adoption / Openstack Little Adoption / Openstack Abandonned it Abandonned it
11. PipelineDSL PipelineDSL • Jenkinsfile Jenkinsfile • Inside a repo Inside a repo • Groovy Groovy • Limited functionality Limited functionality • Initially buggy Initially buggy • Popular for Easy tasks Popular for Easy tasks • (straight, no splits etc) (straight, no splits etc)
12. Jenkins Job DSL Jenkins Job DSL • Groovy Groovy • Flexible Flexible • Well Supported Well Supported • Suitable for more complex Pipelines Suitable for more complex Pipelines
13. Seedjobs Seedjobs • Groovy Groovy • Git Git • Rebuild jobs on Rebuild jobs on commit commit • Projects in Projects in folders folders
14. Example Example
15. Example dsl project Example dsl project job("${foldername}/dashing-dashboard-build") job("${foldername}/dashing-dashboard-build") job("${foldername}/dashing-dashboard-test") job("${foldername}/dashing-dashboard-test") job("${foldername}/dashing-upload2repo") job("${foldername}/dashing-upload2repo") job("${foldername}/dashing-dashboard- job("${foldername}/dashing-dashboard- deploy") deploy") job("${foldername}/dashing-dashboard- job("${foldername}/dashing-dashboard- deploy-prd") deploy-prd") job("${foldername}/dashing-build") job("${foldername}/dashing-build")
16. Larger CI Stacks Larger CI Stacks • Generate Pipelines / Jobs based on Generate Pipelines / Jobs based on config files , config files , • Build libraries Build libraries • CheckoutJob CheckoutJob • DeployJob DeployJob • PackageJob PackageJob • Use Groovy / JobDSL to generate Use Groovy / JobDSL to generate PipelineDSL PipelineDSL
17. Other Tools : Other Tools :
18. Problems solved Problems solved • One job per task, no reuse of jobs with One job per task, no reuse of jobs with different parameters different parameters • All Jobs are in sync, update one = All Jobs are in sync, update one = update all update all • Centrally managed jobs (git) Centrally managed jobs (git)
19. Stop Clicking , Write Code Stop Clicking , Write Code
20. Contact Contact Kris Buytaert Kris Buytaert Kris.Buytaert@inuits.be Kris.Buytaert@inuits.be Further Reading Further Reading @krisbuytaert @krisbuytaert http://www.krisbuytaert.be/blog/ http://www.krisbuytaert.be/blog/ http://www.inuits.be/ http://www.inuits.be/ Inuits Inuits Essensteenweg Essensteenweg 31 31 Brasschaat Brasschaat Belgium Belgium 891.514.231 891.514.231