Infrastructure as Code: Build Pipelines with Docker and Terraform

Many software projects use build pipelines including tools like Jenkins, SonarQube, Artifactory etc. But often those pipeline tools are installed and maintained manually. There are certain risks with this approach and in case of failure it often takes a long time to have a running pipeline again. This session shows how to automate the creation of a build pipeline. With Terraform, a Docker infrastructure is created at AWS, where Jenkins, SonarQube and Artifactory are pre-configured and deployed.

Transcript

1. Kai Tödter
2. Who am I?  Principal Key Expert at Siemens Building Technologies  Web Technology Fan  Open Source Lover  E-mail: kai@toedter.com  Twitter: twitter.com/kaitoedter  Blog: toedter.com/blog 4/10/2018 2 © Kai Tödter, Licensed under a Creative Commons Attribution 4.0 International License.
3. Show Hands! 4/10/2018 © Kai Tödter, Licensed under a Creative Commons Attribution 4.0 International License. 3
4. Outline  Build Pipeline with Docker  Setting up  Jenkins  SonarQube  Artifactory  Terraform  AWS Deployment 4/10/2018 © Kai Tödter, Licensed under a Creative Commons Attribution 4.0 International License. 4
5. Demo (Local Docker)  https://192.168.99.100 4/10/2018 © Kai Tödter, Licensed under a Creative Commons Attribution 4.0 International License. 5
6. How to run the Demo?  Install Docker  git clone https://github.com/toedter/cd-pipeline  cd docker  Edit the docker host in cd-pipeline.sh or .bat  When using Windows: Make sure *.sh files and plugins.txt have Unix line endings  Run cd-pipeline.sh or .bat 4/10/2018 © Kai Tödter, Licensed under a Creative Commons Attribution 4.0 International License. 6
7. DevOps CAMS  Culture  Automation  Measurement  Sharing 4/10/2018 © Kai Tödter, Licensed under a Creative Commons Attribution 4.0 International License. 7
8. 4/10/2018 © Kai Tödter, Licensed under a Creative Commons Attribution 4.0 International License. 8 Automation
9. What can be automated  Testing  Build Pipelines  Build Infrastructure  Deployments  Monitoring  Configuration  … 4/10/2018 © Kai Tödter, Licensed under a Creative Commons Attribution 4.0 International License. 9
10. Infrastructure as Code  Automation of infrastructure creation  Servers, OS, software, storage, networking, …  Stored in code repositories  Lots of tools available  SaltStack, Puppet, Chef, Terraform, Ansible, …  Infrastructure as Code is the approach “treating infrastructure like software” 4/10/2018 © Kai Tödter, Licensed under a Creative Commons Attribution 4.0 International License. 10
11. Creating a Build Pipeline  Jenkins  Continuous Integration  SonarQube  Internal Quality  Artifactory  Binary Artifact Repository 4/10/2018 © Kai Tödter, Licensed under a Creative Commons Attribution 4.0 International License. 11
12. 4/10/2018 © Kai Tödter, Licensed under a Creative Commons Attribution 4.0 International License. 12
13. Docker  Docker images for  Jenkins, Sonarqube, Artifactory, nginx  Extra container for tools configuration  Docker Compose  for defining and running the multi-container pipeline tools 4/10/2018 © Kai Tödter, Licensed under a Creative Commons Attribution 4.0 International License. 13
14. Dockerfile (nginx) FROM nginx COPY nginx-selfsigned.crt /etc/ssl/ COPY nginx-selfsigned.key /etc/ssl/ COPY index.html /usr/share/nginx/html/ COPY *.png /usr/share/nginx/html/ COPY nginx.conf /etc/nginx 4/10/2018 © Kai Tödter, Licensed under a Creative Commons Attribution 4.0 International License. 14
15. Docker Compose version: '3.2' networks: cd-tools-network: driver: bridge services: sonarqubedb: image: postgres container_name: sonarqube-db environment: - POSTGRES_USER=sonar - POSTGRES_PASSWORD=sonar volumes: - /volumes/postgres/data:/var/lib/postgresql/data 4/10/2018 © Kai Tödter, Licensed under a Creative Commons Attribution 4.0 International License. 15
16. Docker Code Deep Dive 4/10/2018 © Kai Tödter, Licensed under a Creative Commons Attribution 4.0 International License. 16
17. Pipeline as Code  Jenkins => Jenkinsfile  TracisCI => .travis.yml  Gitlab CI => .gitlab-ci.yml 4/10/2018 © Kai Tödter, Licensed under a Creative Commons Attribution 4.0 International License. 17
18. Jenkinsfile pipeline { agent { docker { image 'java:openjdk-8' args '--network=docker_cd-tools-network' } } stages { stage('build + tests') { steps { sh './gradlew test' } } … 4/10/2018 © Kai Tödter, Licensed under a Creative Commons Attribution 4.0 International License. 18
19. 4/10/2018 © Kai Tödter, Licensed under a Creative Commons Attribution 4.0 International License. 19 Terraform
20. Terraform  Write  Infrastructure as code  Plan  Preview changes before applying  Create  Always reproducible infrastructure 4/10/2018 © Kai Tödter, Licensed under a Creative Commons Attribution 4.0 International License. 20
21. Pipeline, Terraform and AWS  Create Security Groups  Configure SSH keys  Create EC2 instance  Provide scripts that are executed after creation  Output IP address 4/10/2018 © Kai Tödter, Licensed under a Creative Commons Attribution 4.0 International License. 21
22. Terraform Code Deep Dive 4/10/2018 © Kai Tödter, Licensed under a Creative Commons Attribution 4.0 International License. 22
23. Demo (AWS)  https://<ip address from terraform output> 4/10/2018 © Kai Tödter, Licensed under a Creative Commons Attribution 4.0 International License. 23
24. What’ Next?  DNS entries for all tools  Let’s Encrypt Certificates  Backup/Restore  Monitoring with Prometheus and Grafana  Support of Nexus and Teamscale  AWS Spot instances  Single & Multi host deployments 4/10/2018 © Kai Tödter, Licensed under a Creative Commons Attribution 4.0 International License. 24
25. NextGen Pipeline Demo (AWS) 4/10/2018 © Kai Tödter, Licensed under a Creative Commons Attribution 4.0 International License. 25
26. 4/10/2018 © Kai Tödter, Licensed under a Creative Commons Attribution 4.0 International License. 26 Discussion
27. Links  Github Repo: https://github.com/toedter/cd-pipeline  Terraform: https://www.terraform.io/ 4/10/2018 © Kai Tödter, Licensed under a Creative Commons Attribution 4.0 International License. 27
28. License  This work is licensed under a Creative Commons Attribution 4.0 International License.  See http://creativecommons.org/licenses/by/4.0/ 4/10/2018 © Kai Tödter, Licensed under a Creative Commons Attribution 4.0 International License. 28