Create your CI/CD environment with Google Cloud Build

Jenkins, Travis, ... there are a lot of tools to create your own CI environment and public clouds like Google Cloud or AWS provide services to try to do it easily. You can always use a virtual machine with the third party tool you want, but there are more options.

This talk is about my experience with Google Cloud Build after using it as a CI/CD environment at Koliseo.

Transcript

1. MAD · NOV 23-24 · 2018 Create your CI/CD environment with Google Cloud Build Carlos Coloma (@ccescribano) Developer, otaku, addicted to sports. MAD · NOV 23-24 · 2018
2. MAD · NOV 23-24 · 2018 Who am I? ▪ Google Certified Cloud Architect ▪ Google Cloud Platform and Spring Framework trainer ▪ Mr Wolf at Extrema Sistemas ∘ Other people call me “full stack” and stuff ▪ I am NOT considered an expert on CI/CD ∘ I know things, and I share things. Just that.
3. MAD · NOV 23-24 · 2018 What this talk is NOT about ▪ Google Cloud marketing ▪ Advanced tips for CI/CD architectures ▪ A containers talk ∘ We will rather focus on Cloud Build itself ▪ A Happy API/Documentation Reading Session
4. MAD · NOV 23-24 · 2018 What this talk IS about ▪ Basic concepts to start a real CI/CD system ▪ Strengths of the Cloud Build service ▪ Things to consider when designing a CI/CD system And in general, my own experience
5. MAD · NOV 23-24 · 2018 Life with CI/CD is clean and simple
6. MAD · NOV 23-24 · 2018 Theory (a.k.a. The World of Unicorns and Rainbows) ▪ Build ▪ Test ▪ Deploy (optional) ▪ Profit!
7. MAD · NOV 23-24 · 2018 Real world ▪ Different programming languages in your pipeline ▪ Runtime versions (node, JVM, python) ▪ Source code hosting ▪ Deployment target: On-prem, IaaS, Containers, PaaS, FaaS ▪ Environments: test, qa, prod ▪ Security: Authentication and authorization ▪ CI/CD error notifications
8. MAD · NOV 23-24 · 2018 Building a pipeline is like trying to drink lemonade through your eyes It doesn’t matter the CI/CD system you use Photo by Nery Montenegro on Unsplash
9. MAD · NOV 23-24 · 2018 Our goal is to look for the way that hurts less* *Also known as the Less Shitty Way
10. MAD · NOV 23-24 · 2018 Let’s start Hello World
11. MAD · NOV 23-24 · 2018 Hello world project 1. Source on GitHub 2. Google Cloud Build config ▪ Dockerfile ▪ cloudbuild.yaml (see next slides) 3. A trigger listening to changes in master
12. MAD · NOV 23-24 · 2018 Google Cloud Build YAML
13. MAD · NOV 23-24 · 2018 Steps ▪ Independent containers ▪ Share code using /workspace ▪ Basic attributes ∘ Name ⇒ Cloud Builder ⇒ Container image to use ∘ Args ∘ Env ∘ Entrypoint ▪ Example steps: - name: ‘gcr.io/cloud-builders/npm’ args: [‘install’]
14. MAD · NOV 23-24 · 2018 Additional config ▪ Timeout ▪ Disk size ▪ Machine type ▪ Substitutions ▪ Images ▪ Secrets ▪ Artifacts
15. MAD · NOV 23-24 · 2018 Additional config ▪ Timeout ▪ Disk size ▪ Machine type ▪ Substitutions ▪ Images ▪ Secrets ▪ Artifacts
16. MAD · NOV 23-24 · 2018 Real world requirements
17. MAD · NOV 23-24 · 2018 Real world requirements ▪ Security ▪ Notifications ▪ Environments ▪ Secrets ▪ Custom Cloud Builders
18. MAD · NOV 23-24 · 2018 Security ▪ Authentication ∘ Cloud Build uses a Service Account ∘ <id>@cloudbuild.gserviceaccount.com · It rotates keys automagically ∘ Maybe we don’t have to configure anything ▪ Authorization ∘ Google Cloud IAM ∘ Granular control
19. MAD · NOV 23-24 · 2018 Real world requirements ▪ Security ▪ Notifications ▪ Environments ▪ Secrets ▪ Custom Cloud Builders
20. MAD · NOV 23-24 · 2018 Notifications ▪ There is a Pub/Sub topic for Cloud Build ▪ Dispatches multiple types of notifications ▪ A subscriber can listen to these events ∘ You can use Cloud Functions to implement it
21. MAD · NOV 23-24 · 2018 Real world requirements ▪ Security ▪ Notifications ▪ Environments ▪ Secrets ▪ Custom Cloud Builders
22. MAD · NOV 23-24 · 2018 ▪ Development, CI, sandbox, production ▪ The difference is in the config files and the target infrastructure ▪ We can use ∘ Environment variables ∘ Substitutions Environments
23. MAD · NOV 23-24 · 2018 Environment variables ▪ steps: - name: ‘ubuntu’ entrypoint: ‘bash’ args: [‘./my-build-script’] env: [‘MY_ENVIRONMENT_VARIABLE=helloworld’] ▪ Later, the script will resolve the environment variable
24. MAD · NOV 23-24 · 2018 Substitutions ▪ steps: - name: ‘gcr.io/cloud-builders/gcloud’ args: [‘functions’, ‘deploy’, ‘${_NAME}’, ‘--runtime’, ‘nodejs8’, ‘--trigger-http’] substitutions: _NAME: “helloworld-sandbox” #default value ▪ gcloud builds submit --config=cloudbuild.yaml --substitutions=_NAME=helloworld-prod
25. MAD · NOV 23-24 · 2018 Real world requirements ▪ Security ▪ Notifications ▪ Environments ▪ Secrets ▪ Custom Cloud Builders
26. MAD · NOV 23-24 · 2018 Secrets ▪ Store sensitive info: certificates, user/pass pairs, etc ∘ Do it on your own, from scratch ∘ Third party systems like Vault ∘ KMS: A system on Google Cloud designed to keep secrets
27. MAD · NOV 23-24 · 2018 Secrets
28. MAD · NOV 23-24 · 2018 Secrets
29. MAD · NOV 23-24 · 2018 Using Secrets steps: - name: ‘gcr.io/cloud-builders/gradle’ entrypoint: ‘bash’ args: [‘-c’, ‘gradle install -Duser=foo -Dpassword=$$PASSWORD’] secretEnv: [‘PASSWORD’] secrets: kmsKeyName: <KMS key to decrypt the PASSWORD> secretEnv: PASSWORD: <base64-encoded encrypted secret>
30. MAD · NOV 23-24 · 2018 Using Secrets steps: - name: ‘gcr.io/cloud-builders/gradle’ entrypoint: ‘bash’ args: [‘-c’, ‘gradle install -Duser=foo -Dpassword=$$PASSWORD’] secretEnv: [‘PASSWORD’] secrets: kmsKeyName: <KMS key to decrypt the PASSWORD> secretEnv: PASSWORD: <base64-encoded encrypted secret>
31. MAD · NOV 23-24 · 2018
32. MAD · NOV 23-24 · 2018 Real world requirements ▪ Security ▪ Notifications ▪ Environments ▪ Secrets ▪ Custom Cloud Builders
33. MAD · NOV 23-24 · 2018 Custom Cloud Builders ▪ steps: - name: ‘gcr.io/cloud-builders/npm’ args: [‘install’] ▪ Tons of Cloud Builders available ∘ 16 provided by Google Cloud: go, npm, gradle, yarn... ∘ +35 provided by the community: terraform, skaffold, scala, firebase... ∘ You can use any container image
34. MAD · NOV 23-24 · 2018 Custom Cloud Builders ▪ If no standard builder fits the bill, you can create your own. ▪ Create a Docker container image with everything you want ∘ That’s just a Dockerfile. Google doesn’t do anything here. ▪ Upload it to any registry ∘ gcloud builds submit . --tag ‘gcr.io/<PROJECTID>/<IMAGEID>’ ▪ Use it in your cloudbuild.yaml!
35. MAD · NOV 23-24 · 2018 How is Cloud Build different from managing my own IaaS CI/CD environment? Remember, this is SaaS
36. MAD · NOV 23-24 · 2018 What if my CI/CD environment is down? ▪ Travis, Jenkins, even Google Cloud Build can be down when you need them ▪ You should be able to deploy, even when the CI system is unavailable ▪ Use Cloud Build without the Cloud Build Service ∘ That was a serious sentence. In all seriousness.
37. MAD · NOV 23-24 · 2018 What if my CI/CD environment is down? ▪ Thanks to docker you can launch the CI pipeline in a local machine ∘ Remember, steps are containers ▪ What you will need: ∘ Docker ∘ The Google Cloud SDK installed, incl. the cloud-build-local component ∘ A user with the same permissions of the Cloud Build Service Account ▪ To launch your pipeline in your local environment: ∘ cloud-build-local --dryrun=false --substitutions=__ENV=sandbox .
38. MAD · NOV 23-24 · 2018 Cloud Build can take a long time to build “My CI/CD pipeline is fast” - said noone, ever
39. MAD · NOV 23-24 · 2018 Speeding up your builds ▪ Check your machine size ∘ You can use machineType and diskSizeGb in cloudbuild.yaml to scale your system up ▪ Create custom, more efficient builders ▪ Run steps concurrently (see next slide) ▪ Cache containers (not covered) ▪ Cache directories (not covered)
40. MAD · NOV 23-24 · 2018 Run steps concurrently Sequential steps steps: - name: foo ... - name: bar ... - name: baz ... Concurrent steps steps: - name: foo id: A ... - name: bar waitFor: [‘A’] ... - name: baz waitFor: [‘A’] ...
41. MAD · NOV 23-24 · 2018 Wrap-up
42. MAD · NOV 23-24 · 2018 Google Cloud Build ▪ Google Cloud Build is a powerful tool, but starting a CI pipeline from scratch is hard ▪ SaaS means that you don’t maintain any server ▪ Every step is a container, even if the final artifact does not need to be a container image ▪ Can be run locally ▪ Security can be configured through IAM and KMS ▪ Free tier: 120 minutes per day, free of cost
43. MAD · NOV 23-24 · 2018 Create your CI/CD system with Google Cloud Build Carlos Coloma (@ccescribano) Developer, otaku, addicted to sports. MAD · NOV 23-24 · 2018 Thanks