CircleCI Tips and Tricks

This is originally a presentation I made for an internal guild session at Moneytree, Japan.

I shared some tricks I have been using in order to maximize our experiences building out CI/CD pipelines with CircleCI.

Transcript

1. CircleCI ‘Tips & Tricks’ Things I’ve learned while fumbling around CircleCI. Kelvin Tay (ktay@moneytree.jp)
2. Disclaimer: I’m not a CircleCI expert! There may be better ways to tweak your CircleCI setup :) Also, we may switch to using GitHub Actions?
3. Anatomy of a CircleCI config Visualizing the body parts of a configuration workflow Job Job Job Command Commands in a job share same execution context 🐳 Command Job Orbs Executors
4. Caches, Workspaces, Artifacts? Connecting the dots jobs
5. Party Tricks How to 1. visualize the workflow 2. check your config 3. inspect tools available 4. keep calm while waiting on CI build 5. keep things dry like biscuits a. Parameters b. Matrix builds c. Orbs 6. take the road less travelled a. Scheduled Workflows b. Tag Workflows 7. do Serverless with CircleCI
6. How to Visualize the CI workflow Try https://circle.mikul.as/ [redacted] Example 1 [redacted] Example 2
7. How to Check & Test your Config Use the CircleCI local CLI, Luke! # validate .circleci/config.yml $ circleci config validate # run a local job $ circleci config process .circleci/config.yml > process.yml $ circleci local execute -c process.yml --job ruby_test-2_7 $ circleci local execute -c process.yml --job node_test-14
8. Hold on! (Caveats) - Not available for machine executor (also, Docker must be installed on local machine) - Steps like save_cache or persist_to_workspace are skipped - Need to inject encrypted environment variables (e.g., contexts) 👉 https://circleci.com/docs/2.0/local-cli/#limitations-of-running-jobs-locally
9. How to Inspect Tools available CircleCI images are publicly available 🐳 # check out $ docker run cimg/base:2020.01 jq --version $ docker run cimg/ruby:2.5 which openssl $ docker run circleci/ruby:2.5 curl https://example.com
10. Quick Tip! CircleCI’s newer images are hosted under the cimg organization in Docker Hub instead So use cimg/ruby rather than circleci/ruby for a quicker build*, for example. https://circleci.com/blog/announcing-our-next-generation-convenience-images-smaller-faster-more-deterministic/
11. How to Keep Calm and wait on the CI build Browser Notification while you brew that coffee ☕
12. How to Keep your config.yml DRY Use parameters like function arguments # parameters can be declared in both # commands and jobs! commands: deploy_xxx: parameters: region: type: string steps: - run: name: deploy XXX on << parameters.username >> command: | AWS_REGION=<< parameters.region >> aws ... jobs: deploy: steps: - deploy_xxx: region: ap-northeast-1 # JP - deploy_xxx: region: ap-southeast-2 # AU
13. Configuring Parameters - can set defaults - various types available: - string - boolean - enum - executor - env_var_name - steps (yes, even steps! think pre / post hooks) 👉https://circleci.com/docs/2.0/reusing-config/#using-the-parameters-declaration
14. Step Parameters Useful for pre / post hooks! commands: deploy_xxx: parameters: region: type: string post_deploy: type: steps defaults: [] steps: - run: name: deploy XXX on << parameters.username >> command: | AWS_REGION=<< parameters.region >> aws ... jobs: deploy: steps: - deploy_xxx: region: ap-northeast-1 # JP post_deploy: - run: echo “Quick! Monitor this service!” - deploy_xxx: region: ap-southeast-2 # AU
15. How to Keep your config.yml DRY Utilize matrix builds. 👍 - libraries testing against language versions [link redacted] - Independent and yet, similar jobs workflows: build_test_deploy jobs: - unit_test - deploy: matrix: parameters: client: [A, B, C] region: [jp, au] exclude: # client A is AU only - client : A region: jp # client C is JP only - client : C region: au
16. Trade-offs for Matrix Builds Consider time to spin up runtime context vs DRYable job / command’s time to complete! Parallel jobs are unlikely to be run exactly at same time; expect some delays. Scenario: deploying multiple Lambda functions on multiple AWS regions: Matrix ✔: Time to spin up execution: 2~3 mins Time to deploy Lambda per region: 5 minutes Matrix ❌: Time to spin up execution: 2~3 mins Time to deploy Lambda per region: 5 secs
17. How to Keep your config.yml DRY Orbs: shareable, reusable across projects ‘Dependencies’ / ‘packages’ you can import - {Language} Orb: - install deps incl. caching - run tests - e.g., ruby/rspec-test - Slack Orb: - notify on Slack - AWS CLI Orb: 🎖 - setup AWS credentials for AWS CLI - install AWS CLI - Serverless Framework Orb: - Install Serverless CLI - deployment
18. How to do Scheduled Workflow - Nightly builds / releases - Smoke tests workflows: schedule_sns_certificates_check: triggers: - schedule: # 00:00 UTC (09:00 JST) every 1st of the month cron: "0 0 1 * *" filters: branches: only: - master jobs: - check_expiring_certificates: matrix: parameters: country: ['au', 'jp'] stage: ['development', 'staging', 'production']
19. How to do Tag Workflow - libraries (tag -> version release) workflows: release: jobs: - document: filters: tags: only: /^v\d+\.\d+\.\d+/ # strictly on tag events only branches: ignore: /.*/ - release_artifacts: requires: - document filters: tags: only: /^v\d+\.\d+\.\d+/ # strictly on tag events only branches: ignore: /.*/
20. Case for Tagging: Moneytree’s private Ruby gem https://circleci.com/blog/publishing-to-github-relea ses-via-circleci/
21. How to do Serverless? ⚠ More about Serverless framework than CircleCI Context: 1. AWS Lambdas run on specific Linux a. https://hub.docker.com/_/amazonlinux/ 2. Native dependencies / extensions thus should be compiled against this runtime environment. 3. Serverless uses LambCI images for invoking local functions for AWS a. LambCI images mirror AWS Linux 4. Serverless package plugins also uses LambCI images to install dependencies a. Python b. Ruby
22. How to do Serverless? Challenges faced for SmartData: Packaging and Deploying for our AWS Lambdas (Ruby) via Serverless Challenge: 1. CircleCI base images differ from AWS Linux 2. Native extensions compiled in CircleCI build may not work a. e.g., mt-slogger’s ruby requires oj gem which has bindings on C
23. How to do Serverless! Previous Solution: - used machine build on CircleCI Steps: 1. Switched to machine build 2. Install all dependencies manually a. Ruby b. Node, and thus Serverless c. Python, and thus AWS CLI
24. 15 mins for deploy job not scalable in future: - support deploy multi-client - multi-region?
25. How we deployed for Serverless (Ruby) New Solution: Using LambCI images as executor for deploy job https://github.com/moneytree/mt-engage/pull/190 orbs: aws-cli: circleci/aws-cli@1.0.0 node: circleci/node@2.1.1 jobs: ... deploy_serverless: docker: image: lambci/build-ruby:2.5 environment: # compile extensions in this env # else, plugin installs deps via docker! CROSS_COMPILE_EXTENSIONS: false steps: - checkout - attach_to_workspace: at: ~/.aws # AWS setup via previous job - node/install - node/install-packages # Serverless - run: command: | serverless deploy --profile … --region …
26. 15 5 mins for deploy job!
27. Have fun automating builds! Also, come join SmartData! 😛
28. Other Useful Links / References - https://academy.circleci.com/ - https://circleci.com/blog - https://github.com/circleCI-Public/ - https://hub.docker.com/u/cimg OR https://hub.docker.com/u/circleci