DroneCI - Container native continuous Integration / Deployment

Drone is an open source Continuous Delivery platform which utilizes containers to automate testing and release workflows.
In this session, we will explore the core concepts of Drone as well as how one can deploy it on AWS.
Continuing onward - we will take a deep dive, on how we can leverage AWS to automatically scale Drone based on the workload.

Transcript

1. Drone 13.08.2018 Patrick Jahns Container native continuous Integration / Delivery
2. 2 • Software Engineer • ❤ Clean Code // DevOps // SRE • Solutions Architect at ownCloud patrick_jahns patrickjahns mail@patrickjahns.de
4. 4 Drone CI
5. 5 Drone CI • Open Source (https://github.com/drone) - started in 2014 • Easy to install & maintain (docker pull drone/drone) • Simple YAML Configuration (superset of docker-compose.yml) • Container native CI/CD platform (everything runs within containers) • Isolated builds ( workspace / network ) • Integrates with several VCS Providers ( Github, Gitlab, Bitbucket, Gitea … ) • Rich set of official plugins (any container can be a plugin) • Management via cli / REST Api ( but also has a WebUI ) • Execute locally with “drone exec” • Provides prometheus metrics
6. 6 Drone CI Server SERVICES WORKSPACE STEP1 git clone STEP2 make STEP3 publish Agent
7. 7 Drone - Concepts • Pipeline – Steps to be executed • Workspace – Ephemeral – Shared between individual pipeline steps • Services – Containers that are available during the pipeline execution – Yaml key == hostname of service • Conditions – Ability to specify conditions when a pipeline/step will be executed Basic Concepts Examples: https://github.com/patrickjahns/drone-examples
8. 8 Drone - Concepts Groups • Shared workspace • Run on the same Agent Matrix Builds • Completely seperated builds • Can be scheduled on different Agents Parallel builts // Concurrency Examples: https://github.com/patrickjahns/drone-examples
9. 9 Drone - Concepts • Secrets are injected as environment variables • Ability to limit secret per build event ( push / pull-request / tag ) • Ability to limit secrets to specific images/plugins • Secrets can be stored in various backends ( database / “global secrets file”* / vault* ) • For public repositories –> gated builds (changes to .drone.yml need approval ) Secrets *) requires enterprise subscription Examples: https://github.com/patrickjahns/drone-examples
10. 10 Drone - Concepts • Plugins are “just” containers => complete freedom and language agnostic • Large ecosystem of readily available plugins => http://plugins.drone.io/ • Anything missing => easy to build your own plugin – Example from ownCloud Plugins
11. 11 Drone - Concepts • Run your CI pipeline locally • Ability to debug // introspect if anything fails on the remote server • Example: Drone Exec
12. Drone@AWS
13. 13 Drone@AWS • Easy Deployment – Instance where drone server runs ( EC2 / ECS / Kubernetes ) – Database Server ( RDS ? ) – Instance(s) where drone Agents run • Various Plugins for direct interaction with AWS services available – Example: AWS Cloudformation, AWS ECR, AWS ECS, AWS Elastic Beanstalk, AWS Lambda more at http://plugins.drone.io/ – Most plugins require secret and access_key => currently not possible to use IAM Profiles
14. 14 Drone@AWS Publishing Container to ECR • plugins/ecr can publish to ECR via a ACCESS_KEY & Secret Using Images from ECR in drone pipeline • Global Registry File *) • Periodic Job Running on Agent Host Instance ( aws ecr get-login --no-include-email ) • Pipeline step with mounted docker-socket that logs into aws ecr and pulls the image *) requires enterprise subscription http://docs.drone.io/setup-global-registry-credentials/#elastic-container-registry-ecr Working with AWS ECR
15. Scaling DroneCI
16. 16 Scaling Drone - Autoscaler • CI/CD volumes are highly dependant on ”daily-business” • Fast feedback cycles are required for developer velocity • CI/CD requires compute power -> easily scaleable with cloud providers
17. 17 Scaling Drone - Autoscaler • Support for AWS, DigitalOcean, Google, HetznerCloud • Planned to support Azure, Packet.net, Scaleway • Simple service connected to Drone server • Hooked into Drone CLI, e.g. “drone server create” • Checks the Drone queue in a loop • Launch servers based on a cloud-init config • Start Drone agent via remote Docker connection (secured by TLS) • Unregister Drone agent if not needed anymore • Destroy server instance after a minimal amount of time • Provides Prometheus metrics
18. 18 Server Autoscaler Agent Agent Agent Agent Scaling Drone - Autoscaler
19. 19 Scaling Drone – Autoscaler@AWS • Requirements: – VPC, Subnet, Security Group – EC2 / ECS where autoscaler can be additionally started • Caveats – Agents either need to have a public-ip associate // or a NAT needs to be defined – It is currently limited to On Demand instances – Pull Request for Spot Instances will follow – Currently not yet possible to attach instance roles
21. 21 0 5000 10000 15000 20000 25000 30000 35000 Nov 6-Nov 11-Nov 16-Nov 21-Nov 26-Nov Dec 5-Dec 12-Dec 17-Dec 22-Dec 27-Dec Jan 5-Jan 11-Jan 16-Jan 21-Jan 26-Jan 31-Jan 4-Feb 9-Feb 14-Feb 19-Feb 24-Feb Mar 5-Mar 10-Mar 15-Mar 20-Mar 25-Mar 30-Mar 3-Apr 8-Apr 13-Apr 18-Apr 26-Apr 1-May 7-May 12-May 17-May 22-May 28-May 1-Jun 6-Jun 11-Jun 16-Jun commulated runtime time to finish time to finish (including queue wait)
22. DroneCI Outlook
23. 23 Drone - Outlook • Official support for Arm / Arm64 • Experimental support for windows containers • Support for multi-machine fan-in / fan-out • Support for multi-machine, multi-architecture • Hooks (http/REST) to override default behavior i.e. secrets, registry credentials, approval workflows – Example for AWS – Trigger a Lambda Function to fetch secrets from AWS SecretsManager / ParameterStore RoadMap / Topics for drone 0.9
24. 24 Links Slides & SourceCode • https://github.com/patrickjahns/awsmeetup-20180813-drone • https://github.com/patrickjahns/drone-examples Drone • Drone Github – https://github.com/drone • Drone documentation - http://docs.drone.io/ • Autoscaler - https://github.com/drone/autoscaler • Autoscaler documentation - https://autoscale.drone.io/ • Official Drone plugins - http://plugins.drone.io/ • Drone community - https://discourse.drone.io/
25. Thank You!