Continuous Entropy

A look into Yieldbot's build and deployment pipelines
Boston DevOps Meetup - May 2016

Transcript

1. Continuous Entropy A look into Yieldbot’s build and deployment pipelines. Anthony Spring Boston DevOps Meetup - May 2016
2. History Architecture Pipeline Generation Building artifacts Traditionally deployments Containerized deployments
3. History 2014 - Single monolithic deployment, once per day, several hours, human intervention the norm. Early 2015 - Decomposition of the Monolith into independently built and deployed services. (“Traditional Deploy”) Late 2015 - Start to containerize services with deployment into Mesos. Early 2016 - Full speed a head on containerization. (“Microservices Deploy”) Late 2016 - All remnants of the Monolithic are gone. All but a few services will be running inside of Mesos.
4. Architecture Apache Mesos with Marathon - used to run generic and specialized on-demand containers as slaves. Artifactory - Artifact storage — highly available and geolocated. Jenkins - bare bones master and very few physical slaves. Key plugins: Job DSL Plugin, Swarm
5. Pipeline Generation All Jenkins jobs are handled in an Infrastructure-As-Code manner, with all of their components stored in source control. The jobs are made up of two pieces: Erubis template containing Job DSL JSON-based job configuration Jobs are realized by a small utility that applies the job configuration against the set of job templates, with the end result being Groovy that is fed into the Job DSL plugin. Allows for changes to be quickly iterated upon for all jobs of a pipeline Jobs can be configured as a team desires Auto or manual version bump and deploy into Artifactory. Any combination of push button or auto deployment into environments.
7. –Yieldbot Speak “Seeder jobs watch for notifications from GitHub to run Rowlf against Dogchow to generate Groovy fed into the Job DSL plugin.”
8. Enablement Pro - A single job is all that is necessary to make a Jenkins master viable. We create this job with Chef when standing up the host. Pro - Creating a new job with an existing job type is as easy as pushing a single JSON file, and the pipeline is available in less than a minute. Pro - Repositories with a well-defined structure can have jobs automatically created for them. Con - Creating a new job type template can be cumbersome due to the Job DSL and abstracting the necessary pieces to Erubis.
9. So now what? We have this nifty process for creating jobs, but they need to do something?
 
 Lets say Hello to DrTeeth!
10. What is DrTeeth? Allows for a standardized method of building, testing, versioning and storing artifacts, with full customizability for edge cases. The standardization for each language was created between the primary language users and the Core team. “Oldest” tool in the chain, predating the IAC pipelines.
11. DrTeeth CLI
12. Building a project
13. Great, how do we deploy them? Electric Mayhem enters stage left. The “Traditional Deployment” jam is the first song on the set list.
14. Members of the Band Chef Creates a “Capistrano” like directory hierarchy Create the service runner pieces for the application, with dummy settings, and starts the application. Registers the service with Consul. Creates necessary Sensu checks.
15. Members of the Band Electric Mayhem Opinionated wrapper around SSHkit Capistrano like symlink based deployments allowing for fast deployments and rollbacks.
16. Putting it all together A Chef cookbook is created and deployed to manage the lifecycle of the application for the host(s) that will run the application. A job definition is created in Dogchow. This creates the build and deploy pipeline for the application based upon the configured type. The next push to the corresponding application repository will start the CI/CD pipeline.
17. Poof `git push` happens The pipeline picks up the commit DrTeeth builds, tests, and deploys the resulting artifact. Electric Mayhem takes over Warming the destination hosts with the artifact Updates the necessary symlinks to bring the artifact into play Restarts/HUPs the underlying process to pickup the latest artifact 1 Executes any necessary smoke tests. 1 In the first deployment the dummy placeholder app is replaced with the actual app, the related Consul/Sensu checks become healthy as the app starts up, and the app is ready to rock.
18. Enablement Pro - Very fast to rollback and roll forward an application. Pro - Standardized build and deployment makes working across products very easy. Pro - Standardized application configuration for service discovery and alerting. Con - Electric Mayhem needs to be “told” about the deployment requirements of an application. Con - Knowledge of Chef is required. Con - Requires some braining to plan out where an application should live in terms of availability, resources and cost.
19. –Engineering “Lets microservice all the things with containers!”
20. Microservices Orchestration Apache Mesos - “Apache Mesos abstracts CPU, memory, storage, and other compute resources away from machines (physical or virtual), enabling fault-tolerant and elastic distributed systems to easily be built and run effectively.” Apache Marathon - “Marathon is a production-grade container orchestration platform for Mesosphere's Datacenter Operating System (DCOS) and Apache Mesos.” HubSpot Singularity - “Singularity is an API and web application for running and scheduling Apache Mesos tasks — including long running processes, scheduled jobs, and one-off tasks.” 1 http://mesos.apache.org/ 2 https://mesosphere.github.io/marathon/ 3 https://github.com/HubSpot/Singularity
21. 10,000 Mile View Frameworks for running containers within Mesos. Ability to configure the necessary resources of an application on a job by job basis. (CPU/Memory) Ability to configure container lifecycle to guarantee a certain level of availability in the event of deployment or failure.
22. See what we did there? No need to “self” manage HA of services. No Chef. No underlying host resource management. http://www.grok-interactive.com/assets/2014/05/happy-developer.gif
23. First things first How do we get jobs into the frameworks? Infrastructure as Code approach by leveraging a pair of tools to take job definitions from source control and push them into the respective framework. https://github.com/yieldbot/scooter https://github.com/yieldbot/singularity-py
24. Build/Deployment v2 DrTeeth still builds the artifact, with the additional step of getting it into a container with the necessary runtime tools and pushing that container into Artifactory. After the container is generated, we update the M/S job definitions with the necessary artifact information and push that up to source control. The M/S IAC job pipelines pick up the change and push the updated configuration into the respective framework.
25. Putting it all together v2 A job definition is created in Dogchow. This creates the build and deploy pipeline for the application based upon the configured type. A M/S job definition is created in Skeeter. This creates the service for the underlying application. Any deployment failures by the frameworks result in emails/notifications alerting the product owner of the issue.
26. Whats Next? Currently evaluating replacing Jenkins Job DSL plugin with the newly revamped Jenkins Pipeline Plugin for a subset of the pipelines. Look at alternatives for deployment visualization, orchestration, and verification.
27. Anthony Spring Core Infrastructure Architect/ Mindreader aspring@yieldbot.com @devopsanthony