Mesos / Marathon / Docker at HolidayCheck (Docker Meetup Munich)

An overview of the new continuous deployment infrastructure at HolidayCheck.com using Mesos, Marathon, Bamboo, HAPRoxy and Docker

Transcript

1. Mesos, Marathon and Docker at HolidayCheck Docker Meetup Munich, February 23 2015 Christoph Neuroth @c089
2. disclaimer: me doing ops Image sources http://www.tyan.com/archive/products/assets/images/Ta_plusATXd.gif http://swizzler121.deviantart.com/art/Cardboard-Box-Upside-Down-49340358
3. I have the bestest clients :3 find /etc/apache2/ | xargs wc -l # … # ¯\_(ツ)_/¯ # … 7145 total
4. The Deployment Mutex (d’awwwww!) https://www.flickr.com/photos/roland/1924450950/
5. new service architecture • now: painful deployment, scaling complicated, manual labor, … • goal: continuous delivery of services, simplified scaling, technology-agnosticism
7. “Apache Mesos abstracts CPU, memory, storage, and other compute resources away from machines (physical or virtual), enabling fault-tolerant and elastic distributed systems to easily be built and run effectively.”
8. • allows to run tasks • a task can be “start a docker container” • tasks managed by mesos applications called “frameworks” • offers resources to frameworks
11. “Marathon is an Apache Mesos framework for long-running applications. Given that you have Mesos running as the kernel for your datacenter, Marathon is the init or upstart daemon.”
12. • implemented as mesos framework • allows to run long-running tasks on your mesos cluster • detects failing services and restarts them • deploying as easy as sending a POST request • scaling as easy as sending a PUT request
13. { "id": "/product/service/my-app", "container": { "type": "DOCKER", "docker": { "image": "group/image", "network": "BRIDGE", "portMappings":[ {"containerPort":8080}] } }, "env": {“ELASTICSEARCH_HOST”: “es.example.com“}, "healthChecks": [ {"path": "/health" }], "cpus": 1.5, "mem": 256.0, "instances": 3 }
16. • HAProxy routes requests to services • needs configuration to find services • Bamboo uses Marathon API to listen for state changes • rewrites HAProxy config from template and triggers reload when services are created, scaled, etc. (no API :sadface:)
17. Bamboo: PUT /api/services { “id":"/app-1", "acl":"hdr(host) -i app-1.example.com” } { “id":"/app-2", “acl”:”path_beg /app-2/” }
20. • problem: deploying is too complicated • deploying requires to call two APIs • plenty of options aka room for failure • plenty of power aka room for fucking up • error reporting of API is less than optimal • solution: hide complexity and limit options
21. • small io.js CLI application to simplify deployment $ hc-deploy -a /project/branchName -i docker-registry.hc.ag/team/ imageName:version -e ‘{“FOO":"bar"}' -m http://staging-marathon -b http:// staging-bamboo/ • HTTP interface curl http://staging/deploy -X POST -d ‘{“appId”:”/project/branchName”, …}’
22. • follow the 12factor guidelines, especially statelessness and configuration via environment variables • some custom guidelines, e.g. allow app to respond to a given PATH_PREFIX environment variable • build docker image and push to internal registry Missing piece: services to deploy ;)
23. • migration / adoption “plan” • regular tech demos for teams • workshops to help teams migrate their services to fit into the new architecture • test cluster open for experimentation by all teams from day one • dogfooding small services
24. I showed some work in progress to teams and you won’t believe what happened next • github webhook + jenkins • deploys every branch • custom dashboard with links to all instances
25. • gliderlabs/logspout • runs on every mesos slave • forwards stdout/err to central syslog • elastichsearch/logstash/kibana Logging
26. • 0.x components • If you don’t plan on having lots of services, it’s probably way too complex • haproxy can’t be reconfigured without reload -> need to restart / problematic with keep-alive connections • It’s awesome when it works, but hard to debug (mesos does a very bad job at reporting errors back to marathon, especially regarding docker foo) • deployments are not super fast (marathon bug / docker registry performance) There’s always a but…
27. • Questions? • Wanna docker at work? HolidayCheck is hiring in Munich (and near Konstanz, next to pretty lake) • socrates-conference.de August 27-30 • twitter/github: c089