Building a Docker Based Platform

My talk at the Railshock meetup on Feb 4th in Zurich, Switzerland about the requirements of a flexible container platform and the possible open source building blocks. Finishing by a quick demo of CloudGear, my concrete implementation of such a platform.

https://www.cloudgear.net
http://www.meetup.com/rubyonrails-ch/events/219735348/

Transcript

1. Building a Docker Based Platform Georg Kunz, CloudGear 4. Feb. 2015
2. About Me Ruby/Rails freelancer and founder of CloudGear, previous web architect @localch Interests: Rails/Ruby/Go/Cloud/Docker/PaaaS/Infrastructure Web: http://georgkunz.com https://www.cloudgear.net Twitter: @geku
3. Agenda • Goal • Requirements and Building Blocks • Implementation (CloudGear) • Demo • Advice
4. Platform Goal • “Heroku” Workflow • New application in minutes • Run everything • Simple scalability • High availability • On VMs and bare metal
5. Abstract Platform to operate microservice architectures and allow continuous delivery.
6. Reality Check What we would like to ship What we have to ship Images by Derell Licht https://flic.kr/p/pnSzL licensed under CC BY-ND 2.0 and Chris https://flic.kr/p/5TtPFZ licensed under CC BY 2.0
7. Requirements • Scheduling • Orchestration • Discovery • Routing • Stateful Services • Networking • Observability • Workflow • Image Handling • Deployment • Service Management
8. Scheduling • Resource management • Decide where a service runs • Based on resource availability and constraints, e.g. - nodes with 2GB memory available - nodes tagged frontend or - nodes with service X
9. Scheduling: Tools • Apache Mesos • Google Kubernetes • CoreOS Fleet • Docker Swarm http://mesos.apache.org/ http://kubernetes.io/ https://github.com/coreos/fleet https://github.com/docker/swarm/
10. Orchestration • Service • states • scaling • registration • dependencies e.g. web app and DB
11. Orchestration like a Procfile but service level
12. Orchestration: Tools • Kubernetes • Marathon • Fleet • Docker compose/fig http://kubernetes.io/ https://github.com/mesosphere/marathon https://github.com/coreos/fleet http://www.fig.sh/
13. Cluster State • Scheduling/Orchestration requires persistent cluster state • Solution: distributed storage • Zookeeper • etcd • Consul https://raftconsensus.github.io/ http://zookeeper.apache.org/ http://etcd.io https://www.consul.io/
14. Discovery • Services run on arbitrary node and port • “Lookup table” required • Louse coupling preferred • Example: resolve host and port of database
15. Discovery: Solutions • DNS • ENV variables • Ambassador pattern • Proxy/Balancer https://docs.docker.com/articles/ ambassador_pattern_linking/
16. Discovery: Tools • Docker links (only localhost) • Consul (DNS or REST) • SkyDNS (based on etcd) • Smartstack by AirBNB • etcd (REST) https://www.consul.io/ https://github.com/skynetservices/skydns http://nerds.airbnb.com/smartstack-service- discovery-cloud/ http://etcd.io
17. Routing • Expose services to public • Load balancing & redundancy • Domain based routing • SSL • Internal TCP based routing
18. Routing: Tools • Configuration based (consul-template/confd) • nginx (HTTP) • HAProxy (TCP) • Vulcand (etcd backend) • Hipache (very basic) https://github.com/hashicorp/consul-template https://github.com/kelseyhightower/confd http://www.vulcanproxy.com/ https://github.com/hipache/hipache
19. Stateful Services • Managing persistence (volumes) • Redundant storage • Backups • Migration • Restore
20. Persistence: Tools • ClusterHQ (ZFS based) • Ceph (object storage like AWS S3) • other cluster storage solutions • Not many projects for Docker https://github.com/clusterhq/flocker http://ceph.com/
21. Network • Goal: no constraints for services (any port useable) • Ideal case: each Container has public routable IP • Advantages: • direct container to container connections • each port directly accessible
22. Network: Problems • Not working because • IPv4 limitations • dependent on environment • Hence • default Docker setup: NAT
23. Network: Solutions • NAT (port mapping) • Overlay network • Docker solutions: flannel / Weave / SocketPlane • IPv6 (recently merged) https://github.com/coreos/flannel http://weave.works/ http://socketplane.io/
24. Observability • Logs, metrics, monitoring • Aggregated by service • Know what is running where
25. Observability: Tools • Logspout and remote Syslog • Logstash + Kibana • InfluxDB + Grafana or Graphite • cAdvisor (container metrics) • Docker built in (open PR) https://github.com/progrium/logspout http://www.elasticsearch.org/overview/logstash/ http://www.elasticsearch.org/overview/kibana/ http://grafana.org/docs/features/influxdb/ https://github.com/google/cadvisor
26. Workflow: Deployment • GIT based • Different strategies: • restart (with downtime / maintenance) • rolling • canary (one instance updated) • blue/green (2x instances) • Rollback • Migrations (other one-off tasks)
27. Deployment: Tools • GIT (pretty simple) • gitreceive by Progrium • No tools for deployment strategies/ rollback https://github.com/progrium/gitreceive
28. Workflow: Image Handling • Image building • Dockerfile • Heroku Buildpack • Distribution • Docker Registry (private or commercial) • Image based (export/import) https://coreos.com/products/enterprise-registry/
29. Image Handling: Tools • Building • Buildpacks: buildstep by Progrium • Docker build for Dockerfiles • Registries: • Docker project • Docker or CoreOS SaaS https://github.com/progrium/buildstep
30. Workflow: Service Mgmt. • Create/delete/clone applications • App configuration • Auditing (timeline) • One-off jobs and console => Layer on top of everything else
31. Service Mgmt.: Tools • Deis • OpenShift http://deis.io/ https://www.openshift.com/
32. Typical Architecture Infrastructure Platform ... Node Node Node Node Config Store Config Store Config Store Master Service Service Service Service Service Service Service Service Service Service Service Service Service Service Controller Client Web UI API GIT Server Router Docker Image Registry CLI GIT Provisioner Config Agent Network Agent Network Agent Network Agent Network Agent Docker Daemon Docker Daemon Docker Daemon Docker Daemon
33. CloudGear http://www.cloudgear.net
34. CloudGear • Based around Consul • Benefits: • service discovery built in • capable K/V store • built in health checks https://www.consul.io/
35. CloudGear • “Glue code” written in Ruby • Pluggable scheduler & deployment strategies • DNS based service discovery (Consul) • Custom router (nginx and GO app) • Simple network, hence NAT
36. CloudGear • Workflow • GIT based deployment with Dockerfiles • CLI tool running on node
37. Internals • Describe and configure service • Distribute to multiple nodes (aka scale) • Pull image • Start on defined nodes • Register in Consul
38. Demo
39. Outlook • Service manifest (like Fig) • Network manager (IPTables setup etc.) • Basic volume manager • Private Docker registry for distribution
40. Advice I • Images • choose base wisely and use few (OS security updates) • Understand what you’re running, most images are playground projects!
41. Advice II • Prepare today • adopt building blocks, not full stack, e.g. Consul or only dev environment • Shopify limited itself after first trial with Mesos • Conclusion: simplicity is king Shopify post: http://bit.ly/1uNeLqE
42. Just Remember This Slide • Check out Consul.io - it’s pretty cool! • Please follow @cloudgear_net or sign up on the mailing list. • Become CloudGear Early Adopter https://www.consul.io/ https://www.cloudgear.net/subscriptions/new
43. ?