Docker at scale with Mesos

Transcript

1. Docker at scale with Mesos Phạm Tuấn Anh Lead engineer @ VCCloud Platform Services
2. $ whoami • Lead engineer at VCCloud Platform Services • Build & work with large-scale/production platform services for about 6 years • Services: Storage, DNS, CDN, DDoS Protection, Logging, Private PaaS
3. Agenda 1. What is Mesos? 2. How to run docker containers on Mesos 3. Lessons learned 4. Q&A
4. 1. What is Mesos? 2. How to run docker containers on Mesos 3. Lessons learned 4. Q&A
9. https://blog.twitter.com/2016/overview-of-the-twitter-cloud-platform-compute
10. https://mesosphere.com/blog/2015/04/23/apple-details-j-a-r-v-i-s-the-mesos- framework-that-runs-siri/
12. Powered by Mesos: • Apple • Atlassian • CERN • Cisco • Cloudflare • Foursquare • Groupon • HubSpot • Netflix • Opera • Paypal • Shopee • Shopify • Saleforce • Twitter • Uber • Vimeo • Verizon • Weibo • Yelp • Xiaomi http://mesos.apache.org/documentation/latest/powered-by-mesos/
13. Solomon Hykes, Founder & CTO of Docker, said at DockerCon EU 12/2014: Mesos is the “gold standard” for large-scale production clusters running containers
14. Mesos is: • Open source, top-level Apache project • Proven at massive scale • Multi-resource (CPU, RAM, Disk, GPU) scheduling • Simplified operational model • Cross platform • Native Docker support
17. • PaaS: Marathon, Aurora, Kubernetes, Swarm • Big Data: Hadoop, Spark, Storm • Database: Cassandra, ArangoDB, • ElasticSearch, Chronos, Jenkins, Kafka… Mesos Frameworks:
18. 1. What is Mesos? 2. How to run docker containers on Mesos 3. Lessons learned 4. Q&A
19. • Apache Aurora • Google Kubernetes • HubSpot Singularity • Mesosphere Marathon
26. Features: • Start, stop, scale, update, rollback apps • Rolling deploy / restart • Constraints • Health checks • App dependencies • Highly available, no SPoF
30. 1. What is Mesos? 2. How to run docker containers on mesos 3. Lessons learned 4. Q&A
31. – Immutable Infrastructure / CodeShip “Some of the major challenges today when building infrastructure are predictability, scalability and automated recovery.”
32. • Container style • Service discovery • Networking Lessons Learned • Managing secrets • App deployment • Docker annoyances
33. Choose a container style: • "thin" single-process container • "fat" container • sshd in container
34. • Container style • Service discovery • Networking Lessons Learned • Managing secrets • App deployment • Docker annoyances
35. Services Discovery • Mesos-DNS: • <app_id>.<app_group>.marathon.mesos • record types: A / SRV • simple & stateless
37. • Container style • Service discovery • Networking Lessons Learned • Managing secrets • App deployment • Docker annoyances
38. Networking • Project Calico: • IP-per-container • Pure Layer 3
42. • Container style • Service discovery • Networking Lessons Learned • Managing secrets • App deployment • Docker annoyances
43. Managing Secrets • ansible-vault
47. • Container style • Service discovery • Networking Lessons Learned • Managing secrets • App deployment • Docker annoyances
48. App deployment • ansible-playbook • marathon.json • docker tag name = git commit hash • 1 mesos cluster (3 VMs) per dev • deploy to servers → deploy to marathon
50. marathon.json
59. https://flask-hello.marathon.mesos.vn/
60. Monitoring • Container’s RAM used
61. Stateful Containers • Local persistent volumes
62. • Container style • Service discovery • Networking Lessons Learned • Managing secrets • App deployment • Docker annoyances
63. Docker • restart / upgrade docker without stopping the running containers • old images / tags cleanup • container’s stdout / stderr log files • docker pull • docker hub
64. "Normal" container vs cloud-native container: • No manual steps • Service discovery friendly • Config files should be optional • Using env vars for config
65. 1. What is Mesos? 2. How to run docker containers on Mesos 3. Lessons learned 4. Q&A
67. Links • Mesos: http://mesos.apache.org/ • Marathon: https://mesosphere.github.io/marathon/ • Project Calico: https://www.projectcalico.org/ • Mesos-DNS: https://github.com/mesosphere/mesos-dns • A Healthy Platform Checklist: https://gist.github.com/andy-pham/ 04dedff872b33c07a31277afb900950b