DCOS for java devs

Container orchestration on Apache Mesos - DC/OS for java devs

In the era of docker, big data and microservices it is really important to distribute your applications reasonable across your cluster and keep a good overview of all of your applications. Because of this, cluster management software like Apache Mesos and DC/OS are very important and popular. In this session we will implement a java based service with spring boot and will start him locally and with docker-compose. Then we will deploy our service to DC/OS and see how easy it is to scale, upgrade, monitor and do other fancy stuff. And the best thing for the demo: We will have a nice UI and super cool CLI.

Transcript

1. 1 Container orchestrierung on Mesos DC/OS for Java developers @unterstein @dcos @jaxcon
2. © 2016 Mesosphere, Inc. All Rights Reserved. 3 Johannes Unterstein Distributed Applications Engineer @unterstein
3. © 2016 Mesosphere, Inc. All Rights Reserved. 4 The story
4. © 2016 Mesosphere, Inc. All Rights Reserved. 5 Java developer Alice builds new `App with Chuck Norris jokes` and put it into production in three easy steps. https://git.io/vXUoy The story
5. © 2016 Mesosphere, Inc. All Rights Reserved. 6 Step 1 Runs on my machine
6. © 2016 Mesosphere, Inc. All Rights Reserved. 7 Step 2 Runs on my container engine
7. © 2016 Mesosphere, Inc. All Rights Reserved. 8 Step 3 Runs on my datacenter
8. © 2016 Mesosphere, Inc. All Rights Reserved. 9 Works on my machine!
9. © Gerard Julien/AFP Run everything in containers!
10. © 2016 Mesosphere, Inc. All Rights Reserved. 11 Wait! What about production?
11. © 2015 Mesosphere, Inc. All Rights Reserved. What about container scheduling? 12
12. © 2015 Mesosphere, Inc. All Rights Reserved. What about resource management? 13
13. © 2015 Mesosphere, Inc. All Rights Reserved. What about service management? 14
14. © 2016 Mesosphere, Inc. All Rights Reserved. 15 CONTAINER SCHEDULING RESOURCE MANAGEMENT SERVICE MANAGEMENT - Load Balancing - Readiness Checking Orchestration
15. © 2016 Mesosphere, Inc. All Rights Reserved. 16 CONTAINER SCHEDULING - Placement - Replication/Scaling - Resurrection - Rescheduling - Rolling Deployment - Upgrades - Downgrades - Collocation RESOURCE MANAGEMENT - Memory - CPU - GPU - Volumes - Ports - IPs - Images/Artifacts SERVICE MANAGEMENT - Labels - Groups/Namespaces - Dependencies - Load Balancing - Readiness Checking Orchestration
16. © 2016 Mesosphere, Inc. All Rights Reserved. 17 Modern datacenter
17. Hadoop Database Services Spark memcached
18. Hadoop Database Services Spark memcached
19. © 2015 Mesosphere, Inc. All Rights Reserved. 20 A naive approach to handling varied app requirements: static partitioning. Maintaining sufficient headroom to handle peak workloads on all partitions leads to poor utilization overall. KEEP IT STATIC time
20. © 2015 Mesosphere, Inc. All Rights Reserved. 21 Multiple frameworks can use the same cluster resources, with their share adjusting dynamically. SHARED RESOURCES time
21. © 2016 Mesosphere, Inc. All Rights Reserved. 22
22. © 2015 Mesosphere, Inc. All Rights Reserved. 23 THE BIRTH OF MESOS TWITTER TECH TALK The grad students working on Mesos give a tech talk at Twitter. March 2010 APACHE INCUBATION Mesos enters the Apache Incubator. Spring 2009 CS262B Ben Hindman, Andy Konwinski and Matei Zaharia create “Nexus” as their CS262B class project. MESOS PUBLISHED Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center is published as a technical report. September 2010 December 2010 DC/OS April 2016
23. © 2015 Mesosphere, Inc. All Rights Reserved. 24 Sharing resources between batch processing frameworks • Hadoop • MPI • Spark What does an operating system provide? • Resource management • Programming abstractions • Security • Monitoring, debugging, logging TECHNOLOGY VISION
24. © 2015 Mesosphere, Inc. All Rights Reserved. • A top-level Apache project • A cluster resource negotiator • Scalable to 10,000s of nodes • Fault-tolerant, battle-tested • An SDK for distributed apps • Native Docker support 25 Apache Mesos
25. © 2015 Mesosphere, Inc. All Rights Reserved. 26 ARCHITECTURE MESOS FUNDAMENTALS
26. © 2016 Mesosphere, Inc. All Rights Reserved. 2-Level Scheduling 27 Launch Tasks Scheduler(s) Allocation User Work Resource Offers
27. © 2015 Mesosphere, Inc. All Rights Reserved. 28 ARCHITECTURE MESOS FUNDAMENTALS • Agents advertise resources to Master • Master offers resources to Framework • Framework rejects/uses resources • Agents report task status to Master
28. © 2015 Mesosphere, Inc. All Rights Reserved. 29 ARCHITECTURE MESOS FUNDAMENTALS • Agents advertise resources to Master • Master offers resources to Framework • Framework rejects/uses resources • Agents report task status to Master
29. © 2015 Mesosphere, Inc. All Rights Reserved. 30 ARCHITECTURE MESOS FUNDAMENTALS • Agents advertise resources to Master • Master offers resources to Framework • Framework rejects/uses resources • Agents report task status to Master
30. © 2015 Mesosphere, Inc. All Rights Reserved. 31 ARCHITECTURE MESOS FUNDAMENTALS • Agents advertise resources to Master • Master offers resources to Framework • Framework rejects/uses resources • Agents report task status to Master
31. © 2015 Mesosphere, Inc. All Rights Reserved. 32 ARCHITECTURE MESOS FUNDAMENTALS • Agents advertise resources to Master • Master offers resources to Framework • Framework rejects/uses resources • Agents report task status to Master
32. © 2016 Mesosphere, Inc. All Rights Reserved. 33 Persistence
33. © 2016 Mesosphere, Inc. All Rights Reserved. 34 • Default Sandbox Simple to use, Task failures • Persistent Volumes Task failures, (permanent) Node failures • Distributed File System/External Storage Node failures, non-local writes Storage Options
34. © 2015 Mesosphere, Inc. All Rights Reserved. 35
35. Datacenter Operating System (DC/OS) Distributed Systems Kernel (Mesos) DC/OS ENABLES MODERN DISTRIBUTED APPS Big Data + Analytics Engines Microservices (in containers) Streaming Batch Machine Learning Analytics Functions & Logic Search Time Series SQL / NoSQL Databases Modern App Components Distributed systems kernel to abstract resources Ecosystem of frameworks & apps Consistent architecture to run on top of kernel User Interface (GUI & CLI) Core system services (e.g., distributed init, cron, service discovery, package mgt & installer, storage) Any Infrastructure (Physical, Virtual, Cloud) 36
36. 37 THE BASICS DC/OS is … • 100% open source (ASL2.0) + A big, diverse community • An umbrella for ~30 OSS projects + Roadmap and designs + The build tool chain + Docs and tutorials • Not limited in any way • Familiar, with a few new features
38. 39 THE UNIVERSE
40. © 2016 Mesosphere, Inc. All Rights Reserved. 42 Questions? https://git.io/vXUoy