Upgrade to Pro — share decks privately, control downloads, hide ads and more …

DCOS for java devs jughh

DCOS for java devs jughh

Container orchestration on Apache Mesos - DC/OS for java devs

In the era of docker, big data and microservices it is really important to distribute your applications reasonable across your cluster and keep a good overview of all of your applications. Because of this, cluster management software like Apache Mesos and DC/OS are very important and popular. In this session we will implement a java based service with spring boot and will start him locally and with docker-compose. Then we will deploy our service to DC/OS and see how easy it is to scale, upgrade, monitor and do other fancy stuff. And the best thing for the demo: We will have a nice UI and super cool CLI.

7c6d957f56a234a73c0a321bbdd5ebee?s=128

Johannes Unterstein

November 12, 2016
Tweet

More Decks by Johannes Unterstein

Other Decks in Technology

Transcript

  1. 1 Container orchestrierung on Mesos DC/OS for Java developers @unterstein

    @dcos #jughh
  2. © 2016 Mesosphere, Inc. All Rights Reserved. 2 Johannes Unterstein

    Software Engineer @Mesosphere @unterstein
  3. © 2016 Mesosphere, Inc. All Rights Reserved. 3 The story

  4. © 2016 Mesosphere, Inc. All Rights Reserved. 4 Java developer

    Alice builds new `App with Chuck Norris jokes` and put it into production in three easy steps. Code & slides @ https://git.io/vXUoy The story
  5. © 2016 Mesosphere, Inc. All Rights Reserved. 5 Step 1

    Runs on my machine
  6. © 2016 Mesosphere, Inc. All Rights Reserved. 6 Step 2

    Runs on my container engine
  7. © 2016 Mesosphere, Inc. All Rights Reserved. 7 Step 3

    Runs on my datacenter
  8. © 2016 Mesosphere, Inc. All Rights Reserved. 8 Works on

    my machine!
  9. © Gerard Julien/AFP Run everything in containers!

  10. © 2016 Mesosphere, Inc. All Rights Reserved. 10 Wait! What

    about production?
  11. © 2015 Mesosphere, Inc. All Rights Reserved. What about container

    scheduling? 11
  12. © 2015 Mesosphere, Inc. All Rights Reserved. What about resource

    management? 12
  13. © 2015 Mesosphere, Inc. All Rights Reserved. What about service

    management? 13
  14. © 2016 Mesosphere, Inc. All Rights Reserved. 14 CONTAINER SCHEDULING

    RESOURCE MANAGEMENT SERVICE MANAGEMENT - Load Balancing - Readiness Checking Orchestration
  15. © 2016 Mesosphere, Inc. All Rights Reserved. 15 CONTAINER SCHEDULING

    - Placement - Replication/Scaling - Resurrection - Rescheduling - Rolling Deployment - Upgrades - Downgrades - Collocation RESOURCE MANAGEMENT - Memory - CPU - GPU - Volumes - Ports - IPs - Images/Artifacts SERVICE MANAGEMENT - Labels - Groups/Namespaces - Dependencies - Load Balancing - Readiness Checking Orchestration
  16. © 2016 Mesosphere, Inc. All Rights Reserved. 16 Modern datacenter

  17. Hadoop Database Services Spark memcached

  18. Hadoop Database Services Spark memcached

  19. © 2015 Mesosphere, Inc. All Rights Reserved. 19 A naive

    approach to handling varied app requirements: static partitioning. Maintaining sufficient headroom to handle peak workloads on all partitions leads to poor utilization overall. KEEP IT STATIC time
  20. © 2015 Mesosphere, Inc. All Rights Reserved. 20 Multiple frameworks

    can use the same cluster resources, with their share adjusting dynamically. SHARED RESOURCES time
  21. © 2016 Mesosphere, Inc. All Rights Reserved. 21

  22. © 2015 Mesosphere, Inc. All Rights Reserved. 22 THE BIRTH

    OF MESOS TWITTER TECH TALK The grad students working on Mesos give a tech talk at Twitter. March 2010 APACHE INCUBATION Mesos enters the Apache Incubator. Spring 2009 CS262B Ben Hindman, Andy Konwinski and Matei Zaharia create “Nexus” as their CS262B class project. MESOS PUBLISHED Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center is published as a technical report. September 2010 December 2010 DC/OS April 2016
  23. © 2015 Mesosphere, Inc. All Rights Reserved. 23 Sharing resources

    between batch processing frameworks • Hadoop • MPI • Spark What does an operating system provide? • Resource management • Programming abstractions • Security • Monitoring, debugging, logging TECHNOLOGY VISION
  24. © 2015 Mesosphere, Inc. All Rights Reserved. • A top-level

    Apache project • A cluster resource negotiator • Scalable to 10,000s of nodes • Fault-tolerant, battle-tested • An SDK for distributed apps • Native Docker support 24 Apache Mesos
  25. © 2015 Mesosphere, Inc. All Rights Reserved. 25 ARCHITECTURE MESOS

    FUNDAMENTALS
  26. © 2016 Mesosphere, Inc. All Rights Reserved. 2-Level Scheduling 26

    Launch Tasks Scheduler(s) Allocation User Work Resource Offers
  27. © 2015 Mesosphere, Inc. All Rights Reserved. 27 ARCHITECTURE MESOS

    FUNDAMENTALS • Agents advertise resources to Master • Master offers resources to Framework • Framework rejects/uses resources • Agents report task status to Master
  28. © 2015 Mesosphere, Inc. All Rights Reserved. 28 ARCHITECTURE MESOS

    FUNDAMENTALS • Agents advertise resources to Master • Master offers resources to Framework • Framework rejects/uses resources • Agents report task status to Master
  29. © 2015 Mesosphere, Inc. All Rights Reserved. 29 ARCHITECTURE MESOS

    FUNDAMENTALS • Agents advertise resources to Master • Master offers resources to Framework • Framework rejects/uses resources • Agents report task status to Master
  30. © 2015 Mesosphere, Inc. All Rights Reserved. 30 ARCHITECTURE MESOS

    FUNDAMENTALS • Agents advertise resources to Master • Master offers resources to Framework • Framework rejects/uses resources • Agents report task status to Master
  31. © 2015 Mesosphere, Inc. All Rights Reserved. 31 ARCHITECTURE MESOS

    FUNDAMENTALS • Agents advertise resources to Master • Master offers resources to Framework • Framework rejects/uses resources • Agents report task status to Master
  32. © 2016 Mesosphere, Inc. All Rights Reserved. 32 Persistence

  33. © 2016 Mesosphere, Inc. All Rights Reserved. 33 • Default

    Sandbox Simple to use, Task failures • Persistent Volumes Task failures, (permanent) Node failures • Distributed File System/External Storage Node failures, non-local writes Storage Options
  34. © 2015 Mesosphere, Inc. All Rights Reserved. 34

  35. Datacenter Operating System (DC/OS) Distributed Systems Kernel (Mesos) DC/OS ENABLES

    MODERN DISTRIBUTED APPS Big Data + Analytics Engines Microservices (in containers) Streaming Batch Machine Learning Analytics Functions & Logic Search Time Series SQL / NoSQL Databases Modern App Components Any Infrastructure (Physical, Virtual, Cloud) 35
  36. 36 THE BASICS DC/OS is … • 100% open source

    (ASL2.0) + A big, diverse community • An umbrella for ~30 OSS projects + Roadmap and designs + The build tool chain + Docs and tutorials • Not limited in any way • Familiar, with a few new features
  37. 37

  38. 38 THE UNIVERSE

  39. 39

  40. © 2016 Mesosphere, Inc. All Rights Reserved. 40 Questions? Code:

    https://git.io/vXUoy Psssssssst … … we are hiring! http:// grnh.se/ ie76ru
  41. © 2015 Mesosphere, Inc. All Rights Reserved. 41 Well …

  42. © 2015 Mesosphere, Inc. All Rights Reserved. 42 … let`s

    talk about data
  43. © 2016 Mesosphere, Inc. All Rights Reserved. 43 Persistence Data

  44. © 2016 Mesosphere, Inc. All Rights Reserved. 44 Big Data

  45. © 2016 Mesosphere, Inc. All Rights Reserved. 45 Fast Data

  46. © 2016 Mesosphere, Inc. All Rights Reserved. 46 Questions? Code:

    https://git.io/vXUoy Psssssssst … … we are hiring! http:// grnh.se/ ie76ru