From Big Data to Micro Service

A journey with Adam, Akka, Cassandra, DCOS, Spark and Spark Notebook. Tutorial at Devoxx in Antwerp, see also http://cfp.devoxx.be/2015/talk/YHI-5063/From_Big_Data_to_Micro_Service:_A_journey_with_Adam,_Cassandra,_DCOS,_Play!_2,_Spark,_Spark_Notebook_and_Spray

Transcript

1. © 2015 Mesosphere, Inc. All Rights Reserved. FROM BIG DATA TO MICRO SERVICE:
 A JOURNEY WITH ADAM, AKKA, CASSANDRA, DCOS, SPARK AND SPARK NOTEBOOK 1 Devoxx, Antwerp | Michael Hausenblas | 2015-11-10
2. © 2015 Mesosphere, Inc. All Rights Reserved. WORKLOADS* … 2 *) kudos to Timothy St. Clair, @timothysc batch streaming PaaS MapReduce
3. © 2015 Mesosphere, Inc. All Rights Reserved. APACHE MESOS 3
4. © 2015 Mesosphere, Inc. All Rights Reserved. • A top-level ASF project • A cluster resource negotiator • Scalable to 10,000s of nodes but also useful for a handful of nodes • Fault-tolerant, battle-tested • An SDK for distributed apps • Native Docker support 4 Apache Mesos
5. © 2015 Mesosphere, Inc. All Rights Reserved. ARCHITECTURE 5 Mesos http://mesos.berkeley.edu/mesos_tech_report.pdf *) now: agent *
6. © 2015 Mesosphere, Inc. All Rights Reserved. RESOURCES 6 Mesos • resource == anything a task/executor consumes in order to do their work • standard resources: cpu, mem, disk, ports • DRF
7. © 2015 Mesosphere, Inc. 7
8. © 2015 Mesosphere, Inc. 8
9. © 2015 Mesosphere, Inc. 9
10. © 2015 Mesosphere, Inc. 10
11. © 2015 Mesosphere, Inc. 11
12. © 2015 Mesosphere, Inc. 12
13. © 2015 Mesosphere, Inc. 13
14. © 2015 Mesosphere, Inc. 14
15. © 2015 Mesosphere, Inc. 15
16. © 2015 Mesosphere, Inc. 16
17. © 2015 Mesosphere, Inc. 17
18. © 2015 Mesosphere, Inc. 18
19. © 2015 Mesosphere, Inc. 19
20. © 2015 Mesosphere, Inc. 20
21. © 2015 Mesosphere, Inc. 21
22. © 2015 Mesosphere, Inc. 22
23. © 2015 Mesosphere, Inc. All Rights Reserved. MEET THE DATACENTER OPERATING SYSTEM (DCOS) 23
24. © 2015 Mesosphere, Inc. All Rights Reserved. LOCAL OS VS. DISTRIBUTED OS 24 http://bitly.com/os-vs-dcos
25. © 2015 Mesosphere, Inc. All Rights Reserved. DCOS IS A DISTRIBUTED OPERATING SYSTEM 25 • kernel (Apache Mesos, written in C++) scales to 10,000 of nodes • fault-tolerant in all components, rolling upgrades throughout • containers first class citizens (LXC, Docker) • local OS per node (+container enabled) • scheduling (long-lived, batch) • service discovery, monitoring, logging, debugging
26. © 2015 Mesosphere, Inc. All Rights Reserved. DCOS HIGH LEVEL VIEW 26 Any Service or Container Any Infrastructure Mesosphere DCOS Your favorite services, container formats, and those yet to come Build apps once on DCOS, and run it anywhere Runs distributed apps anywhere 
 as simply as running apps on your laptop
27. © 2015 Mesosphere, Inc. All Rights Reserved. DCOS BENEFITS 27 • Run stateless services such as Web servers, app servers (via Marathon) and stateful services like Spark, Kafka, HDFS, Cassandra, ArangoDB etc. together on one cluster • Dynamic partitioning of your cluster, depending on your needs (business requirements) • Increased utilization (10% → 80% an more)
28. © 2015 Mesosphere, Inc. All Rights Reserved. DCOS ARCHITECTURE 28 https://docs.mesosphere.com/getting-started/dcosarchitecture/
29. © 2015 Mesosphere, Inc. All Rights Reserved. 29
30. © 2015 Mesosphere, Inc. All Rights Reserved. 30
31. © 2015 Mesosphere, Inc. All Rights Reserved. DEMO TIME! 31
32. © 2015 Mesosphere, Inc. All Rights Reserved. REFERENCES 32 Look up stuff: FAQ: help.mesosphere.com Docs: docs.mesosphere.com Chat with us and others from the community: Slack: chat.mesosphere.com Intercom (embedded in DCOS) IRC: #mesos on irc.freenode.net
33. © 2015 Mesosphere, Inc. All Rights Reserved. SEE ALSO … 33 • @mhausenblas • mhausenblas.info • @mesosphere • mesosphere.io/product http://shop.oreilly.com/product/9781939902184.do http://p24e.io http://shop.oreilly.com/product/0636920039952.do https://manning.com/books/mesos-in-action
34. © 2015 Mesosphere, Inc. All Rights Reserved. WE ARE HIRING, WORLDWIDE … San Francisco New York Hamburg https://mesosphere.com/careers/