Tweet
Tweet

Slide 1

Slide 1 text

© 2015 Mesosphere, Inc. All Rights Reserved. FILESYSTEMS, SQL AND NOSQL WITH
 APACHE MESOS 1 Michael Hausenblas, Developer & Cloud Advocate | 2015-10-05 | CloudOpen Dublin, Ireland

Slide 2

Slide 2 text

© 2015 Mesosphere, Inc. All Rights Reserved. MOTIVATION 2

Slide 3

Slide 3 text

© 2015 Mesosphere, Inc. All Rights Reserved. PARTITIONING 3

Slide 4

Slide 4 text

© 2015 Mesosphere, Inc. All Rights Reserved. PARTITIONING 4

Slide 5

Slide 5 text

© 2015 Mesosphere, Inc. All Rights Reserved. STATELESS VS
 STATEFUL 5

Slide 6

Slide 6 text

© 2015 Mesosphere, Inc. All Rights Reserved. DEFINITIONS 6 • A centralized service runs on one machine • A distributed service runs on two or more machines that are connected via a network • A stateless service persists (no significant) state • A stateful service persists state

Slide 7

Slide 7 text

© 2015 Mesosphere, Inc. All Rights Reserved. EXAMPLES 7 # a stateless, centralized service cat /dev/random # a stateless, distributed service (Marathon) http PUT http://1.2.3.4:8080/v2/apps < app.json # a stateful, centralized service (redirect to file) echo 'lolwut' > ./some.dat # a stateful, distributed service (HDFS) hadoop fs -put ./some.dat hdfs:///user/me/some.dat

Slide 8

Slide 8 text

© 2015 Mesosphere, Inc. All Rights Reserved. CLASSIFICATION 8

Slide 9

Slide 9 text

© 2015 Mesosphere, Inc. All Rights Reserved. MESOS 101 9

Slide 10

Slide 10 text

© 2015 Mesosphere, Inc. All Rights Reserved. ARCHITECTURE 10 Mesos http://mesos.berkeley.edu/mesos_tech_report.pdf *) now: agent *

Slide 11

Slide 11 text

© 2015 Mesosphere, Inc. All Rights Reserved. RESOURCES 11 Mesos • resource == anything a task/executor consumes in order to do their work • standard resources: cpu, mem, disk, ports • define your own, for example:
 --attributes='host_os:ubuntu;zone:us_east;rack:42'

Slide 12

Slide 12 text

© 2015 Mesosphere, Inc. All Rights Reserved. RESOURCE
 OFFERS 12 Mesos http://mesos.berkeley.edu/mesos_tech_report.pdf

Slide 13

Slide 13 text

© 2015 Mesosphere, Inc. All Rights Reserved. RESOURCES
 AND 
 STATEFUL
 SERVICES 13 Mesos • prior to Mesos 0.23 no storage primitives existed • issues for stateful services • resource reservation • data persistence

Slide 14

Slide 14 text

© 2015 Mesosphere, Inc. All Rights Reserved. WORKAROUNDS
 FOR STATEFUL SERVICES 14 Mesos resource reservation → static reservations, e.g.:
 mesos-slave --resources=cpu(crate):0.5;mem(crate):2048 • operator-heavy • inflexible • requires Mesos agent restart

Slide 15

Slide 15 text

© 2015 Mesosphere, Inc. All Rights Reserved. WORKAROUNDS
 FOR STATEFUL SERVICES 15 Mesos data persistence → store in out-of-band location, e.g.:
 path.data: /var/crate/data # in crate.yml • no Mesos agent garbage collection • not accounted for by Mesos • setup ahead of time via local volume, NFS-mount of SAN/NAS or DFS such as HDFS, Ceph or Quobyte

Slide 16

Slide 16 text

© 2015 Mesosphere, Inc. All Rights Reserved. DYNAMIC
 RESERVATION 16

Slide 17

Slide 17 text

© 2015 Mesosphere, Inc. All Rights Reserved. OVERVIEW 17 Dynamic Reservation See MESOS-2018 and docs/reservation • solves resource reservation issue • enables re-launch on same agent w/ guaranteed allocation • via framework or operator

Slide 18

Slide 18 text

© 2015 Mesosphere, Inc. All Rights Reserved. PERSISTENT
 VOLUMES 18

Slide 19

Slide 19 text

© 2015 Mesosphere, Inc. All Rights Reserved. OVERVIEW 19 Persistent Volumes See MESOS-1554 and docs/persistent-volume • solves data persistence issue • avoids garbage collection in the sandbox • can only exist on reserved resources (access/security)

Slide 20

Slide 20 text

© 2015 Mesosphere, Inc. All Rights Reserved. MEET THE
 DATA
 CENTER
 OPERATING
 SYSTEM 20

Slide 21

Slide 21 text

© 2015 Mesosphere, Inc. All Rights Reserved. LOCAL OS VS. DISTRIBUTED OS 21 http://bitly.com/os-vs-dcos

Slide 22

Slide 22 text

© 2015 Mesosphere, Inc. All Rights Reserved. DCOS IS A DISTRIBUTED OPERATING SYSTEM 22 • kernel (Apache Mesos, written in C++) scales to 10,000 of nodes • fault-tolerant in all components, rolling upgrades throughout • containers first class citizens (LXC, Docker) • local OS per node (+container enabled) • scheduling (long-lived, batch) • service discovery, monitoring, logging, debugging

Slide 23

Slide 23 text

© 2015 Mesosphere, Inc. All Rights Reserved. 23

Slide 24

Slide 24 text

© 2015 Mesosphere, Inc. All Rights Reserved. 24

Slide 25

Slide 25 text

© 2015 Mesosphere, Inc. DCOS CE Architecture 25 https://docs.mesosphere.com/getting-started/dcosarchitecture/

Slide 26

Slide 26 text

© 2015 Mesosphere, Inc. All Rights Reserved. BENEFITS 26 DCOS • Run stateless services such as Web servers, app servers (via Marathon) and stateful services like Crate, Kafka, HDFS, Cassandra, ArangoDB etc. together on one cluster • Dynamic partitioning of your cluster, depending on your needs (business requirements) • Increased utilization (10% → 80% an more)

Slide 27

Slide 27 text

© 2015 Mesosphere, Inc. All Rights Reserved. MESOSPHERE IS HIRING, WORLDWIDE … San Francisco New York Hamburg https://mesosphere.com/careers/

Slide 28

Slide 28 text

© 2015 Mesosphere, Inc. All Rights Reserved. Q & A 28 • @mhausenblas • mhausenblas.info • @mesosphere • mesosphere.io/product Kudos for material in this talk go out to:
 David Greenberg and Joe Stein http://shop.oreilly.com/product/0636920039952.do https://manning.com/books/mesos-in-action