Tweet
Tweet

Slide 1

Slide 1 text

© 2016 Mesosphere, Inc. All Rights Reserved. SCALING DATA ENGINEERING 1 Michael Hausenblas | 2016-11-17 | Big Data Spain, Madrid

Slide 2

Slide 2 text

© 2015 Mesosphere, Inc. All Rights Reserved. MOTIVATION 2

Slide 3

Slide 3 text

© 2015 Mesosphere, Inc. All Rights Reserved. AIRLINES 3

Slide 4

Slide 4 text

© 2015 Mesosphere, Inc. All Rights Reserved. LOGISTICS 4

Slide 5

Slide 5 text

© 2015 Mesosphere, Inc. All Rights Reserved. HEALTH
 CARE 5

Slide 6

Slide 6 text

© 2015 Mesosphere, Inc. All Rights Reserved. TRADERS 6

Slide 7

Slide 7 text

© 2015 Mesosphere, Inc. All Rights Reserved. FARMERS 7

Slide 8

Slide 8 text

© 2015 Mesosphere, Inc. All Rights Reserved. CITIES 8 © 2014, Wired magazine

Slide 9

Slide 9 text

© 2015 Mesosphere, Inc. All Rights Reserved. YOU 9

Slide 10

Slide 10 text

© 2015 Mesosphere, Inc. All Rights Reserved. TRENDS,
 OPPORTUNITIES, AND CHALLENGES 10

Slide 11

Slide 11 text

© 2016 Mesosphere, Inc. All Rights Reserved. ACTIONABLE INSIGHTS,
 ANYONE? 11

Slide 12

Slide 12 text

© 2016 Mesosphere, Inc. All Rights Reserved. IMPEDANCE MISMATCH 12

Slide 13

Slide 13 text

© 2016 Mesosphere, Inc. All Rights Reserved. TOWARDS 100% 
 SELF SERVICE 13

Slide 14

Slide 14 text

© 2016 Mesosphere, Inc. All Rights Reserved. CLOUD, CONTAINERS
 AND DEVOPS 14

Slide 15

Slide 15 text

© 2015 Mesosphere, Inc. All Rights Reserved. FAST AND BIG DATA … 15 *) kudos to Timothy St. Clair, @timothysc batch streaming PaaS MapReduce

Slide 16

Slide 16 text

© 2015 Mesosphere, Inc. All Rights Reserved. CHALLENGES 16 • Set up and operation of data pipeline components • Dealing with back-pressure: elasticity (static vs. dynamic partitioning) • Efficient usage of resources (utilization/TCO)

Slide 17

Slide 17 text

© 2015 Mesosphere, Inc. All Rights Reserved. TOOLING 17

Slide 18

Slide 18 text

© 2015 Mesosphere, Inc. All Rights Reserved. • Apache Kafka • ØMQ, RabbitMQ, Disque (Redis-based), etc. • fluentd, Logstash, Flume • Akka streams • cloud-only: AWS SQS, Google Cloud Pub/Sub • see also queues.io MESSAGE QUEUES & ROUTERS 18

Slide 19

Slide 19 text

© 2015 Mesosphere, Inc. All Rights Reserved. STREAM PROCESSING PLATFORMS 19 • Apache Storm • Apache Spark • Apache Samza • Apache Flink • Concord • cloud-only: AWS Kinesis, Google Cloud Dataflow • see also my webinar on stream processing

Slide 20

Slide 20 text

© 2015 Mesosphere, Inc. All Rights Reserved. TIME SERIES DATASTORES 20 • InfluxDB • OpenTSDB • KairosDB • Prometheus

Slide 21

Slide 21 text

© 2016 Mesosphere, Inc. All Rights Reserved. TIME FOR A NEW KIND OF OPERATING SYSTEM 21

Slide 22

Slide 22 text

© 2016 Mesosphere, Inc. All Rights Reserved. SINGLE MACHINE APPLICATION 22 hardware OS app

Slide 23

Slide 23 text

© 2016 Mesosphere, Inc. All Rights Reserved. DISTRIBUTED APPLICATION 23 hardware OS app hardware OS hardware OS hardware OS hardware OS hardware OS hardware OS

Slide 24

Slide 24 text

© 2016 Mesosphere, Inc. All Rights Reserved. DISTRIBUTED OS + DISTRIBUTED APP 24 hardware OS app hardware OS hardware OS hardware OS hardware OS hardware OS hardware OS distributed OS

Slide 25

Slide 25 text

© 2016 Mesosphere, Inc. All Rights Reserved. LOCAL OS
 VS
 DISTRIBUTED OS 25

Slide 26

Slide 26 text

© 2016 Mesosphere, Inc. All Rights Reserved. DC/OS ARCHITECTURE 26

Slide 27

Slide 27 text

© 2016 Mesosphere, Inc. All Rights Reserved. DC/OS BENEFITS 27 • One cluster for • stateless services such as Web servers & app servers • stateful services like PostgreSQL, MemSQL, Kafka, Cassandra, etc. • elastic data processing via Spark, Storm/Heron, Akka, etc. • CI/CD, for example Jenkins/Marathon • Dynamic partitioning of your cluster, depending on your needs • Increased utilization (10% → 80%+)

Slide 28

Slide 28 text

© 2016 Mesosphere, Inc. All Rights Reserved. BUT WHAT ABOUT HUMANS? 28

Slide 29

Slide 29 text

© 2016 Mesosphere, Inc. All Rights Reserved. 29 appops The person who writes an app is also the person responsible for operating the app in prod.

Slide 30

Slide 30 text

© 2016 Mesosphere, Inc. All Rights Reserved. 30 It's not about provisioning
 a VM or installing a DC/OS cluster or replacing a faulty HDD …
 
 … this would be on the infrastructure team. appops

Slide 31

Slide 31 text

© 2016 Mesosphere, Inc. All Rights Reserved. 31 human fault tolerance UX matters! protect people from themselves

Slide 32

Slide 32 text

© 2016 Mesosphere, Inc. All Rights Reserved. DEMO TIME! 32

Slide 33

Slide 33 text

© 2015 Mesosphere, Inc. All Rights Reserved. 33 A SIMPLE DATA PIPELINE https://api.github.com/orgs/$ORG/events

Slide 34

Slide 34 text

© 2015 Mesosphere, Inc. All Rights Reserved. 34 A SIMPLE DATA PIPELINE $ dcos package install marathon-lb $ dcos package install --options=config.json influxdb $ dcos package install grafana $ dcos marathon app add fetcher.json $ curl fetcher.marathon.l4lb.thisdcos.directory:80/start

Slide 35

Slide 35 text

© 2016 Mesosphere, Inc. All Rights Reserved. TAKE HOME MESSAGES 35 • Try to have short feedback loops • Containers and 'The Cloud' make deployment easy, leverage it! • Technology is the simple part of the solution:
 big data technologies won't fix your broken culture

Slide 36

Slide 36 text

© 2016 Mesosphere, Inc. All Rights Reserved. Q & A 36 • @mhausenblas • mhausenblas.info • michael@dcos.io https://dcos.io