Developing Elastic Data Pipelines

Webinar, eSynergySolutions, see also video at https://www.youtube.com/watch?v=9y1qHY3JMko

Transcript

1. © 2016 Mesosphere, Inc. All Rights Reserved. DEVELOPING ELASTIC DATA PIPELINES 1 Michael Hausenblas, Developer & Cloud Advocate | 2016-03-14 | Webinar
2. © 2016 Mesosphere, Inc. All Rights Reserved. MOTIVATION 2
3. © 2016 Mesosphere, Inc. All Rights Reserved. AIRLINES 3
4. © 2016 Mesosphere, Inc. All Rights Reserved. LOGISTICS 4
5. © 2016 Mesosphere, Inc. All Rights Reserved. HEALTH
 CARE 5
6. © 2016 Mesosphere, Inc. All Rights Reserved. TRADERS 6
7. © 2016 Mesosphere, Inc. All Rights Reserved. CITIES 7 © 2014, Wired magazine
8. © 2016 Mesosphere, Inc. All Rights Reserved. YOU 8
9. © 2016 Mesosphere, Inc. All Rights Reserved. THE
 TOOLBOX 9
10. © 2016 Mesosphere, Inc. All Rights Reserved. LET'S TALK ABOUT WORKLOADS* … 10 *) kudos to Timothy St. Clair, @timothysc batch streaming PaaS MapReduce
11. © 2016 Mesosphere, Inc. All Rights Reserved. • Apache Kafka • ØMQ, RabbitMQ, Disque (Redis-based), etc. • fluentd, Logstash, Flume • Akka streams • cloud-only: AWS SQS, Google Cloud Pub/Sub • see also queues.io MESSAGE QUEUES & ROUTERS 11
12. © 2016 Mesosphere, Inc. All Rights Reserved. APACHE KAFKA 12 • High-throughput, distributed, persistent publish-subscribe messaging system • Originates from LinkedIn • Typically used as buffer/de-coupling layer in online stream processing Message queues & routers kafka.apache.org
13. © 2016 Mesosphere, Inc. All Rights Reserved. FLUENTD 13 Message queues & routers www.fluentd.org
14. © 2016 Mesosphere, Inc. All Rights Reserved. STREAM PROCESSING PLATFORMS 14 • Apache Storm • Apache Spark • Apache Samza • Apache Flink • Concord • cloud-only: AWS Kinesis, Google Cloud Dataflow • see also my webinar on stream processing
15. © 2016 Mesosphere, Inc. All Rights Reserved. APACHE STORM 15 • Distributed, fault-tolerant stream- processing platform • Guaranteed message processing (replaying messages on failure) • Concepts: tuples, streams, spouts, bolts, topologies Stream processing platforms storm.apache.org
16. © 2016 Mesosphere, Inc. All Rights Reserved. APACHE SPARK 16 Stream processing platforms spark.apache.org Spark SQL Spark Streaming MLlib
 (machine learning) Spark core (RDD) GraphX
 (graph processing) Mesos Filesystem (local, HDFS, S3) or data store (HBase, Cassandra, Elasticsearch, etc.) YARN Standalone
17. © 2016 Mesosphere, Inc. All Rights Reserved. TIME SERIES DATASTORES 17 • InfluxDB • OpenTSDB • KairosDB • Prometheus • see also iot-a.info
18. © 2016 Mesosphere, Inc. All Rights Reserved. OPENTSDB 18 • Distributed time series database on top HBase • Store, index, query & plot metrics • Extremely scalable • Low-level monitoring Time series datastores opentsdb.net
19. © 2016 Mesosphere, Inc. All Rights Reserved. INFLUXDB 19 • No-dependency, time series database written in Go • SQLish query language (incl. regex, fan out) • Single node or Raft-based distributed node mode Time series datastores influxdb.com
20. © 2016 Mesosphere, Inc. All Rights Reserved. CHALLENGES 20 The Toolbox • Distributed systems are hard • Set up and operation of components • One (static) cluster per component • Efficient usage of cluster resources (TCO)
21. © 2016 Mesosphere, Inc. All Rights Reserved. MEET THE DATACENTER
 OPERATING
 SYSTEM
 21
22. © 2016 Mesosphere, Inc. All Rights Reserved. 22
23. © 2016 Mesosphere, Inc. All Rights Reserved. BENEFITS 23 DCOS • Run stateless services (Web server, app server, etc.) and Big Data services like Kafka, Spark, or Cassandra together on one cluster • Dynamic partitioning of your cluster, depending on your business requirements • Increased utilization (10% → 80%++)
24. © 2016 Mesosphere, Inc. All Rights Reserved. INFINITY 24 DCOS https://mesosphere.com/infinity/
25. © 2016 Mesosphere, Inc. All Rights Reserved. ELASTIC DATA PIPELINES BY
 EXAMPLE 25
26. © 2016 Mesosphere, Inc. All Rights Reserved. A SIMPLE DATA PIPELINE 26 Examples https://github.com/mesosphere/cassandra-kairosdb-tutorial
27. © 2016 Mesosphere, Inc. All Rights Reserved. HYBRID DATA PIPELINE 27 Examples https://mesosphere.com/blog/2015/11/18/dcos-time-series-demo
28. © 2016 Mesosphere, Inc. All Rights Reserved. HANDS- ON … 28 Examples
29. © 2016 Mesosphere, Inc. All Rights Reserved. Q & A 29 • @mhausenblas • mhausenblas.info • @mesosphere • mesosphere.com