From Big Data to Fast Data (DevFest)

An introduction to Apache Spark.

Transcript

1. From Big Data to Fast Data Apache Spark Stefano Baghino / GDG Milano
2. Hi! •  Collaborator @ GDG Milano •  Software Engineer @ DATABIZ I stare at shiny rectangles and push buttons. Sometimes good things happen and people pay me for this!
3. Wait! Who are you? •  Are you a lambda dancer? •  Do you have a PB of doges?
4. Agenda Big Data? Fast Data? What do we have now? How can we do better? What is Spark? What does it do? How does it work? And also code, somewhere here and there.
5. Big Data Huge and diverse datasets. Just for today, size matters. You need a cluster, you need horizontal scalability. New problems require new solutions.
6. What about Fast Data? You have a massive, endless stream of data. And you can’t wait for a batch computation to complete. You must have an answer now.
7. What do we have now? MapReduce is one of the leading cluster computing frameworks. It scales beautifully, but... But it does I/O at each step. And the computational model is too restrictive.
8. Meet Apache Spark It works with your cluster. Simple, expressive API with multi-language support. Computes and caches in-memory. It’s fast!
9. Fast? Really? Yup! Hadoop 102.5 TB Spark 100 TB Spark 1 PB Elapsed Time 72’ 23’ 234’ # Cores 50400 6592 6080 Rate/Node 0.67 GB/min 20.7 GB/min 22.5 GB/min Source: https://databricks.com/blog/2014/10/10/spark-petabyte-sort.html
10. Enjoy the API anyway you want Java Scala Python
11. How does it play with my cluster? Local Standalone Hadoop Mesos
12. A lot of interest around it All major Big Data players are supporting it. Available on the cloud from Spark makers, Databricks. Also available for Click to Deploy on Google Compute Engine*. * https://cloud.google.com/solutions/hadoop/click-to-deploy
13. Where do things go?
14. Introducing the RDD Say hi to Spark’s main data abstraction, the Resilient Distributed Dataset From the user’s point of view, the RDD is effectively a collection, hiding all the details of its distribution throughout the cluster.
15. CODE!!!
16. The anatomy of an RDD
17. What about resilience? When you have a large cluster, things fail often. You must have a way to recover from failures. RDDs hide a little secret.
18. The true nature of an RDD
19. What can I do with an RDD? Transformations Produce a new RDD, extending the graph at each step (map, flatMap, filter and the like) Actions They are “terminal” operations, executing the graph (like collect or reduce)
20. The execution model 1.  Create a DAG of RDDs to represent computation 2.  Create the logical execution plan for the DAG 3.  Schedule and execute individual tasks
21. The execution model in action Let’s look at the process in slow motion. Let’s count distinct names grouped by their initial.
22. First, create the logical DAG
23. Then, create the execution plan
24. So, how is it resilient? RDDs are “just” a lazy, immutable representation of computation. If something goes wrong, Spark will simply try again.
25. The ecosystem
26. The ecosystem
27. Spark Streaming
28. “Mini-batching” These “mini-batches” are DStreams (discretized streams). It’s just a collection of RDDs. Create your DStreams from streaming sources. Apply transformations just as you would with RDDs.
29. CODE!!! Available on Github: https://github.com/stefanobaghino/spark-twitter-stream-example
30. Homework! •  Caching •  Shared variables •  Partitioning optimization •  DataFrames •  Know the API •  Learn the ecosystem
31. THANKS! Psst... we’re hiring! Java? Scala? Big Data? Front End? Come talk to me afterwards. @stefanobaghino stefano.baghino@databiz.it