In-memory computing with Akka, Spark and Cassandra

Transcript

2. In-memory computing with Akka, Spark and Cassandra Natalino Busa Data Platform Architect at Ing
3. Why Fast Data? 1. Relevant up-to-date information. 2. Delivers actionable events.
4. Why Big Data? 1. Analyze and model 2. Learn, cluster, categorize, organize facts
5. Big Data and Fast Data population: events, transactions, sessions, customers, etc
6. SMACK S Spark/Scala M Mesos A Akka C Cassandra K Kafka
7. high available on multiple data centers tunable consistency very scalable, low latency masterless architecture, all nodes are equal flexible data model (more on this later) 2D key-value store hashed access on partition keys sorted on clustering keys Cassandra
8. Cassandra: Write replication Application Murmur3 hashing. Hashes (token) are partitioned on the nodes Consistent: Nreads + Nwrites > Nreplicas Trade availability for consistency Serialization (Paxos) since 2.0 Compaction and Repartitioning happen in the background
9. Application replication factor 3 Cassandra: Write replication Murmur3 hashing. Hashes (token) are partitioned on the nodes Consistent: Nreads + Nwrites > Nreplicas Trade availability for consistency Serialization (Paxos) since 2.0 Compaction and Repartitioning happen in the background
10. Application DC1: replication factor 3 DC2: replication factor 3 Cassandra: Write replication
11. Cassandra Queries Can you do ….. filtering: limited transformations: limited aggregations: limited joins: no Speed comes with a cost. … or shouldn’t it? It’s NoSQL man.
12. Cassandra-Spark Connector Cassandra: Store all the data Spark: Analyze all the data DC1: replication factor 3 DC2: replication factor 3 DC2: replication factor 3 + Spark Executors Transactional Analytics
13. Cassandra-Spark Connector Cassandra: Store all the data Spark: Analyze all the data DC1: replication factor 3 DC2: replication factor 3 DC2: replication factor 3 + Spark Executors Transactional Analytics Application
14. Cassandra-Spark Stack Cassandra: Store all the data Spark: Distributed Data Processing Executors and Workers Cluster Resource Allocators: Mesos, Yarn, Spark Connector: Data locality, Reduce Shuffling RDD’s to Cassandra Partitions DC2: replication factor 3 + Spark Executors Analytics
15. Cassandra-Spark Stack Cassandra-Spark Connector Cassandra Spark Streaming SQL MLlib Graphx Spark SQL: SQL library on Spark RDDs MLlib: Spark Machine Learning library Graphx: Spark Graph library Spark Streaming: scheduling micro batches on Spark DC2: replication factor 3 + Spark Executors
16. Cassandra-Spark Stack: Use cases Cassandra-Spark Connector Cassandra Spark Streaming SQL MLlib Graphx • Sanitize, Validate • Transform and Enrich: ◦ Model Scoring, Look-up-tables, UDFs • Schema migration, Data Conversion • Analytics: Stats and Model building • Load/Save to other sources
17. Cassandra-Spark Stack Cassandra-Spark Connector Cassandra Spark Streaming SQL MLlib Graphx Extract Data Create Models, Enrich, Transform Fetch from other Sources: Kafka Fetch from other Sources: DB’s, Files API’s REST resources API’s (model scoring) Analytics, Statistics, Data Science, Model Training Solr
18. 18 Real Time APIs Streaming Data Data Sources, Files, DB extracts Batched Data Training, Scoring and Exposing models
19. 19 Real Time APIs Streaming Data Data Sources, Files, DB extracts Batched Data Training, Scoring and Exposing models
20. 20 Real Time APIs Streaming Data Data Sources, Files, DB extracts Batched Data Training, Scoring and Exposing models
21. Cassandra+Akka+Spark: Machine Learning Fast writes 2D Data Structure Replicated Tunable consistency Multi-Data centers C* Akka Spark Very Fast processing Distributed, Scalable computing Actor-based Pipelines Actor state can be persisted Supervision strategies Ad-Hoc Queries Joins, Aggregate User Defined Functions Machine Learning, Advanced Stats and Analytics
22. Data Science: Anomaly Detection An outlier is an observation that deviates so much from other observations as to arouse suspicion that it was generated by a different mechanism. Hawkins, 1980
23. Data Science: Anomaly Detection Distance Based Density Based
24. Example: Analyze gowalla check-ins year | month | day | time | uid | lat | lon | ts | vid ------+-------+-----+------+--------+----------+-----------+--------------------------+--------- 2010 | 9 | 14 | 91 | 853 | 40.73474 | -73.87434 | 2010-09-14 00:01:31+0000 | 917955 2010 | 9 | 14 | 328 | 4516 | 40.72585 | -73.99289 | 2010-09-14 00:05:28+0000 | 37160 2010 | 9 | 14 | 344 | 2964 | 40.67621 | -73.98405 | 2010-09-14 00:05:44+0000 | 956870 Check-ins dataset Venues dataset vid | name | lat | long ------+-------+-----+------+--------+----------+----------- +--------------------------+--------- 754108 | My Suit NY | 40.73474 | -73.87434 249755 | UA Court Street Stadium 12 | 40.72585 | -73.99289 6919688 | Sky Asian Bistro | 40.67621 | -73.98405
25. Data Science: clustering venues Weekly visitors patterns! Madison Square, Apple Store, Radio City Music Hall Thursdays, Fridays, Saturdays are busy Statue of Liberty, Jacob K. Javits Convention Center, Whole Foods Market (Columbus Circle) Not popular on midweek Intuition:
26. Data Science: clustering with k-means Histograms components as dimensions Similar histograms would occupy similar places in the feature space How do I compare histograms: - EMD - Chi-squared distance - Space transformation (DCT) Intuition:
27. K-Means: Featurize data + cluster val weekly_visits = checkins_venues.select("vid","ts") .map(row => (row.getLong("vid"), vectorize_time(s.getTimestamp("ts")) .reduceByKey(_ + _) .mapValues(_ => featurize_histogram(_._1)) val numClusters = 15 val numIterations = 100 val clusters = KMeans.train(weekly_visits, numClusters, numIterations) PairRDDs, weekly patterns per venue cluster similar weekly patterns
28. How to use it 1) Classification Classify venues to given groups 2) Anomaly Detection Detect shift in the clustering assignment for a given venue for a given week Keep monitoring weekly change in patterns, when it happens trigger a signal week 26 week 27 Action
29. MLlib KMeans.train(FeaturesRDD, numClusters, numIterations) MLlib Truly distributed algorithm Classify venues to given groups Millions of datapoints Limited amount of clusters
30. 30 Real Time APIs Streaming Data Data Sources, Files, DB extracts Batched Data Training, Scoring and Exposing models
31. Training vs Scoring: Latency budget • Akka: millisecond response • Spark: in-memory data models Train: Spark Score: Spark Train: Spark Score: Akka slow: minutes fast: millisecs Model Scoring Model Training slow: minutes
32. Akka Mixed Load Cassandra Cluster Akka: Web API for dynamic data flows
33. Akka: asynchronous & event-driven • Fill the wait with other requests • React to events and continue where you left • Utilize your hardware to the max: CPUs, Bandwidth
34. Akka: actors • Messages from/to other actors • Internal state not shared • Create other actors • Supervised by its parent actor
35. class TemperatureActor extends Actor { var temperature = 0.0 def receive = { case value: Double => temperature = max(temperature, value) case "get" => sender ! temperature } } Akka: actors
36. Scala / Akka: a WEB API reactive framework Actor A Actor B Actor C msg 1 msg 2 msg 3 msg 4 • it scales horizontally (can run in cluster mode) • maximum use of the available cores/memory 1. processing is non-blocking, threads are re-used 2. can parallelize computing power across many actors Very fast: 1000’s messages/sec Very reliable: auto recovery
37. Mesos Spark Streaming SQL MLlib Graphx DBs ES C*
38. 38 Real Time APIs Streaming Data Data Sources, Files, DB extracts Batched Data Summary:
39. Thanks!