Data at the Speed of your Users

Apache Cassandra and Spark for simple, distributed, near real-time stream processing.

Transcript

1. Data at the Speed of your Users Apache Cassandra and Spark for simple, distributed, near real-time stream processing. GOTO Copenhagen 2014
2. Rustam Aliyev Solution Architect at . ! ! @rstml
3. Big Data? Photo: Flickr / Watches En Masse
4. " Volume # Variety $ Velocity
5. Velocity = Near Real Time
6. Near Real Time?
7. 0.5 sec ≤ ≤ 60 sec Near Real Time
8. Use Cases Photo: Flickr / Swiss Army / Jim Pennucci
9. Web Analytics Dynamic Pricing Recommendation Fraud Detection
10. Architecture Photo: Ilkin Kangarli / Baku Haydar Aliyev Center
11. Architecture Goals Low Latency High Availability Horizontal Scalability Simplicity
12. Stream Processing % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % Collection Processing Storing Delivery
13. Stream Processing % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % Collection ! ! Spark  ! Cassandra Delivery
14. Cassandra Distributed Database Photo: Flickr / Hypostyle Hall / Jorge Láscar
15. Data Model
16. Partition Cell 1 Cell 2 … Cell 3 Partition Key
17. Partition os: Android storage: 32GB version: 4.4 weight: 130g sort order on disk Nexus 5
18. Table os: Android storage: 32GB version: 4.4 weight: 130g Nexus 5 os: iOS storage: 64GB version: 8.0 weight: 129g iPhone 6
19. Distribution
20. 0000 8000 4000 C000 2000 6000 E000 A000 3D97 Nexus 5
21. 0000 8000 4000 C000 2000 6000 E000 A000 9C4F iPhone 6 3D97
22. Replication
23. 0000 8000 4000 C000 2000 6000 E000 A000 3D97 9C4F 1 replica
24. 0000 8000 4000 C000 2000 6000 E000 A000 3D97 9C4F 9C4F 3D97 2 replicas
25. Spark Distributed Data Processing Engine Photo: Flickr / Sparklers / Alexandra Compo / CreativeCommons
26. Fast In-memory
27. Logistic Regression Running Time (s) 1000 2000 3000 4000 Number of Iterations 1 5 10 20 30 Spark Hadoop
28. Easy
29. map ! reduce
30. map filter groupBy sort union join leftOuterJoin rightOuterJoin reduce count fold reduceByKey groupByKey cogroup cross zip sample take first partitionBy mapWith pipe save 
 ...
31. RDD Resilient Distributed Datasets Node 1 Node 2 Node 3 Node 1 Node 2 Node 3
32. Operator DAG groupBy join filter map Disk RDD Memory RDD
33. Spark Streaming Micro-batching
34. RDD DStream Data Stream
35. Spark + Cassandra DataStax Spark Cassandra Connector
36. https://github.com/datastax/spark-cassandra-connector
37.         M M M  Cassandra Spark Worker Spark Master & Worker
38. Demo ! ! Twitter Analytics
39. Cassandra Data Model
40. ALL: 7139 2014-09-21: 220 2014-09-20: 309 2014-09-19: 129 sort order #hashtag
41. CREATE  TABLE  hashtags  (          hashtag  text,          interval  text,          mentions  counter,          PRIMARY  KEY((hashtag),  interval)   )  WITH  CLUSTERING  ORDER  BY  (interval  DESC);  
42. Processing Data Stream
43. import  com.datastax.spark.connector.streaming._   ! val  sc  =  new  SparkConf()      .setMaster("spark://127.0.0.1:7077")      .setAppName("Twitter-­‐Demo")      .setJars("demo-­‐assembly-­‐1.0.jar"))      .set("spark.cassandra.connection.host",  "127.0.0.1")   ! val  ssc  =  new  StreamingContext(sc,  Seconds(2))   ! val  stream  =  TwitterUtils.      createStream(ssc,  None,  Nil,  storageLevel  =  StorageLevel.MEMORY_ONLY_SER_2)   ! val  hashTags  =  stream.flatMap(tweet  =>      tweet.getText.toLowerCase.split("  ").      filter(tags.contains(Seq("#iphone",  "#android"))))   ! val  tagCounts  =  hashTags.map((_,  1)).reduceByKey(_  +  _)   ! val  tagCountsAll  =  tagCounts.map{     case  (tag,  mentions)  =>  (tag,  mentions,  "ALL")   }   !
44. import  com.datastax.spark.connector.streaming._   ! val  sc  =  new  SparkConf()      .setMaster("spark://127.0.0.1:7077")      .setAppName("Twitter-­‐Demo")      .setJars("demo-­‐assembly-­‐1.0.jar"))      .set("spark.cassandra.connection.host",  "127.0.0.1")   ! val  ssc  =  new  StreamingContext(sc,  Seconds(2))   ! val  stream  =  TwitterUtils.      createStream(ssc,  None,  Nil,  storageLevel  =  StorageLevel.MEMORY_ONLY_SER_2)   ! val  hashTags  =  stream.flatMap(tweet  =>      tweet.getText.toLowerCase.split("  ").      filter(tags.contains(Seq("#iphone",  "#android"))))   ! val  tagCounts  =  hashTags.map((_,  1)).reduceByKey(_  +  _)   ! val  tagCountsAll  =  tagCounts.map{     case  (tag,  mentions)  =>  (tag,  mentions,  "ALL")   }   !
45. import  com.datastax.spark.connector.streaming._   ! val  sc  =  new  SparkConf()      .setMaster("spark://127.0.0.1:7077")      .setAppName("Twitter-­‐Demo")      .setJars("demo-­‐assembly-­‐1.0.jar"))      .set("spark.cassandra.connection.host",  "127.0.0.1")   ! val  ssc  =  new  StreamingContext(sc,  Seconds(2))   ! val  stream  =  TwitterUtils.      createStream(ssc,  None,  Nil,  storageLevel  =  StorageLevel.MEMORY_ONLY_SER_2)   ! val  hashTags  =  stream.flatMap(tweet  =>      tweet.getText.toLowerCase.split("  ").      filter(tags.contains(Seq("#iphone",  "#android"))))   ! val  tagCounts  =  hashTags.map((_,  1)).reduceByKey(_  +  _)   ! val  tagCountsAll  =  tagCounts.map{     case  (tag,  mentions)  =>  (tag,  mentions,  "ALL")   }   !
46. ! val  ssc  =  new  StreamingContext(sc,  Seconds(2))   ! val  stream  =  TwitterUtils.      createStream(ssc,  None,  Nil,  storageLevel  =  StorageLevel.MEMORY_ONLY_SER_2)   ! val  hashTags  =  stream.flatMap(tweet  =>      tweet.getText.toLowerCase.split("  ").      filter(tags.contains(Seq("#iphone",  "#android"))))   ! val  tagCounts  =  hashTags.map((_,  1)).reduceByKey(_  +  _)   ! val  tagCountsAll  =  tagCounts.map{     case  (tag,  mentions)  =>  (tag,  mentions,  "ALL")   }   ! tagCountsAll.saveToCassandra(     "demo_ks",  "hashtags",  Seq("hashtag",  "mentions",  "interval"))   ! ssc.start()   ssc.awaitTermination()  
47. ! val  ssc  =  new  StreamingContext(sc,  Seconds(2))   ! val  stream  =  TwitterUtils.      createStream(ssc,  None,  Nil,  storageLevel  =  StorageLevel.MEMORY_ONLY_SER_2)   ! val  hashTags  =  stream.flatMap(tweet  =>      tweet.getText.toLowerCase.split("  ").      filter(tags.contains(Seq("#iphone",  "#android"))))   ! val  tagCounts  =  hashTags.map((_,  1)).reduceByKey(_  +  _)   ! val  tagCountsByDay  =  tagCounts.map{     case  (tag,  mentions)  =>  (tag,  mentions,  DateTime.now.toString("yyyyMMdd"))   }   ! tagCountsByDay.saveToCassandra(     "demo_ks",  "hashtags",  Seq("hashtag",  "mentions",  "interval"))   ! ssc.start()   ssc.awaitTermination()  
48. ! val  ssc  =  new  StreamingContext(sc,  Seconds(2))   ! val  stream  =  TwitterUtils.      createStream(ssc,  None,  Nil,  storageLevel  =  StorageLevel.MEMORY_ONLY_SER_2)   ! val  hashTags  =  stream.flatMap(tweet  =>      tweet.getText.toLowerCase.split("  ").      filter(tags.contains(Seq("#iphone",  "#android"))))   ! val  tagCounts  =  hashTags.map((_,  1)).reduceByKey(_  +  _)   ! val  tagCountsAll  =  tagCounts.map{     case  (tag,  mentions)  =>  (tag,  mentions,  "ALL")   }   ! tagCountsAll.saveToCassandra(     "demo_ks",  "hashtags",  Seq("hashtag",  "mentions",  "interval"))   ! ssc.start()   ssc.awaitTermination()  
49. Questions ?