Scala + Google Dataflow = Serverless Spark

喜歡 Spark 簡潔強大的 Scala 語法，但又覺得自己開一個 Spark Cluster 太多設定要弄懂嗎？一起來看看 Scio，由 Spotify 開發的 Google Cloud Dataflow Scala API。

Transcript

1. Scala + Google Dataflow = Serverless Spark @pishen
2. Google Dataflow? a Spark/Hadoop-like service with Serverless design on Google Cloud Platform https://cloud.google.com/dataflow/
3. Traditional Cluster EMR Spark EC2 Subnet setting? SSH setting? Firewall setting? Jobs How to see the dashboard? Memory setting? Routing table? What ports? Where is my key? How many GB do I have? Jumper? Tunneling?
4. Google Dataflow Job GCE (Google's EC2)
5. Google Dataflow Job GCE (Google's EC2) Subnet setting? SSH setting? Firewall setting? How to see the dashboard? Memory setting? Routing table? What ports? Where is my key? How many GB do I have? Jumper? Tunneling?
6. Google Dataflow Job GCE (Google's EC2) Subnet setting? SSH setting? Firewall setting? How to see the dashboard? Memory setting? Routing table? What ports? Where is my key? How many GB do I have? Jumper? Tunneling?
9. Google Dataflow
10. def main() = { ... sc.textFile() .map() .reduce() ... } Traditional Submission EMR Spark assembly run! spark-submit
11. def main() = { ... sc.textFile() .map() .reduce() ... sc.submit() } run! Google Dataflow API JSON DAG Job
12. Traditional Planning source rdd r1 val rdd = source.map(...) val r1 = rdd.count()
13. Traditional Planning source rdd r1 val rdd = source.map(...) val r1 = rdd.count()
14. Traditional Planning source rdd r1 val rdd = source.map(...) val r1 = rdd.count()
15. Traditional Planning source rdd r1 val rdd = source.map(...) val r1 = rdd.count()
16. Traditional Planning source rdd r1 val rdd = source.map(...) val r1 = rdd.count()
17. Traditional Planning source rdd r1 r2 val rdd = source.map(...) val r1 = rdd.count() val r2 = rdd.reduce(...)
18. Traditional Planning source rdd r1 r2 val rdd = source.map(...) val r1 = rdd.count() val r2 = rdd.reduce(...)
19. Traditional Planning source rdd r1 r2 val rdd = source.map(...) val r1 = rdd.count() val r2 = rdd.reduce(...)
20. Traditional Planning source rdd r1 r2 val rdd = source.map(...) val r1 = rdd.count() val r2 = rdd.reduce(...)
21. Traditional Planning source rdd r1 r2 val rdd = source.map(...) val r1 = rdd.count() val r2 = rdd.reduce(...) actions
22. Traditional Planning source rdd r1 val rdd = source.map(...) val r1 = rdd.count() rdd.cache()
23. Traditional Planning source rdd r1 val rdd = source.map(...) val r1 = rdd.count() rdd.cache()
24. Traditional Planning source rdd r1 val rdd = source.map(...) val r1 = rdd.count() rdd.cache()
25. Traditional Planning source rdd r1 val rdd = source.map(...) val r1 = rdd.count() rdd.cache()
26. Traditional Planning source rdd val rdd = source.map(...) val r1 = rdd.count() rdd.cache() r1
27. Traditional Planning source rdd r1 r2 val rdd = source.map(...) val r1 = rdd.count() val r2 = rdd.reduce(...) rdd.cache()
28. Traditional Planning source rdd r1 r2 val rdd = source.map(...) val r1 = rdd.count() val r2 = rdd.reduce(...) rdd.cache()
29. Traditional Planning source rdd r1 r2 val rdd = source.map(...) val r1 = rdd.count() val r2 = rdd.reduce(...) rdd.cache()
30. Traditional Planning source rdd r1 r2 val rdd = source.map(...) val r1 = rdd.count() val r2 = rdd.reduce(...) rdd.cache()
31. Google Dataflow source rdd r1 r2 val rdd = source.map(...) val r1 = rdd.count() val r2 = rdd.reduce(...) sc.submit()
32. Google Dataflow source rdd r1 r2 val rdd = source.map(...) val r1 = rdd.count() val r2 = rdd.reduce(...) sc.submit()
33. Google Dataflow source rdd r1 r2 val rdd = source.map(...) val r1 = rdd.count() val r2 = rdd.reduce(...) sc.submit()
34. Google Dataflow source rdd r1 r2 val rdd = source.map(...) val r1 = rdd.count() val r2 = rdd.reduce(...) sc.submit()
35. Google Dataflow source rdd r1 r2 val rdd = source.map(...) val r1 = rdd.count() val r2 = rdd.reduce(...) No actions at all! sc.submit() Doesn't need cache()
36. Other Features • Cloud Shuffle • Streaming • Built-in I/O Transforms
37. Cloud Storage Datastore BigQuery Dataflow Pub/Sub Cloud Storage Datastore BigQuery Pub/Sub I/O Transforms
38. Looks perfect, right?
39. Dataflow The Ugly Part Apache Beam https://beam.apache.org/
40. Dataflow Beam Java Beam Python Beam Go The Ugly Part 1st priority
41. The Ugly Part Beam Java Beam Python Beam Go
42. The Ugly Part sc.textFile("s3://bucket/*").flatMap(_.split("[^\\p{L}]+")) Spark p.apply(TextIO.read().from("gs://bucket/*")) .apply("ExtractWords", FlatMapElements.into(TypeDescriptors.strings()) .via((String word) -> Arrays.asList(word.split("[^\\p{L}]+")))) Beam Java (p | beam.io.ReadFromText('gs://bucket/*') | 'ExtractWords' >> beam.FlatMap(lambda x: re.findall(r'[A-Za-z\']+', x))) Beam Python lines := textio.Read(s, "gs://bucket/*") words := beam.ParDo(s, func(line string, emit func(string)) { for _, word := range wordRE.FindAllString(line, -1) { emit(word) } }, lines) Beam Go
45. Dataflow Beam Java Beam Python Beam Go Scio
46. sc.textFile("s3://bucket/*").flatMap(_.split("[^\\p{L}]+")) Spark sc.textFile("gs://bucket/*").flatMap(_.split("[^\\p{L}]+")) Scio
47. sc.textFile("s3://bucket/*").flatMap(_.split("[^\\p{L}]+")) Spark sc.textFile("gs://bucket/*").flatMap(_.split("[^\\p{L}]+")) Scio SparkContext ScioContext RDD[String] SCollection[String]
48. .collect .count .distinct .filter .flatMap .groupBy .keyBy .map .max .reduce .sortBy .take .zipWithIndex .count .distinct .filter .flatMap .groupBy .keyBy .map .max .reduce .take RDD SCollection actions
49. RDD def count(): Long SCollection def count(): SCollection[Long] SCollection with single element
50. .countByKey .groupByKey .join .mapValues .reduceByKey .rightOuterJoin .countByKey .groupByKey .intersectByKey .join .mapValues .maxByKey .reduceByKey .rightOuterJoin .sumByKey PairRDDFunctions PairSCollectionFunctions
51. Spark val size: Long = rdd1.count val bSize: Broadcast[Long] = sc.broadcast(size) rdd2.map(x => x + bSize.value) val size: SCollection[Long] = scollection1.count val sSize: SideInput[Long] = size.asSingletonSideInput scollection2 .withSideInputs(sSize) .map { case (x, s) => x + s(sSize)) Scio
52. The Power of Scala Macros @BigQueryType.fromQuery("SELECT user, ad FROM click_logs") class Row sc.typedBigQuery[Row]() .groupBy(row => row.user) .mapValues(_.map(row => row.ad).sorted.takeRight(100))
53. The Power of Scala Macros @BigQueryType.fromQuery("SELECT user, ad FROM click_logs") class Row(user: String, ad: Long) sc.typedBigQuery[Row]() .groupBy(row => row.user) .mapValues(_.map(row => row.ad).sorted.takeRight(100))
54. The Power of Scala Macros @BigQueryType.fromQuery("SELECT user, ab FROM click_logs") class Row sc.typedBigQuery[Row]() .groupBy(row => row.user) .mapValues(_.map(row => row.ab).sorted.takeRight(100)) Compile error: Field 'ab' not found.
55. 5 steps to run a Scio job
56. Step 0 1. Open a GCP project 3. Enable Dataflow API 2. Give your credit card to Google $300 free trial~!
57. Step 1 brew cask install google-cloud-sdk Install Google Cloud SDK Setup gcloud gcloud init gcloud auth application-default login
58. Step 2 libraryDependencies ++= Seq( "com.spotify" %% "scio-core" % "0.5.2", "org.apache.beam" % "beam-runners-google-cloud-dataflow-java" % "2.4.0" ) https://github.com/spotify/scio/wiki/Getting-Started#sbt-project-setup
59. Step 3 def main(args: Array[String]): Unit = { val sc = ScioContext { val opts = PipelineOptionsFactory.as(classOf[DataflowPipelineOptions]) opts.setRegion("asia-east1") opts.setProject("your-project-id") opts.setRunner(classOf[DataflowRunner]) opts } sc.textFile("gs://your-bucket/*.txt") .flatMap(_.split(" ")) .countByValue() .saveAsTextFile("gs://output-bucket/output-folder/") sc.close() }
60. Step 4 sbt run
61. Step 5 Check the Dataflow web console on GCP
62. sc.thankYou()