Funktionale Parallelität in Scala

Transcript

1. Funktionale Parallelität in Scala
2. Procedural Parallelism vs.
   Functional Parallelism
3. How
   vs.
   What
4. A uniform API for parallelism
5. map / / transform from a -> b flatMap / / transform a -> a1..ax filter / / only use certain values zip / / combine computations reduce / / transform a1..ax -> b
6. Divide and Conquer
7. Parallel Sum
8. Technical View
9. Slice P P P W W
10. API View
11. def sum(numbers:Seq[Long]) =
    numbers.reduce(_+_)
    
12. A Scala ForkJoin Interface
13. def sum(numbers:Seq[Long]) =
    numbers.par.reduce(_+_)
14. Async IO Processing
15. Reduce Blocking -> Increase parallelism
17. 0 20 40 60 80 Deal2 Deal1 Deal5 Deal4 Deal3 Deals Berlin
19. App Index Page Deal 1 Deal 2 Deal X HTML Parser Output Gen
20. App Index Get D1 Get D2 Get DX Result Parse D1 Parse D2 Parse DX Merge Time
21. Futures
22. java.util.concurrent
25. scala.concurrent
26. map flatMap filter forEach zip andThen ...
27. val f = for { a ˡ Future(10 / 2) / / 10 / 2 = 5 b ˡ Future(a + 1) / / 5 + 1 = 6 c ˡ Future(a - 1) / / 5 - 1 = 4 if c > 3 } yield b * c
28. Back to the example
29. App Index Get D1 Get D2 Get DX Result Parse D1 Parse D2 Parse DX Merge Time
30. A chain of dependent operations happening in the future
    
31. def dealOverviewData(city: String) = for
    { links
    <- dealLinksForCity(city) dealPriceAndDiscounts
    <- fetchAndExtractDeals(links)
    if !dealPriceAndDiscounts.isEmpty } yield dealPriceAndDiscounts
32. App Index Get D1 Get D2 Get DX Result Parse D1 Parse D2 Parse DX Merge Time
33. def dealLinksForCity(city: String) = { WS.url(url+city).get() .map(_.body.toString)
    .map(extractLinks) }
34. App Index Get D1 Get D2 Get DX Result Parse D1 Parse D2 Parse DX Merge Time
35. def dealOverviewData(city: String) = for
    { links
    <- dealLinksForCity(city) dealPriceAndDiscounts
    <- fetchAndExtractDeals(links)
    if !dealPriceAndDiscounts.isEmpty } yield dealPriceAndDiscounts “Parallel Looping”
36. def fetchAndExtractDeals(links:List[String]) = ....
    links.map(fetchAndExtractDealData)
    ....
37. def fetchAndExtractDealData(link: String) = { WS.url(link).get() .map(_.body.toString) .map(extractDealData) }
38. App Index Get D1 Get D2 Get DX Result Parse D1 Parse D2 Parse DX Merge Time
39. def fetchAndExtractDeals(links:List[String]) = Future.sequence(
    links.map(fetchAndExtractDealData)
    )
40. def dealOverviewData(city: String) = for
    { links
    <- dealLinksForCity(city) dealPriceAndDiscounts
    <- fetchAndExtractDeals(links)
    if !dealPriceAndDiscounts.isEmpty } yield dealPriceAndDiscounts
41. https:/ /github.com/ tobnee/play-async-ws- demo
42. Parallel (Distributed) Batch Processing
43. MapReduce
44. map (k1,v1) ˠ list(k2,v2) reduce (k2,list(v2)) ˠ list(v3)
45. Distributed
    WordCount http:/ /commons.wikimedia.org/wiki/File:Rosetta_Stone.JPG
46. map(String key, String value): for each word w in value: EmitIntermediate(w, "1");
47. reduce(String key, Iterator values): int result = 0; for each v in values: result += ParseInt(v); Emit(AsString(result));
48. http:/ /de.wikipedia.org/wiki/Datei:Mapreduce_(Ville_Tuulos).png
49. Lets get real
50. 1 package org.myorg; 2 3 import java.io.IOException; 4 import java.util.*; 5 6 import org.apache.hadoop.fs.Path; 7 import org.apache.hadoop.conf.*; 8 import org.apache.hadoop.io.*; 9 import org.apache.hadoop.mapreduce.*; 10 import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; 11 import org.apache.hadoop.mapreduce.lib.input.TextInputFormat; 12 import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; 13 import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat; 14 15 public class WordCount { 16 17 public static class Map extends Mapper<LongWritable, Text, Text, IntWritable> { 18 private final static IntWritable one = new IntWritable(1); 19 private Text word = new Text(); 20 21 public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException { 22 String line = value.toString(); 23 StringTokenizer tokenizer = new StringTokenizer(line); 24 while (tokenizer.hasMoreTokens()) { 25 word.set(tokenizer.nextToken()); 26 context.write(word, one); 27 } 28 } 29 } 30 31 public static class Reduce extends Reducer<Text, IntWritable, Text, IntWritable> { 32 33 public void reduce(Text key, Iterable<IntWritable> values, Context context) 34 throws IOException, InterruptedException { 35 int sum = 0; 36 for (IntWritable val : values) { 37 sum += val.get(); 38 } 39 context.write(key, new IntWritable(sum)); 40 } 41 } 42 43 public static void main(String[] args) throws Exception { 44 Configuration conf = new Configuration(); 45 46 Job job = new Job(conf, "wordcount"); 47 48 job.setOutputKeyClass(Text.class); 49 job.setOutputValueClass(IntWritable.class); 50 51 job.setMapperClass(Map.class); 52 job.setReducerClass(Reduce.class); 53 54 job.setInputFormatClass(TextInputFormat.class); 55 job.setOutputFormatClass(TextOutputFormat.class); 56 57 FileInputFormat.addInputPath(job, new Path(args[0])); 58 FileOutputFormat.setOutputPath(job, new Path(args[1])); 59 60 job.waitForCompletion(true); 61 } 62 63 }
51. FP equivalent in Scala
52. val lines = fromTextFile("hdfs:/ /in/...") ! val counts = lines.flatMap(line => line.split(" ")) .map(word => (word, 1)) .groupByKey .combine(_+_) ! persist(counts.toTextFile("hdfs:/ /out/...", overwrite=true))
53. We have seen classic examples of parallel computations MapReduce ForkJoin Async Workflows
54. Some of these concepts can be expressed in a uniform, monadic way This style of programming is well suited for functional languages like Scala, Haskell, Clojure or F#