Parallel Collections Overview

Slide 1

Slide 1 text

Scala Parallel Collections Aleksandar Prokopec EPFL

Slide 2

Slide 2 text

No content

Slide 3

Slide 3 text

Scala collections for { s <- surnames n <- names if s endsWith n } yield (n, s) McDonald

Slide 4

Slide 4 text

Scala collections for { s <- surnames n <- names if s endsWith n } yield (n, s) 1040 ms

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

Scala parallel collections for { s <- surnames n <- names if s endsWith n } yield (n, s)

Slide 7

Slide 7 text

Scala parallel collections for { s <- surnames.par n <- names.par if s endsWith n } yield (n, s)

Slide 8

Slide 8 text

Scala parallel collections for { s <- surnames.par n <- names.par if s endsWith n } yield (n, s) 2 cores 575 ms

Slide 9

Slide 9 text

Scala parallel collections for { s <- surnames.par n <- names.par if s endsWith n } yield (n, s) 4 cores 305 ms

Slide 10

Slide 10 text

for comprehensions surnames.par.flatMap { s => names.par .filter(n => s endsWith n) .map(n => (n, s)) }

Slide 11

Slide 11 text

for comprehensions nested parallelized bulk operations surnames.par.flatMap { s => names.par .filter(n => s endsWith n) .map(n => (n, s)) }

Slide 12

Slide 12 text

Nested parallelism

Slide 13

Slide 13 text

Nested parallelism parallel within parallel composition surnames.par.flatMap { s => surnameToCollection(s) // may invoke parallel ops }

Slide 14

Slide 14 text

Nested parallelism going recursive def vowel(c: Char): Boolean = ...

Slide 15

Slide 15 text

Nested parallelism going recursive def vowel(c: Char): Boolean = ... def gen(n: Int, acc: Seq[String]): Seq[String] = if (n == 0) acc

Slide 16

Slide 16 text

Slide 17

Slide 17 text

Slide 18

Slide 18 text

Nested parallelism going recursive def vowel(c: Char): Boolean = ... def gen(n: Int, acc: Seq[String]): Seq[String] = if (n == 0) acc else for (s <- gen(n - 1, acc); c <- 'a' to 'z') yield if (s.length == 0) s + c else if (vowel(s.last) && !vowel(c)) s + c else if (!vowel(s.last) && vowel(c)) s + c

Slide 19

Slide 19 text

Slide 20

Slide 20 text

Slide 21

Slide 21 text

Nested parallelism going recursive def vowel(c: Char): Boolean = ... def gen(n: Int, acc: ParSeq[String]): ParSeq[String] = if (n == 0) acc else for (s <- gen(n - 1, acc); c <- 'a' to 'z') yield if (s.length == 0) s + c else if (vowel(s.last) && !vowel(c)) s + c else if (!vowel(s.last) && vowel(c)) s + c else s gen(5, ParArray(""))

Slide 22

Slide 22 text

Slide 23

Slide 23 text

Slide 24

Slide 24 text

Slide 25

Slide 25 text

So, I just use par and I’m home free?

Slide 26

Slide 26 text

How to think parallel

Slide 27

Slide 27 text

Character count use case for foldLeft val txt: String = ... txt.foldLeft(0) { case (a, ‘ ‘) => a case (a, c) => a + 1 }

Slide 28

Slide 28 text

6 5 4 3 2 1 0 Character count use case for foldLeft txt.foldLeft(0) { case (a, ‘ ‘) => a case (a, c) => a + 1 } going left to right - not parallelizable! A B C D E F _ + 1

Slide 29

Slide 29 text

Character count use case for foldLeft txt.foldLeft(0) { case (a, ‘ ‘) => a case (a, c) => a + 1 } going left to right – not really necessary 3 2 1 0 A B C _ + 1 3 2 1 0 D E F _ + 1 _ + _ 6

Slide 30

Slide 30 text

Character count in parallel txt.fold(0) { case (a, ‘ ‘) => a case (a, c) => a + 1 }

Slide 31

Slide 31 text

Character count in parallel txt.fold(0) { case (a, ‘ ‘) => a case (a, c) => a + 1 } 3 2 1 1 A B C _ + 1 3 2 1 1 A B C : (Int, Char) => Int

Slide 32

Slide 32 text

Character count fold not applicable txt.fold(0) { case (a, ‘ ‘) => a case (a, c) => a + 1 } 3 2 1 3 A B C _ + _ 3 3 3 2 1 3 A B C ! (Int, Int) => Int

Slide 33

Slide 33 text

Character count use case for aggregate txt.aggregate(0)({ case (a, ‘ ‘) => a case (a, c) => a + 1 }, _ + _)

Slide 34

Slide 34 text

3 2 1 1 A B C Character count use case for aggregate txt.aggregate(0)({ case (a, ‘ ‘) => a case (a, c) => a + 1 }, _ + _) _ + _ 3 3 3 2 1 3 A B C _ + 1

Slide 35

Slide 35 text

Character count use case for aggregate aggregation  element 3 2 1 1 A B C _ + _ 3 3 3 2 1 3 A B C txt.aggregate(0)({ case (a, ‘ ‘) => a case (a, c) => a + 1 }, _ + _) B _ + 1

Slide 36

Slide 36 text

Character count use case for aggregate aggregation  aggregation aggregation  element 3 2 1 1 A B C _ + _ 3 3 3 2 1 3 A B C txt.aggregate(0)({ case (a, ‘ ‘) => a case (a, c) => a + 1 }, _ + _) B _ + 1

Slide 37

Slide 37 text

Word count another use case for foldLeft txt.foldLeft((0, true)) { case ((wc, _), ' ') => (wc, true) case ((wc, true), x) => (wc + 1, false) case ((wc, false), x) => (wc, false) }

Slide 38

Slide 38 text

Word count initial accumulation txt.foldLeft((0, true)) { case ((wc, _), ' ') => (wc, true) case ((wc, true), x) => (wc + 1, false) case ((wc, false), x) => (wc, false) } 0 words so far last character was a space “Folding me softly.”

Slide 39

Slide 39 text

Word count a space txt.foldLeft((0, true)) { case ((wc, _), ' ') => (wc, true) case ((wc, true), x) => (wc + 1, false) case ((wc, false), x) => (wc, false) } “Folding me softly.” last seen character is a space

Slide 40

Slide 40 text

Word count a non space txt.foldLeft((0, true)) { case ((wc, _), ' ') => (wc, true) case ((wc, true), x) => (wc + 1, false) case ((wc, false), x) => (wc, false) } “Folding me softly.” last seen character was a space – a new word

Slide 41

Slide 41 text

Slide 42

Slide 42 text

Word count in parallel “softly.“ “Folding me “ P1 P2

Slide 43

Slide 43 text

Word count in parallel “softly.“ “Folding me “ wc = 2; rs = 1 wc = 1; ls = 0  P1 P2

Slide 44

Slide 44 text

Word count in parallel “softly.“ “Folding me “ wc = 2; rs = 1 wc = 1; ls = 0  wc = 3 P1 P2

Slide 45

Slide 45 text

Word count must assume arbitrary partitions “g me softly.“ “Foldin“ wc = 1; rs = 0 wc = 3; ls = 0  P1 P2

Slide 46

Slide 46 text

Word count must assume arbitrary partitions “g me softly.“ “Foldin“ wc = 1; rs = 0 wc = 3; ls = 0  P1 P2 wc = 3

Slide 47

Slide 47 text

Word count initial aggregation txt.par.aggregate((0, 0, 0))

Slide 48

Slide 48 text

Word count initial aggregation txt.par.aggregate((0, 0, 0)) # spaces on the left # spaces on the right #words

Slide 49

Slide 49 text

Word count initial aggregation txt.par.aggregate((0, 0, 0)) # spaces on the left # spaces on the right #words ””

Slide 50

Slide 50 text

Word count aggregation  aggregation ... }, { case ((0, 0, 0), res) => res case (res, (0, 0, 0)) => res ““ “Folding me“  “softly.“ ““ 

Slide 51

Slide 51 text

Word count aggregation  aggregation ... }, { case ((0, 0, 0), res) => res case (res, (0, 0, 0)) => res case ((lls, lwc, 0), (0, rwc, rrs)) => (lls, lwc + rwc - 1, rrs) “e softly.“ “Folding m“ 

Slide 52

Slide 52 text

Word count aggregation  aggregation ... }, { case ((0, 0, 0), res) => res case (res, (0, 0, 0)) => res case ((lls, lwc, 0), (0, rwc, rrs)) => (lls, lwc + rwc - 1, rrs) case ((lls, lwc, _), (_, rwc, rrs)) => (lls, lwc + rwc, rrs) “ softly.“ “Folding me” 

Slide 53

Slide 53 text

Word count aggregation  element txt.par.aggregate((0, 0, 0))({ case ((ls, 0, _), ' ') => (ls + 1, 0, ls + 1) ”_” 0 words and a space – add one more space each side

Slide 54

Slide 54 text

Word count aggregation  element txt.par.aggregate((0, 0, 0))({ case ((ls, 0, _), ' ') => (ls + 1, 0, ls + 1) case ((ls, 0, _), c) => (ls, 1, 0) ” m” 0 words and a non-space – one word, no spaces on the right side

Slide 55

Slide 55 text

Word count aggregation  element txt.par.aggregate((0, 0, 0))({ case ((ls, 0, _), ' ') => (ls + 1, 0, ls + 1) case ((ls, 0, _), c) => (ls, 1, 0) case ((ls, wc, rs), ' ') => (ls, wc, rs + 1) ” me_” nonzero words and a space – one more space on the right side

Slide 56

Slide 56 text

Word count aggregation  element txt.par.aggregate((0, 0, 0))({ case ((ls, 0, _), ' ') => (ls + 1, 0, ls + 1) case ((ls, 0, _), c) => (ls, 1, 0) case ((ls, wc, rs), ' ') => (ls, wc, rs + 1) case ((ls, wc, 0), c) => (ls, wc, 0) ” me sof” nonzero words, last non-space and current non-space – no change

Slide 57

Slide 57 text

Word count aggregation  element txt.par.aggregate((0, 0, 0))({ case ((ls, 0, _), ' ') => (ls + 1, 0, ls + 1) case ((ls, 0, _), c) => (ls, 1, 0) case ((ls, wc, rs), ' ') => (ls, wc, rs + 1) case ((ls, wc, 0), c) => (ls, wc, 0) case ((ls, wc, rs), c) => (ls, wc + 1, 0) ” me s” nonzero words, last space and current non-space – one more word

Slide 58

Slide 58 text

Word count in parallel txt.par.aggregate((0, 0, 0))({ case ((ls, 0, _), ' ') => (ls + 1, 0, ls + 1) case ((ls, 0, _), c) => (ls, 1, 0) case ((ls, wc, rs), ' ') => (ls, wc, rs + 1) case ((ls, wc, 0), c) => (ls, wc, 0) case ((ls, wc, rs), c) => (ls, wc + 1, 0) }, { case ((0, 0, 0), res) => res case (res, (0, 0, 0)) => res case ((lls, lwc, 0), (0, rwc, rrs)) => (lls, lwc + rwc - 1, rrs) case ((lls, lwc, _), (_, rwc, rrs)) => (lls, lwc + rwc, rrs) })

Slide 59

Slide 59 text

Word count using parallel strings? txt.par.aggregate((0, 0, 0))({ case ((ls, 0, _), ' ') => (ls + 1, 0, ls + 1) case ((ls, 0, _), c) => (ls, 1, 0) case ((ls, wc, rs), ' ') => (ls, wc, rs + 1) case ((ls, wc, 0), c) => (ls, wc, 0) case ((ls, wc, rs), c) => (ls, wc + 1, 0) }, { case ((0, 0, 0), res) => res case (res, (0, 0, 0)) => res case ((lls, lwc, 0), (0, rwc, rrs)) => (lls, lwc + rwc - 1, rrs) case ((lls, lwc, _), (_, rwc, rrs)) => (lls, lwc + rwc, rrs) })

Slide 60

Slide 60 text

Word count string not really parallelizable scala> (txt: String).par

Slide 61

Slide 61 text

Word count string not really parallelizable scala> (txt: String).par collection.parallel.ParSeq[Char] = ParArray(…)

Slide 62

Slide 62 text

Word count string not really parallelizable scala> (txt: String).par collection.parallel.ParSeq[Char] = ParArray(…) different internal representation!

Slide 63

Slide 63 text

Word count string not really parallelizable scala> (txt: String).par collection.parallel.ParSeq[Char] = ParArray(…) different internal representation! ParArray

Slide 64

Slide 64 text

Word count string not really parallelizable scala> (txt: String).par collection.parallel.ParSeq[Char] = ParArray(…) different internal representation! ParArray  copy string contents into an array

Slide 65

Slide 65 text

Conversions going parallel // `par` is efficient for... mutable.{Array, ArrayBuffer, ArraySeq} mutable.{HashMap, HashSet} immutable.{Vector, Range} immutable.{HashMap, HashSet}

Slide 66

Slide 66 text

Conversions going parallel // `par` is efficient for... mutable.{Array, ArrayBuffer, ArraySeq} mutable.{HashMap, HashSet} immutable.{Vector, Range} immutable.{HashMap, HashSet} most other collections construct a new parallel collection!

Slide 67

Slide 67 text

Conversions going parallel sequential parallel Array, ArrayBuffer, ArraySeq mutable.ParArray mutable.HashMap mutable.ParHashMap mutable.HashSet mutable.ParHashSet immutable.Vector immutable.ParVector immutable.Range immutable.ParRange immutable.HashMap immutable.ParHashMap immutable.HashSet immutable.ParHashSet

Slide 68

Slide 68 text

Conversions going parallel // `seq` is always efficient ParArray(1, 2, 3).seq List(1, 2, 3, 4).seq ParHashMap(1 -> 2, 3 -> 4).seq ”abcd”.seq // `par` may not be... ”abcd”.par

Slide 69

Slide 69 text

Custom collections

Slide 70

Slide 70 text

Custom collection class ParString(val str: String)

Slide 71

Slide 71 text

Custom collection class ParString(val str: String) extends parallel.immutable.ParSeq[Char] {

Slide 72

Slide 72 text

Custom collection class ParString(val str: String) extends parallel.immutable.ParSeq[Char] { def apply(i: Int) = str.charAt(i) def length = str.length

Slide 73

Slide 73 text

Custom collection class ParString(val str: String) extends parallel.immutable.ParSeq[Char] { def apply(i: Int) = str.charAt(i) def length = str.length def seq = new WrappedString(str)

Slide 74

Slide 74 text

Custom collection class ParString(val str: String) extends parallel.immutable.ParSeq[Char] { def apply(i: Int) = str.charAt(i) def length = str.length def seq = new WrappedString(str) def splitter: Splitter[Char]

Slide 75

Slide 75 text

Slide 76

Slide 76 text

Custom collection splitter definition class ParStringSplitter(var i: Int, len: Int) extends Splitter[Char] {

Slide 77

Slide 77 text

Custom collection splitters are iterators class ParStringSplitter(i: Int, len: Int) extends Splitter[Char] { def hasNext = i < len def next = { val r = str.charAt(i) i += 1 r }

Slide 78

Slide 78 text

Custom collection splitters must be duplicated ... def dup = new ParStringSplitter(i, len)

Slide 79

Slide 79 text

Custom collection splitters know how many elements remain ... def dup = new ParStringSplitter(i, len) def remaining = len - i

Slide 80

Slide 80 text

Custom collection splitters can be split ... def psplit(sizes: Int*): Seq[ParStringSplitter] = { val splitted = new ArrayBuffer[ParStringSplitter] for (sz <- sizes) { val next = (i + sz) min ntl splitted += new ParStringSplitter(i, next) i = next } splitted }

Slide 81

Slide 81 text

Word count now with parallel strings new ParString(txt).aggregate((0, 0, 0))({ case ((ls, 0, _), ' ') => (ls + 1, 0, ls + 1) case ((ls, 0, _), c) => (ls, 1, 0) case ((ls, wc, rs), ' ') => (ls, wc, rs + 1) case ((ls, wc, 0), c) => (ls, wc, 0) case ((ls, wc, rs), c) => (ls, wc + 1, 0) }, { case ((0, 0, 0), res) => res case (res, (0, 0, 0)) => res case ((lls, lwc, 0), (0, rwc, rrs)) => (lls, lwc + rwc - 1, rrs) case ((lls, lwc, _), (_, rwc, rrs)) => (lls, lwc + rwc, rrs) })

Slide 82

Slide 82 text

Word count performance txt.foldLeft((0, true)) { case ((wc, _), ' ') => (wc, true) case ((wc, true), x) => (wc + 1, false) case ((wc, false), x) => (wc, false) } new ParString(txt).aggregate((0, 0, 0))({ case ((ls, 0, _), ' ') => (ls + 1, 0, ls + 1) case ((ls, 0, _), c) => (ls, 1, 0) case ((ls, wc, rs), ' ') => (ls, wc, rs + 1) case ((ls, wc, 0), c) => (ls, wc, 0) case ((ls, wc, rs), c) => (ls, wc + 1, 0) }, { case ((0, 0, 0), res) => res case (res, (0, 0, 0)) => res case ((lls, lwc, 0), (0, rwc, rrs)) => (lls, lwc + rwc - 1, rrs) case ((lls, lwc, _), (_, rwc, rrs)) => (lls, lwc + rwc, rrs) }) 100 ms cores: 1 2 4 time: 137 ms 70 ms 35 ms

Slide 83

Slide 83 text

Hierarchy GenTraversable GenIterable GenSeq Traversable Iterable Seq ParIterable ParSeq

Slide 84

Slide 84 text

Hierarchy def nonEmpty(sq: Seq[String]) = { val res = new mutable.ArrayBuffer[String]() for (s <- sq) { if (s.nonEmpty) res += s } res }

Slide 85

Slide 85 text

Hierarchy def nonEmpty(sq: ParSeq[String]) = { val res = new mutable.ArrayBuffer[String]() for (s <- sq) { if (s.nonEmpty) res += s } res }

Slide 86

Slide 86 text

Hierarchy def nonEmpty(sq: ParSeq[String]) = { val res = new mutable.ArrayBuffer[String]() for (s <- sq) { if (s.nonEmpty) res += s } res } side-effects! ArrayBuffer is not synchronized!

Slide 87

Slide 87 text

Hierarchy def nonEmpty(sq: ParSeq[String]) = { val res = new mutable.ArrayBuffer[String]() for (s <- sq) { if (s.nonEmpty) res += s } res } side-effects! ArrayBuffer is not synchronized! ParSeq Seq

Slide 88

Slide 88 text

Hierarchy def nonEmpty(sq: GenSeq[String]) = { val res = new mutable.ArrayBuffer[String]() for (s <- sq) { if (s.nonEmpty) res.synchronized { res += s } } res }

Slide 89

Slide 89 text

Accessors vs. transformers some methods need more than just splitters foreach, reduce, find, sameElements, indexOf, corresponds, forall, exists, max, min, sum, count, … map, flatMap, filter, partition, ++, take, drop, span, zip, patch, padTo, …

Slide 90

Slide 90 text

Slide 91

Slide 91 text

Slide 92

Slide 92 text

Slide 93

Slide 93 text

Builders building a sequential collection 1 2 3 4 5 6 7 Nil 2 4 6 Nil ListBuilder += += += result

Slide 94

Slide 94 text

How to build parallel?

Slide 95

Slide 95 text

Combiners building parallel collections trait Combiner[-Elem, +To] extends Builder[Elem, To] { def combine[N <: Elem, NewTo >: To] (other: Combiner[N, NewTo]): Combiner[N, NewTo] }

Slide 96

Slide 96 text

Combiners building parallel collections trait Combiner[-Elem, +To] extends Builder[Elem, To] { def combine[N <: Elem, NewTo >: To] (other: Combiner[N, NewTo]): Combiner[N, NewTo] } Combiner Combiner Combiner

Slide 97

Slide 97 text

Slide 98

Slide 98 text

Slide 99

Slide 99 text

Parallel arrays 1, 2, 3, 4 5, 6, 7, 8 2, 4 6, 8 3, 1, 8, 0 2, 2, 1, 9 8, 0 2, 2 merge merge merge copy allocate 2 4 6 8 8 0 2 2

Slide 100

Slide 100 text

Parallel hash tables ParHashMap

Slide 101

Slide 101 text

Parallel hash tables ParHashMap 0 1 2 4 5 7 8 9 e.g. calling filter

Slide 102

Slide 102 text

Parallel hash tables ParHashMap 0 1 2 4 5 7 8 9 ParHashCombiner ParHashCombiner e.g. calling filter 0 5 1 7 9 4

Slide 103

Slide 103 text

Parallel hash tables ParHashMap 0 1 2 4 5 7 8 9 ParHashCombiner 0 1 4 ParHashCombiner 5 7 9

Slide 104

Slide 104 text

Parallel hash tables ParHashMap 0 1 2 4 5 7 8 9 ParHashCombiner 0 1 4 ParHashCombiner 5 9 5 7 0 1 4 7 9

Slide 105

Slide 105 text

Parallel hash tables ParHashMap ParHashCombiner ParHashCombiner How to merge? 5 7 0 1 4 9

Slide 106

Slide 106 text

5 7 8 9 1 4 0 Parallel hash tables buckets! ParHashCombiner ParHashCombiner 0 1 4 9 7 5 ParHashMap 2 0 = 00002 1 = 00012 4 = 01002

Slide 107

Slide 107 text

Parallel hash tables ParHashCombiner ParHashCombiner 0 1 4 9 7 5 combine

Slide 108

Slide 108 text

Parallel hash tables ParHashCombiner ParHashCombiner 9 7 5 0 1 4 ParHashCombiner no copying!

Slide 109

Slide 109 text

Parallel hash tables 9 7 5 0 1 4 ParHashCombiner

Slide 110

Slide 110 text

Parallel hash tables 9 7 5 0 1 4 ParHashMap

Slide 111

Slide 111 text

Custom combiners for methods returning custom collections new ParString(txt).filter(_ != ‘ ‘) What is the return type here?

Slide 112

Slide 112 text

Custom combiners for methods returning custom collections new ParString(txt).filter(_ != ‘ ‘) creates a ParVector!

Slide 113

Slide 113 text

Custom combiners for methods returning custom collections new ParString(txt).filter(_ != ‘ ‘) creates a ParVector! class ParString(val str: String) extends parallel.immutable.ParSeq[Char] { def apply(i: Int) = str.charAt(i) ...

Slide 114

Slide 114 text

Custom combiners for methods returning custom collections class ParString(val str: String) extends immutable.ParSeq[Char] with ParSeqLike[Char, ParString, WrappedString] { def apply(i: Int) = str.charAt(i) ...

Slide 115

Slide 115 text

Slide 116

Slide 116 text

Slide 117

Slide 117 text

Custom combiners for methods returning custom collections class ParStringCombiner extends Combiner[Char, ParString] {

Slide 118

Slide 118 text

Custom combiners for methods returning custom collections class ParStringCombiner extends Combiner[Char, ParString] { var size = 0

Slide 119

Slide 119 text

Custom combiners for methods returning custom collections class ParStringCombiner extends Combiner[Char, ParString] { var size = 0 size

Slide 120

Slide 120 text

Custom combiners for methods returning custom collections class ParStringCombiner extends Combiner[Char, ParString] { var size = 0 val chunks = ArrayBuffer(new StringBuilder) size

Slide 121

Slide 121 text

Custom combiners for methods returning custom collections class ParStringCombiner extends Combiner[Char, ParString] { var size = 0 val chunks = ArrayBuffer(new StringBuilder) size chunks

Slide 122

Slide 122 text

Custom combiners for methods returning custom collections class ParStringCombiner extends Combiner[Char, ParString] { var size = 0 val chunks = ArrayBuffer(new StringBuilder) var lastc = chunks.last size chunks

Slide 123

Slide 123 text

Slide 124

Slide 124 text

Slide 125

Slide 125 text

Slide 126

Slide 126 text

Custom combiners for methods returning custom collections ... def combine[U <: Char, NewTo >: ParString] (other: Combiner[U, NewTo]) = other match { case psc: ParStringCombiner => sz += that.sz chunks ++= that.chunks lastc = chunks.last this }

Slide 127

Slide 127 text

Custom combiners for methods returning custom collections ... def combine[U <: Char, NewTo >: ParString] (other: Combiner[U, NewTo]) lastc chunks lastc chunks

Slide 128

Slide 128 text

Custom combiners for methods returning custom collections ... def result = { val rsb = new StringBuilder for (sb <- chunks) rsb.append(sb) new ParString(rsb.toString) } ...

Slide 129

Slide 129 text

Custom combiners for methods returning custom collections ... def result = ... lastc chunks StringBuilder

Slide 130

Slide 130 text

Custom combiners for methods expecting implicit builder factories // only for big boys ... with GenericParTemplate[T, ParColl] ... object ParColl extends ParFactory[ParColl] { implicit def canCombineFrom[T] = new GenericCanCombineFrom[T] ...

Slide 131

Slide 131 text

Custom combiners performance measurement txt.filter(_ != ‘ ‘) new ParString(txt).filter(_ != ‘ ‘)

Slide 132

Slide 132 text

txt.filter(_ != ‘ ‘) new ParString(txt).filter(_ != ‘ ‘) 106 ms Custom combiners performance measurement

Slide 133

Slide 133 text

txt.filter(_ != ‘ ‘) new ParString(txt).filter(_ != ‘ ‘) 106 ms 1 core 125 ms Custom combiners performance measurement

Slide 134

Slide 134 text

txt.filter(_ != ‘ ‘) new ParString(txt).filter(_ != ‘ ‘) 106 ms 1 core 125 ms 2 cores 81 ms Custom combiners performance measurement

Slide 135

Slide 135 text

txt.filter(_ != ‘ ‘) new ParString(txt).filter(_ != ‘ ‘) 106 ms 1 core 125 ms 2 cores 81 ms 4 cores 56 ms Custom combiners performance measurement

Slide 136

Slide 136 text

1 core 125 ms 2 cores 81 ms 4 cores 56 ms t/ms proc 125 ms 1 2 4 81 ms 56 ms Custom combiners performance measurement

Slide 137

Slide 137 text

1 core 125 ms 2 cores 81 ms 4 cores 56 ms t/ms proc 125 ms 1 2 4 81 ms 56 ms def result (not parallelized) Custom combiners performance measurement

Slide 138

Slide 138 text

Custom combiners tricky! • two-step evaluation – parallelize the result method in combiners • efficient merge operation – binomial heaps, ropes, etc. • concurrent data structures – non-blocking scalable insertion operation – we’re working on this

Slide 139

Slide 139 text

Future work coming up • concurrent data structures • more efficient vectors • custom task pools • user defined scheduling • parallel bulk in-place modifications

Slide 140

Slide 140 text

Thank you! Examples at: git://github.com/axel22/sd.git