Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Scala Collections: Why Not? - Paul Phillps

Scala Collections: Why Not? - Paul Phillps

4867789d19ac436c6e5a4e79b956c4b0?s=128

NewCircle Training

January 29, 2014
Tweet

Transcript

  1. A Scala Corrections Library Paul Phillips paulp@improving.org Source: Dilbert. Your

    parents might have read it.
  2. – R. Buckminster Fuller “When I'm working on a problem,

    I never think about beauty. I think only how to solve the problem.” ! “But when I have finished, if the solution is not beautiful, I know it is wrong.” (syntax highlighting donated by paulp)
  3. “When I'm working on a problem, I never think about

    beauty. I think only how to solve the problem.” ! “But when I have finished, if the solution is not beautiful, I know it is wrong.” trait ParSeqViewLike[ +T, +Coll <: Parallel, +CollSeq, +This <: ParSeqView[T, Coll, CollSeq] with ParSeqViewLike[T, Coll, CollSeq, This, ThisSeq], +ThisSeq <: SeqView[T, CollSeq] with SeqViewLike[T, CollSeq, ThisSeq] ] extends GenSeqView[T, Coll] with GenSeqViewLike[T, Coll, This] with ParIterableView[T, Coll, CollSeq] with ParIterableViewLike[T, Coll, CollSeq, This, ThisSeq] with ParSeq[T] with ParSeqLike[T, This, ThisSeq] – R. Buckminster Fuller
  4. The Winding Stairway • Five years on scala • Rooting

    for scala/typesafe • But I quit a dream job... • ...because I lost faith
  5. Credentials

  6. Credentials, cont.

  7. Should you care? • I offer my credentials only to

    bear witness to my credibility • I suspect I have written more scala code than anyone else, ever. • What’s visible in compiler/library represents only a small fraction of it
  8. The early bird gets the can of worms • I

    don’t wish to make slagging a full-time gig • Tonight let’s focus less on what’s wrong with scala and more on how to do it better • Video from last night will be available if that’s your thing
  9. Is Scala too complex? • Yes. • Is anyone fooled

    by specious comparisons of language grammar size? Who cares? • Half the time when someone hits a bug they can’t tell whether it is a bug in scala or the expected behavior • That definitely includes me
  10. • A meme is going around that scala is too

    complex • Option A: Own it • Option B: Address it • Option C: Obscure it Perceived Problem Option C
  11. // A fictional idealized version of the genuine method def

    map[B](f: (A) 㱺 B): Map[B] ! // The laughably labeled "full" signature def map[B, That](f: ((A, B)) 㱺 B) (implicit bf: CanBuildFrom[Map[A, B], B, That]): That Thus is born the “use case” neither has any basis in reality!
  12. // markers to distinguish Map's class type parameters scala> class

    K ; class V defined class K, V ! scala> val host = typeOf[Map[K, V]] host: Type = Map[K,V] ! scala> val method = host member TermName("map") method: Symbol = method map ! // Correct signature for map has FOUR distinct identifiers scala> method defStringSeenAs (host memberType method) res0: String = \ def map[B, That](f: ((K, V)) => B) (implicit bf: CBF[Map[K,V],B,That]): That the true name of map
  13. • Maybe you’re thinking “So it’s a bug. Bugs get

    fixed.” • “As soon as the situation is known, of course it will be fixed? At the very least it will be marked somehow?” • Nope! Your time has no value. 44 MONTHS
  14. Today

  15. map “map” Signature def map[B](f: A => B): F[B] def

    map[B, That](f: A => B) (implicit bf: CanBuildFrom[Repr, B, That]): That Elegance Among the purest and most reusable abstractions known to computing science <—- Not this. Advantages Can reason abstractly about code Can map a BitSet to a BitSet without typing “toBitSet” Spokespicture Slightly Caricatured
  16. // Fancy, we get a Bitset back! scala> BitSet(1, 2,

    3) map (_.toString.toInt) res0: BitSet = BitSet(1, 2, 3) ! // Except… scala> BitSet(1, 2, 3) map (_.toString) map (_.toInt) res1: SortedSet[Int] = TreeSet(1, 2, 3) ! // Um… scala> (BitSet(1, 2, 3) map identity)(1) <console>:21: error: type mismatch; found : Int(1) required: CanBuildFrom[BitSet,Int,?] (BitSet(1, 2, 3) map identity)(1) ^ // What’s going on? The docs said map is A => B ! // The primary docs can’t be wrong, can they? The Bitset Gimmick
  17. scala> def f[T](x: T) = (x, new Object) f: [T](x:

    T)(T, Object) ! scala> SortedSet(1 to 10: _*) res0: SortedSet[Int] = TreeSet(1, 2, 3, 4, 5, 6, 7, 8, 9, 10) ! scala> SortedSet(1 to 10: _*) map (x => f(x)._1) res1: SortedSet[Int] = TreeSet(1, 2, 3, 4, 5, 6, 7, 8, 9, 10) ! // What happened to the “Sorted” in my SortedSet? scala> SortedSet(1 to 10: _*) map f map (_._1) res2: Set[Int] = Set(5, 10, 1, 6, 9, 2, 7, 3, 8, 4) similarly
  18. scala> val f: Int => Int = _ % 3

    f: Int => Int = <function1> ! scala> val g: Int => Int = _ => System.nanoTime % 1000000 toInt g: Int => Int = <function1> ! scala> Set(3, 6, 9) map f map g res0: Set[Int] = Set(633000) ! scala> Set(3, 6, 9) map (f andThen g) res1: Set[Int] = Set(305000, 307000, 308000) and in a similar vein
  19. Java Interop: the cruelest joke • It’s impossible to call

    scala’s map from java! • See all the grotesque details at SI-4389 “I played with it until it got too tedious. I think the signatures work fine. What does not work is that the variances of CanBuildFrom cannot be modelled in Java, so types do not match. And it seems Java does not even let me override with a cast. So short answer: You can't call these things from Java because instead of declaration side variance you have only a broken wildcard system.” ! — Martin Odersky WONTFIX
  20. Laundry List
 Lightning Round!

  21. • Implementation details infest everything • And every detail is

    implementation-defined • Casts (556 explicit asInstanceOf) and suppression of variance checks abound • Specificity rules render contravariance useless • Implicit selection and type inference inextricably bound - so type inference is largely frozen because any change will break existing code
  22. • Inheritance of implementation is the hammer for every nail…

    • …yet “final” and “private”, critical for a hope of correctness under inheritance, are almost unknown • All manner of behavior is ill-defined, which means users have to program defensively and compiler team cannot optimize • Can xs filter (_ => true) return xs?
  23. // WHY infer this utterly useless type? scala> List(1, 2)

    ::: List(3, 4.0) res0: List[AnyVal] = List(1, 2, 3.0, 4.0) ! scala> PspList(1, 2) ::: PspList(3, 4.0) <console>:23: error: type mismatch; found : PspList[Int] required: PspList[Double] Why is covariance such an object of worship? Types exist so we don’t have to live like this!
  24. Type Inference + Variance ——————————

  25. scala> trait Ord[-A] defined trait Ord ! scala> implicit def

    x1: Ord[Any] = { println("Any") ; ??? } ; implicit def x2: Ord[List[Double]] = { println("List[Double]") ; ??? } x1: Ord[Any] x2: Ord[List[Double]] ! // An implementation isn’t the only thing missing here scala> implicitly[Ord[List[Double]]] Any scala.NotImplementedError: an implementation is missing Speaking of variance
  26. Abstracting over mutability • An inherited implementation is ALWAYS wrong

    somewhere!! • Example: how do you write "drop" so it's reusable?! • In a mutable class, drop MUST NOT share, but in an immutable class, drop MUST share! • Half the overrides in collections exist to stave off the incorrectness which looms above. This is nuts.! • Not to mention “Map”, “Set”, etc. in three namespaces
  27. % ack --no-filename 'def slice\(' src/library/ ! 1 override def

    slice(from: Int, until: Int): Iterator[A] = 2 def slice(from: Int, until: Int): Iterator[A] = { 3 def slice(from: Int, until: Int): Repr = 4 def slice(from: Int, until: Int): Repr = { 5 def slice(from: Int, until: Int): Repr = { 6 def slice(start: Int): PagedSeq[T] = slice(start, UndeterminedEnd) 7 def slice(unc_from: Int, unc_until: Int): Repr 8 override /*IterableLike*/ def slice(from: Int, until: Int): Vector[A] = 9 override /*TraversableLike*/ def slice(from: Int, until: Int): Repr = { 10 override def slice(_start: Int, _end: Int): PagedSeq[T] = { 11 override def slice(from1: Int, until1: Int): IterableSplitter[T] = 12 override def slice(from1: Int, until1: Int): SeqSplitter[T] = 13 override def slice(from: Int, until: Int) = { 14 override def slice(from: Int, until: Int) = { 15 override def slice(from: Int, until: Int): List[A] = { 16 override def slice(from: Int, until: Int): Repr = self.slice(from, until) 17 override def slice(from: Int, until: Int): Repr = { 18 override def slice(from: Int, until: Int): Stream[A] = { 19 override def slice(from: Int, until: Int): String = { 20 override def slice(from: Int, until: Int): This = 21 override def slice(from: Int, until: Int): This = 22 override def slice(from: Int, until: Int): Traversable[A] 23 override def slice(from: Int, until: Int): WrappedString = { 24 override def slice(unc_from: Int, unc_until: Int): Repr = { How many ways are there to write ‘slice’ ?
  28. scala> List(1, 2, 3).toSet() res0: Boolean = false ! scala>

    123456789.round res1: Int = 123456792 ! scala> List(1, 2, 3) contains "your mom" res2: Boolean = false ! scala> def sum[T](xs: Iterable[T]): Int = xs.map(_.hashCode).sum Some perennial favorites
  29. Two complementary ways to define Set[A]. Complementary - and NOT

    the same thing! sets Intensional Extensional Specification Membership test Members Variance Set[-A] Set[+A] Defining Signature A => Boolean Iterable[A] Size Unknowable Known Duplicates(*) Meaningless Disallowed
  30. scala> class xs[A] extends Set[A] error: class xs has 4

    unimplemented members. ! // Intensional/extensional, conflated. // Any possibility of variance eliminated. def iterator: Iterator[A] def contains(elem: A): Boolean // What are these doing in the interface? // Why can I define a Seq without them? def -(elem: A): Set[A] def +(elem: A): Set[A] What's going on here?
  31. tyranny of the interface • Mandating "def size: Int" for

    all collections is the fast track to Glacialville! • Countless times have I fixed xs.size != 0 • Collections are both worlds: all performance/ termination trap, no exploiting of size information! • A universal size method must be SAFE and CHEAP
  32. Psp Collections • So here is a little of what

    I would do differently • I realized since agreeing to this talk that I may have to go cold turkey to escape scala’s orbit. It’s just too frustrating to use. • Which means this may never go anywhere • But you can have whatever gets done
  33. trait Collections { type CC[+X] // the overarching container type

    (in scala: any covariant collection, e.g. List, Vector) type Min[+X] // least type constructor which can be reconstituted to CC[X] (scala: GenTraversableOnce) type Opt[+X] // the container type for optional results (in scala: Option) type CCPair[+X] // some representation of a divided CC[A] (at simplest, (CC[A], CC[A])) type ~>[-V1, +V2] // some means of composing operations (at simplest, Function1) ! type Iso[A] = CC[A] ~> CC[A] // e.g. filter, take, drop, reverse, etc. type Map[-A, +B] = CC[A] ~> CC[B] // e.g. map, collect type FlatMap[-A, +B] = CC[A] ~> Min[B] // e.g. flatMap type Grouped[A, DD[X]] = CC[A] ~> CC[DD[A]] // e.g. sliding type Fold[-A, +R] = CC[A] ~> R // e.g. fold, but also subsumes all operations on CC[A] type Flatten[A] = CC[Min[A]] ~> CC[A] // e.g. flatten type Build[A] = Min[A] ~> CC[A] // for use in e.g. sliding, flatMap type Pure[A] = A ~> CC[A] // we may not need ! trait Relations[A] { type MapTo[+B] = Map[A, B] // an alias incorporating the known A type FoldTo[+R] = Fold[A, R] // another one type This = CC[A] // the CC[A] under consideration type Twosome = CCPair[A] // a (CC[A], CC[A]) representation type Self = Iso[A] // a.k.a. CC[A] => CC[A], e.g. tail, filter, reverse type Select = FoldTo[A] // a.k.a. CC[A] => A, e.g. head, reduce, max type Find = FoldTo[Opt[A]] // a.k.a. CC[A] => Opt[A], e.g. find type Split = FoldTo[Twosome] // a.k.a. CC[A] => (CC[A], CC[A]), e.g. partition, span } } Conceptual Integrity
  34. “Do not multiply entities unnecessarily” • mutable / immutable •

    Seq / Set / Map • parallel / sequential • view / regular 24 Combinations!
  35. Surface Area Reduced 96% • A Set is a Seq

    without duplicates. • A Map is a Set paired with a function K => V. • A mutable collection has nothing useful in common with an immutable collection. Write your own mutable collections. • If we can’t get sequential collections right, we have no hope of parallel collections. Write your own parallel collections. • “Views” should be how it always works.
  36. scala> def f(xs: Iterable[Int]) = xs.size f: (xs: Seq[Int])Int !

    // O(1) scala> f(Set(1)) res0: Int = 1 ! // O(n) scala> f(List(1)) res1: Int = 1 ! // O(NOES) scala> f(Stream continually 1) <ctrl-C> predictability: size matters
  37. SizeInfo / \ Atomic Bounded / \ Infinite Precise Asking

    the right question
  38. scala> val xs = Foreach from BigInt(1) xs: Foreach[BigInt] =

    unfold from 1 ! scala> xs.size <console>:22: error: value size is not a member of Foreach[BigInt] xs.size ^ ! scala> xs.sizeInfo res0: SizeInfo = <inf> ! scala> (xs.m take 10000).sizeInfo res1: SizeInfo = 10000 ! scala> (xs.m take 10000 map (_ + 1)).sizeInfo res2: SizeInfo = 10000 Don’t ask unanswerable questions unless you enjoy hearing lies
  39. // It’s 2014 and our language still allows this? scala>

    List(1, 2, 3) contains "1" res0: Boolean = false ! scala> PspList(1, 2, 3) contains "1" <console>:23: error: type mismatch; found : String("1") required: Int PspList(1, 2, 3) contains "1" ^ the joy of the invariant leaf
  40. // Us scala> timed(Indexed.to(1, 1000000).m map (_ + 1) \

    map (_ + 1) map (_ + 1) drop 999999 take 1 sum) Elapsed: 1.427 ms res0: Int = 1000003 ! // Them scala> timed((1 to 1000000).view map (_ + 1) \ map (_ + 1) map (_ + 1) drop 999999 take 1 sum) Elapsed: 201.825 ms res1: Int = 1000003 What’s a view?
  41. scala> "abc" map (_.toInt.toChar) res1: String = abc ! scala>

    "abc" map (_.toInt) map (_.toChar) res2: IndexedSeq[Char] = Vector(a, b, c) ! // psp to the rescue scala> "abc".m map (_.toInt) map (_.toChar) res3: psp.core.View[String,Char] = view of abc ! scala> res3.force res4: String = abc Save map!