Counting At Scale with Scalding

Slides from a talk I gave at Coursera on using approximate data structures for scaling operations on very large datasets

Transcript

1. Counting (Approximately) at Scale With Algebird and Scalding
2. Who am I? What I do? • Who: • Ian O’Connell @0x138 • Engineer @ Twitter • Build data tools and products • e.g. Algebird/Scalding/Summingbird/ Storehaus/Bijection…
3. This talk 1. Sample real world problems & solutions A. Hyper Log Log B. Count Min Sketch 2. How Algebraic concepts helps us get more from these 3. Scalding / using the above 4. But I want it exact! 5. Other applications?
4. Four more years Retweets: 763,570
5. Best photo ever. #oscars Retweets: 3,381,051
6. Problems • How many unique users viewed each tweet? • How many of those are really twitter junkies and viewed both tweets?
7. Naïve approach • Set’s and Set intersection • Set[Long] of User Id’s per tweet • 10^8 tweets/day • 10^8 users
8. HyperLogLog • Approximate Set Cardinality • Associative & Commutative • Space is ∝error
9. HyperLogLog M1 M2 M3 Mm
10. HyperLogLog Update x
11. HyperLogLog Update x h(x) hash of x
12. HyperLogLog Update x h(x) hash of x (Y1,Y2) split on m bits
13. HyperLogLog Update x) (Y1,Y2) split on m bits Y1 Y2
14. HyperLogLog Update (Y1,Y2) ts Y1 Y2 M1 M2 M3 Mm
15. HyperLogLog Update (Y1,Y2) ts Y1 Y2 M1 M2 M3 Mj Mm
16. HyperLogLog Update (Y1,Y2) Y1 Y2 Mj
17. HyperLogLog Update (Y1,Y2) Y1 Y2 Mj Z(Y2) = p Leading Zeros
18. HyperLogLog Update Mj Z(Y2) = p ing os Max Mj
19. HyperLogLog Read M1 M2 M3 Alpha * Harmonic Mean(M1..Mj) Mm
20. HyperLogLog Read Alpha * Harmonic Mean(M1..Mj) Estimated Cardinality
21. HyperLogLog • For cardinalities > 10^9 • Error ~ 2% => 1.5kb of space • 10^8 Tweets per day, @ Max(1.5kb/tweet) • 150GB/day uncompressed
22. Spokane, Washington
23. House of cards
24. Problem • How many views of the tweet the day it was posted? between 5 and 7? • How many views in Spokane of the tweet? • How many of those users were using an iPhone 6? • Key: (TweetId, Geo, Device, Minute)
25. Challenge • 1 Month of storage: • Tweets: 10^9/month • Geo’s: 10^5 • Devices: 10^2 • Time: 10^4 minutely buckets ~10^20 keys/month
26. Count Min Sketch j i
27. Count Min Sketch Update (x, y)
28. Count Min Sketch Update (x, y) (h1(x), y) (h2(x), y) (hj(x), y)
29. Count Min Sketch Update (x, y) (h1(x), y) (h2(x), y) (hj(x), y) i j
30. Count Min Sketch Update (h1(x), y) (h2(x), y) (hj(x), y) j i
31. Reading Count Min Sketch x
32. Reading Count Min Sketch x h1(x) h2(x) hj(x)
33. Reading Count Min Sketch i j h1(x) h2(x) hj(x) x
34. Reading Count Min Sketch (h1(x), y) (h2(x), y) (hj(x), y) j i
35. Reading Count Min Sketch i
36. Reading Count Min Sketch i Min
37. Reading Count Min Sketch i Min
38. Reading Count Min Sketch i Min Estimated value
39. Count Min Sketch Sketchyness • Estimate: x • Total count N • Width: w • Depth: d
40. Count Min Sketch Sketchyness • Upper bound: x • Lower bound: max(0, x - ((e/w) * N)) • Probability: 1 - 1/ed
41. And back to the problem • Can make a 160GB cms or more per month • Take 10 hash functions • Width is ~10^9 • Upper bound looks like: 10^-9 * N • At N = 1 trillion ==> lower bound: -3000. • With probability: 99.99%
42. Making these accessible • Don’t want any special code in Scalding/ Summingbird for ease of use • We use them outside of hadoop/data processing • Generic testing ability
43. Monoids • Have a well defined plus under associativity • (a + b) + c == a + (b + c) • Not needed: (a + b) + c == (b + a) + c (commutativity) • Have a zero, such that a + 0 == a
44. Monoids • Divide and conquer in calculations • Unlocks parallelism in hadoop
45. Commutativity • Map side aggregation! • Not everything is commutative however concatenation, division
46. Scalding • API for running a sequence of Map reduce jobs • Based on the Scala collections API • Type safe, easily testable • Stable/robust
47. Scalding • Map-like functions • map, flatMap, filter • Grouping functions • sum, mapGroup, toList, sumByKey, reduce, sortedTake, head, min • Join functions • join, leftJoin, rightJoin, sketchJoin
48. Scalding • def sumByKey[K, V](implicit ev: T <:< (K, V), ord: Ordering[K], plus: Semigroup[V])
49. Scalding
50. But I want it exact • Having exact results in big data is a myth • Everything has error rates in it • Reporting from devices • Logging pipelines • Corrupted Lzo/Serialization
51. Other applications of CMS? • CMS’s are great • Heavy Hitters, associative! • Sketch Map, not… but top N. Often what you want even if its less accurate — (related headlines on twitter) • Storehaus cache filter with a CMS to efficiently only cache frequent keys