Using Monoids for large scale aggregation - Scala.io, Lyon 2017

In this talk, you will see, how Monoids acts as a powerful abstraction to build distributed stats aggregation system. You will also see a high level architecture of how an in-house system named able was built based on this premise.

Transcript

1. Using Monoids for Large Scale Aggregates Scala.io 2017
2. Sriram Ramachandrasekaran @brewkode Principal Engineer, Indix
3. Ashwanth Kumar @_ashwanthkumar Principal Engineer, Indix
5. class Crawler { def crawl(url: String) { agent.doCrawl(url) metric.count(“urls_crawled”, 1L) } }
6. 1 1 1 1 1 1 1 1 1 +
7. 1 1 1 1 1 1 1 1 1 4 1 1 1 1 1 + +
8. 1 1 1 1 1 1 1 1 1 4 1 1 1 1 1 8 1 + = 9 Total URLs Crawled + +
9. 1 1 1 1 1 1 1 1 1 4 1 1 1 1 1 8 1 = 9 Total URLs Crawled (1+1+1+1)+(1+1+1+1)+1 = 4+4+1 (4+5) = (8+1) = 9 + + +
10. (1+1+1+1)+(1+1+1+1)+1 = 4+4+1 (4+5) = (8+1) = 9 1 1 1 1 1 1 1 1 1 4 1 1 1 1 1 8 1 = 9 Total URLs Crawled Associativity + + +
11. class Crawler { def crawl(url: String) { val page = agent.doCrawl(url) metric.average(“response_times”, page.responseTime) } }
12. 1200 3600 4800
13. 1200 3600 4800 [1200,1] [3600,1] [4800,1] +
14. 1200 3600 4800 [1200,1] [3600,1] [4800,1] [4800,2] [4800,1] + +
15. 1200 3600 4800 [1200,1] [3600,1] [4800,1] [4800,2] [4800,1] = 3200 Average Response Time + +
16. Generalizing Sum and Average • Takes 2 numbers and produces another number (binary operation) - Add: simple add of two numbers - Average: maintain two values - sum and count & “adds” each of them • Ordering of operations don’t matter (commutative) • Grouping of operations don’t matter (associative) • Ignores 0s
17. Abstraction • We are dealing with Sets • Associative binary operations • Identity element exists (for additions - it’s zero)
18. Abstraction • We are dealing with Sets • Associative binary operations • Identity element exists (for additions - it’s zero) = Monoid
19. Abstraction • We are dealing with Sets • Associative binary operations • Identity element exists (for additions - it’s zero) • Add Commutativity to the mix = Commutative Monoid
20. Aggregations at Scale • Associative and Commutative ◦ Makes it an EMBARRASSINGLY PARALLEL* problem • User Queries are handled via Scatter Gather ◦ Reduce on individual nodes ◦ Re-reduce on the results and return as the response
21. Talk is cheap, show me the code!
22. Introducing Abel
23. • Monoid based aggregations • Durable delivery via Kafka • Persistence via RocksDB • User queries handled via Scatter Gather • Scala all the way Abel twitter/algebird ashwanthkumar/suuchi
24. stats.service.ix Count(“a”, 1L) Count(“c”, 1L) Unique(“ua”, “a”) Count(“b”, 1L) Abel Data Flow Average(“a”, 5682)
25. Abel Internals Metric = Key * Aggregate (Monoid) case class Metric[T <: Aggregate[T]] (key: Key, value: T with Aggregate[T])
26. Abel Internals Metric = Key * Aggregate (Monoid) trait Aggregate[T <: Aggregate[_]] { self: T => def plus(another: T): T def show: JsValue }
27. Abel Internals Key = Name * Tags * Time case class Time(time: Long, granularity: Long) case class Key(name:String, tags:SortedSet[String], time:Time = Time.Forever)
28. Abel Internals client.send(Metric(Key( name = “unique-url-per-hour”, tags = SortedSet(“www.amazon.com”), time = Time.ThisHour ), UniqueCount(“http://...”))
29. client.send(Metrics( “unique-urls”, tag(“site:www.amazon.com”) * (perday | forever) * now, UniqueCount(“http://...”) )) Abel Internals To find Unique count of URLs crawled per site for every day and forever.
30. client.send(Metrics( “unique-urls”, (tag(“site:www.amazon.com”) | `#`) * (perday | forever) * now, UniqueCount(“http://...”) )) Abel Internals To find Unique count of URLs crawled per site and across sites for every day and forever.
31. client.send(Metrics( “unique-urls”, (tag(“site:www.amazon.com”) | `#`) * (perday | forever) * now, UniqueCount(“http://...”) )) Abel Internals To find Unique count of URLs crawled per site and across sites for every day and forever. It is implemented as a Ring.
32. stats.service.ix Count(“a”, 1L) Count(“c”, 1L) Abel v1 Average(“a”, 5682)
33. stats.service.ix Count(“a”, 1L) Count(“c”, 1L) Unique(“ua”, “a”) Count(“b”, 1L) Abel v1 Average(“a”, 5682)
34. A stats.service.ix 1.1.1.1 1.1.1.2 1.1.1.3 aggregate.plus 1.1.1.2 Count(“a”, 1L) Average(“a”, 5682) Count(“c”, 1L) Count(“b”, 1L) Abel in Distributed Mode aggregate.plus 1.1.1.1 aggregate.plus 1.1.1.3 DNS based Load Balancing Unique(“ua”, “a”)
35. Scatter Gather - Average (123, 8) (3, 1) (12303, 24) Reduce Reduce Reduce
36. (123, 8) (3, 1) (12303, 24) Re-reduce (12429, 33) = 376.6 Scatter Gather - Average
37. • Monoid based aggregations • Durable delivery via Kafka • Persistence via RocksDB • User queries handled via Scatter Gather • Scala all the way Abel twitter/algebird ashwanthkumar/suuchi
38. Monoid Cheatsheet Stat / Metric Type Abstraction Count of Urls Sum Average Response Time Sum with Count & Total Unique count of urls crawled HyperLogLog HTTP Response Code Distribution Count-Min Sketch Top K Websites with poor response time Heap with K elements Website response times percentiles QTree (loosely based on q-digest) Histogram of response times Array(to model bins) and slotwise Sum
39. Credits VinothKumar Raman @eventaken Swathi Ravichandran @swathrav Thank you
40. Meta
41. Ring • Abelian group under Addition ◦ Associative ◦ Commutative ◦ Identity ◦ Inverse • Monoid under multiplication ◦ Associative ◦ Multiplicative Identity • Multiplication is distributive with respect to addition ◦ (a + b) . c = (ac + bc) Right Distributivity ◦ a . (b + c) = (ab + ac) Left Distributivity
44. suuchi toolkit for building distributed function shipping applications github.com/ashwanthkumar/suuchi
45. Slides designed by www.swathiravichandran.com