Using Monoids for large scale aggregation - Scala.io, Lyon 2017

In this talk, you will see, how Monoids acts as a powerful abstraction to build distributed stats aggregation system. You will also see a high level architecture of how an in-house system named able was built based on this premise.

Transcript

1. Using Monoids for Large Scale
   Aggregates
   Scala.io 2017
   

   View Slide

2. Sriram
   Ramachandrasekaran
   @brewkode
   Principal Engineer, Indix
   

   View Slide

3. Ashwanth
   Kumar
   @_ashwanthkumar
   Principal Engineer, Indix
   

   View Slide

4. View Slide

5. class Crawler {
   def crawl(url: String) {
   agent.doCrawl(url)
   metric.count(“urls_crawled”, 1L)
   }
   }
   

   View Slide

6. 1 1 1 1 1 1 1 1 1
   +
   

   View Slide

7. 1 1 1 1 1 1 1 1 1
   4 1 1 1 1 1
   +
   +
   

   View Slide

8. 1 1 1 1 1 1 1 1 1
   4 1 1 1 1 1
   8 1
   +
   = 9
   Total URLs Crawled
   +
   +
   

   View Slide

9. 1 1 1 1 1 1 1 1 1
   4 1 1 1 1 1
   8 1
   = 9
   Total URLs Crawled
   (1+1+1+1)+(1+1+1+1)+1 = 4+4+1
   (4+5) = (8+1) = 9
   +
   +
   +
   

   View Slide

10. (1+1+1+1)+(1+1+1+1)+1 = 4+4+1
    (4+5) = (8+1) = 9
    1 1 1 1 1 1 1 1 1
    4 1 1 1 1 1
    8 1
    = 9
    Total URLs Crawled
    Associativity
    +
    +
    +
    

    View Slide

11. class Crawler {
    def crawl(url: String) {
    val page = agent.doCrawl(url)
    metric.average(“response_times”,
    page.responseTime)
    }
    }
    

    View Slide

12. 1200 3600 4800
    

    View Slide

13. 1200 3600 4800 [1200,1] [3600,1] [4800,1]
    +
    

    View Slide

14. 1200 3600 4800 [1200,1] [3600,1] [4800,1]
    [4800,2] [4800,1]
    +
    +
    

    View Slide

15. 1200 3600 4800 [1200,1] [3600,1] [4800,1]
    [4800,2] [4800,1]
    = 3200
    Average Response Time
    +
    +
    

    View Slide

16. Generalizing Sum and Average
    ● Takes 2 numbers and produces another number (binary
    operation)
    - Add: simple add of two numbers
    - Average: maintain two values - sum and count &
    “adds” each of them
    ● Ordering of operations don’t matter (commutative)
    ● Grouping of operations don’t matter (associative)
    ● Ignores 0s
    

    View Slide

17. Abstraction
    ● We are dealing with Sets
    ● Associative binary operations
    ● Identity element exists (for additions - it’s zero)
    

    View Slide

18. Abstraction
    ● We are dealing with Sets
    ● Associative binary operations
    ● Identity element exists (for additions - it’s zero)
    = Monoid
    

    View Slide

19. Abstraction
    ● We are dealing with Sets
    ● Associative binary operations
    ● Identity element exists (for additions - it’s zero)
    ● Add Commutativity to the mix
    = Commutative Monoid
    

    View Slide

20. Aggregations at Scale
    ● Associative and Commutative
    ○ Makes it an EMBARRASSINGLY PARALLEL* problem
    ● User Queries are handled via Scatter Gather
    ○ Reduce on individual nodes
    ○ Re-reduce on the results and return as the response
    

    View Slide

21. Talk is cheap, show me the code!
    

    View Slide

22. Introducing Abel
    

    View Slide

23. ● Monoid based aggregations
    ● Durable delivery via Kafka
    ● Persistence via RocksDB
    ● User queries handled via Scatter Gather
    ● Scala all the way
    Abel
    twitter/algebird
    ashwanthkumar/suuchi
    

    View Slide

24. stats.service.ix
    Count(“a”, 1L) Count(“c”, 1L)
    Unique(“ua”, “a”)
    Count(“b”, 1L)
    Abel Data Flow
    Average(“a”, 5682)
    

    View Slide

25. Abel Internals
    Metric = Key * Aggregate (Monoid)
    case class Metric[T <: Aggregate[T]] (key: Key, value: T with Aggregate[T])
    

    View Slide

26. Abel Internals
    Metric = Key * Aggregate (Monoid)
    trait Aggregate[T <: Aggregate[_]] { self: T =>
    def plus(another: T): T
    def show: JsValue
    }
    

    View Slide

27. Abel Internals
    Key = Name * Tags * Time
    case class Time(time: Long, granularity: Long)
    case class Key(name:String, tags:SortedSet[String],
    time:Time = Time.Forever)
    

    View Slide

28. Abel Internals
    client.send(Metric(Key(
    name = “unique-url-per-hour”,
    tags = SortedSet(“www.amazon.com”),
    time = Time.ThisHour
    ), UniqueCount(“http://...”))
    

    View Slide

29. client.send(Metrics(
    “unique-urls”,
    tag(“site:www.amazon.com”) *
    (perday | forever) * now,
    UniqueCount(“http://...”)
    ))
    Abel Internals
    To find Unique count of URLs crawled per site for
    every day and forever.
    

    View Slide

30. client.send(Metrics(
    “unique-urls”,
    (tag(“site:www.amazon.com”) | `#`) *
    (perday | forever) * now,
    UniqueCount(“http://...”)
    ))
    Abel Internals
    To find Unique count of URLs crawled per site and across sites for
    every day and forever.
    

    View Slide

31. client.send(Metrics(
    “unique-urls”,
    (tag(“site:www.amazon.com”) | `#`) *
    (perday | forever) * now,
    UniqueCount(“http://...”)
    ))
    Abel Internals
    To find Unique count of URLs crawled per site and across sites for
    every day and forever.
    It is implemented
    as a Ring.
    

    View Slide

32. stats.service.ix
    Count(“a”, 1L) Count(“c”, 1L)
    Abel v1
    Average(“a”, 5682)
    

    View Slide

33. stats.service.ix
    Count(“a”, 1L) Count(“c”, 1L)
    Unique(“ua”, “a”)
    Count(“b”, 1L)
    Abel v1
    Average(“a”, 5682)
    

    View Slide

34. A
    stats.service.ix
    1.1.1.1
    1.1.1.2
    1.1.1.3
    aggregate.plus
    1.1.1.2
    Count(“a”, 1L) Average(“a”, 5682) Count(“c”, 1L)
    Count(“b”, 1L)
    Abel in Distributed Mode
    aggregate.plus
    1.1.1.1
    aggregate.plus
    1.1.1.3
    DNS based
    Load
    Balancing
    Unique(“ua”, “a”)
    

    View Slide

35. Scatter Gather - Average
    (123, 8)
    (3, 1)
    (12303, 24)
    Reduce
    Reduce
    Reduce
    

    View Slide

36. (123, 8)
    (3, 1)
    (12303, 24)
    Re-reduce
    (12429, 33)
    =
    376.6
    Scatter Gather - Average
    

    View Slide

37. ● Monoid based aggregations
    ● Durable delivery via Kafka
    ● Persistence via RocksDB
    ● User queries handled via Scatter Gather
    ● Scala all the way
    Abel
    twitter/algebird
    ashwanthkumar/suuchi
    

    View Slide

38. Monoid Cheatsheet
    Stat / Metric Type Abstraction
    Count of Urls Sum
    Average Response Time Sum with Count & Total
    Unique count of urls crawled HyperLogLog
    HTTP Response Code Distribution Count-Min Sketch
    Top K Websites with poor response time Heap with K elements
    Website response times percentiles QTree (loosely based on q-digest)
    Histogram of response times Array(to model bins) and slotwise Sum
    

    View Slide

39. Credits
    VinothKumar Raman
    @eventaken
    Swathi Ravichandran
    @swathrav
    Thank you
    

    View Slide

40. Meta
    

    View Slide

41. Ring
    ● Abelian group under Addition
    ○ Associative
    ○ Commutative
    ○ Identity
    ○ Inverse
    ● Monoid under multiplication
    ○ Associative
    ○ Multiplicative Identity
    ● Multiplication is distributive with respect to addition
    ○ (a + b) . c = (ac + bc) Right Distributivity
    ○ a . (b + c) = (ab + ac) Left Distributivity
    

    View Slide

42. View Slide

43. View Slide

44. suuchi
    toolkit for building distributed
    function shipping applications
    github.com/ashwanthkumar/suuchi
    

    View Slide

45. Slides designed by
    www.swathiravichandran.com
    

    View Slide