Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Using Monoids for large scale aggregation - Scala.io, Lyon 2017

Sriram
November 02, 2017

Using Monoids for large scale aggregation - Scala.io, Lyon 2017

In this talk, you will see, how Monoids acts as a powerful abstraction to build distributed stats aggregation system. You will also see a high level architecture of how an in-house system named able was built based on this premise.

Sriram

November 02, 2017
Tweet

More Decks by Sriram

Other Decks in Technology

Transcript

  1. Using Monoids for Large Scale
    Aggregates
    Scala.io 2017

    View Slide

  2. Sriram
    Ramachandrasekaran
    @brewkode
    Principal Engineer, Indix

    View Slide

  3. Ashwanth
    Kumar
    @_ashwanthkumar
    Principal Engineer, Indix

    View Slide

  4. View Slide

  5. class Crawler {
    def crawl(url: String) {
    agent.doCrawl(url)
    metric.count(“urls_crawled”, 1L)
    }
    }

    View Slide

  6. 1 1 1 1 1 1 1 1 1
    +

    View Slide

  7. 1 1 1 1 1 1 1 1 1
    4 1 1 1 1 1
    +
    +

    View Slide

  8. 1 1 1 1 1 1 1 1 1
    4 1 1 1 1 1
    8 1
    +
    = 9
    Total URLs Crawled
    +
    +

    View Slide

  9. 1 1 1 1 1 1 1 1 1
    4 1 1 1 1 1
    8 1
    = 9
    Total URLs Crawled
    (1+1+1+1)+(1+1+1+1)+1 = 4+4+1
    (4+5) = (8+1) = 9
    +
    +
    +

    View Slide

  10. (1+1+1+1)+(1+1+1+1)+1 = 4+4+1
    (4+5) = (8+1) = 9
    1 1 1 1 1 1 1 1 1
    4 1 1 1 1 1
    8 1
    = 9
    Total URLs Crawled
    Associativity
    +
    +
    +

    View Slide

  11. class Crawler {
    def crawl(url: String) {
    val page = agent.doCrawl(url)
    metric.average(“response_times”,
    page.responseTime)
    }
    }

    View Slide

  12. 1200 3600 4800

    View Slide

  13. 1200 3600 4800 [1200,1] [3600,1] [4800,1]
    +

    View Slide

  14. 1200 3600 4800 [1200,1] [3600,1] [4800,1]
    [4800,2] [4800,1]
    +
    +

    View Slide

  15. 1200 3600 4800 [1200,1] [3600,1] [4800,1]
    [4800,2] [4800,1]
    = 3200
    Average Response Time
    +
    +

    View Slide

  16. Generalizing Sum and Average
    ● Takes 2 numbers and produces another number (binary
    operation)
    - Add: simple add of two numbers
    - Average: maintain two values - sum and count &
    “adds” each of them
    ● Ordering of operations don’t matter (commutative)
    ● Grouping of operations don’t matter (associative)
    ● Ignores 0s

    View Slide

  17. Abstraction
    ● We are dealing with Sets
    ● Associative binary operations
    ● Identity element exists (for additions - it’s zero)

    View Slide

  18. Abstraction
    ● We are dealing with Sets
    ● Associative binary operations
    ● Identity element exists (for additions - it’s zero)
    = Monoid

    View Slide

  19. Abstraction
    ● We are dealing with Sets
    ● Associative binary operations
    ● Identity element exists (for additions - it’s zero)
    ● Add Commutativity to the mix
    = Commutative Monoid

    View Slide

  20. Aggregations at Scale
    ● Associative and Commutative
    ○ Makes it an EMBARRASSINGLY PARALLEL* problem
    ● User Queries are handled via Scatter Gather
    ○ Reduce on individual nodes
    ○ Re-reduce on the results and return as the response

    View Slide

  21. Talk is cheap, show me the code!

    View Slide

  22. Introducing Abel

    View Slide

  23. ● Monoid based aggregations
    ● Durable delivery via Kafka
    ● Persistence via RocksDB
    ● User queries handled via Scatter Gather
    ● Scala all the way
    Abel
    twitter/algebird
    ashwanthkumar/suuchi

    View Slide

  24. stats.service.ix
    Count(“a”, 1L) Count(“c”, 1L)
    Unique(“ua”, “a”)
    Count(“b”, 1L)
    Abel Data Flow
    Average(“a”, 5682)

    View Slide

  25. Abel Internals
    Metric = Key * Aggregate (Monoid)
    case class Metric[T <: Aggregate[T]] (key: Key, value: T with Aggregate[T])

    View Slide

  26. Abel Internals
    Metric = Key * Aggregate (Monoid)
    trait Aggregate[T <: Aggregate[_]] { self: T =>
    def plus(another: T): T
    def show: JsValue
    }

    View Slide

  27. Abel Internals
    Key = Name * Tags * Time
    case class Time(time: Long, granularity: Long)
    case class Key(name:String, tags:SortedSet[String],
    time:Time = Time.Forever)

    View Slide

  28. Abel Internals
    client.send(Metric(Key(
    name = “unique-url-per-hour”,
    tags = SortedSet(“www.amazon.com”),
    time = Time.ThisHour
    ), UniqueCount(“http://...”))

    View Slide

  29. client.send(Metrics(
    “unique-urls”,
    tag(“site:www.amazon.com”) *
    (perday | forever) * now,
    UniqueCount(“http://...”)
    ))
    Abel Internals
    To find Unique count of URLs crawled per site for
    every day and forever.

    View Slide

  30. client.send(Metrics(
    “unique-urls”,
    (tag(“site:www.amazon.com”) | `#`) *
    (perday | forever) * now,
    UniqueCount(“http://...”)
    ))
    Abel Internals
    To find Unique count of URLs crawled per site and across sites for
    every day and forever.

    View Slide

  31. client.send(Metrics(
    “unique-urls”,
    (tag(“site:www.amazon.com”) | `#`) *
    (perday | forever) * now,
    UniqueCount(“http://...”)
    ))
    Abel Internals
    To find Unique count of URLs crawled per site and across sites for
    every day and forever.
    It is implemented
    as a Ring.

    View Slide

  32. stats.service.ix
    Count(“a”, 1L) Count(“c”, 1L)
    Abel v1
    Average(“a”, 5682)

    View Slide

  33. stats.service.ix
    Count(“a”, 1L) Count(“c”, 1L)
    Unique(“ua”, “a”)
    Count(“b”, 1L)
    Abel v1
    Average(“a”, 5682)

    View Slide

  34. A
    stats.service.ix
    1.1.1.1
    1.1.1.2
    1.1.1.3
    aggregate.plus
    1.1.1.2
    Count(“a”, 1L) Average(“a”, 5682) Count(“c”, 1L)
    Count(“b”, 1L)
    Abel in Distributed Mode
    aggregate.plus
    1.1.1.1
    aggregate.plus
    1.1.1.3
    DNS based
    Load
    Balancing
    Unique(“ua”, “a”)

    View Slide

  35. Scatter Gather - Average
    (123, 8)
    (3, 1)
    (12303, 24)
    Reduce
    Reduce
    Reduce

    View Slide

  36. (123, 8)
    (3, 1)
    (12303, 24)
    Re-reduce
    (12429, 33)
    =
    376.6
    Scatter Gather - Average

    View Slide

  37. ● Monoid based aggregations
    ● Durable delivery via Kafka
    ● Persistence via RocksDB
    ● User queries handled via Scatter Gather
    ● Scala all the way
    Abel
    twitter/algebird
    ashwanthkumar/suuchi

    View Slide

  38. Monoid Cheatsheet
    Stat / Metric Type Abstraction
    Count of Urls Sum
    Average Response Time Sum with Count & Total
    Unique count of urls crawled HyperLogLog
    HTTP Response Code Distribution Count-Min Sketch
    Top K Websites with poor response time Heap with K elements
    Website response times percentiles QTree (loosely based on q-digest)
    Histogram of response times Array(to model bins) and slotwise Sum

    View Slide

  39. Credits
    VinothKumar Raman
    @eventaken
    Swathi Ravichandran
    @swathrav
    Thank you

    View Slide

  40. Meta

    View Slide

  41. Ring
    ● Abelian group under Addition
    ○ Associative
    ○ Commutative
    ○ Identity
    ○ Inverse
    ● Monoid under multiplication
    ○ Associative
    ○ Multiplicative Identity
    ● Multiplication is distributive with respect to addition
    ○ (a + b) . c = (ac + bc) Right Distributivity
    ○ a . (b + c) = (ab + ac) Left Distributivity

    View Slide

  42. View Slide

  43. View Slide

  44. suuchi
    toolkit for building distributed
    function shipping applications
    github.com/ashwanthkumar/suuchi

    View Slide

  45. Slides designed by
    www.swathiravichandran.com

    View Slide