Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Why we built a distributed system - DSConf, Pune 2018

Sriram
April 21, 2018

Why we built a distributed system - DSConf, Pune 2018

In the first edition of DSConf, we spoke about our reasons for why we built an in-house distributed system and how Suuchi - toolkit to build such systems evolved.

Sriram

April 21, 2018
Tweet

More Decks by Sriram

Other Decks in Technology

Transcript

  1. Why we built a distributed
    system
    DSConf 2018

    View Slide

  2. Sriram
    Ramachandrasekaran
    Principal Engineer, Indix
    https://github.com/brewkode

    View Slide

  3. 1B+
    Products
    67K+
    Brands
    2.5B+
    Offers
    6K+
    Categories

    View Slide

  4. Crawl Parse Dedup Classify Extract Match Index
    Data Pipeline @ Indix

    View Slide

  5. Crawl Parse Dedup Classify Extract Match Index
    Data Pipeline @ Indix

    View Slide

  6. Desirable Properties
    ● Handle Scale - order of TBs
    ● Fault Tolerant
    ● Operability

    View Slide

  7. Traditionally...
    ● Tiered architecture
    ● Scale individual tiers
    ● Until...

    View Slide

  8. Traditionally...
    ● Tiered architecture
    ● Scale individual tiers
    ○ Web Tier
    ○ Service Tier
    ● Until...

    View Slide

  9. Traditionally...
    ● Tiered architecture
    ● Scale individual tiers
    ○ Web Tier
    ○ Service Tier
    ● Until...

    View Slide

  10. Essentially, we are looking
    to Scale data systems

    View Slide

  11. BigTable,
    2006
    Dynamo,
    2007
    Cassandra,
    2008
    Voldemort,
    2009
    rise of KV Stores
    distributed, replicated, fault-tolerant, sorted*

    View Slide

  12. Service Service Service
    Distributed Data Store

    View Slide

  13. Service Service Service
    Distributed Data Store
    Latency

    View Slide

  14. Distributed Service

    View Slide

  15. Distributed Service
    Data locality
    kills
    latency
    Increases
    Application
    Complexity

    View Slide

  16. Just having a distributed store
    isn’t enough!
    We need something more...

    View Slide

  17. boils down to...
    Distributed Data Store
    +
    CoProcessors
    (Bigtable / HBase)
    …run arbitrary code “next” to each shard

    View Slide

  18. Distributed Data Store + CoProcessors
    (Bigtable / HBase)
    - Business logic upgrade is painful
    - CoProcessors are not services, more an afterthought
    - Failure semantics are not well established
    - More applications means multiple coproc or single bloated coproc
    - Noisy neighbours / Impedance due to a shared datastore

    View Slide

  19. Applications need to
    OWN
    Scaling

    View Slide

  20. In-house Vs Off-the-shelf
    In-house Off-the-shelf
    Features Subset Superset
    Moving parts Fully Controllable Community Controlled
    Ownership Implicit Acquired / Cultural
    Upfront cost High Low
    Expertise Hired / Retained / Nurtured Community

    View Slide

  21. Ashwanth
    Kumar
    Principal Engineer, Indix
    https://github.com/ashwanthkumar

    View Slide

  22. पांग

    Communication
    key=”foo”
    key=”bar”
    key=”baz”
    Request Routing
    Sync / Async
    Replication
    Replication
    Data Sharding
    Cluster Membership
    Primitives in a Distributed System

    View Slide

  23. Introducing Suuchi
    DIY kit for building distributed systems
    github.com/ashwanthkumar/suuchi

    View Slide

  24. Suuchi
    Provides support for ...
    - underlying communication channel
    - routing queries to appropriate member
    - detecting your cluster members
    - replicating your data based on your strategy
    - local state via embedded KV store per node (optionally)
    github.com/ashwanthkumar/suuchi

    View Slide

  25. Communication
    + HandleOrForward
    + Scatter Gather
    uses http/2 with streaming

    View Slide

  26. Sharding / Routing
    + Consistent Hash Ring
    - Your own sharding technique?
    node 2
    node 1
    node 3
    node 4
    Consistent hashing and random trees: Distributed
    caching protocols for relieving hot spots on the
    World Wide Web

    View Slide

  27. Sharding / Routing
    + Consistent Hash Ring
    - Your own sharding technique?
    node 2
    node 1
    node 3
    node 4
    Consistent hashing and random trees: Distributed
    caching protocols for relieving hot spots on the
    World Wide Web

    View Slide

  28. Sharding / Routing
    + Consistent Hash Ring
    - Your own sharding technique?
    node 2
    node 1
    node 3
    node 4
    Consistent hashing and random trees: Distributed
    caching protocols for relieving hot spots on the
    World Wide Web

    View Slide

  29. Sharding / Routing
    + Consistent Hash Ring
    - Your own sharding technique?
    Consistent hashing and random trees: Distributed
    caching protocols for relieving hot spots on the
    World Wide Web
    node 2
    node 3
    node 4

    View Slide

  30. Membership
    static dynamic
    fault tolerance in case of
    node/process failure
    scaling up/down needs downtime
    of the system

    View Slide

  31. Replication
    Provides high availability for write heavy
    systems at the cost of consistency
    sync async*
    every request is successful only if
    all the replicas succeeded

    View Slide

  32. Storage
    + KeyValue
    + RocksDB
    - Your own abstraction?
    embedded KV store from FB for server
    workloads

    View Slide

  33. Suuchi @ Indix
    ● HTML Archive
    ○ Handles 1000+ tps - write heavy system
    ○ Stores 120 TB of url & timestamp indexed HTML pages
    ● Stats (as Monoids) Aggregation System
    ○ Approximate real-time aggregates
    ○ Timeline & windowed queries
    ● Real time scheduler for our Crawlers
    ○ Prioritising which next batch of urls to crawl
    ○ Helps crawl 20+ million urls per day

    View Slide

  34. Ringpop
    from
    Uber,
    2015
    Gizzard
    from
    Twitter,
    2011
    Slicer from
    Google,
    2016
    Suuchi, 2016
    Idea behind Suuchi
    Membership, Request Routing, Sharding etc.

    View Slide

  35. Slides designed by
    www.swathiravichandran.com

    View Slide