Why we built a distributed system - DSConf, Pune 2018

In the first edition of DSConf, we spoke about our reasons for why we built an in-house distributed system and how Suuchi - toolkit to build such systems evolved.

Transcript

1. Why we built a distributed
   system
   DSConf 2018
   

   View Slide

2. Sriram
   Ramachandrasekaran
   Principal Engineer, Indix
   https://github.com/brewkode
   

   View Slide

3. 1B+
   Products
   67K+
   Brands
   2.5B+
   Offers
   6K+
   Categories
   

   View Slide

4. Crawl Parse Dedup Classify Extract Match Index
   Data Pipeline @ Indix
   

   View Slide

5. Crawl Parse Dedup Classify Extract Match Index
   Data Pipeline @ Indix
   

   View Slide

6. Desirable Properties
   ● Handle Scale - order of TBs
   ● Fault Tolerant
   ● Operability
   

   View Slide

7. Traditionally...
   ● Tiered architecture
   ● Scale individual tiers
   ● Until...
   

   View Slide

8. Traditionally...
   ● Tiered architecture
   ● Scale individual tiers
   ○ Web Tier
   ○ Service Tier
   ● Until...
   

   View Slide

9. Traditionally...
   ● Tiered architecture
   ● Scale individual tiers
   ○ Web Tier
   ○ Service Tier
   ● Until...
   

   View Slide

10. Essentially, we are looking
    to Scale data systems
    

    View Slide

11. BigTable,
    2006
    Dynamo,
    2007
    Cassandra,
    2008
    Voldemort,
    2009
    rise of KV Stores
    distributed, replicated, fault-tolerant, sorted*
    

    View Slide

12. Service Service Service
    Distributed Data Store
    

    View Slide

13. Service Service Service
    Distributed Data Store
    Latency
    

    View Slide

14. Distributed Service
    

    View Slide

15. Distributed Service
    Data locality
    kills
    latency
    Increases
    Application
    Complexity
    

    View Slide

16. Just having a distributed store
    isn’t enough!
    We need something more...
    

    View Slide

17. boils down to...
    Distributed Data Store
    +
    CoProcessors
    (Bigtable / HBase)
    …run arbitrary code “next” to each shard
    

    View Slide

18. Distributed Data Store + CoProcessors
    (Bigtable / HBase)
    - Business logic upgrade is painful
    - CoProcessors are not services, more an afterthought
    - Failure semantics are not well established
    - More applications means multiple coproc or single bloated coproc
    - Noisy neighbours / Impedance due to a shared datastore
    

    View Slide

19. Applications need to
    OWN
    Scaling
    

    View Slide

20. In-house Vs Off-the-shelf
    In-house Off-the-shelf
    Features Subset Superset
    Moving parts Fully Controllable Community Controlled
    Ownership Implicit Acquired / Cultural
    Upfront cost High Low
    Expertise Hired / Retained / Nurtured Community
    

    View Slide

21. Ashwanth
    Kumar
    Principal Engineer, Indix
    https://github.com/ashwanthkumar
    

    View Slide

22. पांग
    ப
    Communication
    key=”foo”
    key=”bar”
    key=”baz”
    Request Routing
    Sync / Async
    Replication
    Replication
    Data Sharding
    Cluster Membership
    Primitives in a Distributed System
    

    View Slide

23. Introducing Suuchi
    DIY kit for building distributed systems
    github.com/ashwanthkumar/suuchi
    

    View Slide

24. Suuchi
    Provides support for ...
    - underlying communication channel
    - routing queries to appropriate member
    - detecting your cluster members
    - replicating your data based on your strategy
    - local state via embedded KV store per node (optionally)
    github.com/ashwanthkumar/suuchi
    

    View Slide

25. Communication
    + HandleOrForward
    + Scatter Gather
    uses http/2 with streaming
    

    View Slide

26. Sharding / Routing
    + Consistent Hash Ring
    - Your own sharding technique?
    node 2
    node 1
    node 3
    node 4
    Consistent hashing and random trees: Distributed
    caching protocols for relieving hot spots on the
    World Wide Web
    

    View Slide

27. Sharding / Routing
    + Consistent Hash Ring
    - Your own sharding technique?
    node 2
    node 1
    node 3
    node 4
    Consistent hashing and random trees: Distributed
    caching protocols for relieving hot spots on the
    World Wide Web
    

    View Slide

28. Sharding / Routing
    + Consistent Hash Ring
    - Your own sharding technique?
    node 2
    node 1
    node 3
    node 4
    Consistent hashing and random trees: Distributed
    caching protocols for relieving hot spots on the
    World Wide Web
    

    View Slide

29. Sharding / Routing
    + Consistent Hash Ring
    - Your own sharding technique?
    Consistent hashing and random trees: Distributed
    caching protocols for relieving hot spots on the
    World Wide Web
    node 2
    node 3
    node 4
    

    View Slide

30. Membership
    static dynamic
    fault tolerance in case of
    node/process failure
    scaling up/down needs downtime
    of the system
    

    View Slide

31. Replication
    Provides high availability for write heavy
    systems at the cost of consistency
    sync async*
    every request is successful only if
    all the replicas succeeded
    

    View Slide

32. Storage
    + KeyValue
    + RocksDB
    - Your own abstraction?
    embedded KV store from FB for server
    workloads
    

    View Slide

33. Suuchi @ Indix
    ● HTML Archive
    ○ Handles 1000+ tps - write heavy system
    ○ Stores 120 TB of url & timestamp indexed HTML pages
    ● Stats (as Monoids) Aggregation System
    ○ Approximate real-time aggregates
    ○ Timeline & windowed queries
    ● Real time scheduler for our Crawlers
    ○ Prioritising which next batch of urls to crawl
    ○ Helps crawl 20+ million urls per day
    

    View Slide

34. Ringpop
    from
    Uber,
    2015
    Gizzard
    from
    Twitter,
    2011
    Slicer from
    Google,
    2016
    Suuchi, 2016
    Idea behind Suuchi
    Membership, Request Routing, Sharding etc.
    

    View Slide

35. Slides designed by
    www.swathiravichandran.com
    

    View Slide