Suuchi - FifthElephant, Bengaluru 2017

Slides were designed by my wife - Swathi Ravichandran
www.swathiravichandran.com | @swathrav

Transcript

1. Suuchi toolkit to build distributed systems
2. Sriram Ramachandrasekaran Principal Engineer, Indix https://github.com/brewkode
3. 950M+ Products 50K+ Brands 2B+ Offers 7.5K+ Categories
4. Crawl Parse Dedup Classify Extract Match Index Data Pipeline @ Indix
5. Crawl Parse Dedup Classify Extract Match Index Data Pipeline @ Indix
6. Desirable Properties • Handle Scale - order of TBs • Fault Tolerant • Ease of operations - less moving parts
7. Traditionally... • Tiered architecture • Scale individual tiers • Until...
8. Traditionally... • Tiered architecture • Scale individual tiers ◦ Web Tier ◦ Service Tier • Until...
9. Traditionally... • Tiered architecture • Scale individual tiers ◦ Web Tier ◦ Service Tier • Until...
10. Essentially, we are looking to Scale data systems
11. BigTable, 2006 Dynamo, 2007 Cassandra, 2008 Voldemort, 2009 rise of KV Stores distributed, replicated, fault-tolerant, sorted*
12. Service Service Service Distributed Data Store
13. Service Service Service Distributed Data Store Latency
14. Distributed Service
15. Distributed Service Data locality kills latency Increases Application Complexity
16. Just having a distributed store isn’t enough! We need something more...
17. boils down to... Distributed Data Store + CoProcessors (Bigtable / HBase) …run arbitrary code “next” to each shard
18. Distributed Data Store + CoProcessors (Bigtable / HBase) - Business logic upgrade is painful - CoProcessors are not services, more an afterthought - Failure semantics are not well established - More applications means multiple coproc or single bloated coproc - Noisy neighbours / Impedance due to a shared datastore
19. Applications need to OWN Scaling
20. In-house Vs Off-the-shelf In-house Off-the-shelf Features Subset Superset Moving parts Fully Controllable Community Controlled Ownership Implicit Acquired / Cultural Upfront cost High Low Expertise Hired / Retained / Nurtured Community
21. पांग ப Communication key=”foo” key=”bar” key=”baz” Request Routing Sync / Async Replication Replication Data Sharding Cluster Membership
22. Introducing Suuchi DIY kit for building distributed systems github.com/ashwanthkumar/suuchi
23. Suuchi Provides support for ... - underlying communication channel - routing queries to appropriate member - detecting your cluster members - replicating your data based on your strategy - local state via embedded KV store per node (optionally) github.com/ashwanthkumar/suuchi
24. Communication + HandleOrForward + Scatter Gather uses http/2 with streaming
25. Sharding / Routing + Consistent Hash Ring - Your own sharding technique? node 2 node 1 node 3 node 4 Consistent hashing and random trees: Distributed caching protocols for relieving hot spots on the World Wide Web
26. Membership static dynamic fault tolerance in case of node/process failure scaling up/down needs downtime of the system
27. Replication sync async provides very high availability for write systems at the cost of eventual consistency every request is successful only if all the replicas succeeded
28. Storage + KeyValue + RocksDB - Your own abstraction? embedded KV store from FB for server workloads
29. Getting started • gRPC Service using Protobuf2 • Generate stubs & implement them • Connect using Suuchi “Server” abstraction
30. Server Abstraction • Pluggable membership mechanism • Pluggable routing strategy • Pluggable replication method
31. Suuchi @ Indix • HTML Archive ◦ Handles 1000+ tps - write heavy system ◦ Stores 120 TB of url & timestamp indexed HTML pages • Stats Aggregation System ◦ Approximate real-time aggregates ◦ Timeline & windowed queries • Real time scheduler for our Crawlers ◦ Prioritising which next batch of urls to crawl ◦ Helps crawl 20+ million urls per day
32. Thank you