Future Patterns in Data Ecosystem

It is important for anyone building a software platform to have a good pulse on the patterns of use that the system would see — this is one of the most important aspects that determines the value it would generate. The world of Big Data, Analytics & Data Science is fast-moving, with the "in vogue" tools changing almost every year. While most of us have grown accustomed to adopting to the changes in tooling, what often catch us unawares are the paradigm shifts within the ecosystem. This talk will walk through some such emerging patterns in the world of Big Data and go over how they're manifested at Flipkart — which has been an early adopter of many of these patterns.

Transcript

1. Architect, Flipkart Future Patterns in Data Ecosystem Siddhartha Reddy
2. Data as a 1st class citizen in SDLC Pattern #1
3. Data availability for Analytics & Data Science ✤ Product teams develop applications
 ☛ store data in databases
 ☛ log information for debugging ✤ A central data team
 ☛ pulls data from the databases
 ☛ parses log files
 ☛ loads into a data warehouse
4. ✤ This is broken ✤ No contract between product and data teams
 ☛ frequent breakages ✤ Data team spends most time
 ☛ fixing issues & catching up to changes ✤ Availability of data in logs
 ☛ is a matter of luck ✤ Tight coupling, no cohesiveness ✤ Does not scale ✤ Data is an afterthought
5. “Data is the only true IP of an internet company.” –Amod Malviya, CTO, Flipkart
7. Data availability for Analytics & Data Science ✤ Product teams develop applications
 ☛ store data in databases
 ☛ log information for debugging
 ☛ push data to a central data repository ✤ The central data team
 ☛ can take a nice long vacation
8. At Flipkart ✤ Dart: Central ingestion service ✤ Multiple modes of ingestion: HTTP service, daemon running alongside applications, bulk ingestion etc. ✤ Push, not pull ✤ Responsibility of ensuring ingestion pipeline is healthy is with product teams, not Dart team ✤ Audit: Ability to match ingested data with data in databases
10. Data as a 1st class citizen in SDLC ✤ Apache ✤ (Facebook) ✤ Apache
11. Primary users will be machines,
 not humans Pattern #2
12. At Flipkart ✤ Analytics ✤ Systemic Consumption ✤ Recommendations / Personalisation ✤ Inventory Planning ✤ Fraud Detection ✤ Pricing Engine ✤ …
13. Real-time processing takes
 centre stage Pattern #3
14. Real-time ✤ Daily
 (or weekly, or monthly) ✤ Hourly ✤ A few minutes ✤ A few seconds ✤ Instantaneous
15. ✤ Apache ✤ Apache — Streaming ✤ Apache ✤ AWS ✤ Google Cloud Dataflow ✤ Druid
16. At Flipkart ✤ Challenges ✤ windowing ✤ out-of-order data ✤ out-of-phase streams ✤ state mutations in streams ✤ Storm based “recipes” ✤ Joins (symmetric, asymmetric) ✤ Aggregations ✤ Bootstrapping ✤ Multiple
17. Convergence between Batch and Stream processing Pattern #4
18. ✤ Batch Processing: Scale & power ✤ Stream Processing: Freshness
19. Lambda Architecture
20. At Flipkart ✤ Batch Processing: Hadoop + Vertica ✤ Stream Processing: Storm + ElasticSearch ✤ Query: Apache
21. But the workflow is… broken ✤ Write batch-processing pipelines ✤ Then write stream-processing pipelines ✤ When logic needs to be updated?
 Update both!
22. ✤ Twitter’s Summingbird ✤ Apache — Streaming ✤ Google Cloud Dataflow
23. Thank you Siddhartha Reddy sid@flipkart.com