Streaming Ingestion & Processing at Flipkart

4dcec668adc940079e59c9fce1666253?s=47 Siddhartha Reddy
May 15, 2015
Technology
0
280

Streaming Ingestion & Processing at Flipkart

Presented at the Bangalore Hadoop Meetup held on 15th May 2015.

4dcec668adc940079e59c9fce1666253?s=128

Siddhartha Reddy

May 15, 2015
Tweet

More Decks by Siddhartha Reddy

See All by Siddhartha Reddy
4dcec668adc940079e59c9fce1666253?s=48 sids
1
140
4dcec668adc940079e59c9fce1666253?s=48 sids
6
4.8k

Other Decks in Technology

See All in Technology
A645e7c3a6635ead8fa323c583769591?s=48 hololab
0
210
69f5ab96892df4d8cbe32189f2ed2d9d?s=48 heyinc
2
10k
79dcb04101764d7bf66f898dd446838e?s=48 tsukubachallenge
0
230
3499a1d71fa70b8ee44816ca9e7329fe?s=48 bells17
5
870
9e7a653ed313b3da4b263a772bfc5026?s=48 clustervr
0
320
53850955f15249a1a9dc49df6113e400?s=48 line_developers
0
140
A64ac825509279a770c13c6fc517f11a?s=48 kawaguti
2
430
C9237e9aa6c8fe028a484c9621991423?s=48 arahabica
0
190
Ce2ffa10331de7d89e06eabe1d231aef?s=48 hornets79
1
110
6557f271a0b301f453f773be43cf876f?s=48 miyamoto_hiroki
1
200
9e7a653ed313b3da4b263a772bfc5026?s=48 clustervr
0
340
Dd672f13e7e17714700544cb355cbaba?s=48 kyoshimoto
2
190

Featured

See All Featured
0c6fd6a08d0f898159cda7cafcacf07f?s=48 afnizarnur
175
14k
0bce06bd06ee7a2c4f5500f25dcf7f18?s=48 paulrobertlloyd
70
3.3k
5f50f94346fbcb23706f0707169317a4?s=48 aarron
253
36k
A415db3ac9e9668acbc1fb36eead84ac?s=48 mthomps
37
2.2k
A60068bce2e73de3a37ca9d2dbe36092?s=48 bryan
100
11k
D8fc2580cfaca035f666d9e4ee79a7f7?s=48 mongodb
23
3.7k
79acae287842acf29084005533a7fb3b?s=48 hursman
103
9k
B83d5b590577969ee79a5ad845411a7d?s=48 ammeep
652
54k
20bfe76b3d6105641f879fe45cfc9272?s=48 bkeepers
319
53k
C9b18d9dff88a9dd2393364c2b3b21bd?s=48 colly
183
14k
9cde37f47e4a800ea081ea42de8d749a?s=48 dougneiner
55
5.3k
Ef32e0b1ac5082450dc8c1fca3a26b5c?s=48 cherdarchuk
65
250k

Transcript

  1. Streaming Ingestion & Processing at Flipkart Siddhartha Reddy @sids

  2. Flipkart Data Platform (an oversimplified view)

  3. Streaming Ingestion

  4. Choices • push, not pull • schemas & validations

  5. Streaming Ingestion v1.0

  6. None

  7. • Push 㱺 accountability (with source teams) • good call!

    • Schemas 㱺 contracts for consumers • can make assumptions that are assured to be true • Insufficient tooling 㱺 too many “ingestion frameworks” • adopt some frameworks & offer as tools! • Synchronous error handling 㱺 complexity • accept all data

  8. Streaming Ingestion v2.0

  9. Stream Processing

  10. An Example

  11. Streaming Joins: Example It works! But… how do we deal

    with lookup failures?

  12. Streaming Joins: Handling Failures

  13. None
  14. None

  15. Streaming Joins: Bootstrapping With a little help from MR friends

  16. Streaming Joins: But… The example that doesn’t really work correctly

  17. Streaming Joins

  18. In summary • Streaming Ingestion: push, schemas & validation, HTTP

    service, local daemon, change data capture • Streaming Joins: indexing, lookup tables, map-joins, retry queue, batch re-driver sid@flipkart.com