Hadoop at AdRoll

http://www.meetup.com/hadoopsf/events/187173842/

Transcript

1. Hadoop @ AdRoll Wesley Chow
2. Retargeting
3. Pixel “fires” Serve ad? Ad served
4. 7/2011 - ~50GB/day 4/2013 - ~5TB/day 10/2013 - ~20TB/day 06/2014 - ~50TB/day
5. Big Data !   50 billion bid requests/day !   2 PB total compressed data !   15 TB/day compressed, ~50 TB uncompressed !   2 TB/day per engineer !   2.5 engineers actively managing (including interns)
6. Simple Storage Service (S3) Elastic MapReduce Elastic Compute Cloud DynamoDB
7. How do we get from here…
8. …to here?
11. low latency high accuracy
12. Lambda Architecture
13. Lambda Architecture
14. Lambda Architecture
15. Lambda Architecture
16. Lambda Architecture
17. Batch Layer !   Once a day (high latency) !   Exactly once semantics (high accuracy)
18. Speed Layer !   Real-time computation (low latency) !   At-least-once semantics (low accuracy)
19. Serving Layer !   Merge batch results with real-time results !   Gets us soft real-time analytics with hard end-of-day numbers !   Directly powers our dashboard, billing, reports, etc.
20. Lambda Architecture
21. Problems?
22. Collisions! Batch vs. real-time – different interpretation of the same data
23. Example !   Storm sees a stream of events – order not guaranteed ! Hadoop sees the same events – order guaranteed !   Disorder leads to different conclusions !   Same conversion attributed to different ads
24. Store results separately then merge when reading
25. HBase Column Families ! row_key : column_family : qualifier => value ! Hadoop writes into the “hadoop” column family !   Storm writes into the “storm” column family
26. HBase Column Families Example: !   20140611_adroll : storm : attribution => 8 !   20140611_adroll : hadoop : attribution => 10 When reading: !   “storm” is used when hadoop data is unavailable !   “hadoop” is the source of truth
27. Another problem: burstiness Bursty write traffic caused by MapReduce jobs
28. Bulk load instead of Put/Delete
29. HBase Bulk Loading ! MapReduce outputs region-aligned HFiles (StoreFiles) ! HFiles are then ingested by HBase, skipping the WAL !   Minimize latency between HFile generation and load
30. Extra Benefits !   Decouple MapReduce failures from HBase failures
31. Extra Benefits ! HFiles are great for keeping a secondary cluster up to date !   Cluster in another region !   Hot backup !   On catastrophic failure, just point everything to the secondary
32. Extra Benefits !   One event (pixel, impression, click, etc.) may generate a write into multiple tables !   Fairly easy to generate HFiles for each multiple tables in one pass ! Denormalize aggressively with little cost
33. Bulk loading… just do it!
35. Questions?
36. Thanks! wesley.chow@adroll.com We’re hiring! adroll.com/careers