Modern data pipelines in AdTech - life in the trenches

"There are various tasks that the modern data pipelines approach helps us solve in different domains, including advertising. Modern data pipelines allow us to process data in a more efficient manner with a diverse set of data transformation tools for both batch and streaming data processing. AdTech is a traditional industry that constantly changes and innovates. Today, it draws a lot of attention as we’re expanding the reach and movement toward a cookieless world.

In this talk, you will learn how to use modern data pipelines for reporting and analytics, as well as the case of historical data reprocessing in AdTech. We’ll dive deeper into each case, exploring the problem itself, implementation, challenges, and future improvements. In cases like business rule changes or errors in past data, we need to re-process our historical data, and it’s not a trivial task as it requires a lot of time, precision, and computational resources for each step. Due to this, a whole section of the talk will be devoted to approaches to historical data reprocessing and data lifecycle management."

Presented at QCon London 2022 (London, UK), QCon Plus 2022, O'Reilly Data Superstream 2022, Codemotion Spain 2022, Big Data Tech Warsaw 2023, Codementors meetup

Transcript

1. Modern Data Pipelines in AdTech - life in the trenches
2. Roksolana Diachuk •Big Data Developer at Captify •Diversity & Inclusion ambassador at Captify •Women Who Code Kyiv Data Engineering Lead •Speaker
3. Agenda 1. What is AdTech? 2. Data pipelines in AdTech 3. Practical examples 4. Historical data reprocessing 5. Conclusions
4. AdTech methodologies deliver the right content at the right time to the right consumer AdTech
5. What Captify does? Captify’s technologies unite to collect, connect and categorise billions of real-time search events from 2.3 billion consumers.
7. Data pipelines in AdTech •Reporting
8. Data pipelines in AdTech •Reporting •Insights
9. Data pipelines in AdTech •Reporting •Insights •Data costs attribution
10. Data pipelines in AdTech •Reporting •Insights •Data costs attribution •Users audiences building
11. Data pipelines in AdTech •Reporting •Insights •Data costs attribution •Users audiences building •All kinds of data processing/storage
12. Reporting
13. Reporting Data provider Transformer Loader
14. Data ingestion
15. Data ingestion Schema Dates parsing S3 lister
16. Data loading
17. Metadata handling First upload vs scheduled ones Schema de fi nition Data loading
18. Challenges •Diverse data types
19. •Time dependency Challenges •Diverse data types
20. •Time dependency Challenges •Diverse data types •External data storage
21. •Time dependency Challenges •Diverse data types •External data storage •Constant connection with end users
22. Data costs attribution
23. Log-level data Mapper Ingestor Transformer Data costs attribution Data costs calculator
24. Data costs attribution
25. Attribution data source Standard feed Segment feed
26. Attribution data source Impressions Clicks Conversions
27. Data costs attribution Mapping Ingestion Transformation Data costs calculator
28. Data costs attribution Mapping Ingestion Transformation Data costs calculator S3 lister Uni fi ed schema Data costs Feeds join
29. Challenges •Processing and storing really large data volumes (!)
30. •Failures handling Challenges •Processing and storing really large data volumes (!)
31. •Failures handling Challenges •Processing and storing really large data volumes (!) •Historical data reprocessing
32. Historical data reprocessing
33. Business use case
34. Attribution pipeline
35. standardfeed.transformer.Config.feedPeriod: “P30D” standardfeed.transformer.Config.startDateTime: 2022-03-01T00:00 Reprocessing mechanism val minTime = currentDay.minus(config.feedPeriod) 
 listFiles.filter(file => file.eventDateTime isAfter minTime)
36. Reprocessing Production version Reprocessed version is_impression=true is_impression=false is_impression=true is_impression=false
37. Reprocessing Production version Reprocessed version is_impression=true is_impression=false is_impression=true is_impression=false
38. Reprocessing Production version Reprocessed version
39. Production version Reprocessed version attribution_ reprocessed attribution_ prod Reporting
40. Reporting Reprocessed table Dropped partitions
41. Future with Delta lake Time travel Keeping track of changes Schema enforcement
42. Parquet files => Delta files Spark tables => Delta tables … Leveraging data versions through Delta tables history Vacuum unsuitable data Future with Delta lake
43. Challenges •Computing resources
44. •Speed of processing Challenges •Computing resources
45. •Speed of processing Challenges •Computing resources •Complexity
46. •Speed of processing Challenges •Computing resources •Complexity •High cost of the errors
48. Conclusions 1. AdTech is an exciting domain for big data
49. 2. There is more than one approach to leveraging data Conclusions 1. AdTech is an exciting domain for big data
50. 2. There is more than one approach to leveraging data Conclusions 3. There is always a room for improvement 1. AdTech is an exciting domain for big data
51. dead_ fl owers22 roksolana-d roksolanadiachuk roksolanad My contact info
52. Stand With Ukraine