Big Data at AdTech

"AdTech is a very innovative sphere and brings more and more attention now with the removal of cookies. Captify is one of the industry players and is the largest independent holder of user search data outside of Google. But how do companies like Captify leverage big data to find solutions for advertising problems? Which problems can be solved using big data? And where’s Scala in all of that? You will find out in this talk"
Presented at Functional Scala 2021 (London, UK)

Transcript

2. Roksolana Diachuk • Big Data Developer at Captify • Diversity & Inclusion ambassador at Captify • Women Who Code Kyiv Data Engineering Lead • Speaker
3. Agenda 1. What is big data? 2. What Captify does? 3. Big data tasks at Captify 4. Practical examples 5. Conclusions
4. Big Data
5. 5 VS
7. AdTech AdTech methodologies deliver the right content at the right time to the right consumer
8. Captify’s technologies unite to collect, connect and categorize billions of real-time search events from 2.3bn consumers. What Captify does
10. Where’s Scala?
11. Big data tasks • Reporting • Insights • Data costs attribution • Users audiences building • All kinds of data processing/storage
12. Reporting
13. Reporting Data provider Transformer Loader
14. Data ingestion
15. S3 lister Dates parsing Schema Data ingestion
16. Data loading Data loading
17. Data loading Data loading Metadata handling First upload vs scheduled ones Schema de f inition
18. Challenges • Diverse data types • Time dependency • External data storage • Constant connection with end users
19. Insights
20. Insights Search data Impala Loader ES Loader Keywords pro f iler
21. Keywords pro f iler
22. Keywords pro f iler Extracting geographical data Extracting demographics
23. Data loading Data loading
24. Data loading Data loading ES service Time-bounded user pro f iles building
25. Challenges • Same as reporting • Large data volume in Elastic search • Cross-team collaboration
26. Data costs attribution
27. Data costs attribution Log-level data Mapper Ingestion Transformer
28. Data costs attribution
29. Data costs attribution Mapping Ingestion Transformation
30. Data costs attribution Mapping Ingestion Transformation S3 lister Uni f ied schema Data costs calculation
31. Challenges • Processing and storing really large data volumes (!) • Failures handling • Historical data re-processing
33. dead_flowers22 roksolana-d roksolanadiachuk roksolanad My contact info