Gestión de eventos de aplicación en la nube

Como parte de nuestro desarrollo en seedtag una de las necesidades que tuvimos que afrontar fue la de crear un sistema que
nos permitiera gestionar y generar reportes con la información que pasaba por nuestro sistema basado en microservicios para
poder controlar desde el negocio cosas tan importantes la evolución de nuestro producto y su facturación.

En esta charla veremos las diferentes opciones que planteamos para realizar esta tarea, las que funcionaron y las que tuvimos que modificar para acabar con un sistema que actualmente procesa más de 3 TB al día de eventos (y sigue creciendo día a día) utilizando herramientas de Google Cloud.

Transcript

1. @Laura_Morillo @Laura_Morillo Events Analysis at seedtag SSP
2. @Laura_Morillo • Twitter: @laura_morillo • GDG Madrid Organizer, WTM Lead • Startup lover. • Tech Lead at Seedtag. • Google Developer Expert (Cloud) • +12 years working with different technologies.
3. @Laura_Morillo • Twitter: @laura_morillo • GDG Madrid Organizer, WTM Lead • Startup lover. • Tech Lead at Seedtag. • Google Developer Expert (Cloud) • +12 years working with different technologies.
4. @Laura_Morillo SSP? . . .
5. @Laura_Morillo SSP? SSP
6. @Laura_Morillo 1. Events!
7. @Laura_Morillo • Requests received • Requests sent to third parties • Responses received: bids, no bids, timeouts, errors • Prices for bids • Winner impressions • Paid price and amount to pay to the publisher • ... Events Report Debug • Exact information sent to every third party • Exact information in third party response • Creatives • Exact response from SSP • ...
8. @Laura_Morillo 2. Solutions!
9. @Laura_Morillo Production Architecture
10. @Laura_Morillo Report Events Report consolidation
11. @Laura_Morillo Option 1: Sidecar container
12. @Laura_Morillo Option 1: Sidecar container SSP Event Service
13. @Laura_Morillo Option 1: Advantages • Dividing SSP and Event logic • Using existing architecture • No vendor lock-in
14. @Laura_Morillo Option 1: Disadvantages • Handle Pod Lifecycle • Adding persistence • Events not persisted immediately
15. @Laura_Morillo Option 2: BigQuery Streaming
16. @Laura_Morillo Option 2: BigQuery Streaming SSP
17. @Laura_Morillo Option 2: Advantages • Simplicity • Events immediately stored • Events ready to be queried • Low Maintenance Cost
18. @Laura_Morillo Option 2: Disadvantages • Calculate cost expectation • Connection issues
19. @Laura_Morillo Option 3: Kafka + Spark
20. @Laura_Morillo Spark Structured Streaming: • Fast • Scalable • Fault-tolerant • Exactly-once
21. @Laura_Morillo Option 3: Kafka + Spark SSP
22. @Laura_Morillo Option 3: Advantages • Existing infrastructure • Cost control • Data immediately stored
23. @Laura_Morillo Option 3: Disadvantages • Many pieces involved • Event size
24. @Laura_Morillo Winner... Option 1: Sidecar container Option 2: Bigquery Streaming Option 3: Kafka + Spark
25. @Laura_Morillo Winner... Option 1: Sidecar container Option 2: Bigquery Streaming Option 3: Kafka + Spark
26. @Laura_Morillo Reasons: • Fastest to implement and test • Small team • No infrastructure team
27. @Laura_Morillo Complete process SSP
28. @Laura_Morillo @Laura_Morillo Events Analysis at seedtag SSP
29. @Laura_Morillo @Laura_Morillo
30. @Laura_Morillo Winner... Option 1: Sidecar container Option 2: Bigquery Streaming Option 3: Kafka + Spark
31. @Laura_Morillo Winner... Option 1: Sidecar container Option 2: Bigquery Streaming Option 3: Kafka + Spark
32. @Laura_Morillo SSP Complete process
33. @Laura_Morillo Spark Structured Streaming: • Fast • Scalable • Fault-tolerant • Exactly-once
34. @Laura_Morillo Importer Input
35. @Laura_Morillo Importer Output
36. @Laura_Morillo BigQuery
37. @Laura_Morillo
38. @Laura_Morillo @Laura_Morillo Events Analysis at seedtag SSP
39. @Laura_Morillo thanks! Any questions? We are hiring! @laura_morillo laura.morillovelarde@gmail.com Presentation template by SlidesCarnival