Pragmatic streaming, beaming to BigQuery

Our experience with Apache Beam on Cloud Dataflow and streaming to BigQuery.

Transcript

1. Shop smart, feel great Pragmatic Streaming, beaming to BigQuery at vex
2. Big Data/Cloud Architect @ vente-exclusive.com Google Developer Expert - follow @alexvb Alex Van Boxel
3. Product Manager @ vente-exclusive.com follow @bodschut Bob De Schutter
4. The views and opinions expressed in this talk are those of the authors and do not necessarily reflect the official position of Vente-Exclusive or Google
5. The Monolith Production database Reporting database BigQuery Cloud Pub/Sub Nightly table transfers Cloud Storage Channel interactions (CAL events) .Net Applications Tableau Server Microsoft Excel Cloud Dataflow
6. Why?
7. Platform
8. Cloud Google Edge POP Cloud Load Balancing Cloud CDN Cloud Firewall Rules Cloud Armor Google Edge POP Cloud Armor Google Edge POP Cloud Armor Cloud DNS Identity-Aware Proxy Google Edge POP Cloud Armor europe-west1 zone-b Kubernetes Engine zone-c Kubernetes Engine zone-d Kubernetes Engine europe-west2/3/4 (future) zone-x Kubernetes Engine zone-y Kubernetes Engine zone-z Kubernetes Engine Storage Images
9. stream, stream, stream your flow gently down the stream Data - Technology
10. Technology - Apache Beam
11. Technology - Cloud Dataflow
12. Technology - Beam on Cloud DataFlow Amazing Uptimes Magic Updates
13. Technology - Apache Beam Windows and the famous large window experiment Source: Medium: Cloud Dataflow and large beam windows
14. Technology - Backbone
15. stream, stream, stream your flow gently down the stream Architecture
16. Streaming - Data Backbone Entities Cloud Pub/Sub Quantum Microservices Third Party Cloud Pub/Sub Cloud Pub/Sub Calvin Kubernetes Engine
17. Streaming - Backup Backup Cloud Dataflow Backup BigQuery Entities Cloud Pub/Sub Quantum Microservices Coldline Backup Cloud Storage
18. Streaming - Activity Dead Letter BigQuery Clean Cloud Dataflow Cloud Pub/Sub Calvin Kubernetes Engine Cloud Pub/Sub Session Cloud Dataflow Raw BigQuery Realtime BigQuery Sessionized BigQuery
19. Streaming - Entities Entities Cloud Pub/Sub Quantum Microservices Entity Stream - BQ Cloud Dataflow Entity Stream - BT Cloud Dataflow Entity Stream - ES Cloud Dataflow Clean Cloud Dataflow Central Store Cloud Bigtable Entities Cloud Pub/Sub Data Lake BigQuery Search
20. Streaming - KPI Entities Cloud Pub/Sub KPI - x Cloud Dataflow KPI results Cloud Bigtable KPI - y Cloud Dataflow KPI results Cloud Bigtable KPI - z Cloud Dataflow KPI results Cloud Bigtable KPI - a Cloud Dataflow KPI results Cloud Bigtable
21. Streaming - KPI refocus > Realtime aggregates Entities Cloud Pub/Sub Customer RT Cloud Dataflow KPI results Cloud Bigtable KPI results Cloud Bigtable Mail RT Cloud Dataflow KPI results Cloud Bigtable Other RT KPI results Cloud Bigtable
22. Spanner for KPI's Streaming - KPI refocus > Business aggregates Entities Cloud Pub/Sub Quantum Microservices Entity Stream - BQ Cloud Dataflow Entity Stream - BT Cloud Dataflow Entity Stream - ES Cloud Dataflow Search Elastic on GCP Central Store Cloud Bigtable Entities Cloud Pub/Sub Data Lake BigQuery Search Entity Stream - SP Cloud Dataflow KPI Cloud Spanner
23. Time for some self reflection Reflection
24. Reflection PubSub Bad: It's no Kafka Good: It's no Kafka Reliable Scalable Zero Maintenance
25. Reflection PubSub Bad: It's no Kafka Good: It's no Kafka Reliable Scalable Zero Maintenance
26. Reflection PubSub Bad: It's no Kafka Good: It's no Kafka Reliable Scalable Zero Maintenance
27. Reflection Apache Beam Spark... whatever Beam Portable Reliable Scalable Zero Maintenance
28. Reflection Apache Beam Spark... whatever Beam Portable Reliable Scalable Zero Maintenance
29. Reflection Apache Beam Spark... whatever Beam Portable Reliable Scalable Zero Maintenance
30. Current Architecture Reflection - Mongo Change Streams Entities Cloud Pub/Sub Quantum Microservices Entities Cloud Pub/Sub Entity Stream Cloud Dataflow Central Store Cloud Bigtable Entity Stream - BQ Cloud Dataflow Entity Stream - EL Cloud Dataflow Entity Stream - SP Cloud Dataflow
31. Reflection Contracts
32. Enable real-time decision making Consuming the stream
33. Consuming the stream Different database technologies for different purposes Reporting data -- real-time! -- full change history -- self-service BI -- excel report generation -- permission based -- GDPR! -- SaaS -- Standard SQL language -- High-availability -- low-latency -- real-time -- PaaS -- backup + history -- search (backoffice only atm) -- future: site search -- low-latency -- real-time -- distributed -- Standard SQL -- complex KPI’s BigQuery BigTable Elastic Spanner?
34. Data lake: end-user data consumption Google BigQuery as analytics data warehouse Columnar server-less operation API or UI access Pay for amount of data travelled ($5/Tb) audit logs billing details available Standard SQL compliant fine-grained permissions support for nested and array types storage is cheap BigQuery
35. BigQuery as real-time data warehouse Projects, datasets, tables and views Data Lake Raw entity events (with audit info) raw tracking events table Entity Data Core business entities Controlled views on raw data Aggregated Data Pre-aggregated KPI’s with scheduled refreshes No nesting, tableau ready No analyst access Custom Data Custom datasets Per analyst / use case analyst access tables (mostly) views Events Data parsed and cleaned tracking events
36. BigQuery as real-time data warehouse DEMO
37. Shop smart, feel great THANK YOU twitter @alexvb @bodschut