Real Time Data Processing

Pada tech talk ini dijelaskan proses tim data memproses data secara real time pada aplikasi vidio.com

Transcript

1. Inside KMK Real Time Data Processing Teguh Riadi
2. KMK DATA ARCHITECTURE
4. Pub/Sub Plenty DATA INGESTION
5. Pub/Sub Plenty DATA INGESTION DATA STREAMING Dataflow
6. Pub/Sub Plenty DATA INGESTION DATA STREAMING Dataflow GCS Big Query DATA LAKE
7. Pub/Sub Plenty DATA INGESTION DATA STREAMING Dataflow GCS Big Query DATA LAKE
8. Pub/Sub Plenty DATA INGESTION DATA STREAMING Dataflow GCS Big Query DATA LAKE ETL DATA WAREHOUSE
9. Pub/Sub Plenty DATA INGESTION DATA STREAMING Dataflow GCS Big Query DATA LAKE ETL DATA WAREHOUSE VISUALIZATION Zeppelin
10. Pub/Sub Plenty DATA INGESTION DATA STREAMING Dataflow GCS Big Query DATA LAKE ETL DATA WAREHOUSE VISUALIZATION Zeppelin
11. KMK REAL TIME PROCESSING
12. LIVE STREAMING VIDEO
13. LIVE STREAMING VIDEO
14. LIVE STREAMING VIDEO NON LIVE STREAMING VIDEO
15. LIVE STREAMING VIDEO ON DEMAND
16. WHY WE NEED REAL TIME PROCESSING ?
17. Pub/Sub Plenty DATA INGESTION DATA STREAMING Dataflow Big Query DATA LAKE ETL DATA WAREHOUSE BATCH PROCESSING BEFORE USING REAL TIME PROCESSING
18. CASE : Vidio Alltime Count
19. DATA INGESTION DATA STREAMING DATA WAREHOUSE & LAKE BEFORE USING REAL TIME PROCESSING CASE : Vidio Alltime Count
20. DATA INGESTION DATA STREAMING DATA WAREHOUSE & LAKE ETL 2 ETL 1 ETL 3 BEFORE USING REAL TIME PROCESSING CASE : Vidio Alltime Count
21. DATA INGESTION DATA STREAMING DATA WAREHOUSE & LAKE ETL 2 ETL 1 ETL 3 ETL 4 BEFORE USING REAL TIME PROCESSING CASE : Vidio Alltime Count Every 5 min
22. DATA INGESTION DATA STREAMING DATA WAREHOUSE & LAKE ETL 2 ETL 1 ETL 3 ETL 4 BEFORE USING REAL TIME PROCESSING CASE : Vidio Alltime Count Every 5 min Not Fault Tolerance Not Real Time
23. Dataflow Pub/Sub REAL TIME PROCESSING
24. Pub/Sub Plenty DATA INGESTION DATA STREAMING Dataflow GCS Big Query DATA LAKE ETL DATA WAREHOUSE VISUALIZATION Zeppelin REAL TIME PROCESSING BATCH PROCESSING
25. REAL TIME PROCESSING CHALLENGE
26. CHALLENGE LIVE STREAMING & VIDEO COUNT VIEWER ACCURACY
27. CHALLENGE LIVE STREAMING & VIDEO COUNT VIEWER ACCURACY
28. CHALLENGE LIVE STREAMING & VIDEO COUNT VIEWER ACCURACY DATAFLOW SPIKE
29. CHALLENGE LIVE STREAMING & VIDEO COUNT VIEWER ACCURACY DATAFLOW SPIKE DATAFLOW SYSTEM LAG
30. CHALLENGE LIVE STREAMING & VIDEO COUNT VIEWER ACCURACY DATAFLOW SPIKE DATAFLOW SYSTEM LAG UN ACKNOWLEDGE MESSAGE
31. CHALLENGE LIVE STREAMING & VIDEO COUNT VIEWER ACCURACY DATAFLOW SPIKE DATAFLOW SYSTEM LAG UN ACKNOWLEDGE MESSAGE LATE DATA
32. CHALLENGE LIVE STREAMING & VIDEO COUNT VIEWER ACCURACY DATAFLOW SPIKE DATAFLOW SYSTEM LAG UN ACKNOWLEDGE MESSAGE LATE DATA
33. CHALLENGE LIVE STREAMING & VIDEO COUNT VIEWER ACCURACY GROWING DATA DATAFLOW SPIKE DATAFLOW SYSTEM LAG UN ACKNOWLEDGE MESSAGE LATE DATA
34. 350 K Request / Minutes 2.5 M Request / Minutes Plenty GROWING DATA
35. 350 K Request / Minutes 2.5 M Request / Minutes 127 K Events / Minutes 648 K Events / Minutes Plenty Live streaming GROWING DATA
36. 350 K Request / Minutes 2.5 M Request / Minutes 127 K Events / Minutes 648 K Events / Minutes 368 Events / Minutes 3.2 K Events / Minutes Plenty Live streaming Video GROWING DATA * August 2018 Data
37. 1 Request
38. Dataflow Spike
39. Dataflow System Lag
40. Un Acknowledge Message
41. Late Data
42. HOW WE TACKLE IT ?
43. ADD MORE DATAFLOW PIPELINE
44. ADD MORE DATAFLOW PIPELINE Dataflow Pub/Sub
45. ADD MORE DATAFLOW PIPELINE Dataflow Pub/Sub
46. ADD MORE DATAFLOW PIPELINE Dataflow Dataflow Pub/Sub Dataflow Pub/Sub Dataflow
47. ADD MORE DATAFLOW PIPELINE SPLIT PUBSUB TOPIC
48. SPLIT PUBSUB TOPIC Pub/Sub Ahoy.events
49. SPLIT PUBSUB TOPIC Pub/Sub Ahoy.events
50. SPLIT PUBSUB TOPIC Pub/Sub Ahoy.events Pub/Sub Ahoy.events Ahoy.events.livestreaming Ahoy.events.video.start
51. SPLIT PUBSUB TOPIC Pub/Sub Ahoy.events Pub/Sub Ahoy.events Ahoy.events.livestreaming Ahoy.events.video.start Dataflow Dataflow Dataflow Dataflow
52. ADD MORE DATAFLOW PIPELINE SPLIT PUBSUB EVENT ETL
53. Pub/Sub Plenty DATA INGESTION DATA STREAMING Dataflow REAL TIME PROCESSING
54. Pub/Sub Plenty DATA INGESTION DATA STREAMING Dataflow Big Query DATA LAKE ETL DATA WAREHOUSE REAL TIME PROCESSING BATCH PROCESSING
55. ADD MORE DATAFLOW PIPELINE SPLIT PUBSUB EVENT ETL MONITORING
56. MONITORING
57. MONITORING
58. MONITORING
59. THANK YOU
60. Pub/Sub Dataflow Big Query Batch Layer Speed Layer Serving Layer ETL 1 2 3 4 5