AppsFlyer Data Architecture

Transcript

1. Dis$lling  insights  @                                         Arnon  Rotem-­‐Gal-­‐Oz   Chief  Data  Officer  
4. Kafka Columnar Database (Redshift- evaluating Vertica) IMDG (Ignite - evaluating Geode) Secor Spark Aggregations SparkSQL (evaluating Drill, Presto) SQL SQL Raw (sequence files) DW (parquet files) DM (Aggregations) Application dashboard Self-serve BI (TBD) Spark ETL Spark Spark ML Latest Events Scoring exploration Agg. logic Internal tools installs clicks inapp launches Accounts
5. Data’s  hierarchy  of  needs*   *With  apologies  to  Maslow   Acted   upon   presented   Dis$lled   Usable   Accessible   Exist  
6. Exist  
7. Kafka Columnar Database (Redshift- evaluating Vertica) IMDG (Ignite - evaluating Geode) Secor Spark Aggregations SparkSQL (evaluating Drill, Presto) SQL SQL Raw (sequence files) DW (parquet files) DM (Aggregations) Application dashboard Self-serve BI (TBD) Spark ETL Spark Spark ML Latest Events Scoring exploration Agg. logic Internal tools installs clicks inapp launches Accounts
8. Kafka Columnar Database (Redshift- evaluating Vertica) IMDG (Ignite - evaluating Geode) Secor Spark Aggregations SparkSQL (evaluating Drill, Presto) SQL SQL Raw (sequence files) DW (parquet files) DM (Aggregations) Application dashboard Self-serve BI (TBD) Spark ETL Spark Spark ML Latest Events Scoring exploration Agg. logic Internal tools installs clicks inapp launches Accounts
9. Working  off   of  RAW  data    
10. “Mal$ng”   Just  slap  SQL  on  everything     Accessible  
11. Kafka Columnar Database (Redshift- evaluating Vertica) IMDG (Ignite - evaluating Geode) Secor Spark Aggregations SparkSQL (evaluating Drill, Presto) SQL SQL Raw (sequence files) DW (parquet files) DM (Aggregations) Application dashboard Self-serve BI (TBD) Spark ETL Spark Spark ML Latest Events Scoring exploration Agg. logic Internal tools installs clicks inapp launches Accounts
12. Fermen$ng   Usable  
13. Kafka Columnar Database (Redshift- evaluating Vertica) IMDG (Ignite - evaluating Geode) Secor Spark Aggregations SparkSQL (evaluating Drill, Presto) SQL SQL Raw (sequence files) DW (parquet files) DM (Aggregations) Application dashboard Self-serve BI (TBD) Spark ETL Spark Spark ML Latest Events Scoring exploration Agg. logic Internal tools installs clicks inapp launches Accounts
14. Dis$lling     Dis$lled  
15. Kafka Columnar Database (Redshift- evaluating Vertica) IMDG (Ignite - evaluating Geode) Secor Spark Aggregations SparkSQL (evaluating Drill, Presto) SQL SQL Raw (sequence files) DW (parquet files) DM (Aggregations) Application dashboard Self-serve BI (TBD) Spark ETL Spark Spark ML Latest Events Scoring exploration Agg. logic Internal tools installs clicks inapp launches Accounts
16. RT  insights     Predic$ve       Prescrip$ve     Dashboards     whatnot     presented  
17. Sidetrack:    On  use  of  Spark  
18. Hadoop  &  Mesos  
20. Land  data  in  a  queue  
21. All  data  is     $me-­‐series  
22. Enrich  with  foreign    keys  before  persis$ng  
23. Analyze  and     balance  jobs  
24. Not  everything  is     big  data  
25. We’re  hiring….   jobs@appsflyer.com