Atlas

 Atlas

GO-JEK’s real-time geospatial visualization platform

122c67ce0fdc3f270e8cd9017dca8068?s=128

Ravi Suhag

April 18, 2018
Tweet

Transcript

  1. ATLAS GO-JEK’s real-time geospatial visualization platform Ravi Suhag @ravi_suhag suhag.ravi@gmail.com

  2. ONE APP FOR ALL YOUR NEEDS Established in 2010 as

    a motorcycle ride-hailing phone based service, GO-JEK has evolved into an on-demand provider of transport and other lifestyle services. GO-JEK?
  3. Designer turned Developer turned Data Journalist turned Data Visualizer turned

    Data Engineer turning…. And me?
  4. What is Atlas? Why we built it? Architecture Challenges Approach

    Agenda
  5. At GO-JEK, location is built into the fabric of all

    our products 2,000,000,000 + GPS points daily
  6. What is Atlas? Geospatial data analysis Maps to actionable insights

    Automated metric pipeline Extensible viz layer architecture
  7. Why Atlas? Real-time information of the resource distribution. e.g. current

    supply and demand Rich exploratory interfaces to tackle multidimensional data for experimental analysis. and more…
  8. None
  9. Foundation LAYERS METRICS Choropleth Points Route Demand Supply Conversion rate

  10. { name: 'choropleth', description: 'S2 cell analysis', dimensions:[ { name:

    'color', description: 'Color of the bars', type: 'number', required: 'false', identifier: { title: 'S2 Cell Id', description: 'S2 cell on the choropleth', type: 's2id' } }, { name: 'elevation', description: 'Height of the bars', type: 'number', required: 'false', identifier: { title: 'S2 Cell Id', description: 'S2 cell on the choropleth', type: 's2id' } } ] } Layer schema
  11. { name: "demand", description: "Demand by S2 ids", frequency: 60000,

    status: "online", delay: 0, scale: { factor: 1, name: "linear" } tags: [{ name: "service_type", type: "string", description: "Service Type", filter: true, classification: "enum" }, { name: "s2id", type: "s2id", description: "S2 id of the cell", filter: false, classification: "enum" }], "fields": [{ name: "unique_customers", type: "number", aggregation: "sum", description: "No. of unique customers" }] } Metric schema
  12. Layer slection: Pick your visualization layer from heatmaps, data clusters,

    2D/3D choropleths, and more. Metric selection: Flexible mapping of dimensions and data metrics allow exploratory analysis. START DATE 15/07/2017 END DATE 15/07/2017 START TIME 15/07/2017 END TIME 15/07/2017 COLOR ACCEPTANCE RATE FACTOR 0.04 FACTOR 0.04 ALTITUDE CONVERSION RATE SERVICE TYPE GO CAR SERVICE AREA JAKARTA 2D MAP CHOROPLETH HEATMAP ROUTES MAP TYPE REALTIME TIME
  13. Cell Panel: Explore detailed data of layer’s cell. e.g. number

    stats, histogram. Each layer can interface with cell panel and provide detailed information about the cells (e.g. S2 Id). 32 UNIQUE DRIVERS START TIME 15/07/2017 ROLL UP H O U R D A Y W E E K M O N T H END TIME 15/07/2017 12 EFFECTIVE DSR C O N V E R S I O N R A T E D E M A N D S U P P L Y R A T I O 3 4 9 4 4 0 5 9 0 4 9 9 0 9
  14. Make beautiful data- driven maps. Focus on certain properties like

    roads, area boundaries etc
  15. Automated metric pipeline How do you support 15+ teams to

    analyse realtime metrics without any development effort? How does the data flow?
  16. DATA STREAM - 10+ Kafka clusters - 6 Billion+ events

    DAGGERS DATABASE DATLANTIS ENIGMA COSMOS ATLAS Data stream
  17. DATA STREAM - Flink powered jobs - SQL DIY Interface

    - Time Series Database DAGGERS DATABASE DATLANTIS ENIGMA COSMOS ATLAS Data aggregation
  18. DATA STREAM - 15+ Teams - <1 min from metric

    to visualization DAGGERS DATABASE DATLANTIS ENIGMA COSMOS ATLAS Data visualization
  19. DATA STREAM - Metric store - User Census DAGGERS DATABASE

    DATLANTIS ENIGMA COSMOS ATLAS Enigma
  20. DATA STREAM - Metric definitions - User configurations DAGGERS DATABASE

    DATLANTIS ENIGMA COSMOS ATLAS Cosmos
  21. DATA STREAM - Viz Platform - Layer definitions DAGGERS DATABASE

    DATLANTIS ENIGMA COSMOS ATLAS Atlas
  22. Realtime ? What does it mean for different metrics and

    how does it effect performance?
  23. Metric frequency Demand 60 sec 90 sec Supply Aggregation window

  24. { name: "demand", description: "Demand by S2 ids", frequency: 60000,

    status: "online", delay: 0, scale: { factor: 1, name: "linear" } tags: [{ name: "service_type", type: "string", description: "Service Type", filter: true, classification: "enum" }], "fields": [{ name: "unique_customers", type: "number", aggregation: "sum", description: "No. of unique customers" }] } Metric schema
  25. ENIGMA COSMOS ATLAS S t a r t s t

    r e a m i n g t o s u b c r i b e r s o f m e t r i c X S t a r t s t r e a m i n g t o s u b c r i b e r s o f m e t r i c Y R e g i s t e r c e n s u s f o r M e t r i c X a t a g g r e g a t i o n f r e q u e n c y S e t P o l l e r f o r M e r t i c Y a t a g g . f r e q u e n c y C h e c k i f M e t r i c Y i s a l r e a d y r e g i s t e r e d i f n o t r e g i s t e r i t I f M e t r i c X i s a l r e a d y r e g i s t e r e d a d d c l i e n t t o s u b s c r i b e r s C l i e n t A s u b s c r i b e s t o m e t r i c Y C l i e n t B s u b s c r i b e s t o m e t r i c X C l i e n t A s u b s c r i b e s t o m e t r i c X
  26. Metric delay d Time Demand Conversion Rate Now -1 min

    -15 min
  27. { name: "demand", description: "Demand by S2 ids", frequency: 60000,

    status: "online", delay: 20, scale: { factor: 1, name: "linear" } tags: [{ name: "service_type", type: "string", description: "Service Type", filter: true, classification: "enum" }], "fields": [{ name: "unique_customers", type: "number", aggregation: "sum", description: "No. of unique customers" }] } Metric schema
  28. Timeline synchronization d When different metrics are aggregated in different

    timelines, how do you visualize them in single frame?
  29. Metric delay d Time Demand Conversion Rate Now -1 min

    (latest) -15 min (earliest)
  30. Does just building the tool solve the problem? User Adoption

  31. Let’s talk! Ravi Suhag @ravi_suhag medium.com/@ravisuhag