Architectural Design for Interactive Visualization

Visualisation for data science requires an interactive visualisation setup which works at scale. In this talk, we will explore the key architectural design considerations for such a system and illustrate using examples the four key tradeoffs in this design space - rendering for data scale, computation for interaction speed, adaptive to data complexity and responsive to data velocity.

Transcript

1. Architecture Design for Interactive Visualisation Amit Kapoor amitkaps.com Bargava Subramanian bargava.com 1
2. Exemplar 2
3. Layers of Abstraction — Raw Data — Transform Layer — Visual Layer — Interaction Layer 3
4. Interactive Data Visualisation0 0 Grammar of Visualisation 4
5. Architectural Design Trade-offs 1. Rendering for Data Scale 2. Computation for Interaction Speed 3. Adaptive to Data Complexity 4. Responsive to Data Velocity 5
6. 1. Rendering for Data Scale "How do you render interactive visualization when there are millions or billions of data points?" 6
7. Visualise a Million Points Show all the Data Same order as the number of pixels on my MacBook Air: 1400 x 900 Problems with overplotting 7
8. Visualise a Million Points Sample the Data Sampling can be effective (with overweighting unusual values) Require multiple plots or careful tuning parameters 8
9. Visualise a Million Points Model the Data Models are great as they scale nicely. But, visualisation is needed to answer the question: “I don’t know, what I don’t know.” 9
10. Visualise a Million Points Bin the Data Reduce the size of the data to complement the pixel resolution on screen 10
11. “Visualising data at scale is the process of creating generalized histograms” 11
12. Layers of Visualisation 12
13. Rendering: How many data points? — SVG: ~10^3 — Canvas: ~10^4 — Web.gl: ~10^6 13
14. Web.gl based Rendering: Deck.gl1 1 UK Road Accident Data using deck.gl 14
15. Bin and Web.gl Rendering 15
16. Visualise more than Million Points Bin-Summarize-Smooth Shift the transfer of data from raw data to binned- summarised-smoothed data 16
17. BigVis: Bin-Summarize-Smooth-Visualise 17
18. Why not sent an image of the Data: DataShader3 3 DataShader: Turn largest data to image, accurately 18
19. DataShader: Encode-Bin-Summarise-ColorMap 19
20. 2. Computation for Interaction Speed "How do you reduce the latency of the query at the interaction layer, so that the user can interact with the visualisation?" 20
21. Aggregation & In-Memory Cubes e.g imMems4 4 ImMems Demo 21
22. In-memory Cubes 22
23. Approximate Query Processing e.g. VerdictDB5 5 Speed Comparison: AQP is 50x+ faster 23
24. Approximate Query Processing 24
25. Challenges in Visualise Uncertainty 6 6 25
26. Faster Databases e.g. MapD7 7 TweetMap Demo by MapD 26
27. Faster Databases 27
28. 3. Adaptive to Data Complexity Choosing a good visualisation design for a singular dataset is possible after a few experiments and iteration. But how do you ensure that the visualisation will adapt to the variety, volume and edge cases in the real data? 28
29. Responsive Visualisation to Space & Data8 8 Responsive Data Visualisation - Nick Rabinowitz 29
30. Handling High Cardinality e.g. Facet-Dive9 9 FacetDive from PAIR Research 30
31. Handling High Cardinality e.g. DataComb10 10 Datacomb: Interactive Table Plot 31
32. Dimension Reduction e.g. Embedding Projector11 11 Embedding Projector 32
33. 4. Responsive to Data Velocity "How do you trade-offs between real-time vs. near real-time data and its impact on refreshing visualization" 33
34. Optimizing for near real-time visual refreshes 34
35. Architectural Design Trade-offs 1. Rendering for Data Scale 2. Computation for Interaction Speed 3. Adaptive to Data Complexity 4. Responsive to Data Velocity 35
36. Architecture Design for Interactive Visualisation Amit Kapoor amitkaps.com Bargava Subramanian bargava.com 36