Upgrade to Pro — share decks privately, control downloads, hide ads and more …

To a Billion & Beyond: How to Visually Explore, Compare & Share Large Quantitative Datasets with HiGlass

To a Billion & Beyond: How to Visually Explore, Compare & Share Large Quantitative Datasets with HiGlass

Features and patterns in large quantitative datasets are difficult to explore, compare, and communicate. We present the HiGlass application and library for interactive visualization and comparison of multiple multiscale datasets. Using a tile-based API, we demonstrate how data can be dynamically rendered on the client irrespective of its size on the server. For comparison between datasets we demonstrate multiple interactively selectable modes for synchronized panning, zooming, and value scaling. Finally, to share interactive views into the data, we store and disseminate the visualization as a combination of data location, aesthetics, and synchronizations.

Ab6e0a2ba101a2eb0a975b1ef915e85a?s=128

Fritz Lekschas

July 10, 2019
Tweet

Transcript

  1. To a Billion and Beyond How to Visually Explore, Compare

    and Share Large Quantitative Datasets with HiGlass Peter Kerpedjiev, Nezar Abdennur, and Fritz Lekschas
  2. Nezar Abdennur PostDoc at MIT @nv1ctus nvictus.me Fritz Lekschas PhD

    Candidate at Harvard @flekschas lekschas.de Peter Kerpedjiev Software Engineer at Zymergen @pkerpedjiev emptypipes.org
  3. Nezar Abdennur PostDoc at MIT @nv1ctus nvictus.me Fritz Lekschas PhD

    Candidate at Harvard @flekschas lekschas.de Peter Kerpedjiev Software Engineer at Zymergen @pkerpedjiev emptypipes.org →
  4. Nezar Abdennur PostDoc at MIT @nv1ctus nvictus.me Fritz Lekschas PhD

    Candidate at Harvard @flekschas lekschas.de Peter Kerpedjiev Software Engineer at Zymergen @pkerpedjiev emptypipes.org
  5. Nezar Abdennur PostDoc at MIT @nv1ctus nvictus.me Fritz Lekschas PhD

    Candidate at Harvard @flekschas lekschas.de Peter Kerpedjiev Software Engineer at Zymergen @pkerpedjiev emptypipes.org
  6. Genomics where we come from

  7. Methods Image taken and slightly adapted from encodeproject.org

  8. Principal Challenges Multiscale Patterns arise at various resolutions Multimodality Different

    data types and synchronized viz Multiple comparable datasets Insights arise from differences Collaborative Exploration Share exploratory state, not the end result Geospatial Data Time Series Data Image Data Genomic Data
  9. Architecture

  10. None
  11. None
  12. None
  13. None
  14. None
  15. None
  16. Python API All demos are available at github.com/higlass/scipy19

  17. HiGlass in Jupyter (see temperature.ipynb)

  18. Tileset API

  19. Tileset API tileset_info() → min_pos & max_pos: relative to the

    scene tile_size: size of the tiles (in pixels) max_zoom: ⌈log 2 (tile mesh size / tile size)⌉ max_width: ⌈tile mesh size / tile size⌉ tiles(tile_ids) ← [<uuid.z.x.y>, …] → {<uuid.z.x.y>: { dense, min_value, max_value, dtype }, ...} Generates or fetches 1D or 2D data tiles
  20. Tileset API tileset_info() → min_pos & max_pos: relative to the

    scene tile_size: size of the tiles (in pixels) max_zoom: ⌈log 2 (tile mesh size / tile size)⌉ max_width: ⌈tile mesh size / tile size⌉ tiles(tile_ids) ← [<uuid.z.x.y>, …] → {<uuid.z.x.y>: { dense, min_value, max_value, dtype }, ...} Base64-encoded raw data Generates or fetches 1D or 2D data tiles
  21. Tileset API Demo (see point-data.ipynb)

  22. View Configs

  23. View Config • Views Shared location and zoom ◦ Layout

    ◦ Tracks • Locks For view synchronization • Globals Server URLs, editability 1. { 2. "views": [ 3. { 4. "uid": "aa", 5. "initialXDomain": [0, 100], 6. "initialYDomain": [0, 100], 7. "layout": { 8. "x": 0, "y": 0, 9. "w": 12, "h": 6, 10. }, 11. "tracks": { 12. "top": [], 13. "left": [], 14. "center": [], 15. "right": [], 16. "bottom": [] 17. } 18. } 19. ], 20. "zoomLocks": { ... }, 21. "locationLocks": { ... }, 22. "valueScaleLocks": { ... }, 23. "editable": true, 24. "zoomFixed": false, 25. "trackSourceServers": ["/api/v1"], 26. "exportViewUrl": "/api/v1/viewconfs" 27. }
  24. View Config • Views Shared location and zoom ◦ Layout

    ◦ Tracks • Locks For view synchronization • Globals Server URLs, editability 1. { 2. "views": [ 3. { 4. "uid": "aa", 5. "initialXDomain": [0, 100], 6. "initialYDomain": [0, 100], 7. "layout": { 8. "x": 0, "y": 0, 9. "w": 12, "h": 6, 10. }, 11. "tracks": { 12. "top": [], 13. "left": [], 14. "center": [], 15. "right": [], 16. "bottom": [] 17. } 18. } 19. ], 20. "zoomLocks": { ... }, 21. "locationLocks": { ... }, 22. "valueScaleLocks": { ... }, 23. "editable": true, 24. "zoomFixed": false, 25. "trackSourceServers": ["/api/v1"], 26. "exportViewUrl": "/api/v1/viewconfs" 27. }
  25. View Config • Views Shared location and zoom ◦ Layout

    ◦ Tracks • Locks For view synchronization • Globals Server URLs, editability 1. { 2. "views": [ 3. { 4. "uid": "aa", 5. "initialXDomain": [0, 100], 6. "initialYDomain": [0, 100], 7. "layout": { 8. "x": 0, "y": 0, 9. "w": 12, "h": 12, 10. }, 11. "tracks": { 12. "top": [], 13. "left": [], 14. "center": [], 15. "right": [], 16. "bottom": [] 17. } 18. } 19. ], 20. "zoomLocks": { ... }, 21. "locationLocks": { ... }, 22. "valueScaleLocks": { ... }, 23. "editable": true, 24. "zoomFixed": false, 25. "trackSourceServers": ["/api/v1"], 26. "exportViewUrl": "/api/v1/viewconfs" 27. } VIEW
  26. 1. { 2. "type": "horizontal-line", 3. "server": "//higlass.io/api/v1", 4. "tilesetUid":

    "OHJakQICQD6gTD7skx4EWA", 5. "uid": "my-very-fancy-line-plot", 6. "options": { 7. "name": "My Very Fancy Line Plot!", 8. ... 9. }, 10. } Track Config • Type general encoding • Server track data source • Tileset UID data identifier • UID track identifier • Options For styling, labeling, etc.
  27. Advanced Features

  28. Advanced Features (see nyc-taxi.ipynb)

  29. Developer Features Modular Codebase Tiling, serving, and rendering are decoupled

    E.g., viewer, server, tileset API, docker Viewer Extensibility Simple plugin architecture for new track types E.g., Epilogos, GeoJSON JavaScript APIs for Integration Library version, React component, JsAPI E.g., HiPiler, Peax, cTracks EPILOGOS GEOJSON HIPILER PEAX
  30. Communication & Collaboration (see genomics.ipynb)

  31. To a Billion and Beyond HiGlass WEB: higlass.io CODE: github.com/higlass/scipy19

    TWITTER: @higlass_io PRESENTERS: Nezar Abdennur @nv1ctus nvictus.me Fritz Lekschas @flekschas lekschas.de This work is supported in part by the National Institutes of Health (U01 CA200059). Scipy Slack Channel #higlass
  32. To a Billion and Beyond HiGlass WEB: higlass.io CODE: github.com/higlass/scipy19

    TWITTER: @higlass_io PRESENTERS: Nezar Abdennur @nv1ctus nvictus.me Fritz Lekschas @flekschas lekschas.de This work is supported in part by the National Institutes of Health (U01 CA200059).
  33. Contributors & Acknowledgements HiGlass Core Contributors: Peter Kerpedjiev, Fritz Lekschas,

    Nezar Abdennur, Chuck McCallum PIs: Nils Gehlenborg, Peter Park, Leonid Mirny, Hanspeter Pfister Co-Authors: Kasper Dinkla, Hendrik Strobelt, Jacob Luber, Scott Ouellette, Alaleh Azhir, Nikhil Kumar, Jeewon Hwang, Soohyun Lee, Burak Alver This work is supported in part by the National Institutes of Health (U01 CA200059).