To a Billion & Beyond: How to Visually Explore, Compare & Share Large Quantitative Datasets with HiGlass

Features and patterns in large quantitative datasets are difficult to explore, compare, and communicate. We present the HiGlass application and library for interactive visualization and comparison of multiple multiscale datasets. Using a tile-based API, we demonstrate how data can be dynamically rendered on the client irrespective of its size on the server. For comparison between datasets we demonstrate multiple interactively selectable modes for synchronized panning, zooming, and value scaling. Finally, to share interactive views into the data, we store and disseminate the visualization as a combination of data location, aesthetics, and synchronizations.

Transcript

1. To a Billion and Beyond
   How to Visually Explore, Compare and Share
   Large Quantitative Datasets with HiGlass
   Peter Kerpedjiev, Nezar Abdennur, and Fritz Lekschas
   

   View Slide

2. Nezar Abdennur
   PostDoc at MIT
   @nv1ctus
   nvictus.me
   Fritz Lekschas
   PhD Candidate at Harvard
   @flekschas
   lekschas.de
   Peter Kerpedjiev
   Software Engineer at Zymergen
   @pkerpedjiev
   emptypipes.org
   

   View Slide

3. Nezar Abdennur
   PostDoc at MIT
   @nv1ctus
   nvictus.me
   Fritz Lekschas
   PhD Candidate at Harvard
   @flekschas
   lekschas.de
   Peter Kerpedjiev
   Software Engineer at Zymergen
   @pkerpedjiev
   emptypipes.org
   →
   

   View Slide

4. Nezar Abdennur
   PostDoc at MIT
   @nv1ctus
   nvictus.me
   Fritz Lekschas
   PhD Candidate at Harvard
   @flekschas
   lekschas.de
   Peter Kerpedjiev
   Software Engineer at Zymergen
   @pkerpedjiev
   emptypipes.org
   
   

   View Slide

5. Nezar Abdennur
   PostDoc at MIT
   @nv1ctus
   nvictus.me
   Fritz Lekschas
   PhD Candidate at Harvard
   @flekschas
   lekschas.de
   Peter Kerpedjiev
   Software Engineer at Zymergen
   @pkerpedjiev
   emptypipes.org
   
   

   View Slide

6. Genomics
   where we come from
   

   View Slide

7. Methods
   Image taken and slightly adapted from encodeproject.org
   

   View Slide

8. Principal Challenges
   Multiscale
   Patterns arise at various resolutions
   Multimodality
   Different data types and synchronized viz
   Multiple comparable datasets
   Insights arise from differences
   Collaborative Exploration
   Share exploratory state, not the end result
   Geospatial Data
   Time Series Data
   Image Data
   Genomic Data
   

   View Slide

9. Architecture
   

   View Slide

10. View Slide

11. View Slide

12. View Slide

13. View Slide

14. View Slide

15. View Slide

16. Python API
    All demos are available at
    github.com/higlass/scipy19
    

    View Slide

17. HiGlass in Jupyter
    (see temperature.ipynb)
    

    View Slide

18. Tileset API
    

    View Slide

19. Tileset API
    tileset_info()
    → min_pos & max_pos: relative to the scene
    tile_size: size of the tiles (in pixels)
    max_zoom: ⌈log
    2
    (tile mesh size / tile size)⌉
    max_width: ⌈tile mesh size / tile size⌉
    tiles(tile_ids)
    ← [, …]
    → {: {
    dense, min_value, max_value, dtype
    }, ...}
    Generates or fetches 1D or 2D data tiles
    

    View Slide

20. Tileset API
    tileset_info()
    → min_pos & max_pos: relative to the scene
    tile_size: size of the tiles (in pixels)
    max_zoom: ⌈log
    2
    (tile mesh size / tile size)⌉
    max_width: ⌈tile mesh size / tile size⌉
    tiles(tile_ids)
    ← [, …]
    → {: {
    dense, min_value, max_value, dtype
    }, ...}
    Base64-encoded raw data
    Generates or fetches 1D or 2D data tiles
    

    View Slide

21. Tileset API Demo
    (see point-data.ipynb)
    

    View Slide

22. View Configs
    

    View Slide

23. View Config
    ● Views
    Shared location and zoom
    ○ Layout
    ○ Tracks
    ● Locks
    For view synchronization
    ● Globals
    Server URLs, editability
    1. {
    2. "views": [
    3. {
    4. "uid": "aa",
    5. "initialXDomain": [0, 100],
    6. "initialYDomain": [0, 100],
    7. "layout": {
    8. "x": 0, "y": 0,
    9. "w": 12, "h": 6,
    10. },
    11. "tracks": {
    12. "top": [],
    13. "left": [],
    14. "center": [],
    15. "right": [],
    16. "bottom": []
    17. }
    18. }
    19. ],
    20. "zoomLocks": { ... },
    21. "locationLocks": { ... },
    22. "valueScaleLocks": { ... },
    23. "editable": true,
    24. "zoomFixed": false,
    25. "trackSourceServers": ["/api/v1"],
    26. "exportViewUrl": "/api/v1/viewconfs"
    27. }
    

    View Slide

24. View Config
    ● Views
    Shared location and zoom
    ○ Layout
    ○ Tracks
    ● Locks
    For view synchronization
    ● Globals
    Server URLs, editability
    1. {
    2. "views": [
    3. {
    4. "uid": "aa",
    5. "initialXDomain": [0, 100],
    6. "initialYDomain": [0, 100],
    7. "layout": {
    8. "x": 0, "y": 0,
    9. "w": 12, "h": 6,
    10. },
    11. "tracks": {
    12. "top": [],
    13. "left": [],
    14. "center": [],
    15. "right": [],
    16. "bottom": []
    17. }
    18. }
    19. ],
    20. "zoomLocks": { ... },
    21. "locationLocks": { ... },
    22. "valueScaleLocks": { ... },
    23. "editable": true,
    24. "zoomFixed": false,
    25. "trackSourceServers": ["/api/v1"],
    26. "exportViewUrl": "/api/v1/viewconfs"
    27. }
    

    View Slide

25. View Config
    ● Views
    Shared location and zoom
    ○ Layout
    ○ Tracks
    ● Locks
    For view synchronization
    ● Globals
    Server URLs, editability
    1. {
    2. "views": [
    3. {
    4. "uid": "aa",
    5. "initialXDomain": [0, 100],
    6. "initialYDomain": [0, 100],
    7. "layout": {
    8. "x": 0, "y": 0,
    9. "w": 12, "h": 12,
    10. },
    11. "tracks": {
    12. "top": [],
    13. "left": [],
    14. "center": [],
    15. "right": [],
    16. "bottom": []
    17. }
    18. }
    19. ],
    20. "zoomLocks": { ... },
    21. "locationLocks": { ... },
    22. "valueScaleLocks": { ... },
    23. "editable": true,
    24. "zoomFixed": false,
    25. "trackSourceServers": ["/api/v1"],
    26. "exportViewUrl": "/api/v1/viewconfs"
    27. }
    VIEW
    

    View Slide

26. 1. {
    2. "type": "horizontal-line",
    3. "server": "//higlass.io/api/v1",
    4. "tilesetUid": "OHJakQICQD6gTD7skx4EWA",
    5. "uid": "my-very-fancy-line-plot",
    6. "options": {
    7. "name": "My Very Fancy Line Plot!",
    8. ...
    9. },
    10. }
    Track Config
    ● Type
    general encoding
    ● Server
    track data source
    ● Tileset UID
    data identifier
    ● UID
    track identifier
    ● Options
    For styling, labeling, etc.
    

    View Slide

27. Advanced Features
    

    View Slide

28. Advanced Features
    (see nyc-taxi.ipynb)
    

    View Slide

29. Developer Features
    Modular Codebase
    Tiling, serving, and rendering are decoupled
    E.g., viewer, server, tileset API, docker
    Viewer Extensibility
    Simple plugin architecture for new track types
    E.g., Epilogos, GeoJSON
    JavaScript APIs for Integration
    Library version, React component, JsAPI
    E.g., HiPiler, Peax, cTracks
    EPILOGOS
    GEOJSON
    HIPILER
    PEAX
    

    View Slide

30. Communication & Collaboration
    (see genomics.ipynb)
    

    View Slide

31. To a Billion and Beyond
    HiGlass
    WEB: higlass.io
    CODE: github.com/higlass/scipy19
    TWITTER: @higlass_io
    PRESENTERS:
    Nezar Abdennur
    @nv1ctus
    nvictus.me
    Fritz Lekschas
    @flekschas
    lekschas.de
    This work is supported in part by the National Institutes of Health (U01 CA200059).
    Scipy Slack Channel
    #higlass
    

    View Slide

32. To a Billion and Beyond
    HiGlass
    WEB: higlass.io
    CODE: github.com/higlass/scipy19
    TWITTER: @higlass_io
    PRESENTERS:
    Nezar Abdennur
    @nv1ctus
    nvictus.me
    Fritz Lekschas
    @flekschas
    lekschas.de
    This work is supported in part by the National Institutes of Health (U01 CA200059).
    

    View Slide

33. Contributors & Acknowledgements
    HiGlass
    Core Contributors:
    Peter Kerpedjiev, Fritz Lekschas, Nezar Abdennur, Chuck McCallum
    PIs:
    Nils Gehlenborg, Peter Park, Leonid Mirny, Hanspeter Pfister
    Co-Authors:
    Kasper Dinkla, Hendrik Strobelt, Jacob Luber, Scott Ouellette, Alaleh
    Azhir, Nikhil Kumar, Jeewon Hwang, Soohyun Lee, Burak Alver
    This work is supported in part by the National Institutes of Health (U01 CA200059).
    

    View Slide