Slide 1

Slide 1 text

To a Billion and Beyond How to Visually Explore, Compare and Share Large Quantitative Datasets with HiGlass Peter Kerpedjiev, Nezar Abdennur, and Fritz Lekschas

Slide 2

Slide 2 text

Nezar Abdennur PostDoc at MIT @nv1ctus nvictus.me Fritz Lekschas PhD Candidate at Harvard @flekschas lekschas.de Peter Kerpedjiev Software Engineer at Zymergen @pkerpedjiev emptypipes.org

Slide 3

Slide 3 text

Nezar Abdennur PostDoc at MIT @nv1ctus nvictus.me Fritz Lekschas PhD Candidate at Harvard @flekschas lekschas.de Peter Kerpedjiev Software Engineer at Zymergen @pkerpedjiev emptypipes.org →

Slide 4

Slide 4 text

Nezar Abdennur PostDoc at MIT @nv1ctus nvictus.me Fritz Lekschas PhD Candidate at Harvard @flekschas lekschas.de Peter Kerpedjiev Software Engineer at Zymergen @pkerpedjiev emptypipes.org

Slide 5

Slide 5 text

Nezar Abdennur PostDoc at MIT @nv1ctus nvictus.me Fritz Lekschas PhD Candidate at Harvard @flekschas lekschas.de Peter Kerpedjiev Software Engineer at Zymergen @pkerpedjiev emptypipes.org

Slide 6

Slide 6 text

Genomics where we come from

Slide 7

Slide 7 text

Methods Image taken and slightly adapted from encodeproject.org

Slide 8

Slide 8 text

Principal Challenges Multiscale Patterns arise at various resolutions Multimodality Different data types and synchronized viz Multiple comparable datasets Insights arise from differences Collaborative Exploration Share exploratory state, not the end result Geospatial Data Time Series Data Image Data Genomic Data

Slide 9

Slide 9 text

Architecture

Slide 10

Slide 10 text

No content

Slide 11

Slide 11 text

No content

Slide 12

Slide 12 text

No content

Slide 13

Slide 13 text

No content

Slide 14

Slide 14 text

No content

Slide 15

Slide 15 text

No content

Slide 16

Slide 16 text

Python API All demos are available at github.com/higlass/scipy19

Slide 17

Slide 17 text

HiGlass in Jupyter (see temperature.ipynb)

Slide 18

Slide 18 text

Tileset API

Slide 19

Slide 19 text

Tileset API tileset_info() → min_pos & max_pos: relative to the scene tile_size: size of the tiles (in pixels) max_zoom: ⌈log 2 (tile mesh size / tile size)⌉ max_width: ⌈tile mesh size / tile size⌉ tiles(tile_ids) ← [, …] → {: { dense, min_value, max_value, dtype }, ...} Generates or fetches 1D or 2D data tiles

Slide 20

Slide 20 text

Tileset API tileset_info() → min_pos & max_pos: relative to the scene tile_size: size of the tiles (in pixels) max_zoom: ⌈log 2 (tile mesh size / tile size)⌉ max_width: ⌈tile mesh size / tile size⌉ tiles(tile_ids) ← [, …] → {: { dense, min_value, max_value, dtype }, ...} Base64-encoded raw data Generates or fetches 1D or 2D data tiles

Slide 21

Slide 21 text

Tileset API Demo (see point-data.ipynb)

Slide 22

Slide 22 text

View Configs

Slide 23

Slide 23 text

View Config ● Views Shared location and zoom ○ Layout ○ Tracks ● Locks For view synchronization ● Globals Server URLs, editability 1. { 2. "views": [ 3. { 4. "uid": "aa", 5. "initialXDomain": [0, 100], 6. "initialYDomain": [0, 100], 7. "layout": { 8. "x": 0, "y": 0, 9. "w": 12, "h": 6, 10. }, 11. "tracks": { 12. "top": [], 13. "left": [], 14. "center": [], 15. "right": [], 16. "bottom": [] 17. } 18. } 19. ], 20. "zoomLocks": { ... }, 21. "locationLocks": { ... }, 22. "valueScaleLocks": { ... }, 23. "editable": true, 24. "zoomFixed": false, 25. "trackSourceServers": ["/api/v1"], 26. "exportViewUrl": "/api/v1/viewconfs" 27. }

Slide 24

Slide 24 text

View Config ● Views Shared location and zoom ○ Layout ○ Tracks ● Locks For view synchronization ● Globals Server URLs, editability 1. { 2. "views": [ 3. { 4. "uid": "aa", 5. "initialXDomain": [0, 100], 6. "initialYDomain": [0, 100], 7. "layout": { 8. "x": 0, "y": 0, 9. "w": 12, "h": 6, 10. }, 11. "tracks": { 12. "top": [], 13. "left": [], 14. "center": [], 15. "right": [], 16. "bottom": [] 17. } 18. } 19. ], 20. "zoomLocks": { ... }, 21. "locationLocks": { ... }, 22. "valueScaleLocks": { ... }, 23. "editable": true, 24. "zoomFixed": false, 25. "trackSourceServers": ["/api/v1"], 26. "exportViewUrl": "/api/v1/viewconfs" 27. }

Slide 25

Slide 25 text

View Config ● Views Shared location and zoom ○ Layout ○ Tracks ● Locks For view synchronization ● Globals Server URLs, editability 1. { 2. "views": [ 3. { 4. "uid": "aa", 5. "initialXDomain": [0, 100], 6. "initialYDomain": [0, 100], 7. "layout": { 8. "x": 0, "y": 0, 9. "w": 12, "h": 12, 10. }, 11. "tracks": { 12. "top": [], 13. "left": [], 14. "center": [], 15. "right": [], 16. "bottom": [] 17. } 18. } 19. ], 20. "zoomLocks": { ... }, 21. "locationLocks": { ... }, 22. "valueScaleLocks": { ... }, 23. "editable": true, 24. "zoomFixed": false, 25. "trackSourceServers": ["/api/v1"], 26. "exportViewUrl": "/api/v1/viewconfs" 27. } VIEW

Slide 26

Slide 26 text

1. { 2. "type": "horizontal-line", 3. "server": "//higlass.io/api/v1", 4. "tilesetUid": "OHJakQICQD6gTD7skx4EWA", 5. "uid": "my-very-fancy-line-plot", 6. "options": { 7. "name": "My Very Fancy Line Plot!", 8. ... 9. }, 10. } Track Config ● Type general encoding ● Server track data source ● Tileset UID data identifier ● UID track identifier ● Options For styling, labeling, etc.

Slide 27

Slide 27 text

Advanced Features

Slide 28

Slide 28 text

Advanced Features (see nyc-taxi.ipynb)

Slide 29

Slide 29 text

Developer Features Modular Codebase Tiling, serving, and rendering are decoupled E.g., viewer, server, tileset API, docker Viewer Extensibility Simple plugin architecture for new track types E.g., Epilogos, GeoJSON JavaScript APIs for Integration Library version, React component, JsAPI E.g., HiPiler, Peax, cTracks EPILOGOS GEOJSON HIPILER PEAX

Slide 30

Slide 30 text

Communication & Collaboration (see genomics.ipynb)

Slide 31

Slide 31 text

To a Billion and Beyond HiGlass WEB: higlass.io CODE: github.com/higlass/scipy19 TWITTER: @higlass_io PRESENTERS: Nezar Abdennur @nv1ctus nvictus.me Fritz Lekschas @flekschas lekschas.de This work is supported in part by the National Institutes of Health (U01 CA200059). Scipy Slack Channel #higlass

Slide 32

Slide 32 text

To a Billion and Beyond HiGlass WEB: higlass.io CODE: github.com/higlass/scipy19 TWITTER: @higlass_io PRESENTERS: Nezar Abdennur @nv1ctus nvictus.me Fritz Lekschas @flekschas lekschas.de This work is supported in part by the National Institutes of Health (U01 CA200059).

Slide 33

Slide 33 text

Contributors & Acknowledgements HiGlass Core Contributors: Peter Kerpedjiev, Fritz Lekschas, Nezar Abdennur, Chuck McCallum PIs: Nils Gehlenborg, Peter Park, Leonid Mirny, Hanspeter Pfister Co-Authors: Kasper Dinkla, Hendrik Strobelt, Jacob Luber, Scott Ouellette, Alaleh Azhir, Nikhil Kumar, Jeewon Hwang, Soohyun Lee, Burak Alver This work is supported in part by the National Institutes of Health (U01 CA200059).