Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Scaling Reproducible Research with Jupyter

Carol Willing
October 15, 2019

Scaling Reproducible Research with Jupyter

Keynote at ECMWF Workshop on Building Reproducible Workflows for Earth Sciences. Reading, UK. #REPWORK19 Looking through the lens of weather forecasting, the keynote discusses tools, processes, and communication that help scale reproducible research. Examples from Project Jupyter include Jupyter Notebook, JupyterLab, JupyterHub, BinderHub, and Binder service. Additional examples from the nteract project, Python, Dagster, and Pangeo.

Carol Willing

October 15, 2019
Tweet

More Decks by Carol Willing

Other Decks in Science

Transcript

  1. @WillingCarol
    Scaling
    Reproducible Research
    Workshop: Building reproducible
    workflows for earth sciences
    ECMWF
    October 15, 2019
    1
    Carol Willing
    @WillingCarol

    View full-size slide

  2. @WillingCarol 2
    San Diego, CA

    View full-size slide

  3. @WillingCarol 3

    View full-size slide

  4. @WillingCarol 4

    View full-size slide

  5. @WillingCarol 5

    View full-size slide

  6. @WillingCarol 6
    Tokyo

    View full-size slide

  7. @WillingCarol
    –Pearl S. Buck
    The typhoon came out of the sea
    first as a deep hollow roar.
    7

    View full-size slide

  8. @WillingCarol 8

    View full-size slide

  9. @WillingCarol 9

    View full-size slide

  10. @WillingCarol 10

    View full-size slide

  11. @WillingCarol
    –Pearl S. Buck
    I was surrounded by the madness,
    the unreason, of uncontrolled,
    undisciplined energy.
    11

    View full-size slide

  12. Copyright: 2019 European
    Union, contains modified
    Copernicus Sentinel data
    2019, processed by
    EUMETSAT
    Super Typhoon
    Hagibis
    View of Super Typhoon
    Hagibis south-west of
    Japan, as captured by
    the Copernicus
    Sentinel-3 satellite on
    08 October at 00:16
    UTC.

    View full-size slide

  13. Title Typhoon Hagibis
    Released 10/10/2019 4:45 pm
    Copyright contains modified Copernicus
    Sentinel data (2019), processed by ESA,
    CC BY-SA 3.0 IGO

    View full-size slide

  14. Source:Twitter

    View full-size slide

  15. A sign is partially submerged as the Tama River floods during Typhoon Hagibis. Source:Getty Images

    View full-size slide

  16. @WillingCarol 16

    View full-size slide

  17. @WillingCarol 17

    View full-size slide

  18. @WillingCarol 18

    View full-size slide

  19. @WillingCarol
    Lives depend on
    19

    View full-size slide

  20. @WillingCarol
    scaling reproducible
    research
    20

    View full-size slide

  21. @WillingCarol
    Tools
    Processes
    Communication
    21

    View full-size slide

  22. @WillingCarol 22

    View full-size slide

  23. Jupyter
    Notebook
    A Jupyter Notebook document with a visualization of measles data.

    View full-size slide

  24. @WillingCarol
    Research
    24
    Jupyter Citations
    Number
    0
    1000
    2000
    3000
    4000
    2015 2016 2017 2018 2019 Projected

    View full-size slide


  25. Millions of
    Notebooks
    https://github.com/trending/jupyter-notebook
    Over 5 million

    on GitHub

    View full-size slide

  26. @WillingCarol 26
    ‣ Growth
    ‣ ACM Award
    ‣ Industry adoption
    ‣ Creative uses
    ‣ Open Source Book

    View full-size slide

  27. @WillingCarol
    JupyterLab
    27

    View full-size slide

  28. 28
    jupyter.org demo

    View full-size slide

  29. 29
    jupyter.org demo

    View full-size slide

  30. @WillingCarol 30
    https://github.com/data-exp-lab/rust-yt-tools/
    npm package @data-exp-lab/yt-tools
    Irber Junior LC. Oxidizing Python: writing
    extensions in Rust [version 1; not peer
    reviewed]. F1000Research 2018, 7(ISCB
    Comm J):955 (poster) (https://doi.org/
    10.7490/f1000research.1115726.1)
    https://github.com/munkm/widgyts
    yt and jupyter
    widgets

    View full-size slide

  31. @WillingCarol 31
    https://towardsdatascience.com/multivolume-
    rendering-in-jupyter-with-ipyvolume-cross-
    language-3d-visualization-64389047634a
    ipyvolume

    View full-size slide

  32. @WillingCarol
    Healthy Best Practices
    32

    View full-size slide

  33. @WillingCarol 33
    Ten Simple Rules for
    Reproducible Research in
    Jupyter Notebooks
    Adam Rule et al.
    https://github.com/jupyter-guide/ten-rules-jupyter
    https://github.com/jupyter-guide/jupyter-guide

    View full-size slide

  34. @WillingCarol
    Keep up with changes
    34

    View full-size slide

  35. @WillingCarol
    Proceed cautiously
    with pseudo-open
    projects
    35

    View full-size slide

  36. @WillingCarol
    Ask why
    36

    View full-size slide

  37. @WillingCarol
    Tools
    Processes
    Communication
    37

    View full-size slide

  38. zero-to-jupyterhub.readthedocs.io

    View full-size slide

  39. @WillingCarol 39
    Papermill
    Parameterize and Run

    View full-size slide

  40. @WillingCarol 40
    Data at scale - Netflix
    https://medium.com/netflix-techblog/notebook-innovation-591ee3221233
    nteract
    Papermill
    Scrapbook
    Bookstore
    Commuter

    View full-size slide

  41. @WillingCarol 41
    https://medium.com/dagster-io/dagster-0-6-0-impossible-princess-898b459375e0
    Pipelines

    View full-size slide

  42. @WillingCarol
    Create a
    Reproducibility
    Pipeline
    42

    View full-size slide

  43. @WillingCarol
    Decouple steps for
    flexibility
    43

    View full-size slide

  44. @WillingCarol
    Plan
    Execute
    Change
    44
    https://jupyterhub-team-compass.readthedocs.io
    https://github.com/jupyterhub/team-compass

    View full-size slide

  45. @WillingCarol
    Tools
    Processes
    Communication
    45

    View full-size slide

  46. @WillingCarol
    Notebooks to web
    46
    https://blog.jupyter.org/and-
    voil%C3%A0-f6a2c08a4a93

    View full-size slide

  47. @WillingCarol 47
    Binder
    mybinder.org
    Binder 2.0 blog post
    elifesciences: Share
    your interactive
    research environment
    Nature article about
    Binder

    View full-size slide

  48. 48
    Juliette Taka

    View full-size slide

  49. 49
    Juliette Taka

    View full-size slide

  50. 50
    Juliette Taka

    View full-size slide

  51. 51
    Juliette Taka

    View full-size slide

  52. 52
    Juliette Taka

    View full-size slide

  53. 53
    Juliette Taka

    View full-size slide

  54. @WillingCarol
    Binder
    54

    View full-size slide

  55. @WillingCarol 55
    Binder
    mybinder.org

    View full-size slide

  56. @WillingCarol 56
    From a
    phone in
    the park!

    View full-size slide

  57. @WillingCarol
    Pangeo
    57
    https://pangeo.io

    View full-size slide

  58. @WillingCarol 58

    View full-size slide

  59. @WillingCarol 59
    https://simexp.github.io/vcog_hps_ad_book/intro.html
    Jupyter Book
    Binder
    Jupyter
    pandas
    scipy
    scikit learn
    matplotlib
    numpy
    seaborn
    Canadian Open Neuroscience Platform

    View full-size slide

  60. @WillingCarol
    Build Communities
    60

    View full-size slide

  61. @WillingCarol
    Leverage solutions
    across disciplines
    62

    View full-size slide

  62. @WillingCarol
    Share binders.
    Foster scientific
    research.
    63

    View full-size slide

  63. @WillingCarol
    Tools
    Processes
    Communication
    64

    View full-size slide

  64. @WillingCarol
    Why strive for
    reproducible research?
    65

    View full-size slide

  65. @WillingCarol
    Reproducible research
    improves prediction
    66

    View full-size slide

  66. @WillingCarol
    prediction =
    impact
    67

    View full-size slide

  67. @WillingCarol 68
    Scaling reproducible
    research improves science
    and our world

    View full-size slide

  68. @WillingCarol 70
    Thank you
    ECMWF Workshop Organizers
    Claudia Vitolo
    Project Jupyter Team
    Min Ragan-Kelly

    View full-size slide

  69. @WillingCarol
    Attributions
    71
    References to published research, projects, and drawings (and marked on slides)
    [2] Statistics: https://fivethirtyeight.com/features/which-city-has-the-most-unpredictable-weather/
    [7, 11] A Bridge for Passing, Pearl S. Buck
    [8, 9, 18] ECMWF
    [12] Copyright: 2019 European Union, contains modified Copernicus Sentinel data 2019, processed by EUMETSAT
    [13] Copyright contains modified Copernicus Sentinel data (2019), processed by ESA, CC BY-SA 3.0 IGO
    [30] Madicken Munk
    [31] Maarten Breddels
    [33] Adam Rule et al.
    [46] Quantstack - Voila
    [48-53] Juliette Taka
    [57] Pangeo
    [58] Lindsey Heagy
    [59] Canadian Open Neuroscience Platform
    Photos
    [2-6, 10, 16-17, 69, 70] Source: Carol Willing and Linnea Willing
    [14] Twitter
    [15] Getty Images
    [55, 56] Kirstie Whitaker
    [23-29, 38, 44, 47, 54, 61] Project Jupyter
    [39-40] nteract and Netflix
    [41] Nick Shrock, Dagster

    View full-size slide

  70. @WillingCarol 72

    View full-size slide