Design for Reproducibility

JupyterCon 2017 Keynote (15 min)

Cite as:
Barba, Lorena A. (2017): Design for Reproducibility. figshare.
https://doi.org/10.6084/m9.figshare.5346277.v1

Abstract:

Stanford Professor Jon Claerbout, reproducible-research grand master, said that: "interactive programs are slavery unless they include the ability to arrive in any previous state by means of a script.” Jupyter was born out of IPython (where the I stands for “interactive”) to offer a solution for creating “reproducible computational narratives.” The tool is both interactive and supports reproducible research, even if there is tension between the two attributes. In this talk, we explore the question of how we may build into the design of our tools (like Jupyter) an enabling capacity to support reproducible research. With better insights on Design for Reproducibility, we might extend the design to our research workflows, with the machine as our active collaborator.

Transcript

1. Design for
   Reproducibility
   @LorenaABarba
   jupytercon
   THE OFFICIAL JUPYTER CONFERENCE
   Aug. 23-25, 2017 – New York, NY
   

   View Slide

2. Jon F. Claerbout
   Professor Emeritus of Geophysics
   
   Stanford University
   
   … pioneered the use of computers in processing
   and filtering seismic exploration data [Wikipedia]
   … from 1991, he required theses to conform to a
   standard of reproducibility.
   

   View Slide

3. Def.— Reproducible research
   Authors provide all the necessary data and
   the computer codes to run the analysis
   again, re-creating the results.
   Adapted from:
   Schwab, M., Karrenbach, N., Claerbout, J. (2000) “Making
   scientific computations reproducible,” Computing in Science
   and Engineering Vol. 2(6):61–67
   

   View Slide

4. View Slide

5. Def.— Replication
   Arriving at the same scientific findings as
   another study, collecting new data
   (possibly with different methods) and
   completing new analyses.
   Roger D. Peng (2011), “Reproducible Research in
   Computational Science” Science, Vol. 334, Issue 6060, pp.
   1226-1227
   

   View Slide

6. “I’ve learned that interactive programs are
   slavery (unless they include the ability to arrive
   in any previous state by means of a script).”
   — Jon Claerbout
   GUIs
   

   View Slide

7. A set of open-source tools
   for interactive and
   exploratory computing.
   

   View Slide

8. Jupyter grant proposal:
   “…the core problem we are trying to solve is
   the collaborative creation of reproducible
   computational narratives.”
   

   View Slide

9. Interactive →← Reproducible
   

   View Slide

10. Interactive →← Reproducible
    

    View Slide

11. View Slide

12. On spreadsheets:
    
    “…the user interface conflates input, output,
    code, and presentation, making testing code and
    discovering bugs difficult.”
    — Philip Stark, Science is ‘show me,’ not ‘trust me’ (2015)
    

    View Slide

13. Why do we care about
    Computational Reproducibility?
    

    View Slide

14. “Science is a conversation”
    —Stephen Downes (“connectivism”)
    ‣ a conversation between scientists and their
    body of knowledge
    ‣ a conversation among scientists
    ‣ a conversation between scientists and
    machines…
    

    View Slide

15. What is a conversation?
    A B
    goal
    engage
    

    View Slide

16. How do we design
    (conversations) for
    reproducibility?
    

    View Slide

17. “Designing the User Interface”
    —Ben Shneiderman, 6th ed.
    Tools that succeed are:
    ‣ comprehensible,
    ‣ predictable, and
    ‣ controllable
    

    View Slide

18. “Designing the User Interface”
    —Ben Shneiderman, 6th ed.
    Those who have authority and responsibility
    must have adequate levels of control.
    Responsibility should guide design.
    

    View Slide

19. Human control ↓↑ Automation
    “Ensuring human control while increasing automation.”
    

    View Slide

20. On 21st-century design:
    
    “…design has expanded from giving form to
    creating systems that support human
    interactions.”
    — Hugh Dubberly & Paul Pangaro, 

    Cybernetics and Design: Conversations for action (2015)
    

    View Slide

21. Conversation builds trust
    A B
    research reproduce
    

    View Slide

22. “I have a button here. I push the button.
    That’s not a conversation.”
    — Paul Pangaro, 

    Rethinking Design Thinking, PICNIC Festival
    Amsterdam (2010)
    Flaticon Madebyoliver CC-BY
    

    View Slide

23. Reproducibility:
    not a one-click solution
    

    View Slide

24. Design for
    Reproducibility
    @LorenaABarba
    jupytercon
    THE OFFICIAL JUPYTER CONFERENCE
    Aug. 23-25, 2017 – New York, NY
    

    View Slide