When Writing It Down Is Not Enough

A condensed version of the earlier talk, delivered as part of the University of Victoria IDEAFest. For the longer, more explained version, please visit https://www.neilernst.net/matrix-blog.html

Transcript

1. When Writing It Down is Not Enough the Era of Computational Notebooks In [3]: In [1]: Neil Ernst Computer Science University of Victoria neilernst.net In [2]: 1
2. About Me !2 Undergrad in Geography/GIS PhD in formal models of software 4 years as software architecture consultant + researcher Currently assistant professor in CS at UVic Study software design and use in the “machine learning” era Extensively study and use notebooks in my research
3. !3 Notebooks Through the Ages
4. !3 Notebooks Through the Ages Leonardo da Vinci
5. !4
6. !5
7. !5 Isaac Newton
8. !6
9. !6 Charles Darwin
10. !7
11. !7 Marie Curie
12. !8
13. !8 Rosalind Franklin
14. !9
15. !9 Margaret Mead
16. !10 Constraints
17. Describe: A temporal record of progress !11 Scientific Notebooks:
18. Explore: a record of key parameters and ‘knob positions’ !12 Scientific Notebooks:
19. Explain: a shareable repository for replication !13 Scientific Notebooks:
20. Afford: portable, cheap and easy to use !14 Scientific Notebooks:
21. But … !15
22. Fragile !16
23. They burn !17 The Burning of the Library of Alexandria’, by Hermann Goll (1876)
24. Idiosyncratic and personal !18
25. Handwriting dependent !19
26. Static and immutable !20
27. !21
28. What is a computational notebook? !22
29. !23
30. !23
31. !24 http://www.apkc.net/external/msc_578d_dm_project.html
32. !24 http://www.apkc.net/external/msc_578d_dm_project.html
33. !25
34. http://intro.syzygy.ca/ !26
35. R Notebooks !27
36. !28 Computational Notebooks
37. Descriptive narrative + execution results !28 Computational Notebooks
38. Descriptive narrative + execution results Interactive rapid feedback on changes* !28 Computational Notebooks
39. Descriptive narrative + execution results Interactive rapid feedback on changes* Connected huge ecosystem of relevant libraries !28 Computational Notebooks
40. Balancing exploration and engineering !29
41. source !30 “most researchers are never taught the equivalent of basic lab skills for research computing” —Software Carpentry
42. source !30 “most researchers are never taught the equivalent of basic lab skills for research computing” —Software Carpentry
43. (Data) Science is about creating software! !31
44. Software Engineering !32 Data Science Hiring Analytics
45. Software Engineering !32 Data Science Examine years of hiring outcomes Hiring Analytics
46. Software Engineering !32 Data Science Examine years of hiring outcomes Hiring Analytics
47. Software Engineering !32 Data Science Examine years of hiring outcomes Apply algorithm to filter resumes Hiring Analytics
48. Software Engineering !32 Data Science Examine years of hiring outcomes Apply algorithm to filter resumes Hiring Analytics
49. It penalized resumes that included the word “women’s,” as in “women’s chess club captain.” And it downgraded graduates of two all-women’s colleges !33
50. Some Engineering Problems with Notebooks !34
51. !35 paper-2-v3-ne-final.docx Longevity and version control
52. !35 paper-2-v3-ne-final.docx Longevity and version control
53. !36 Testing and modularity
54. !37 Notebook carpentry
55. Software Engineering !38 Data Science
56. Software Engineering !38 Data Science Versioned Testable Modular Documented
57. Software Engineering !38 Data Science Versioned Testable Modular Documented Interactive Exploratory Descriptive Connected
58. Software Engineering !38 Data Science Versioned Testable Modular Documented Interactive Exploratory Descriptive Connected
59. → Computational notebooks remain personal and messy and individual → Tradeoff “exploration vs engineering” In [3]: Neil Ernst University of Victoria neilernst.net In [2]: 39