Scientific software, open collaborations and the LSST

Practices vary between scientific domains but all too often the sharing of research software is done on an ad hoc basis between individuals and with little thought about the wider community. In this talk I'll discuss how software development in the sciences could benefit from the collaboration norms of a modern open source project and why LSST is uniquely placed to transform the way that researchers collaborate around research software and data.

Transcript

1. Scientific software, open collaborations and the LSST Arfon Smith @arfon Creative Commons Attribution 3.0 Unported License
2. !
3. What is a GitHub?
12. Astrophysics for non- astronomers
13. tl;dr - technical, but brimming with inefficiencies
14. http://www.flickr.com/photos/blachswan
15. http://www.flickr.com/photos/esoastronomy/
16. http://www.flickr.com/photos/esoastronomy/ http://www.flickr.com/photos/jamiegilbert
17. http://amandabauer.blogspot.com/
20. Diffraction grating Telescope Detector
26. 130 130 1 2048 189 189 258 258 480 562 378 378 493 521 390 397 851 851 247 274 319 319 304 580 493 511 610 636 188 188 228 228 > cat bad_pix_mask.txt
27. Wasteful 2 days work 3 observing runs/week 52 weeks in year 15 year detector lifetime ! 2*3*52*15 = 4680 days (13 years)
28. Wasteful… but the norm 2 days work 3 observing runs/week 52 weeks in year 15 year detector lifetime ! 2*3*52*15 = 4680 days (13 years)
29. How is the Open Source community doing it?
30. Open Source: Ubiquitous culture of reuse
37. Software composed of many components
38. Your software is the thing that is different
39. Reuse, reuse, reuse
40. Open source collaborations Open Source vs Open Collaborations
41. Open source collaborations Open Source: the right to modify, not the right to contribute.
42. Open source collaborations Open Collaborations: a highly collaborative development process and are receptive to contributions of code, documentation, discussion, etc from anyone who shows competent interest.
43. Open source collaborations Open Collaborations: a highly collaborative development process and are receptive to contributions of code, documentation, discussion, etc from anyone who shows competent interest. THIS
44. How do 4000 people work together?
45. The pull request
53. Code first, permission later
57. Open source collaborations Open Source vs Open Collaborations
58. Merged pull requests
63. “open source is… reproducible by necessity” Fernando Perez http://blog.fperez.org/2013/11/an-ambitious-experiment-in-data-science.html
64. Better at collaborating because they have to be
65. (doesn’t have to mean this) Open Public? =
66. Open (within your team, department or institution)
67. Electronic & Available
68. Asynchronous, exposed process
69. Lock-free
70. Low friction collaboration
71. What’s happening in academia today?
72. Collaboration around code
76. Every time this happens the community learns
79. Collaborative authoring
82. Collaborative teaching
86. What could be happening in academia?
87. Where do communities form?
88. Around a shared challenge?
89. Around shared data?
91. 10 ? n Level 1 (continual) Level 2 (periodic)
92. LSST is a project that is inherently open
93. Supernovae Weak lensing Active Galactic Nuclei Solar System Galaxies Transients/variable stars Large-scale structure Stars, Milky Way Strong lensing Informatics and Statistics Dark Energy (DESC)
94. LSST has the opportunity to do something differently
95. Copy Open Source
97. Software composed of many components
98. Your software should be the thing that is different
99. science too! Your software should be the thing that is different
100. So what’s the opportunity?
101. LSST could be an exemplar of open collaborations and open source
102. The biggest challenges are going to require the best tools
103. The best software engineers don’t work in academia Most of the
104. How do we fix this?
105. Credit for software
106. “Academic environments of today do not reward tool builders” Ed Lazowska, OSTP event http://lazowska.cs.washington.edu/MS/MS.OSTP.pdf
115. “publishing a paper about code is basically just advertising” David Donoho http://www.stanford.edu/~vcs/Video.html
117. How to derive meaningful metrics from open contributions?
119. Trust the work of others
125. Discoverability
128. The biggest challenges are going to require the best tools
129. A VISION AND STRATEGY FOR SOFTWARE FOR SCIENCE, ENGINEERING, AND EDUCATION
131. Establish a virtuous cycle# •  6%working%groups,%each%with%# •  3:6%faculty%from%each%ins;tu;on# 14# http://lazowska.cs.washington.edu/MS/MS.OSTP.pdf
132. Barriers are cultural, not technical
133. What can you do today?
134. Establish a virtuous cycle# •  6%working%groups,%each%with%# •  3:6%faculty%from%each%ins;tu;on# 14# http://lazowska.cs.washington.edu/MS/MS.OSTP.pdf
135. Supernovae Weak lensing Active Galactic Nuclei Solar System Galaxies Transients/variable stars Large-scale structure Stars, Milky Way Strong lensing Informatics and Statistics Dark Energy (DESC)
136. Shared tools around shared challenges
137. (Put a licence on it)
139. Treat documentation as a first class entity
140. Share more often (no matter how small)
141. 130 130 1 2048 189 189 258 258 480 562 378 378 493 521 390 397 851 851 247 274 319 319 304 580 493 511 610 636 188 188 228 228 > cat bad_pix_mask.txt > git clone git@github.com:arfon/aat/pixel_masks
143. Open Data is the least we can do
144. LSST has a chance to change how we do astrophysics
145. Will we?
146. Thanks. arfon@github.com @arfon "