– 8 ACM Conferences: • ASPLOS’12, CCS’12, OOPSLA’12, OSDI’12, PLDI’12, SIGMOD’12, SOSP’11, VLDB’12 – 5 Journals • TACO’9, TISSEC’15, TOCS’30, TODS’37, TOPLAS’34 4 EXPERIMENT “Our approach can be applied on ...” “Our implementation can be found at ...” “... we implemented out approach” “code and data can be downloaded from our website”
– 5: The results can be easily reproduced by an independent researcher with at most 15 min of user effort, requiring only standard, freely available tools (C compiler, etc.). – 4: The results can be easily reproduced by an independent researcher with at most 15 min of user effort, requiring some proprietary source packages (MATLAB, etc.). – 3: The results can be reproduced by an independent researcher, requiring considerable effort. – 2: The results could be reproduced by an independent researcher, requiring extreme effort. – 1: The results cannot seem to be reproduced by an independent researcher. – 0: The results cannot be reproduced by an independent researcher. 6 PREVIOUS EXERCISES
machine learning conf. – Why we don’t share the code? 7 PREVIOUS EXERCISES “The time it takes to clean up and document for release” “Dealing with questions from users about the code” “The possibility that your code may be used without citation” “The possibility of patents, or other IP constraints” “Competitors may get an advantage”
that: – Investigators are expected to share with other researchers... – Investigators and grantee are encouraged to share software and inventions... – ... Responsibility that investigators and organizations have as members of the scientific and engineering community, to make results, data and collections available to other researchers. • Industry – Papers with only authors from industry have a low rate or reproducibility 13 RESULTS
be Available Soon • Programmer Left • Bad Backup Practices • Commercial Code • Proprietary Academic Code • Unavailable Subsystems • Multiple Reasons • Intellectual Property • Research vs. Sharing • Security and Privacy • Poor Design • Too Busy to Help So, What Were Their Excuses?
algorithm. I’m not very sure whether it is the final version of the code used in our paper, but it should be at least 99% close. Thank you for your interest in our work. Unfortunately the current system is not mature enough at the moment, so it’s not yet publicly available... I am afraid that the source code was never released. The code was never intended to be released so is not in any shape for general use. (STUDENT) was a graduate student in our program but he left a while back so I am responding instead... Thanks ... Unfortunately, the server in which my implementation was stored had a disk crash in April and three disks crashed simultaneously... The code is owned by (COMPANY), ...is not open-source...You best bet is to reimplement :( Sorry ...sources are not meant to be opensource..I do not have the liberty of making available The source code at my current institution (UNIVERSITY)...
submitted • Build special tools that can run reliably and with reproducible results • Build web sites that allow authors to make their code available to colleagues • Do not follow the bad habits like “publish and forget” style of scientific research 19 RECOMMENDATIONS
release the code. 2. Students will leave, plan for it. 3. Create permanent email addresses. 4. Create project websites. 5. Use a source code control system. 6. Backup your code. 7. Resolve licensing issues. 8. Keep your promises. 9. Plan for longevity. 10. Avoid cool but unusual design. 11. Plan for Reproducible Releases. 21 LESSONS LEARNED
reproducing research that is always, and completely, reproducible. • To share may increase the probabilities of citation. • The sharing specifications will have a positive effect on researchers’ willingness to share. • Sharing specifications can be used as a contract between authors and readers. 36 CONCLUSION
from Testing by Laura Dillon (ACM Distinguished Speakers Program) – http://dsp.acm.org/view_lecture.cfm?lecture_id=108 • Why You Should Share Your Musical Knowledge – http://jazzadvice.com/why-you-should-share-your- musical-knowledge/ • Reproducible Research in Signal Processing – http://rr.epfl.ch/17/1/VandewalleKV09.pdf 38 FURTHER LITERATURE
data that underlie their research publications – http://www.runmycode.org/ • Executable Papers – http://executablepapers.com/ • CDE: Automatically create portable Linux applications (i.e., package, deliver, run). – http://www.pgbovine.net/cde.html 39 FURTHER LITERATURE