Just Do It: Reproducible Research in CFD

Keynote presentation at the International Parallel CFD Conference 2017, Glasgow, Scotland
http://www.strath.ac.uk/engineering/parcfd2017/

—Please cite as:
Barba, Lorena A. (2017): Just Do It: Reproducible Research in CFD. figshare.
https://doi.org/10.6084/m9.figshare.5011751.v1

—Abstract:
Reproducibility hit the mainstream in the last couple of years, after more than two decades of back-alley campaigns. For example, six months ago, the US National Science Foundation (NSF) issued a “Dear Colleague Letter: Encouraging Reproducibility in Computing and Communications Research.” The movement has often been associated with open data and open-source code, without which one could hardly reproduce a previous computational result. But one thing is sharing code and data for a statistical analysis or a bioinformatics workflow; and quite another to achieve reproducible research in parallel CFD. My research group has been practicing open science for years, and we found the hard way that open code is merely a first step. We need to exhaustively document our computational research, to encourage and accept publication of negative results, and to apply defensive tactics against bad code: version control, modular code, testing, and code review. In this talk, I will share our lessons learned from a replication campaign on our own previous study (arXiv:1605.04339, accepted), and make a call to action. The tools and methods require training, but running a lab for reproducibility is your decision. Just do it!

Transcript

1. International Parallel CFD Conference, ParCFD 2017 Just Do It: Reproducible Research in CFD @LorenaABarba
2. Reproducibility hit the mainstream
3. NSF 17-022 Dear Colleague Letter: Encouraging Reproducibility in Computing and Communications Research CISE, October 21, 2016 https://www.nsf.gov/pubs/2017/nsf17022/nsf17022.jsp
4. NSF SBE subcommittee on replicability in science: “reproducibility refers to the ability of a researcher to duplicate results of a prior study using the same materials as were used by the original investigator." “… new evidence is provided by new experimentation, defined in the NSF report as ‘replicability’ “ SBE, May 2015
5. https://www.nih.gov/research-training/rigor-reproducibility When a result can be reproduced by multiple scientists, it validates the original results and readiness to progress to the next phase of research.
6. https://doi.org/10.17226/21915
7. http://sc16.supercomputing.org/2016/03/16/sc16-explores-reproducibility- advanced-computing-student-competition-michela-taufer/
8. SC16 Panel: "Different Architectures, Different Times: Reproducibility and Repeatability in High Performance Computing"
10. https://cos.io/our-services/top-guidelines/
11. Technical Consortium on High Performance Computing New initiative on Reproducibility, led by Barba. https://www.computer.org/web/tchpc
12. Reproducible Research Track (peer reviewed) Lorena A. Barba George Washington University labarba@gwu.edu George K. Thiruvathukal Loyola University Chicago gkt@cs.luc.edu https://www.computer.org/cise/
13. Def.— Reproducible research Authors provide all the necessary data and the computer codes to run the analysis again, re- creating the results. Schwab, M., Karrenbach, N., Claerbout, J. (2000) “Making scientific computations reproducible,” Computing in Science and Engineering Vol. 2(6):61–67
14. Jon F. Claerbout Professor Emeritus of Geophysics Stanford University … pioneered the use of computers in processing and filtering seismic exploration data [Wikipedia] … from 1991, he required theses to conform to a standard of reproducibility.
15. Invited paper at the October 1992 meeting of the Society of Exploration Geophysics http://library.seg.org/doi/abs/10.1190/1.1822162
16. “In 1990, we set this sequence of goals: 1.Learn how to merge a publication with its underlying computational analysis. 2.Teach researchers how to prepare a document in a form where they themselves can reproduce their own research results a year or more later by “pressing a single button”. 3.Learn how to leave finished work in a condition where coworkers can reproduce the calculation including the final illustration by pressing a button in its caption. 4.Prepare a complete copy of our local software environment so that graduating students can take their work away with them to other sites, press a button, and reproduce their Stanford work. 5.Merge electronic documents written by multiple authors (SEP reports). 6.Export electronic documents to numerous other sites (sponsors) so they can readily reproduce a substantial portion of our Stanford research.
17. Think about your latest paper or report …
18. http://lorenabarba.com/gallery/reproducibility-pi-manifesto/ 2012
19. ‣ I teach my graduate students about reproducibility ‣ All our research code (and writing) is under version control ‣ We always carry out verification & validation (and make them public) ‣ For main results, we share data, plotting script & figure under CC-BY ‣ We upload preprint to arXiv at the time of submission to a journal ‣ We release code at the time of submission of a paper to a journal ‣ We add a “Reproducibility” declaration at the end of each paper ‣ I develop a consistent open-science policy & keep an up-to-date web presence Reproducibility PI Manifesto (2012)
20. “private reproducibility” …we can rebuild our own past research results from the precise version of the code that was used to create them.
21. What is Science? ‣ American Physical Society: - Ethics and Values, 1999 "The success and credibility of science are anchored in the willingness of scientists to […] Expose their ideas and results to independent testing and replication by others. This requires the open exchange of data, procedures and materials." https://www.aps.org/policy/statements/99_6.cfm
22. Data and Code Sharing Recommendations ‣ assign a unique identifier to every version of the data and code ‣ describe in each publication the computing environment used ‣ use open licenses and non-proprietary formats ‣ publish under open-access conditions (and/or post pre-prints)
23. Open-source licenses:
    People can coordinate their work freely, within the confines of copyright law, while making access and wide distribution a priority.
24. http://dx.doi.org/10.1073/pnas.1421412111
25. “The key is prevention via the training of more people on techniques for data analysis and reproducible research.” Leek & Peng, PNAS 2015
26. A syllabus for research computing 1. command line utilities in Unix/Linux 2. an open-source scientific software ecosystem (our favorite is Python's) 3. software version control (we like the distributed kind: our favorite is git / GitHub) 4. good practices for scientific software development: code hygiene and testing 5. knowledge of licensing options for sharing software https://barbagroup.github.io/essential_skills_RRC/
28. In parallel, even two runs with identical input data can differ! Different versions of your code, external libraries, even compilers may change results.
29. In HPC, peers may not be able to reproduce, but they will trust the results more if built over a consistent practice of reproducible research.
30. Def.— Replication Arriving at the same scientific findings as another study, collecting new data (possibly with different methods) and completing new analyses. Roger D. Peng (2011), “Reproducible Research in Computational Science” Science, Vol. 334, Issue 6060, pp. 1226-1227
31. Replication study PhD student: Olivier Mesnard
33. Experiments—snake profile lift & drag higher Re, producing a maximum lift coefficient of 1.9 while drag remained approximately the same. At higher angles of attack, the lift gradually decreased while the drag rapidly increased. The lift to drag ratio exhibited similar behavior, producing a maximum lift to drag ratio of 2.7 at 35o due to the peak in lift for the higher Reynolds numbers. The lift increased up to an angle of attack of 35o while exhibiting robust aerodynamic performance by maintaining high lift coefficient between 20 and 60o, and near maximum L/D values over a range of angles of attack between 15 to 40o. Credit: Holden, MSc Thesis, VA Tech (2011)
34. ‣ Immersed boundary method:
 reproduces lift signature, at same angle of attack, but different Reynolds # ‣ This is in 2D -1.5 -1 -0.5 0 0.5 1 1.5 2 2.5 3 0 10 20 30 40 50 CL Angle of Attack (degrees) Re=500 Re=1000 Re=1500 Re=2000 Re=2500 Re=3000 0o 35o Simulations—snake profile lift coefficient
35. Conclusions
    —Leading edge separation without stall —Stronger dorsal vortex —Lift enhancement at AoA=35º
37. Four CFD solvers ‣ cuIBM— Used in the original study, written in C CUDA to exploit GPUs, serial on CPU. Uses the NVIDIA Cusp library for solving sparse linear systems. https://github.com/barbagroup/cuIBM ‣ OpenFOAM— Free and open-source CFD package with a suite of numerical solvers. Core discretization scheme: finite-volume method applied on mesh cells of arbitrary shape. http://www.openfoam.org ‣ IBAMR— A parallel code using the immersed boundary method on Cartesian meshes, with adaptive mesh refinement. https:// github.com/ibamr/ibamr ‣ PetIBM— Our own re-implementation of cuIBM, but for distributed- memory parallel systems. It uses the PETSc library for solving sparse linear systems in parallel. https://github.com/barbagroup/PetIBM
38. Story 1: Meshing and boundary conditions can ruin everything
39. OpenFOAM ‣ Vorticity field at t = 52: angle-of-attack = 35º, Re = 2000. 
 Mesh ~700k triangles created with the free software GMSH.
40. Result ‣ Lift enhancement at 35ºAoA is reproduced
41. … not perfect ‣ time signature of the force coefficients at 30º (top) and 35º AoA (bottom)
42. Story 2: Other researchers' open- source codes have hurdles
43. IBAMR ‣ We first got the boundary conditions wrong … again
44. Results ‣ Using IBAMR in a manner of other IBM codes gave the wrong answer ‣ The “trick” for this code took us months to find (no-slip markers needed inside the body) ‣ Finally the results match
45. … not perfect ‣ time signature of the force coefficients at 30º (top) and 35º AoA (bottom)
46. Story 3: All linear algebra libraries are not created equal
47. cuIBM vs. PetIBM ‣ both written by the same developer, implement the same method ‣ cuIBM: CUDA C, Cusp linear algebra library — used algebraic multigrid preconditioner and conjugate gradient (CG) solver ‣ PetIBM: C++, PETSc linear algebra library — CG solver crashed because of an indefinite preconditioner, so we were forced to switch it for bi-CG stabilized
48. Results ‣ cannot reproduce the lift enhancement at 35º AoA
49. what is going on? ‣ time signature of force coefficients: 
 (top) PetIBM vs cuIBM
 (bottom) two runs with slightly shifted body
50. Wake transition ‣ Vortex merging event changes the wake patter and lift drops
51. Story 4: Different versions of your code, external libraries, even compilers may change results
53. Lessons learned
54. What makes research reproducible? “… authors provide all the necessary data and the computer code to run the analysis again, re-creating the results.” ‣But what data are necessary? - open data & open-source code - actual meshes used, BCs, comprehensive parameter sets - exhaustive records of the process, automated workflows: launch via running scripts, store command-line arguments for every run, capture complete environment settings - post-processing and visualization should be scripted, avoiding GUIs for manipulation of images
55. http://dx.doi.org/10.1371/journal.pcbi.1003285
56. Reproducible Research 10 Simple Rules 1. For every result, keep track of how it was produced 2. Avoid manual data-manipulation steps 3. Archive the exact versions of all external programs used 4. Version-control all custom scripts 5. Record all intermediate results, when possible in standard formats 6. For analyses that include randomness, note underlying random seeds 7. Always store raw data behind plots 8. Generate hierarchical analysis output, allowing layers of increasing detail to be inspected 9. Connect textual statements to underlying results 10.Provide public access to scripts, runs, and results
57. http://dx.doi.org/10.1073/pnas.1421412111
58. “The key is prevention via the training of more people on techniques for data analysis and reproducible research.” Leek & Peng, PNAS 2015
59. https://medium.com/@lorenaabarba
60. http://blogs.nature.com/naturejobs/2017/04/17/techblog-my-digital- toolbox-lorena-barba/
61. Automation: Turn protocols into code GUIs
62. ReproPacks For main results in a paper, we share data, plotting script & figure under CC-BY. File bundle with input data, running scripts, plotting scripts, and figure. We cite our own figure in the caption!
63. DOI Upload Share CC-BY Cite Zip figure files Write paper
64. Version control
65. http://joss.theoj.org
66. http://science.sciencemag.org/content/354/6308/142