How we can solve DM's Documentation 'Problem'

Proposal for LSST DM stack documentation strategy. Presented at LSST2015 in Bremerton WA.

Transcript

1. How we can solve DM’s Documentation ‘Problem’ Jonathan Sick @jonathansick LSST DM (SQuaRE)
2. Principles of Stack Docs 1. Content is versioned with git 2. Static files → no content management system 3. Content lives with code 4. Continuously integrate docstrings/tutorials 5. Maintainable, mechanized content 6. Every stack developer is a doc writer 7. Aligned with the Python ecosystem 8. Be good to the reader 9. Don’t document broken code; fix it 2
3. Let’s use Sphinx • Prose is reStructuredText - reST markup is extensible • Build to static HTML, LaTeX, etc. • Jinja2 templates, bring your own CSS/JavaScript • API Reference generation from docstrings - Python: autodoc/numpydoc - C++: Breathe bridges Doxygen XML • Sphinx is completely extensible (with Python) - Add reST roles (inline) and directives (blocks) 3
4. Versioning Docs 4
5. Versioning Docs 5
6. Versioning Docs 6 dm.lsst.org/docs/v10.1/ Tagged releases get permanent URLs dm.lsst.org/docs/[git SHA1 or branch name]/ Jenkins CI builds make temporary docs
7. Content Strategy 7 Getting Started Overview Install Working with packages Tutorials Processing a single image Processing with the Butler … Stack Packages afw meas_base … Developer Guide Development workflow Code Standards … overview.rst install.rst eups.rst github.com/lsst/afw github.com/lsst/meas_base dev/workflow.rst dev/standards.rst github.com/lsst/single_img_tutorial github.com/lsst/butler_tutorial reST files in docs repo docs live in pkg. repo we’ll provide doc templates for devs
8. Write for the reader. 8
9. Write for the reader. 9
10. Write for the reader. 10 I want to install v10.1 ▾ on Mac OS X Yosemite ▾ from source ▾ . Install the Stack Get Dependencies Install Test I use zsh ▾ .
11. Tutorials/Walk Throughs • Tutorials must teach realistic workflows and best practices; quick hacks not wanted. • Lightweight analysis tutorials - might be backed by Jupyter notebooks - rendered into documentation site • Pipeline development tutorials: - text + git repo - each tutorial stage is tagged 11
12. Package Docs • Package docs live in the package’s repo 1. User guide / lightweight examples 2. API Reference (docstrings) • We will provide a clear template and guidelines 12 package_name Python API Reference C++ API Reference User guide + examples API with examples in docstrings
13. C++ API Docs 13 doxygen I’m sorry. breathe bridges doxygen XML to Sphinx
14. Docstring Formats: Sphinx 14 def foobar(a, b, keyword=None): """Fetches rows from a Bigtable. More paragraphs of documentation. :param a: something. :type a: int :param b: Some other thing with some type. :type b: str :param keyword: An optional variable, that has a much longer explanation than the other args. :type keyword: bool :returns: A dict that means something to you: {'Serak': ('Rigel VII', 'Preparer'), 'Zim': ('Irk', 'Invader'), 'Lrrr': ('Omicron Persei 8', 'Emperor')} More explanation. :rtype: dict :raises IOError: An error occurred accessing something. """
15. Docstring Formats: Google 15 def foobar(a, b, keyword=None): """Fetches rows from a Bigtable. More paragraphs of documentation. Args: a: An int for something. b: Some other thing with some type. keyword: An optional variable, that has a much longer explanation than the other args. Returns: A dict that means something to you: {'Serak': ('Rigel VII', 'Preparer'), 'Zim': ('Irk', 'Invader'), 'Lrrr': ('Omicron Persei 8', 'Emperor')} More explanation. Raises: IOError: An error occurred accessing something. """ with napoleon http://bit.ly/1MBtOwW
16. Docstring Formats: Numpy 16 def foobar(a, b, keyword=None): """Fetches rows from a Bigtable. More paragraphs of documentation. Parameters —————————— a : int An int for something. b : str Some other thing. keyword : str, optional An optional variable, that has a much longer explanation than the other args. Returns ——————— my_dict : dict Something that means something to you:: {'Serak': ('Rigel VII', 'Preparer'), 'Zim': ('Irk', 'Invader'), 'Lrrr': ('Omicron Persei 8', 'Emperor')} Raises —————— IOError: An error occurred accessing something. """ napoleon/numpydoc http://bit.ly/1LjBt2O
17. DRY Docstrings 17 Can C++ docstrings percolate into the Python layer? C++ Only C++ & Python Python Only Need to balance developer efficiency, C++ and Python doc parity, and user experience. Interactive python docstrings matter.
18. Beyond the minimum viable doc • Responsive doc site - Sass / Susy / gulp / responsive [images] - CI of website - Optimize navigation for long pages • Templates for docs and tutorials • Style guide for docs / writing • Continuous integration of examples, tutorials • Integration with community.lsst.org • Academic citation guidance throughout • Even better search of code+docs (Elasticsearch?) 18
19. Culture of Doc Salesforce (Heroku) has an API Design Review Board Solve API issues before shipping Doc & QA Teams are board members 1. Developers present API and fill out the documentation template 2. Achieve consensus over 1. API naming consistency 2. Design idioms/consistency 3. Error codes 4. Usability / documentability 19 Docs as QA