State of the Docs (2016-02-24)

Presented at the LSST Joint Technical Meeting

Transcript

1. State of the Docs Jonathan Sick Software Dev LSST DM, SQuaRE @jonathansick What we’ve done recently and where we’re going.
2. State of the Docs Theme for August 2015 – February 2016 Building an environment for documentation to be written.
3. State of the Docs open source • developer friendly reStructuredText Sphinx readthedocs.org
4. community.lsst.org Main Categories • Announcements • Support • Data Management • DM Notifications • Cross-System • Simulations • Camera • Project / DM Team (*) Open, intra-project and project–community forum. Accounts 160 Topics 345 Posts 1999 (as of 2016-02-19)
5. community.lsst.org • True adoption by the project ‣ Integration with Science Collaborations ‣ ‘Ask LSST Anything’ category is being developed (PST) ‣ Synergy with Project Communications Team Does ‘Announcements’ conflict with News Blog/ Mailing Lists? • Integration with LSST identity management ‣ Project personnel need to be added to LSST and other groups to access privileges and team categories. ‣ We lack integration with Contacts DB to automate this. Future directions.
6. Community Mailbot • Watches for new posts via Discourse API • Broadcasts posts to mailing lists via Mailchimp Notification service for community.lsst.org Data Management DM Notifications dm-devel@lists.lsst.org Support Announcements dm-announce@lists.lsst.org https://github.com/lsst-sqre/community_mailbot
7. Community Mailbot Started version 2 (currently paused) ‣ New configuration system; Redis persistence layer ‣ Fix caching bugs, especially when a Post title changes ‣ Multiple notification channels (i.e., support for Slack & Mailchimp) Support for Slack Ability to notify for every new message; not just new posts Future directions https://github.com/lsst-sqre/community_mailbot
8. rST Design Docs • Write in reStructuredText & Sphinx • Publish with GitHub & readthedocs.org • Approved versions will be archived to Docushare • Converted Word-based Design Docs: ‣ ldm-129.lsst.io (DM Infrastructure Design) ‣ ldm-135.lsst.io (Database Design) ‣ ldm-152.lsst.io (DM Middleware Design) ‣ ldm-230.lsst.io (Automated Operations of DM) Putting our design docs on GitHub & the web.
9. Technical Note Series Native to the web & citeable in literature. More significant than a Confluence page; More accessible than writing a controlled document or academic paper. Since November ’15: • 8+ DM Technotes • 8 SQRE Technotes • 1+ SMTN Technotes Listing: http://ls.st/nre github.com/lsst-sqre/lsst-technote-bootstrap sqr-000.lsst.io
10. Design Docs & Technotes • Discoverable, dynamic listing (and search) of technotes, design docs and other DM documentation projects • Push-button document creation and admin ‣ See ‘LSST the Docs’ • Publishing to NASA/SAO ADS ‣ metadata.yaml → Zenodo → ADS ‣ Python package for uploading to Zenodo (zenodio.lsst.io) • Sphinx HTML/CSS design for LSST ‣ Print styles suitable for Docushare • Sphinx extensions for new content, e.g. ‣ Jupyter notebooks as technotes ‣ BibTeX-like references that are ADS-aware. Future directions.
11. developer.lsst.io • New content ‣ DM Development Workflow with Git, GitHub, JIRA and Jenkins ‣ Writing reStructuredText docs ‣ Using Git Large File Storage (LFS) for Data Repositories ‣ Git Setup and Best Practices, etc. • 1-day collaborative sprint to convert remaining Confluence content • Everyone should edit/contribute to the Developer Guide ‣ Changes merged to master are automatically deployed ‣ Earn value for significant changes with ticket branches! The answer to ‘how do I do this?’ https://github.com/lsst-dm/dm_dev_guide
12. Software Documentation • Former Confluence and Doxygen sites unified as a Sphinx doc project ‣ Overall documentation repo for each product (lsst_apps, qserv, Firefly, etc.) ‣ Package user guides live in package repos (doc/) ‣ Doc sources integrated at build-time • User guides now written in reStructuredText (*) • Changes to API reference production ‣ numpy docstring format for Python (*) ‣ doxygen format still used for C++ Sphinx-based User Guide / API Reference * See Developer Guide
13. LSST the Docs Doc deployment for EUPS-based software GitHub Repositories AWS S3 Bucket ltd-keeper Jenkins Build/CI … … Fastly buildlsstsw.sh ltd-mason Product Doc branch(es) product(s) ltd-keeper-db ltd-keeper-app Package 2 Package n YAML HTTP HTTP HTTP browser developer product-1/ product-2/ product-n/ Package 1 reader HTTP More info: sqr-006.lsst.io & ltd-keeper.lsst.io
14. LSST the Docs One bucket, many versions. lsst_apps/ qserv/ b1/ b2/ b3/ b1/ b2/ https://v1.pipelines.lsst.io https://pipelines.lsst.io https://tickets-dm-9999.pipelines.io b4/ S3 Bucket URLs mapped with Fastly https://v1.qserv.lsst.io https://qserv.lsst.io https://builds.pipelines.lsst.io/b2
15. Software Documentation • API Reference patterns ‣ Useful documentation for SWIG-generated Python APIs • What should package docs look like? ‣ E.g., Astropy packages have: Introduction, Getting Started, User Guide, API Reference • How should (Command line) Tasks be documented? ‣ Present configuration options, including from sub-tasks, and patterns for re-targeting sub-tasks, etc.. • Systems for tutorials ‣ Notebooks, delivery of example datasets, … ‣ Testable and reproducible Content patterns for Sphinx-based documentation.
16. Software Documentation • Docstrings and API references ‣ Part of everyday development work ‣ Plan sprints to catch-up docstring coverage • Higher-level documentation (user guides, tutorials) ‣ Part of developer epics? or ‣ Lead by an editorial team? • Stack Doc ownership and editorial leadership structure? Writing Workflows & Effort