Understanding Documentation Systems - ConFoo Vancouver 2016

This talk will cover the internals of how a system for documenting software works. This talk will include specific examples from the Sphinx documentation generator, and cover the concepts they use to make documentation generation powerful.

The talk aims to impart high-level understanding of the concepts around documentation generation, which will be applicable to most documentation tooling, whether it be Commonmark, RST, or Asciidoc.

Transcript

1. Eric Holscher Confoo Vancouver December 6, 2016 Understanding Documentation Systems
2. Who am I • Co-Founder of Read the Docs • Co-Founder of Write the Docs • I come from the Python world
3. Today • Understand why docs are important • Under the underlying model for documentation tools • Be able to reason more completely about how to extend doc tooling
4. @ericholscher Why Write Docs
5. @ericholscher You will be using your code in 6 months
6. @ericholscher You want people to use your code
7. @ericholscher It makes your code better
8. @ericholscher You want to be a better writer
9. @ericholscher Documentation Systems
10. Documentation Systems • Lightweight Markup Language • Templates • Command Line Interface • Common Output Formats (HTML, PDF)
11. Documentation Systems • Jekyll • Asciidoctor • Sphinx
12. Jekyll • Built on Markdown • A simpler execution model • Great for static sites, blogs, and documentation. • A great number of extensions, and support on GitHub Pages
13. Asciidoctor • Built on Asciidoc markup • Has powerful semantic constructs • Compatible with Docbook XML • Implementation in Ruby, but support for JVM, JS.
14. Sphinx • Built on reStructuredText Markup • Has powerful semantic constructs • Has a lot of primitives for documenting so ware • Has a lot of extensions, especially around testing & documentation source code
15. @ericholscher Lightweight Markup Languages
16. @ericholscher Lightweight Markup Language • Base format to generate other formats from • Readable as plain text • Works well with programmer tools • Works great in code comments/ docstrings
17. @ericholscher Semantic Meaning • The power of HTML & LWML • Semantics mean you are saying what something is, not how to display it • Once you know what it is, you can display it properly • “Separation of Concerns”
18. @ericholscher # HTML (Bad) <b>issue 72</b> # HTML (Good) <span class=“issue”>issue 72</span> # CSS (Good) .issue { text-format: bold; } Classic HTML Example
19. @ericholscher # Bad <font color=“red”>Warning: Don’t do this!</font> # Good <span class=“warning”>Don’t do this!</span> # Best .. warning:: Don’t do this Classic RST Example
20. @ericholscher # Markdown Check out [PEP 8](https:// www.python.org/dev/peps/pep-0008/) # RST Check out :pep:`8` # Asciidoc See pep:8 to get started. Semantic Comparison
21. @ericholscher # Markdown This is an [idempotent](http:// docs.foo.com/glossary#term- idempotent] implementation. # RST This is an :term:`idempotent` implementation. # Asciidoc This is an term:idempotent implementation Semantic Comparison
22. @ericholscher See our image :ref[scatter plot] to see more information. :: include{file=other-file.md} CommonMark Proposal https://talk.commonmark.org/t/generic-directives-plugins- syntax/444
23. @ericholscher Semantic Markup • Shows the intent of your words • Works across output formats • You can style warnings differently in HTML, PDF, ePub, etc.
24. @ericholscher Extendability
25. @ericholscher Inline Markup • Anything that is included in the page content itself • Used for embedding things into the rendered output
26. @ericholscher Inline Markup • Looks like :role:`target` • :pep:`8` will create a link to PEP 8
27. @ericholscher Coding Reference ---------------- Generally we follow :pep:`8` in our code. However, we also have our own :doc:`style-guide` with exceptions. Inline Markup Example
28. @ericholscher Inline Markup Example
29. @ericholscher Page Level Markup • Allows you to nest content inside of them • Great for reference endpoints
30. @ericholscher Page Level Markup • .. directive-name:: • Main source of extendability • A lot of Sphinx’s power comes through Directives
31. @ericholscher .. code-block:: python :linenos: from datetime import datetime time = datetime.now() print time Page Level Example
32. @ericholscher Page Level Example
33. Templates • Allow for extending with specific logic in the markup language • Jinja & Liquid are the major ones
34. {% for speaker in talk.speakers %} * {{ speaker.name }} {% endfor %} Templates
35. reStructuredText Implementation
36. Parts • Reader • Parser • Transformer • Writer
37. How it fits together
38. Reader • Get input and read it into memory • Quite simple, generally don’t need to do much
39. Title ===== Paragraph. Words that have **bold in them**. Reader Example
40. [u'Title', u'=====', u'', u'Paragraph.', u'', u'Words that have **bold in them**.'] Reader Example
41. Reader’s are useful for adding non-filesystem types of input (StringIO, Network)
42. Parser • Takes the input and actually turns it into a Doctree • Handles directives, inline markup, etc. • RST is the only parser implemented in Docutils • Implemented with a lined-based recursive state machine
43. Doctree • AST for Docutils • Source of Truth • Hierarchy with a document at the root node
44. Title ===== Paragraph. Words that have **bold in them**. Doctree Example
45. <document ids="title" names="title" source="test.rst" title="Title"> <title> Title <paragraph> Paragraph. <paragraph> Words that have <strong> bold in them Doctree Example
46. Parsers are used for implementing new RST features or adding new markup languages
47. Transformer • Take the doctree and modify it in place • Allows for full knowledge of the content • Table of Contents • Implemented by traversing nodes
48. <document ids="title" names="title" source="test.rst" title="Title"> <title> Title <paragraph> Paragraph. <paragraph> Words that have <strong> bold in them Transform Example
49. <document ids="title" names="title" source="test.rst" title="Title"> <title> Title <paragraph> Words that have <strong> bold in them <paragraph> Paragraph. Transform Example
50. Writer • Takes the Doctree and writes it to actual files • HTML, XML, PDF, etc. • Implemented with a Visitor pattern
51. How it fits together
52. Extensions
53. Parser • rST • Markdown
54. Markdown Parser • Uses recommonmark • Translates Commonmark Node’s into Docutils Node’s
55. def paragraph(self, block): p = nodes.paragraph() p.line = block.start_line append_inlines(p, block.inline_content) self.current_node.append(p) Markdown Parser
56. ## Markdown Header Hey There Markdown Parser
57. <document source="example.md" title="Markdown Header"> <title> Markdown Header <paragraph> Hey There Markdown Parser
58. Transformer • Table of contents
59. TOC Transformer • Enabled with `.. contents::` directive • Adds a pending node during parsing • Gets properly set with a transform
60. .. contents:: TOC TOC Transformer
61. <topic classes="contents" ids="toc" names="toc"> <title> TOC <pending> .. internal attributes: .transform: docutils.transforms.parts.Contents .details: TOC Transformer
62. <topic classes="contents" ids="toc" names="toc"> <title> TOC <bullet_list> <list_item> <paragraph> <reference ids="id1" refid="getting-started"> Getting Started TOC Transformer
63. Sphinx
64. Sphinx • Builds on top of the standard docutils concepts • Add it’s own abstractions, but uses the same docutils machinery underneath • Goes from single page to project
65. Sphinx Components • Application • Environment • Builder
66. Sphinx Application • Main level of orchestration for Sphinx • Handles configuration & building • Sphinx()
67. Sphinx Environment • Keeps state for all the files for a project • Serialized to disk in between runs • Works as a cache
68. Sphinx Builder • Handles generating output • Where templates & styles are implemented • Allow customization of specific elements
69. Sphinx layers on top of RST to provide full project support
70. Take Aways
71. Knowing how your docs tools work is part of your job
72. Think about your intended use case and outcomes before picking a tool
73. O en times you can save a lot of effort by writing simple extensions to your tools
74. Learn how other languages solve problems, then steal liberally :)
75. Questions? • @ericholscher • eric@ericholscher.com • Come talk to me around the conference