Eric Holscher
Confoo Vancouver
December 6, 2016
Understanding
Documentation Systems
Slide 2
Slide 2 text
Who am I
• Co-Founder of Read the Docs
• Co-Founder of Write the Docs
• I come from the Python world
Slide 3
Slide 3 text
Today
• Understand why docs are important
• Under the underlying model for
documentation tools
• Be able to reason more completely
about how to extend doc tooling
Slide 4
Slide 4 text
@ericholscher
Why Write Docs
Slide 5
Slide 5 text
@ericholscher
You will be using your
code in 6 months
Slide 6
Slide 6 text
@ericholscher
You want people to use
your code
Slide 7
Slide 7 text
@ericholscher
It makes your code
better
Slide 8
Slide 8 text
@ericholscher
You want to be a better
writer
Slide 9
Slide 9 text
@ericholscher
Documentation Systems
Slide 10
Slide 10 text
Documentation Systems
• Lightweight Markup Language
• Templates
• Command Line Interface
• Common Output Formats (HTML, PDF)
Slide 11
Slide 11 text
Documentation Systems
• Jekyll
• Asciidoctor
• Sphinx
Slide 12
Slide 12 text
Jekyll
• Built on Markdown
• A simpler execution model
• Great for static sites, blogs, and
documentation.
• A great number of extensions, and
support on GitHub Pages
Slide 13
Slide 13 text
Asciidoctor
• Built on Asciidoc markup
• Has powerful semantic constructs
• Compatible with Docbook XML
• Implementation in Ruby, but support
for JVM, JS.
Slide 14
Slide 14 text
Sphinx
• Built on reStructuredText Markup
• Has powerful semantic constructs
• Has a lot of primitives for documenting
so ware
• Has a lot of extensions, especially
around testing & documentation source
code
Slide 15
Slide 15 text
@ericholscher
Lightweight
Markup Languages
Slide 16
Slide 16 text
@ericholscher
Lightweight Markup
Language
• Base format to generate other formats
from
• Readable as plain text
• Works well with programmer tools
• Works great in code comments/
docstrings
Slide 17
Slide 17 text
@ericholscher
Semantic Meaning
• The power of HTML & LWML
• Semantics mean you are saying what
something is, not how to display it
• Once you know what it is, you can
display it properly
• “Separation of Concerns”
Slide 18
Slide 18 text
@ericholscher
# HTML (Bad)
issue 72
# HTML (Good)
issue 72
# CSS (Good)
.issue { text-format: bold; }
Classic HTML Example
Slide 19
Slide 19 text
@ericholscher
# Bad
Warning: Don’t do this!
# Good
Don’t do this!
# Best
.. warning:: Don’t do this
Classic RST Example
Slide 20
Slide 20 text
@ericholscher
# Markdown
Check out [PEP 8](https://
www.python.org/dev/peps/pep-0008/)
# RST
Check out :pep:`8`
# Asciidoc
See pep:8 to get started.
Semantic Comparison
Slide 21
Slide 21 text
@ericholscher
# Markdown
This is an [idempotent](http://
docs.foo.com/glossary#term-
idempotent] implementation.
# RST
This is an :term:`idempotent`
implementation.
# Asciidoc
This is an term:idempotent
implementation
Semantic Comparison
Slide 22
Slide 22 text
@ericholscher
See our image :ref[scatter plot] to
see more information.
:: include{file=other-file.md}
CommonMark Proposal
https://talk.commonmark.org/t/generic-directives-plugins-
syntax/444
Slide 23
Slide 23 text
@ericholscher
Semantic Markup
• Shows the intent of your words
• Works across output formats
• You can style warnings differently in
HTML, PDF, ePub, etc.
Slide 24
Slide 24 text
@ericholscher
Extendability
Slide 25
Slide 25 text
@ericholscher
Inline Markup
• Anything that is included in the page
content itself
• Used for embedding things into the
rendered output
Slide 26
Slide 26 text
@ericholscher
Inline Markup
• Looks like :role:`target`
• :pep:`8` will create a link to PEP 8
Slide 27
Slide 27 text
@ericholscher
Coding Reference
----------------
Generally we follow :pep:`8` in our code.
However, we also have our own :doc:`style-guide` with
exceptions.
Inline Markup Example
Slide 28
Slide 28 text
@ericholscher
Inline Markup Example
Slide 29
Slide 29 text
@ericholscher
Page Level Markup
• Allows you to nest content inside of
them
• Great for reference endpoints
Slide 30
Slide 30 text
@ericholscher
Page Level Markup
• .. directive-name::
• Main source of extendability
• A lot of Sphinx’s power comes through
Directives
Slide 31
Slide 31 text
@ericholscher
.. code-block:: python
:linenos:
from datetime import datetime
time = datetime.now()
print time
Page Level Example
Slide 32
Slide 32 text
@ericholscher
Page Level Example
Slide 33
Slide 33 text
Templates
• Allow for extending with specific logic
in the markup language
• Jinja & Liquid are the major ones
Slide 34
Slide 34 text
{% for speaker in talk.speakers %}
* {{ speaker.name }}
{% endfor %}
Templates
Slide 35
Slide 35 text
reStructuredText
Implementation
Slide 36
Slide 36 text
Parts
• Reader
• Parser
• Transformer
• Writer
Slide 37
Slide 37 text
How it fits together
Slide 38
Slide 38 text
Reader
• Get input and read it into memory
• Quite simple, generally don’t need to do
much
Slide 39
Slide 39 text
Title
=====
Paragraph.
Words that have **bold in them**.
Reader Example
Slide 40
Slide 40 text
[u'Title', u'=====', u'', u'Paragraph.', u'', u'Words
that have **bold in them**.']
Reader Example
Slide 41
Slide 41 text
Reader’s are useful for
adding non-filesystem types
of input (StringIO, Network)
Slide 42
Slide 42 text
Parser
• Takes the input and actually turns it
into a Doctree
• Handles directives, inline markup, etc.
• RST is the only parser implemented in
Docutils
• Implemented with a lined-based
recursive state machine
Slide 43
Slide 43 text
Doctree
• AST for Docutils
• Source of Truth
• Hierarchy with a document at the root
node
Slide 44
Slide 44 text
Title
=====
Paragraph.
Words that have **bold in them**.
Doctree Example
Slide 45
Slide 45 text
Title
<paragraph>
Paragraph.
<paragraph>
Words that have
<strong>
bold in them
Doctree Example
Slide 46
Slide 46 text
Parsers are used for
implementing new RST
features or adding new
markup languages
Slide 47
Slide 47 text
Transformer
• Take the doctree and modify it in place
• Allows for full knowledge of the
content
• Table of Contents
• Implemented by traversing nodes
Slide 48
Slide 48 text
Title
<paragraph>
Paragraph.
<paragraph>
Words that have
<strong>
bold in them
Transform Example
Slide 49
Slide 49 text
Title
<paragraph>
Words that have
<strong>
bold in them
<paragraph>
Paragraph.
Transform Example
Slide 50
Slide 50 text
Writer
• Takes the Doctree and writes it to
actual files
• HTML, XML, PDF, etc.
• Implemented with a Visitor pattern
Sphinx
• Builds on top of the standard docutils
concepts
• Add it’s own abstractions, but uses the
same docutils machinery underneath
• Goes from single page to project