Tweet
Tweet

Slide 1

Slide 1 text

A lion, a head, and a dash of YAML Extending Sphinx to automate your documentation FOSDEM 2018 @stephenfin

Slide 2

Slide 2 text

reStructuredText, Docutils & Sphinx 1

Slide 3

Slide 3 text

A little reStructuredText ========================= This document demonstrates some basic features of |rst|. You can use **bold** and *italics*, along with ``literals``. It’s quite similar to `Markdown`_ but much more extensible. CommonMark may one day approach this [1]_, but today is not that day. `docutils`__ does all this for us. .. |rst| replace:: **reStructuredText** .. _Markdown: https://daringfireball.net/projects/markdown/ .. [1] https://talk.commonmark.org/t/444 .. __ http://docutils.sourceforge.net/

Slide 4

Slide 4 text

A little reStructuredText ========================= This document demonstrates some basic features of |rst|. You can use **bold** and *italics*, along with ``literals``. It’s quite similar to `Markdown`_ but much more extensible. CommonMark may one day approach this [1]_, but today is not that day. `docutils`__ does all this for us. .. |rst| replace:: **reStructuredText** .. _Markdown: https://daringfireball.net/projects/markdown/ .. [1] https://talk.commonmark.org/t/444 .. __ http://docutils.sourceforge.net/

Slide 5

Slide 5 text

A little reStructuredText This document demonstrates some basic features of reStructuredText. You can use bold and italics, along with literals. It’s quite similar to Markdown but much more extensible. CommonMark may one day approach this [1], but today is not that day. docutils does all this for us. [1] https://talk.commonmark.org/t/444/

Slide 6

Slide 6 text

A little more reStructuredText ============================== The extensibility really comes into play with directives and roles. We can do things like link to RFCs (:RFC:`2324`, anyone?) or generate some more advanced formatting (I do love me some H\ :sub:`2`\ O). .. warning:: The power can be intoxicating. Of course, all the stuff we showed previously *still works!*. The only limit is your imagination/interest.

Slide 7

Slide 7 text

A little more reStructuredText ============================== The extensibility really comes into play with directives and roles. We can do things like link to RFCs (:RFC:`2324`, anyone?) or generate some more advanced formatting (I do love me some H\ :sub:`2`\ O). .. warning:: The power can be intoxicating. Of course, all the stuff we showed previously *still works!*. The only limit is your imagination/interest.

Slide 8

Slide 8 text

A little more reStructuredText The extensibility really comes into play with directives and roles. We can do things like link to RFCs (RFC 2324, anyone?) or generate some more advanced formatting (I do love me some H 2 O). Warning The power can be intoxicating. Of course, all the stuff we showed previously still works!. The only limit is your imagination/interest.

Slide 9

Slide 9 text

reStructuredText provides the syntax Docutils provides the parsing

Slide 10

Slide 10 text

reStructuredText provides the syntax Docutils provides the parsing Sphinx provides the cross-referencing and file generation

Slide 11

Slide 11 text

Docutils use readers, parsers, transforms, and writers Docutils works with individual files

Slide 12

Slide 12 text

Docutils use readers, parsers, transforms, and writers Docutils works with individual files Sphinx uses readers, writers, transforms, and builders Sphinx works with multiple, cross-referenced files

Slide 13

Slide 13 text

Documentation tool Multiple output formats Extensive cross-referencing support Extensions

Slide 14

Slide 14 text

Documentation tool Multiple output formats Extensive cross-referencing support Extensions

Slide 15

Slide 15 text

sphinx-quickstart sphinx-build sphinx-apidoc sphinx-autogen

Slide 16

Slide 16 text

sphinx-quickstart sphinx-build sphinx-apidoc sphinx-autogen

Slide 17

Slide 17 text

Let’s Get To Extending... 2

Slide 18

Slide 18 text

Current version of Sphinx (1.7.0) - APIs may change Python knowledge is expected Some possible references to OpenStack projects See github.com/stephenfin/fosdem-sphinx-demo for more

Slide 19

Slide 19 text

Extensions are registered via sphinx.application.Application add_builder add_config_value add_domain add_event add_node add_directive add_role connect, disconnect ... (Builders) (Config Values) (Domains) (Events) (docutils Nodes) (Directives) (Interpreted Text Roles, a.k.a. Roles) (Hooks) ...

Slide 20

Slide 20 text

Extensions are registered via sphinx.application.Application add_builder add_config_value add_domain add_event add_node add_directive add_role connect, disconnect ... (Builders) (Config Values) (Domains) (Events) (docutils Nodes) (Directives) (Interpreted Text Roles, a.k.a. Roles) (Hooks) ...

Slide 21

Slide 21 text

Interpreted Text Roles 3 (a.k.a. roles)

Slide 22

Slide 22 text

A little more reStructuredText ============================== The extensibility really comes into play with directives and roles. We can do things like link to RFCs (:RFC:`2324`, anyone?) or generate some more advanced formatting (I do love me some H\ :sub:`2`\ O). .. warning:: The power can be intoxicating. Of course, all the stuff we showed previously *still works!*. The only limit is your imagination/interest.

Slide 23

Slide 23 text

A little more reStructuredText ============================== The extensibility really comes into play with directives and roles. We can do things like link to RFCs (:RFC:`2324`, anyone?) or generate some more advanced formatting (I do love me some H\ :sub:`2`\ O). .. warning:: The power can be intoxicating. Of course, all the stuff we showed previously *still works!*. The only limit is your imagination/interest.

Slide 24

Slide 24 text

def xyz_role(name, rawtext, text, lineno, inliner, options={}, content=[]): # code... def setup(app): app.add_role('xyz', xyz_role) return {'version': '1.0', 'parallel_read_safe': True}

Slide 25

Slide 25 text

Fixes ===== * #2951: Add ``--implicit-namespaces`` PEP-0420 support to apidoc. * Add ``:caption:`` option for sphinx.ext.inheritance_diagram. * #2471: Add config variable for default doctest flags. * Convert linkcheck builder to requests for better encoding handling * #2463, #2516: Add keywords of "meta" directive to search index source/changes.rst

Slide 26

Slide 26 text

Fixes ===== * #2951: Add ``--implicit-namespaces`` PEP-0420 support to apidoc. * Add ``:caption:`` option for sphinx.ext.inheritance_diagram. * #2471: Add config variable for default doctest flags. * Convert linkcheck builder to requests for better encoding handling * #2463, #2516: Add keywords of "meta" directive to search index source/changes.rst

Slide 27

Slide 27 text

Fixes ===== * #2951: Add ``--implicit-namespaces`` PEP-0420 support to apidoc. * Add ``:caption:`` option for sphinx.ext.inheritance_diagram. * #2471: Add config variable for default doctest flags. * Convert linkcheck builder to requests for better encoding handling * #2463, #2516: Add keywords of "meta" directive to search index source/changes.rst

Slide 28

Slide 28 text

Fixes ===== * Add ``--implicit-namespaces`` PEP-0420 support to apidoc (:ghissue:`2951`). * Add ``:caption:`` option for sphinx.ext.inheritance_diagram. * Add config variable for default doctest flags (:ghissue:`2471`). * Convert linkcheck builder to requests for better encoding handling * Add keywords of "meta" directive to search index (:ghissue:`2463`, :ghissue:`2516`) source/changes.rst

Slide 29

Slide 29 text

Fixes ===== * Add ``--implicit-namespaces`` PEP-0420 support to apidoc (:ghissue:`2951`). * Add ``:caption:`` option for sphinx.ext.inheritance_diagram. * Add config variable for default doctest flags (:ghissue:`2471`). * Convert linkcheck builder to requests for better encoding handling * Add keywords of "meta" directive to search index (:ghissue:`2463`, :ghissue:`2516`) source/changes.rst

Slide 30

Slide 30 text

from docutils import nodes BASE_URL = 'https://github.com/sphinx-doc/sphinx/issues/{}' def github_issue(name, rawtext, text, lineno, inliner, options={}, content=[]): refuri = BASE_URL.format(text) node = nodes.reference(rawtext, text, refuri=refuri, **options) return [node], [] def setup(app): app.add_role('ghissue', github_issue) return {'version': '1.0', 'parallel_read_safe': True} ext/issue_role.py

Slide 31

Slide 31 text

from docutils import nodes BASE_URL = 'https://github.com/sphinx-doc/sphinx/issues/{}' def github_issue(name, rawtext, text, lineno, inliner, options={}, content=[]): refuri = BASE_URL.format(text) node = nodes.reference(rawtext, text, refuri=refuri, **options) return [node], [] def setup(app): app.add_role('ghissue', github_issue) return {'version': '1.0', 'parallel_read_safe': True} ext/issue_role.py

Slide 32

Slide 32 text

Fixes ===== * Add ``--implicit-namespaces`` PEP-0420 support to apidoc (:ghissue:`2951`) * Add ``:caption:`` option for sphinx.ext.inheritance_diagram * Add config variable for default doctest flags (:ghissue:`2471`) * Convert linkcheck builder to requests for better encoding handling * Add keywords of "meta" directive to search index (:ghissue:`2463`, :ghissue:`2516`) source/changes.rst

Slide 33

Slide 33 text

Fixes ● Add --implicit-namespaces PEP-0420 support to apidoc (2951) ● Add :caption: option for sphinx.ext.inheritance_diagram ● Add config variable for default doctest flags (2471) ● Convert linkcheck builder to requests for better encoding handling ● Add keywords of “meta” directive to search index (2463, 2516) build/changes.html

Slide 34

Slide 34 text

Directives 4

Slide 35

Slide 35 text

A little more reStructuredText ============================== The extensibility really comes into play with directives and roles. We can do things like link to RFCs (:RFC:`2324`, anyone?) or generate some more advanced formatting (I do love me some H\ :sub:`2`\ O). .. warning:: The power can be intoxicating. Of course, all the stuff we showed previously *still works!*. The only limit is your imagination/interest.

Slide 36

Slide 36 text

A little more reStructuredText ============================== The extensibility really comes into play with directives and roles. We can do things like link to RFCs (:RFC:`2324`, anyone?) or generate some more advanced formatting (I do love me some H\ :sub:`2`\ O). .. warning:: The power can be intoxicating. Of course, all the stuff we showed previously *still works!*. The only limit is your imagination/interest.

Slide 37

Slide 37 text

from docutils import nodes from docutils.parsers.rst import Directive class XYZDirective(Directive): def run(self): section = nodes.section(ids=['test']) section += nodes.title(text='Test') section += nodes.paragraph(text='Hello, world!') return [section] def setup(app): app.add_directive('xyz-directive', XYZDirective) return {'version': '1.0', 'parallel_read_safe': True}

Slide 38

Slide 38 text

Issues ====== Add keywords of "meta" directive to search index (#2463) -------------------------------------------------------- Opened by TimKam :: It would be great to have the keywords of `meta` directives included in the search index. Like this, one can help users who are searching for a synonym of the "correct" term through simply adding synonyms as keywords to a meta directive on the corresponding page. source/issues.rst

Slide 39

Slide 39 text

Issues ====== Add keywords of "meta" directive to search index (#2463) -------------------------------------------------------- Opened by TimKam :: It would be great to have the keywords of `meta` directives included in the search index. Like this, one can help users who are searching for a synonym of the "correct" term through simply adding synonyms as keywords to a meta directive on the corresponding page. source/issues.rst

Slide 40

Slide 40 text

Issues ====== .. github-issue:: 2463 source/issues.rst

Slide 41

Slide 41 text

from docutils import nodes from docutils.parsers.rst import Directive import requests URL = 'https://api.github.com/repos/sphinx-doc/sphinx/issues/{}' def get_issue(issue_id): issue = requests.get(URL.format(issue_id)).json() title = '%s (#%s)' % (issue['title'], issue_id) owner = 'Opened by %s' issue['user']['login'] return issue_id, title, issue['body'], owner ... ext/issue_directive.py

Slide 42

Slide 42 text

... class ShowGitHubIssue(Directive): required_arguments = 1 def run(self): issue = get_issue(self.arguments[0]) section = nodes.section(ids=['github-issue-%s' % issue[0]]) section += nodes.title(text=issue[1]) section += nodes.paragraph(text='Opened by %s' % issue[3]) section += nodes.literal_block(text=issue[2]) return [section] ext/issue_directive.py

Slide 43

Slide 43 text

... class ShowGitHubIssue(Directive): required_arguments = 1 def run(self): issue = get_issue(self.arguments[0]) section = nodes.section(ids=['github-issue-%s' % issue[0]]) section += nodes.title(text=issue[1]) section += nodes.paragraph(text='Opened by %s' % issue[3]) section += nodes.literal_block(text=issue[2]) return [section] ext/issue_directive.py

Slide 44

Slide 44 text

... class ShowGitHubIssue(Directive): required_arguments = 1 def run(self): issue = get_issue(self.arguments[0]) section = nodes.section(ids=['github-issue-%s' % issue[0]]) section += nodes.title(text=issue[1]) section += nodes.paragraph(text='Opened by %s' % issue[3]) section += nodes.literal_block(text=issue[2]) return [section] ext/issue_directive.py

Slide 45

Slide 45 text

... def setup(app): app.add_directive('github-issue', ShowGitHubIssue) return {'version': '1.0', 'parallel_read_safe': True} ext/issue_directive.py

Slide 46

Slide 46 text

Issues ====== .. github-issue:: 2463 source/issues.rst

Slide 47

Slide 47 text

Issues Add keywords of “meta” directive to search index (#2463) Opened by TimKam build/issues.html It would be great to have the keywords of `meta` directives included in the search index. Like this, one can help users who are searching for a synonym of the "correct" term through simply adding synonyms as keywords to a `meta` directive on the corresponding page.

Slide 48

Slide 48 text

class ShowGitHubIssue(Directive): required_arguments = 1 def run(self): issue = get_issue(self.arguments[0]) section = nodes.section(ids=['github-issue-%s' % issue[0]]) section += nodes.title(text=issue[1]) section += nodes.paragraph(text='Opened by %s' % issue[3]) section += nodes.literal_block(text=issue[2]) return [section] ext/issue_directive.py

Slide 49

Slide 49 text

class ShowGitHubIssue(Directive): required_arguments = 1 def run(self): issue = get_issue(self.arguments[0]) section = nodes.section(ids=['github-issue-%s' % issue[0]]) section += nodes.title(text=issue[1]) section += nodes.paragraph(text='Opened by %s' % issue[3]) section += nodes.literal_block(text=issue[2]) return [section] ext/issue_directive.py

Slide 50

Slide 50 text

class ShowGitHubIssue(Directive): required_arguments = 1 def run(self): issue = get_issue(self.arguments[0]) result = statemachine.ViewList() for line in format_issue(issue): result.append(line, '<' + __name__ + '>') node = nodes.section(document=self.state.document) nested_parse_with_titles(self.state, result, node) return node.children ext/issue_directive.py

Slide 51

Slide 51 text

class ShowGitHubIssue(Directive): required_arguments = 1 def run(self): issue = get_issue(self.arguments[0]) result = statemachine.ViewList() for line in format_issue(issue): result.append(line, '<' + __name__ + '>') node = nodes.section(document=self.state.document) nested_parse_with_titles(self.state, result, node) return node.children ext/issue_directive.py

Slide 52

Slide 52 text

def format_issue(issue): num, title, body, owner = issue yield title yield '=' * len(title) yield '' yield '%s ::' % owner yield '' for line in body.splitlines(): yield ' %s' % line if line else '' ext/issue_directive.py

Slide 53

Slide 53 text

Issues Add keywords of “meta” directive to search index (#2463) Opened by TimKam It would be great to have the keywords of `meta` directives included in the search index. Like this, one can help users who are searching for a synonym of the "correct" term through simply adding synonyms as keywords to a `meta` directive on the corresponding page. build/issues.html

Slide 54

Slide 54 text

Events 5

Slide 55

Slide 55 text

builder-inited(app) config-inited(app, config) source-read(app, docname, source) doctree-read(app, doctree) ...

Slide 56

Slide 56 text

from docutils import nodes from docutils.parsers.rst import Directive def builder_inited_handler(app): # code here... def setup(app): app.connect('builder-inited', builder_inited_handler)

Slide 57

Slide 57 text

issues.rst issue-2951.rst issue-2463.rst issue-2516.rst issue-2471.rst ...

Slide 58

Slide 58 text

from docutils import nodes from docutils.parsers.rst import Directive import requests URL = 'https://api.github.com/repos/sphinx-doc/sphinx/issues/' def get_issues(): issues = requests.get(URL).json() for issue in issues: title = '%s (#%s)' % (issue['title'], issue['number']) owner = 'Opened by %s' % issue['user']['login'] yield issue['number'], title, issue['body'], owner ext/issue_event.py

Slide 59

Slide 59 text

... def generate_issue_docs(app): for num, title, body, owner in get_issues(): filename = os.path.join(app.srcdir, 'issues', '%s.rst' % num) with io.open(filename, 'w') as issue_doc: print(title, file=issue_doc) print('=' * len(title), file=issue_doc) print('', file=issue_doc) print('%s ::' % owner, file=issue_doc) print('', file=issue_doc) for line in body.splitlines(): print(' %s' % line if line else '', file=issue_doc) ext/issue_event.py

Slide 60

Slide 60 text

... def setup(app): app.connect('builder-inited', generate_issue_docs) return {'version': '1.0', 'parallel_read_safe': True} ext/issue_event.py

Slide 61

Slide 61 text

Issues ====== .. toctree:: :maxdepth: 1 :glob: issues/* source/index.rst

Slide 62

Slide 62 text

Issues ● Drop special support for rst2pdf (#4463) ● Proposal: Integrate source_suffix and source_parsers (#4474) ● [RFC] Implement delayed resolution in TOC (#4475) ● Not possible to update individual ‘po’ files (#4476) ● Build fails during eclim (aur) build: Babel data files not available (#4481) ● Proposal: Allow to switch parsers on parsing document (#4482) ● Integrate source suffix and source parsers (#4483) ● … build/index.html

Slide 63

Slide 63 text

Enabling Your Extensions 5

Slide 64

Slide 64 text

import os import sys sys.path.insert(0, os.path.abspath('../ext')) extensions = [ 'issue_role', 'issue_directive', 'issue_event', ] source/conf.py

Slide 65

Slide 65 text

import os import sys sys.path.insert(0, os.path.abspath('../ext')) extensions = [ 'issue_role', 'issue_directive', 'issue_event', 'oslo_config.sphinxext', ] source/conf.py

Slide 66

Slide 66 text

Wrap Up 6

Slide 67

Slide 67 text

Extensions are registered via sphinx.application.Application add_builder add_config_value add_domain add_event add_node add_directive add_role connect, disconnect ... (Builders) (Config Values) (Domains) (Events) (docutils Nodes) (Directives) (Interpreted Text Roles, a.k.a. Roles) (Hooks) ...

Slide 68

Slide 68 text

Extensions are registered via sphinx.application.Application add_builder add_config_value add_domain add_event add_node add_directive add_role connect, disconnect ... (Builders) (Config Values) (Domains) (Events) (docutils Nodes) (Directives) (Interpreted Text Roles, a.k.a. Roles) (Hooks) ...

Slide 69

Slide 69 text

Extensions are registered via sphinx.application.Application Builder-specific Extensions (HTML themes, LaTeX templates, …) (Post) Transforms Translators Parsers Search languages ...

Slide 70

Slide 70 text

Fin

Slide 71

Slide 71 text

A lion, a head, and a dash of YAML Extending Sphinx to automate your documentation FOSDEM 2018 @stephenfin

Slide 72

Slide 72 text

References ● Quick reStructuredText ● Docutils Reference Guide ○ reStructuredText Markup Specification ○ reStructuredText Directives ○ reStructuredText Interpreted Text Roles ● Docutils How-Tos ○ Creating reStructuredText Interpreted Text Roles ○ Creating reStructuredText Directives ● Docutils Hacker’s Guide ● Sphinx Tutorial: Writing a simple extension

Slide 73

Slide 73 text

References ● Defining Custom Roles in Sphinx -- Doug Hellmann ● The Power of Sphinx - Integrating Jinja with RST -- Eric Holscher ● Docutils Snippets -- Aurélien Gâteau ● OpenStack + Sphinx In A Tree -- Stephen Finucane ( )