PyConZA 2014: "How Python helps writing documentation less painful" by Nickolas Grigoriadis

We all know writing documentation is an arduous exercise. We all know how useless and frustrating out-of-date or just plain incorrect documentation is. In this talk I'd like to demonstrate how Python can help make writing documentation, keeping it up-to-date and verifying its validity. It can be surprising what happens when other people discover your little-but-well-documented API.

Python has several built-in concepts and standard tools to help making this easier, such as docstrings, help(), Sphinx, Sphinx plugins, Doctests and generating documentation off tests themselves.

I'll focus on not only using Sphinx as a tool, but also on how to structure your application/library/tool so as to make writing accurate documentation as simple as possible.

Transcript

1. Less painful documentation with Python By Nickolas Grigoriadis Internet Solutions
2. This Presentation In this presentation I intend to give you a primer of some built-in documentation abilities of Python, and how to get started using Sphinx-doc. I will also discuss methods we used to make keeping documentation up-to- date a bit easier. This presentation is available at: http://bit.ly/1v3po5z (http://bit.ly/1v3po5z)
3. About Me I'm currently a developer at Internet Solutions, have been writing code since I was 11, and a professional developer for over 10 years now. I'm also a dad...
4. Documentation Sigh, Documentation Reading useful documentation is awesome! Cringe if I have to write it.
5. Useful Documentation What is useful documentation? Targets the right audience. Easy to find relevant information. Technically correct. Complete.
6. Useless Documentation What makes documentation essentially useless? Pretty much the inverse of any of the previous slides points. Incomplete documentation can still be useful. But, incorrect/out-of-date documentation is worse than no documentation!
7. Self Documenting code Ah, that myth I hear every so often. If computers could generate documentation, that carries intent and purpose without any input from its creator, it may as well take over the world. Though, there was some promising and useful work done in this direction. e.g. Doxygen, JavaDoc, etc… These tools can generate documentation that is accurate, but completely devoid of intent and purpose.
8. Self documenting code - cont You have to add comments to your code which ‘helps’ the documentation generator make more readable documentation. Ultimately the documentation often ends up as a documentation of the application internals, whilst useful, not necessarily targeting the right audience or complete.
9. How does Python help? Due in part to how python developed as a language (PEP), and its introspection abilities, Python has several ways of helping generating documentation. There are several useful built-in functions one can use: Returns a list of sub-objects dir() Generates help text on object help() Docstrings Docstrings is the first string literal after a function/class definition.
10. dir() helper function I n [ 2 ] : d i r ( _ _ b u i l t i n _ _ ) [ 7 5 : 1 0 0 ] O u t [ 2 ] : [ ' c o m p i l e ' , ' c o m p l e x ' , ' c o p y r i g h t ' , ' c r e d i t s ' , ' d e l a t t r ' , ' d i c t ' , ' d i r ' , ' d i v m o d ' , ' d r e l o a d ' , ' e n u m e r a t e ' , ' e v a l ' , ' e x e c f i l e ' , ' f i l e ' , ' f i l t e r ' , ' f l o a t ' , ' f o r m a t ' , ' f r o z e n s e t ' , ' g e t _ i p y t h o n ' , ' g e t a t t r ' , ' g l o b a l s ' , ' h a s a t t r ' , ' h a s h ' , ' h e l p ' , ' h e x ' , ' i d ' ]
11. help() function I n [ 3 ] : h e l p ( e n u m e r a t e ) H e l p o n c l a s s e n u m e r a t e i n m o d u l e _ _ b u i l t i n _ _ : c l a s s e n u m e r a t e ( o b j e c t ) | e n u m e r a t e ( i t e r a b l e [ , s t a r t ] ) - > i t e r a t o r f o r i n d e x , v a l u e o f i t e r a b l e | | R e t u r n a n e n u m e r a t e o b j e c t . i t e r a b l e m u s t b e a n o t h e r o b j e c t t h a t s u p p o r t s | i t e r a t i o n . T h e e n u m e r a t e o b j e c t y i e l d s p a i r s c o n t a i n i n g a c o u n t ( f r o m | s t a r t , w h i c h d e f a u l t s t o z e r o ) a n d a v a l u e y i e l d e d b y t h e i t e r a b l e a r g u m e n t . | e n u m e r a t e i s u s e f u l f o r o b t a i n i n g a n i n d e x e d l i s t : | ( 0 , s e q [ 0 ] ) , ( 1 , s e q [ 1 ] ) , ( 2 , s e q [ 2 ] ) , . . . | | M e t h o d s d e f i n e d h e r e : | | _ _ g e t a t t r i b u t e _ _ ( . . . ) | x . _ _ g e t a t t r i b u t e _ _ ( ' n a m e ' ) < = = > x . n a m e | | _ _ i t e r _ _ ( . . . ) | x . _ _ i t e r _ _ ( ) < = = > i t e r ( x ) | | n e x t ( . . . ) | x . n e x t ( ) - > t h e n e x t v a l u e , o r r a i s e S t o p I t e r a t i o n |
12. Basic docstring: I n [ 4 ] : d e f g r e e t ( p e r s o n ) : ' ' ' P r i n t s a g r e e t i n g t o < p e r s o n > ' ' ' p r i n t " G r e e t i n g s , % s " % p e r s o n I n [ 5 ] : h e l p ( g r e e t ) H e l p o n f u n c t i o n g r e e t i n m o d u l e _ _ m a i n _ _ : g r e e t ( p e r s o n ) P r i n t s a g r e e t i n g t o < p e r s o n >
13. Programatically set docstring I n [ 6 ] : d e f g r e e t ( p e r s o n ) : p r i n t " G r e e t i n g s , % s " % p e r s o n g r e e t . _ _ d o c _ _ = " P r o g r a m a t i c a l l y s e t d o c s t r i n g " I n [ 7 ] : h e l p ( g r e e t ) H e l p o n f u n c t i o n g r e e t i n m o d u l e _ _ m a i n _ _ : g r e e t ( p e r s o n ) P r o g r a m a t i c a l l y s e t d o c s t r i n g
14. Docstring that self-generates I n [ 8 ] : P E R S O N S = ( ' m i l k m a n ' , ' r a z ' , ' s e c r e t a g e n t ' ) d e f g r e e t ( p e r s o n ) : ' ' ' P r i n t s a g r e e t i n g t o < p e r s o n > i f < p e r s o n > i s r e c o g n i s e d , e l s e i t t e l l s y o u t o g o a w a y . A l l o w e d c h a r s a r e : { p e r s o n _ l i s t } ' ' ' i f p e r s o n n o t i n C H A R A C T E R S : p r i n t " G o a w a y ! " e l s e : p r i n t " G r e e t i n g s , % s " % p e r s o n g r e e t . _ _ d o c _ _ = g r e e t . _ _ d o c _ _ . f o r m a t ( p e r s o n _ l i s t = P E R S O N S ) I n [ 9 ] : h e l p ( g r e e t ) H e l p o n f u n c t i o n g r e e t i n m o d u l e _ _ m a i n _ _ : g r e e t ( p e r s o n ) P r i n t s a g r e e t i n g t o < p e r s o n > i f < p e r s o n > i s r e c o g n i s e d , e l s e i t t e l l s y o u t o g o a w a y . A l l o w e d c h a r s a r e : ( ' m i l k m a n ' , ' r a z ' , ' s e c r e t a g e n t ' )
15. Docstrings as properties *possibly dangerous I n [ 1 0 ] : c l a s s C l a s s P r o p e r t y ( p r o p e r t y ) : ' ' ' B u i l t i n p r o p e r t y o n l y w o r k s o n i n s t a n c e a t t r i b u t e s ' ' ' d e f _ _ g e t _ _ ( s e l f , c l s , o w n e r ) : r e t u r n s e l f . f g e t . _ _ g e t _ _ ( N o n e , o w n e r ) ( ) c l a s s g r e e t ( ) : ' ' ' U n u s e d D o c s t r i n g f o r C l a s s ' ' ' @ C l a s s P r o p e r t y @ c l a s s m e t h o d d e f _ _ d o c _ _ ( c l s ) : p r i n t " G E N E R A T I N G D O C S T R I N G L A Z I L Y ! \ n " r e t u r n " T h e g e n e r a t e d d o c s t r i n g " I n [ 1 1 ] : h e l p ( g r e e t ) G E N E R A T I N G D O C S T R I N G L A Z I L Y ! H e l p o n c l a s s g r e e t i n m o d u l e _ _ m a i n _ _ : c l a s s g r e e t | T h e g e n e r a t e d d o c s t r i n g
16. reStructuredText http://docutils.sourceforge.net/rst.html (http://docutils.sourceforge.net/rst.html)
17. reST - Formatting reStructuredText is a markup that has been designed to be more capable, it has common formatting rules: C a p t i o n s a r e u n d e r l i n e d - - - - - - - - - - - - - - - - - - - - - - - * B u l l e t l i s t 1 . N e s t e d n u m b e r e d l i s t * I t a l i c * * * B o l d * * ` ` T e x t ` ` . . C o m m e n t e d b l o c k e t c . . .
18. reST - Directives reStructuredText also has directives, which are the more powerful aspect. You can define your own directive_types. Example of a csv-table directive: . . d i r e c t i v e _ t y p e : n a m e : p a r a m a t e r s : c o n t e n t . . c s v - t a b l e : : F r o z e n D e l i g h t s ! : h e a d e r : " T r e a t " , " Q u a n t i t y " , " D e s c r i p t i o n " : w i d t h s : 1 5 , 1 0 , 3 0 " A l b a t r o s s " , 2 . 9 9 , " O n a s t i c k ! " " C r u n c h y F r o g " , 1 . 4 9 , " I f w e t o o k t h e b o n e s o u t , i t w o u l d n ' t b e c r u n c h y , n o w w o u l d i t ? " " G a n n e t R i p p l e " , 1 . 9 9 , " O n a s t i c k ! "
19. Sphinx http://sphinx-doc.org/ (http://sphinx- doc.org/)
20. Simple example Lets say we have a simple class aha in aha.py: I n [ 1 2 ] : ' ' ' C o n t r i v e d ` l s ` ' ' ' i m p o r t o s c l a s s a h a ( ) : ' ' ' A h a ! l i s t s p a t h c o n t e n t s ' ' ' d e f _ _ i n i t _ _ ( s e l f , p a t h ) : s e l f . p a t h = p a t h @ p r o p e r t y d e f e x i s t s ( s e l f ) : ' ' ' C h e c k s w e t h e r p a t h e x i s t s ' ' ' r e t u r n o s . p a t h . e x i s t s ( s e l f . p a t h ) d e f l s ( s e l f ) : ' ' ' R e t u r n s a l i s t o f p a t h c o n t e n t s ' ' ' i f s e l f . e x i s t s : r e t u r n o s . l i s t d i r ( s e l f . p a t h ) e l s e : r e t u r n N o n e
21. Simple Example - cont I n [ 1 3 ] : p l a c e = a h a ( ' d o c s a m p l e ' ) I n [ 1 4 ] : p l a c e . e x i s t s O u t [ 1 4 ] : T r u e I n [ 1 5 ] : p l a c e . l s ( ) O u t [ 1 5 ] : [ ' a h a . p y ' , ' _ _ i n i t _ _ . p y ' ] Lets compare that result to an actual shell l s : I n [ 1 6 ] : ! l s - g o d o c s a m p l e t o t a l 4 - r w - r - - r - - 1 4 2 3 S e p 3 0 1 2 : 4 1 a h a . p y - r w - r - - r - - 1 0 S e p 3 0 1 2 : 4 7 _ _ i n i t _ _ . p y
22. Simple Example - cont Run sphinx-quickstart to create a project.
23. Simple Exampe - cont Now lets build the docs: I n [ 1 9 ] : ! s p h i n x - b u i l d d o c s d o c s / h t m l 1 M a k i n g o u t p u t d i r e c t o r y . . . R u n n i n g S p h i n x v 1 . 2 . 3 l o a d i n g p i c k l e d e n v i r o n m e n t . . . n o t y e t c r e a t e d N o b u i l d e r s e l e c t e d , u s i n g d e f a u l t : h t m l b u i l d i n g [ h t m l ] : t a r g e t s f o r 1 s o u r c e f i l e s t h a t a r e o u t o f d a t e u p d a t i n g e n v i r o n m e n t : 1 a d d e d , 0 c h a n g e d , 0 r e m o v e d l o o k i n g f o r n o w - o u t d a t e d f i l e s . . . n o n e f o u n d p i c k l i n g e n v i r o n m e n t . . . d o n e c h e c k i n g c o n s i s t e n c y . . . d o n e p r e p a r i n g d o c u m e n t s . . . d o n e w r i t i n g a d d i t i o n a l f i l e s . . . ( 0 m o d u l e c o d e p a g e s ) g e n i n d e x s e a r c h c o p y i n g s t a t i c f i l e s . . . W A R N I N G : h t m l _ s t a t i c _ p a t h e n t r y u ' / h o m e / g r i g i / c o d i n g / i p y / d o c s / _ s t a t i c ' d o e s n o t e x i s t d o n e c o p y i n g e x t r a f i l e s . . . d o n e d u m p i n g s e a r c h i n d e x . . . d o n e d u m p i n g o b j e c t i n v e n t o r y . . . d o n e b u i l d s u c c e e d e d , 1 w a r n i n g .
24. Simple Example - cont Lets have a look at the generated docs: Initial docs - DEMO (docs/html1/index.html)
25. Is that it? - Where are my docs? Sphinx doesn’t do any of the magic that Doxygen/JavaDoc does for you. It doesn’t want to constrain what your documentation can be. It allows you to “script” your documentation.
26. Sphinx Plugins Many forms: One can control the documentation generated using directives. Some libraries will generate reST documentation that you can include in the documentation project. . . t o c t r e e : : : m a x d e p t h : 2 i n t r o s t r i n g s ( m a n y m o r e d o c u m e n t s l i s t e d h e r e ) . . i n c l u d e : : b e h a v e / s t e p s _ u s a g e . r s t
27. Important Sphinx Plugins autodoc This extension can import the modules you are documenting, and pull in documentation from docstrings in a semi-automatic way. The docstrings must of course be written in correct reStructuredText. You can then use all of the usual Sphinx markup in the docstrings. autodoc Documentation (sphinx-doc.org/ext/autodoc.html) doctests This extension allows you to test snippets in the documentation in a natural way. It works by collecting specially-marked up code blocks and running them as tests when building the docs. doctest Documentation (sphinx-doc.org/ext/doctest.html)
28. autodoc - helpers I n [ 2 0 ] : ! s p h i n x - a p i d o c - o d o c s d o c s a m p l e C r e a t i n g f i l e d o c s / d o c s a m p l e . r s t . C r e a t i n g f i l e d o c s / m o d u l e s . r s t . I n [ 2 1 ] : % c a t d o c s / d o c s a m p l e . r s t d o c s a m p l e p a c k a g e = = = = = = = = = = = = = = = = = S u b m o d u l e s - - - - - - - - - - d o c s a m p l e . a h a m o d u l e - - - - - - - - - - - - - - - - - - - - . . a u t o m o d u l e : : d o c s a m p l e . a h a : m e m b e r s : : u n d o c - m e m b e r s : : s h o w - i n h e r i t a n c e : M o d u l e c o n t e n t s - - - - - - - - - - - - - - -
29. autodoc - sample Add modules to the root toctree: Then run sphinx-build: . . t o c t r e e : : : m a x d e p t h : 4 m o d u l e s I n [ 2 3 ] : ! s p h i n x - b u i l d d o c s d o c s / h t m l 2 M a k i n g o u t p u t d i r e c t o r y . . . R u n n i n g S p h i n x v 1 . 2 . 3 l o a d i n g p i c k l e d e n v i r o n m e n t . . . n o t y e t c r e a t e d N o b u i l d e r s e l e c t e d , u s i n g d e f a u l t : h t m l b u i l d i n g [ h t m l ] : t a r g e t s f o r 3 s o u r c e f i l e s t h a t a r e o u t o f d a t e u p d a t i n g e n v i r o n m e n t : 3 a d d e d , 0 c h a n g e d , 0 r e m o v e d l o o k i n g f o r n o w - o u t d a t e d f i l e s . . . n o n e f o u n d p i c k l i n g e n v i r o n m e n t . . . d o n e c h e c k i n g c o n s i s t e n c y . . . d o n e p r e p a r i n g d o c u m e n t s . . . d o n e w r i t i n g a d d i t i o n a l f i l e s . . . ( 1 m o d u l e c o d e p a g e s ) _ m o d u l e s / i n d e x g e n i n d e x p y - m o d i n d e x s e a r c h
30. autodoc - sample Lets have a look at the generated docs: Autodoc docs - DEMO (docs/html2/index.html)
31. So, what does this buy us? Some documentation is done in one place (as part of the code). Documentation is a bit easier to keep up-to-date, since it is directly part of the codebase. Documentation is hosted by the codebase, so there is a known place where to look for it. You could auto-generate docs as part of your staging/testing deployment.
32. Keeping docs up to date. Of course, this doesn't mean that docs will automatically be kept up-to-date, so effort must still be taken to keep docs up-to-date. Some methods are: Have more of the docs generate in more specific ways Implement doctests to raise a flag when your docs are out-of-date.
33. Generating docs o tests/code Using some of the demonstrated self-populating docstring methods, you could even have the property lazily run a test to get the output. You can also have a look at the various and helpful sphinx-contrib plugins. (https://bitbucket.org/birkenfeld/sphinx-contrib)
34. Doctests . . t e s t c o d e : : 1 + 1 # t h i s w i l l g i v e n o o u t p u t ! p r i n t 2 + 2 # t h i s w i l l g i v e o u t p u t . . t e s t o u t p u t : : 4 Or the simpler REPL-style: > > > p r i n t 3 + 3 9
35. Done! - for now... Nickolas Grigoriadis, Developer, Internet Solutions grigi (https://github.com/grigi/)