Skyfield and 15 Years of Bad APIs (Brandon Rhodes)

Transcript

1. Skyfield and 15 Years of Bad APIs @brandon_rhodes PyCon Canada August 2013
2. Goal To reflect upon the practice of Python API design through my recent work on the Skyfield astronomy library
6. Elwood Charles Downey et al 1990 Ephem 1993 XEphem
7. e·phem·er·is — A table giving the coordinates of a celestial body at a number of specific times
8. C D T 1 9 : 0 0 : 0 0 4 / 3 0 / 1 9 9 0 | L S T 8 : 1 9 : 5 0 | U T C 0 : 0 0 : 0 0 5 / 0 1 / 1 9 9 0 | | J u l i a n D a t 2 4 4 8 0 1 2 . 5 0 0 0 0 | D a w n 4 : 1 0 | W a t c h | D u s k 2 2 : 1 5 | L i s t i n g o f f | N i t e L n 5 : 5 5 | P l o t o f f | N S t e p 1 | M e n u P l a n e t D a t a | S t p S z R T C L O C K | - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - O C X R . A . D e c A z A l t H L o n g H L a t S u 2 : 3 2 . 3 1 4 : 5 8 2 7 8 : 4 0 1 2 : 3 8 2 2 0 : 2 2 M o 8 : 0 9 . 9 2 1 : 1 1 1 8 6 : 0 6 6 5 : 5 3 1 1 9 : 5 5 1 : 0 4 M e 2 : 4 9 . 4 1 7 : 3 9 2 7 7 : 4 8 1 7 : 2 6 2 1 4 : 0 8 1 : 4 3 V e 2 3 : 4 9 . 4 - 2 : 2 5 2 9 6 : 5 3 - 2 7 : 3 9 2 8 2 : 3 9 - 1 : 3 0 M a 2 2 : 3 9 . 8 - 1 0 : 0 9 3 0 8 : 1 7 - 4 4 : 1 4 2 9 7 : 5 6 - 1 : 4 3 J u 6 : 3 0 . 9 2 3 : 2 3 2 3 5 : 1 3 5 9 : 0 4 1 0 6 : 1 6 0 : 0 8 S a 1 9 : 4 9 . 6 - 2 0 : 5 3 1 7 : 2 4 - 6 5 : 1 4 2 8 9 : 4 5 0 : 1 0
10. © XEphem’s author reserves the right to distribute binaries, but allows free download of the source
11. Missing header (.h) X 1 1 / I n t r i n s i c . h : N o s u c h f i l e o r d i r e c t o r y . . . $ a p t - f i l e s e a r c h X 1 1 / I n t r i n s i c . h l i b x t - d e v : / u s r / i n c l u d e / X 1 1 / I n t r i n s i c . h Missing library (-l) / u s r / b i n / l d : c a n n o t f i n d - l X e x t . . . $ a p t - f i l e s e a r c h l i b X e x t . a l i b x e x t - d e v : / u s r / l i b / i 3 8 6 - l i n u x - g n u / l i b X e x t . a
13. Instead of using a GUI, I wanted to write scripts
14. Inside XEphem’s C-language source code is a computation engine called libastro
15. XEphem PyEphem libastro → libastro
16. “You have my full permission to go with what you have.” — Elwood’s generous reply!
17. PyEphem = C code Wrapper around libastro
18. 1998 Beazley’s Simple Wrapper Interface Generator (SWIG)
19. SWIG Exposed awkward details of C structs b o d y = O b j ( ) b o d y . a n y . t y p e = e p h e m . P L A N E T b o d y . p l . c o d e = e p h e m . S U N e p h e m . c o m p u t e L o c a t i o n ( c i r c u m , b o d y ) p r i n t e p h e m . f o r m a t H o u r s ( o . a n y . r a , 3 6 0 0 0 ) p r i n t e p h e m . f o r m a t D e g r e e s ( o . a n y . d e c , 3 6 0 0 )
20. 2003 Hand-written C that uses Python 2.2 superpowers
21. Python 2.2 made attribute access easy to customize s t a t i c P y G e t S e t D e f b o d y _ g e t s e t [ ] = { { " r a " , g e t _ r a , 0 , " r i g h t a s c e n s i o n " } , { " d e c " , g e t _ d e c , 0 , " d e c l i n a t i o n " } , { " e l o n g " , g e t _ e l o n g , 0 , " e l o n g a t i o n " } , { " m a g " , g e t _ m a g , 0 , " m a g n i t u d e " } , ⋮ }
22. And, I added a pure-Python wrapper on top like h a s h l i b , s q l i t e 3 , and s s l # e p h e m / _ _ i n i t _ _ . p y i m p o r t _ l i b a s t r o ⋮
23. More Pythonic edition of PyEphem m a r s = e p h e m . M a r s ( ) m a r s . c o m p u t e ( ) p r i n t m a r s . r a , m a r s . d e c
24. But, both interfaces were still based on C code and required compilation
25. Early 2000s
26. S u b j e c t : P y E p h e m W i n 3 2 b u i l d e r r o r s S u b j e c t : w i n 3 2 , d o e s P h E p h e m w o r k t h e r e t o o ? S u b j e c t : t r y i n g t o d o w n l o a d b u t I c a n t u n z i p i t
27. Late 2000s
28. S u b j e c t : p y e p h e m o n M a c P P C S u b j e c t : p y E p h e m w o n ’ t b u i l d o n S n o w L e o p a r d S u b j e c t : P y E p h e m … I n s t a l l a t i o n e r r o r i n o p e n s u s e S u b j e c t : P y E p h e m o n U b u n t u 1 0 . 1 0 S u b j e c t : P y e p h e m o n a 6 4 - b i t W i n 7 P C ?
29. Mac Sometimes a problem, but now on MacPorts! Windows p y t h o n s e t u p . p y b d i s t _ w i n i n s t mingw — Open, but quirky Visual Studio Express — Works!
31. C extensions They are difficult to— Install Distribute Maintain
32. I slowly grew open to the idea of an alternative approach
33. An email “I’m interested in ephemeris options for astrology apps. You probably know about the Swiss Ephemeris. Do you know how it compares in accuracy?”
34. libastro Predicts planetary positions using VSOP87 1987
35. Swiss Ephemeris “based upon the DE406,” the JPL Long Ephemeris 1997
36. But wait! # f t p : / / s s d . j p l . n a s a . g o v / p u b / e p h / p l a n e t s / a s c i i / N a m e D a t e M o d i f i e d ⋮ d e 4 0 5 / 1 0 / 7 / 0 7 8 : 0 0 : 0 0 P M d e 4 0 6 / 3 / 2 2 / 1 1 8 : 0 0 : 0 0 P M ⋮ d e 4 2 1 / 2 / 6 / 1 3 7 : 0 0 : 0 0 P M d e 4 2 2 / 8 / 3 / 1 1 8 : 0 0 : 0 0 P M d e 4 2 3 / 3 / 3 0 / 1 0 8 : 0 0 : 0 0 P M ⋮
37. DE421 More recent More accurate “planetary navigation accuracies”
38. Wait—what? DE421 uses the “International Celestial Reference Frame”
39. USNO Circular 179 George H. Kaplan 2005
40. “The … resolutions passed by the International Astronomical Union at its General Assemblies in 1997 and 2000 are the most significant set of international agreements in positional astronomy in several decades and arguably since the Paris conference of 1896.”
41. Time for a rewrite! Not a simple re-implementation But writing, in Python, a complete replacement of the old algorithms used throughout PyEphem
42. Goals Replace entire API Integrate with scientific Python Support vector NumPy computations Avoid re-inventing SciPy wheels
43. Example: Sunrise PyEphem — custom hand-written version of Newton’s Method New approach — an IPython Notebook that finds sunrise with s c i p y . o p t i m i z e
44. Skyfield
45. Tools git Jedi py.test tox
46. py.test p y . t e s t - - p y a r g s s k y f i e l d \ - - d o c t e s t - g l o b = ' * . r s t '
47. py.test a s s e r t c . c a l _ d a t e ( j d ) = = t i m e s c a l e s . c a l _ d a t e ( j d )
48. I distribute tests s e t u p ( . . . p a c k a g e s = [ ' s k y f i e l d ' , ' s k y f i e l d . t e s t s ' ] , . . . )
49. I distribute docs s e t u p ( . . . p a c k a g e _ d a t a = { ' s k y f i e l d ' : [ ' d o c u m e n t a t i o n / * . r s t ' ] } , . . . )
50. Licensing
51. GPL or MIT?
52. GPL F r e e w o r l d C l o s e d w o r l d ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ A w e s o m e ! L e s s a w e s o m e ↑ ↑ Y o u r l i b r a r y → × A l t e r n a t i v e ? R e w r i t e ?
53. MIT/BSD F r e e w o r l d C l o s e d w o r l d ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ A w e s o m e ! C l o s e d a w e s o m e ↑ ↑ Y o u r l i b r a r y → Y o u r l i b r a r y
54. (Look closely — Open Source everywhere!) F r e e w o r l d C l o s e d w o r l d ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ A w e s o m e ! C l o s e d a w e s o m e ↑ ↑ Y o u r l i b r a r y → Y o u r l i b r a r y
55. This is Python’s own model, and Python is increasingly everywhere
56. MIT/BSD The free kind of freedom is what wins
57. Technique
58. single code base that runs on both Python 2 and Python 3
59. Release small projects fast j p l e p h e m s g p 4
60. Carefully monitor my emotions
61. Emotions tell me whether I am programming right
62. Frustration? Clenching teeth? Trapped?
63. Then I might be doing it wrong
64. Frustration often signals inadequate project structure
65. I notice that tiny goals are easier to reach and keep me calm
66. “Test-driven development is a way of managing fear during programming.” — Kent Beck, Test Driven Development By Example
67. Version Control “How soon can I commit so that I can’t lose what I’ve just typed?”
68. Young “How can I split this task into smaller pieces for the computer?” Old “How can I split this task into smaller pieces for me?”
69. Big slog: risks everything! here →→→→→ new feature here → • → • → • → • → new feature Incremental: cheap to revert
70. Always use g i t s t a s h to revert and try again
71. Aim for a Moon Shot Write up an end-to-end example then move straight there
72. The sins of my APIs
73. Sin: Inscrutable names “Explicit is better than implicit” d = ' 2 0 1 2 / 1 1 / 9 ' m = e p h e m . m a r s ( d ) p r i n t ( m . a _ r a , m . a _ d e c ) # S k y f i e l d , i n s t e a d : d = J u l i a n D a t e ( ' 2 0 1 2 / 1 1 / 9 ' ) p = e a r t h ( d ) . o b s e r v e ( m a r s ) . a s t r o m e t r i c ( ) p r i n t ( p . r a , p . d e c )
74. Sin: storing results on object m a r s = e p h e m . M a r s ( ) m a r s . c o m p u t e ( ' 2 0 1 2 / 1 1 / 9 ' ) p r i n t ( m . r a , m . d e c ) # M a r s # . n a m e # . d a t e # . c o m p u t e ( ) → ↘ # . r a ↖ ↓ # . d e c ← ← ← ← ← ← ← ↙ # ⋮
75. # P y E p h e m : p o s i t i o n s = [ ] f o r d a t e i n d a t e s : m a r s . c o m p u t e ( d ) p o s i t i o n s . a p p e n d ( ( m a r s . r a , m a r s . d e c ) ) # S k y f i e l d : c o o r d s = [ m a r s ( d ) . a s t r o m e t r i c ( ) f o r d i n d a t e s ] p o s i t i o n s = [ ( c . r a , c . d e c ) f o r c i n c o o r d s ]
76. Look familiar?
77. l e t t e r s = l i s t ( s e t ( m e s s a g e _ s t r i n g ) ) l e t t e r s . s o r t ( ) p r i n t ' ' . j o i n ( l e t t e r s ) # v s p r i n t ' ' . j o i n ( s o r t e d ( s e t ( m e s s a g e _ s t r i n g ) ) )
78. o u t p u t s = s o r t e d ( i n p u t s ) Making code more functional is a big part of Pythonic
79. Sin: Concealing expense Your API is the only lifeline the programmer has to managing complexity and expense!
80. # P y E p h e m m = e p h e m . m a r s ( ' 2 0 1 2 / 1 1 / 9 ' ) p r i n t ( m . n a m e ) # z e r o w o r k p r i n t ( m . r a , m . d e c ) # c o m p u t e d p r i n t ( m . r i s e _ t i m e ) # e x p e n s i v e !
81. Guideline It’s okay to hide quick conveniences behind properties But expensive operations should always look like calls!
82. m a r s . n a m e # l o o k s c h e a p m a r s . a p p a r e n t ( ) # l o o k s e x p e n s i v e ! Use this difference to train your customers!
83. # “ I c a n u s e t h i s o v e r a n d o v e r a g a i n ! ” m a r s . n a m e # “ L o o k s e x p e n s i v e ; I ’ l l s a v e t h e # r e s u l t t o a l o c a l n a m e i n s t e a d . ” m a r s . a p p a r e n t ( )
84. The big lesson?
85. Write APIs that TEACH!
86. Write APIs that teach You know how your API works and how it can best be used
87. Share as much of that knowledge as possible
88. Write APIs that teach While an API should hide complexity it should also suggest best practices
89. Write APIs that teach r = u r l l i b 2 . u r l o p e n ( u r l ) # b a d r = r e q u e s t s . g e t ( u r l ) # g o o d
90. Why? r = r e q u e s t s . g e t ( u r l ) The very first line of r e q u e s t s code teaches users their first HTTP verb — without their even knowing it
91. PyEphem made every coordinate an attribute — looking exactly the same with no visible relationship! m a r s . a _ r a , m a r s . a _ d e c # a s t r o m e t r i c m a r s . g _ r a , m a r s . g _ d e c # a p p a r e n t g e o c e n t r i c m a r s . r a , m a r s . d e c # a p p a r e n t t o p o c e n t r i c m a r s . a l t , m a r s . a z # a p p a r e n t h o r i z o n t a l
92. Skyfield has you to build a result from smaller operations h e r e = t o r o n t o ( j d ) h e r e . o b s e r v e ( m a r s ) . a s t r o m e t r i c ( ) h e r e . o b s e r v e ( m a r s ) . a p p a r e n t ( ) . e q u a t o r i a l ( ) h e r e . o b s e r v e ( m a r s ) . a p p a r e n t ( ) . h o r i z o n t a l ( )
93. Sin: Confusing functions and methods Python support both procedural and object-based methods, but choosing can be difficult
94. PyEphem m = M a r s ( d a t e ) p r i n t ( c o n s t e l l a t i o n ( m . r a , m . d e c ) ) # T h i s f u n c t i o n c a n O N L Y E V E R b e p a s s e d # a r i g h t a s c e n s i o n a n d d e c l i n a t i o n ; w h y # n o t m a k e i t a c o o r d i n a t e m e t h o d ?
95. So here are some guidelines that I have been using lately
96. 1. Methods constitute Python’s built-in typecheck — the one clean way to limit the types that a function will accept as an argument
97. If you are tempted to start a function with i f i s i n s t a n c e ( … ) then you might want a method instead!
98. 2. f ( x ) should by definition touch only public features of an x
99. If f ( x ) needs internal details, then either make it a method itself, or create a method that does the internal manipulation for it
100. 3. f ( x ) should by definition not mutate the state of x (If you need to mutate state from outside, try the Adapter Pattern!)
101. PyEphem m = M a r s ( d a t e ) p r i n t m . r a # p r i n t s o n e t h i n g t o r o n t o . n e x t _ r i s i n g ( m ) p r i n t m . r a # s o m e t h i n g d i f f e r e n t !
102. 4. Methods are discoverable (Jedi!)
103. Skyfield: New tricks
104. What about when you do need to dynamically compute an attribute?
105. c l a s s S a m p l e ( o b j e c t ) : @ p r o p e r t y d e f l o u d ( s e l f ) : r e t u r n s e l f . m e s s a g e . u p p e r ( )
106. Problem: hidden expense So, we cache
107. c l a s s S a m p l e ( o b j e c t ) : _ l o u d = N o n e @ p r o p e r t y d e f l o u d ( s e l f ) : i f _ l o u d i s N o n e : s e l f . _ l o u d = s e l f . m e s s a g e . u p p e r ( ) r e t u r n s e l f . _ l o u d
108. Problem: makes every access more expensive
109. Solution: dunder-getattr! c l a s s S a m p l e ( o b j e c t ) : d e f _ _ g e t a t t r _ _ ( s e l f , n a m e ) : i f n a m e = = ' l o u d ' : s e l f . l o u d = s e l f . m e s s a g e . u p p e r ( ) r e t u r n s e l f . l o u d r a i s e A t t r i b u t e E r r o r ( )
110. Solution: dunder-getattr For a value frequently accessed in heavy numeric work, _ _ g e t a t t r _ _ ( ) runs only on the first lookup!
111. Trick: scalar style NumPy initially excited me by letting me write scalar code that also works on arrays!
112. d e f f ( x , y ) : r e t u r n s q r t ( x * x + y * y ) p r i n t ( f ( 3 , 4 ) ) # = > 5 x = a r r a y ( [ 3 , 8 , 6 0 ] ) y = a r r a y ( [ 4 , 6 , 8 0 ] ) p r i n t ( f ( x , y ) ) # = > a r r a y ( [ 5 , 1 0 , 1 0 0 ] )
113. # N u m b e r ! j d = t o d a y ( ) p = e a r t h ( j d ) . o b s e r v e ( p l a n e t ) # V e c t o r ! j d = d a t e _ r a n g e ( ' 1 9 8 0 / 1 / 1 ' , ' 2 0 1 0 / 1 / 1 ' , 1 . 0 ) p = e a r t h ( j d ) . o b s e r v e ( p l a n e t )
114. But scalar style is hard to maintain across a large code base! n = c o m p u t e _ n u t a t i o n ( j d ) p = c o m p u t e _ p r e c e s s i o n ( j d . t d b ) f = J 2 0 0 0 _ t o _ I C R S t = e i n s u m ( ' j i n , k j n - > i k n ' , n , p ) t = e i n s u m ( ' i j n , j k - > i k n ' , t , f ) p o s = e i n s u m ( ' i n , i j n - > j n ' , p o s , t ) v e l = e i n s u m ( ' i n , i j n - > j n ' , v e l , t )
115. One last hint Vi Hart says τ = 2ᵰ
116. Q: Should dunder-init import things? # _ _ i n i t _ _ . p y f r o m . e a r t h l i b i m p o r t T o p o s f r o m . p l a n e t s i m p o r t J u p i t e r to allow f r o m s k y f i e l d i m p o r t T o p o s , J u p i t e r
117. Pro Simple way to surface your primary interface
118. Cons Innocent i m p o r t s k y f i e l d winds up importing everything
119. Cons Imports do not match messages f r o m s k y f i e l d i m p o r t T o p o s p r i n t t y p e ( T o p o s ) # = > < c l a s s ' s k y f i e l d . c o o r d i n a t e s . T o p o s ' >
120. Skyfield: decided against Not a one-trick-pony library Teach package structure Can always s k y f i e l d . a p i
121. Opinionated defaults
122. Should Skyfield include an ephemeris by default? s e t u p ( ⋮ i n s t a l l _ r e q u i r e s = [ ' d e 4 2 1 ' ] , )
123. Avoid Contrived tests
124. How do you test both branches of the i f ? d e f f ( . . . ) : ⋮ # 2 0 l i n e s o f c o d e ⋮ i f f i n a l _ d e c i s i o n ( ) : a c t i o n 1 e l s e : a c t i o n 2
125. Do not contrive complex calls to f ( ) that exercise both branches Instead factor the function into smaller pieces that can be exercised separately
126. Final trick Choosing a support forum
127. Mailing list? Web forum? What happens 4 years later when someone has the same question?
128. Answer can be difficult to find within a long discussion thread
129. Answers go out of date
130. Stack Overflow! Answer lives on same page Out-of-date answers get fixed
131. The last mile
132. The last mile Lesson #1
133. New GitHub issue
134. “PyEphem works from ssh but it does not work from the web”
135. I was unhappy Hurts project reputation Issue says “you are wrong”
137. “Can you help me?”
139. “This question is not a good fit to our Q&A format”
140. “Can you help me?”
141. yes
142. i m p o r t s y s o p e n ( ' / t m p / e m e r g e n c y . l o g ' , ' w ' ) . w r i t e ( s t r ( s y s . p a t h ) + ' \ n ' )
143. i m p o r t s y s s y s . p a t h . a p p e n d ( ' / h o m e / a s t r o n o m i a / . l o c a l / l i b 6 4 ' ' / p y t h o n 2 . 6 / s i t e - p a c k a g e s ' ) It worked
144. “FYI the support solution is to keep that emergency code in every script”
146. “Brandon, thanks a lot for your help. PyEphem is great.”
147. The last mile Lesson #2
148. October 2012 — I put NOVAS on PyPI p i p i n s t a l l n o v a s
149. An email “I have been fiddling around with NOVAS_Py-3.1 and have had some problems…” (a list of bugs followed)
150. The temptation Not my library — not my problem Provide the email address of USNO? Forward it to them myself?
152. A s s i s t a n t D i r e c t o r f o r E x p l o r a t i o n . . . @ n a s a . g o v
153. USNO → me → NASA
154. The last mile
155. The last mile In open source, the final component of the API is very often you! You are the the API of last resort
156. The last mile In open source, the final component of the API is very often you! You are the the API of last resort @brandon_rhodes Thank you very much!