Exploring our Python Interpreter

Presented at
* PyCon Ireland 2015
* PyCon Canada 2015
* PythonFOSDEM 2016
* EuroPython 2016

Transcript

1. Exploring our Python Interpreter with CPython 3.6.0a3+ Stephane Wirtel EuroPython 2016 - Bilbao - 07/19/2016 1 / 49
2. I am ... Stephane Wirtel a #python lover not a CPython Core Dev just a contributor of CPython, Gunicorn fellow @thePSF member of the EuroPython Society 1 / 49
3. Reminder It's just an introduction I am not a CPython Core Dev, just a contributor 2 / 49
4. Schedule How to start with CPython ? What's the result of python -c "2 + 2"? Summary 3 / 49
5. How to start ? good question... maybe with ... 4 / 49
6. Start with - DevGuide Developer's Guide Comprehensive resource for contributing to Python Quick Start Grammar Design of the CPython Compiler ... https://docs.python.org/devguide 5 / 49
7. Start with - DevGuide - Mentorship Mentors Provide an open and welcoming place to connect students, programmers and anyone interested in contributing to the Python Core development. Try to maintain contributor interest One mailing list Mentors: Guido Van Rossum Brett Cannon David R. Murray Victor Stinner ... myself ... ;-) ... http://www.pythonmentors.com 6 / 49
8. Start with - DevGuide - Mentorship - Mailing Lists The Mailing Lists Python Bugs Announce: ML/new-bugs-announce Python Bugs: ML/python-bugs-list Python Core Mentorship: ML/core-mentorship Python Dev: ML/python-dev Python Ideas: ML/python-ideas FAT Python Python Speed: ML/speed ML = https://mail.python.org/mailman/listinfo 7 / 49
9. Start with - DevGuide - Mentorship - Mailing Lists - Contributing Step 1 Create a user account on https://bugs.python.org Sign a Contributor Agreement after that, you can ... 8 / 49
10. Start with - DevGuide - Mentorship - Mailing Lists - Contributing Step 2 Improve the docs Fill Issues if you find a bug Test the proposed patches (Review) Write Patches (Fixes, features) be stressed and now, wait 6 months for a review ;-) BUT the process is really slow and laborious :/ We need reviewers and developers FYI: Migrating from hg.python.org to Github PEP512 9 / 49
11. Ok and now? 10 / 49
12. The directories of Python or where can we find the information? 11 / 49
13. Overview of the directories . ├── D o c = = > t h e F i n e M a n u a l . . . ├── G r a m m a r = = > W h e r e t h e g r a m m a r i s d e f i n e d ├── I n c l u d e = = > t h e C H e a d e r s ├── L i b = = > t h e . p y m o d u l e s ├── M o d u l e s = = > t h e . c m o d u l e s ├── O b j e c t s = = > t h e b u i l t i n o b j e c t s ( b o o l , d i c t , s e t , t u p l e , l i s t , u n i c o d e . . . ) ├── P a r s e r = = > g r a m m a r , l e x e r , p a r s e r , c o m p i l e r ├── P r o g r a m s = = > t h e p y t h o n e x e c u t a b l e └── P y t h o n = = > t h e v i r t u a l m a c h i n e 12 / 49
14. About the documentation Language Reference Library Reference Python/C API reference (PyObject, ...) 13 / 49
15. About the documentation Do YOU want to contribute ??? here is a small idea: We need a tutorial on AsyncIO and how to use it because the documentation of AsyncIO is useful but it's not a tutorial but a REFERENCE. 14 / 49
16. I have one question ! 15 / 49
17. yes, really, just one 16 / 49
18. Give me the result of python -c "x = 2 + 2" ? 17 / 49
19. x = 4 18 / 49
20. are you sure ? 19 / 49
21. Answer Command line (CLI) Parsing Compiler Interpreter 20 / 49
22. CLI Command Line # T h e P y t h o n e x e c u t a b l e P r o g r a m / p y t h o n . c : m a i n # T h e P y t h o n l i b r a r y - > M o d u l e s / m a i n . c : P y _ M a i n # I n i t i a l i z e C P y t h o n - > P y I n i t i a l i z e - > P y R u n _ S i m p l e S t r i n g F l a g s - > P y I m p o r t _ A d d M o d u l e - > P y R u n _ S t r i n g F l a g s - > P y P a r s e r _ A S T F r o m S t r i n g O b j e c t - > P y P a r s e r _ P a r s e S t r i n g O b j e c t - > P y A S T _ C o m p i l e O b j e c t - > P y E v a l _ E v a l C o d e 21 / 49
23. CLI Parsing - Lexer Lexer t o k e n s = l e x e r ( ' x = 2 + 2 ' ) > > > s t a t e m e n t = ' x = 2 + 2 ' > > > s t r I O = i o . S t r i n g I O ( s t a t e m e n t ) . r e a d l i n e > > > t o k e n s = t o k e n i z e . g e n e r a t e _ t o k e n s ( s t r I O ) > > > p p ( l i s t ( t o k e n i z e . g e n e r a t e _ t o k e n s ( t o k e n s ) ) ) [ T o k e n I n f o ( t y p e = N A M E , s t r i n g = ' x ' ) , T o k e n I n f o ( t y p e = O P , s t r i n g = ' = ' ) , T o k e n I n f o ( t y p e = N U M B E R , s t r i n g = ' 2 ' ) , T o k e n I n f o ( t y p e = O P , s t r i n g = ' + ' ) , T o k e n I n f o ( t y p e = N U M B E R , s t r i n g = ' 2 ' ) , T o k e n I n f o ( t y p e = E N D M A R K E R , s t r i n g = ' ' ) , ] in P a r s e r / t o k e n i z e r . c -> P y T o k e n i z e r _ F r o m S t r i n g in P a r s e r / p a r s e t o k . c -> p a r s e t o k in L i b / t o k e n i z e . p y 22 / 49
24. CLI Parsing - Lexer Lexer Mysterious async keyword in Python 3.5 > > > i m p o r t k e y w o r d > > > ' a s y n c ' i n k e y w o r d . k w l i s t F a l s e Proof > > > s t a t e m e n t = ' a s y n c = T r u e ' > > > s t r I O = i o . S t r i n g I O ( s t a t e m e n t ) . r e a d l i n e > > > t o k e n s = t o k e n i z e . g e n e r a t e _ t o k e n s ( s t r I O ) > > > p p ( l i s t ( t o k e n s ) ) [ T o k e n I n f o ( t y p e = N A M E , s t r i n g = ' a s y n c ' ) , T o k e n I n f o ( t y p e = O P , s t r i n g = ' = ' ) , T o k e n I n f o ( t y p e = N A M E , s t r i n g = ' T r u e ' ) , T o k e n I n f o ( t y p e = E N D M A R K E R , s t r i n g = ' ' ) ] 23 / 49
25. CLI Parsing - Lexer - Parser Parser a s t = p a r s e r ( t o k e n s ) > > > i m p o r t a s t > > > t r e e = a s t . p a r s e ( ' x = 2 + 2 ' ) > > > a s t . d u m p ( t r e e ) M o d u l e ( b o d y = [ A s s i g n ( t a r g e t s = [ N a m e ( i d = ' x ' ) ] , v a l u e = B i n O p ( l e f t = N u m ( n = 2 ) , o p = A d d ( ) , r i g h t = N u m ( n = 2 ) ) ) ] ) P y t h o n / p y t h o n r u n . c -> P y P a r s e r _ A S T F r o m S t r i n g O b j e c t 24 / 49
26. CLI Parsing Compiler - Compiler Compiler b y t e c o d e s = c o m p i l e r ( a s t ) > > > i m p o r t d i s > > > i m p o r t a s t > > > t r e e = a s t . p a r s e ( ' x = 2 + 2 ' ) > > > c o d e _ o b j e c t = c o m p i l e ( t r e e , ' d u m m y . p y ' , m o d e = ' e x e c ' ) > > > d i s . d i s ( c o d e _ o b j e c t ) 1 0 L O A D _ C O N S T 2 ( 4 ) 3 S T O R E _ N A M E 0 ( x ) 6 L O A D _ C O N S T 1 ( N o n e ) 9 R E T U R N _ V A L U E See PEP-0339 in P y t h o n / c o m p i l e . c -> P y A S T _ C o m p i l e O b j e c t 25 / 49
27. CLI Parsing Compiler - Compiler - Bytecodes Bytecodes compact numeric codes portable code one byte followed by optional parameters used by a software interpreter # d e f i n e P O P _ T O P 1 # d e f i n e R O T _ T W O 2 . . . # d e f i n e B I N A R Y _ A D D 2 3 # d e f i n e B I N A R Y _ S U B S T R A C T 2 4 . . . # d e f i n e S E T U P _ A S Y N C _ W I T H 1 5 4 # d e f i n e F O R M A T _ V A L U E 1 5 5 See in I n c l u d e / o p c o d e . h 26 / 49
28. CLI Parsing Compiler - Compiler - Bytecodes Bytecodes in / t m p / e m p t y . p y # ! / u s r / b i n / e n v p y t h o n # a s u p e r c o m m e n t Here is the bytecode 1 0 L O A D _ C O N S T 0 ( N o n e ) 3 R E T U R N _ V A L U E 27 / 49
29. CLI Parsing Compiler - Compiler - Bytecodes Bytecodes > > > d e f f u n c ( x ) : . . . x = x + 1 . . . r e t u r n x . . . > > > d i s . d i s ( f u n c ) 2 0 L O A D _ F A S T 0 ( x ) 3 L O A D _ C O N S T 1 ( 1 ) 6 B I N A R Y _ A D D 7 S T O R E _ F A S T 0 ( x ) 3 1 0 L O A D _ F A S T 0 ( x ) 1 3 R E T U R N _ V A L U E 28 / 49
30. CLI Parsing Compiler - Compiler - Bytecodes - Peepholer Peepholer b y t e c o d e s = o p t i m i z e r ( b y t e c o d e s ) performs basic peephole optimizations removes the dead branches in P y t h o n / p e e p h o l e . c -> P y C o d e _ O p t i m i z e 29 / 49
31. CLI Parsing Compiler - Compiler - Bytecodes - Peepholer Peepholer x = 2 + 2 1 0 L O A D _ C O N S T 2 ( 4 ) 3 S T O R E _ N A M E 0 ( x ) 6 L O A D _ C O N S T 1 ( N o n e ) 9 R E T U R N _ V A L U E 30 / 49
32. CLI Parsing Compiler - Compiler - Bytecodes - Peepholer Peepholer i f 1 : p r i n t ( ' h e l l o ' ) 1 0 L O A D _ N A M E 0 ( p r i n t ) 3 L O A D _ C O N S T 0 ( ' h e l l o ' ) 6 C A L L _ F U N C T I O N 1 ( 1 p o s i t i o n a l , 0 k e y w o r d p a i r ) 9 P O P _ T O P 1 0 L O A D _ C O N S T 1 ( N o n e ) 1 3 R E T U R N _ V A L U E 31 / 49
33. CLI Parsing Compiler - Compiler - Bytecodes - Peepholer Peepholer i f 0 : p r i n t ( ' h e l l o ' ) 1 0 L O A D _ C O N S T 0 ( N o n e ) 3 R E T U R N _ V A L U E 32 / 49
34. CLI Parsing Compiler Interpreter Interpreter e x e c u t i o n = i n t e r p r e t e r ( b y t e c o d e s ) Virtual Stack Machine Execute the ByteCode 33 / 49
35. CLI Parsing Compiler Interpreter Interpreter a Virtual Stack Machine c l a s s I n t e r p r e t e r : d e f _ _ i n i t _ _ ( s e l f , i n s t r u c t i o n s ) : s e l f . i n s t r u c t i o n s = i n s t r u c t i o n s s e l f . s t a c k = [ ] s e l f . i n s t _ p t r = 0 s e l f . r u n n i n g = T r u e d e f r u n ( s e l f ) : w h i l e s e l f . r u n n i n g : s e l f . e v a l ( s e l f . f e t c h ( ) ) s e l f . i n s t _ p t r + = 1 d e f f e t c h ( s e l f ) : r e t u r n s e l f . i n s t r u c t i o n s [ s e l f . i n s t _ p t r ] d e f e v a l ( s e l f , i n s t r u c t i o n ) : g e t a t t r ( s e l f , ' _ e v a l _ { } ' . f o r m a t ( i n s t r u c t i o n ) ) ( ) 34 / 49
36. CLI Parsing Compiler Interpreter Interpreter a Virtual Stack Machine d e f _ e v a l _ P U S H ( s e l f ) : s e l f . i n s t _ p t r + = 1 s e l f . s t a c k . a p p e n d ( s e l f . i n s t r u c t i o n s [ s e l f . i n s t _ p t r ] ) d e f _ e v a l _ A D D ( s e l f ) : l e f t = s e l f . s t a c k . p o p ( ) r i g h t = s e l f . s t a c k . p o p ( ) r e s u l t = l e f t + r i g h t s e l f . s t a c k . a p p e n d ( r e s u l t ) d e f _ e v a l _ P O P ( s e l f ) : v a l u e = s e l f . s t a c k . p o p ( ) p r i n t ( " P o p p e d % r " % v a l u e ) d e f _ e v a l _ H A L T ( s e l f ) : s e l f . r u n n i n g = F a l s e 35 / 49
37. CLI Parsing Compiler Interpreter Interpreter Execution of my custom Bytecode P U S H 5 | - - - - - - - - - - - | P U S H 3 | | P U S H 1 0 | - - - - - - - - - - - | A D D A D D P O P 36 / 49
38. CLI Parsing Compiler Interpreter Interpreter Execution of my custom Bytecode - - > P U S H 5 | - - - - - - - - - - - | P U S H 3 | 5 | P U S H 1 0 | - - - - - - - - - - - | A D D A D D P O P 37 / 49
39. CLI Parsing Compiler Interpreter Interpreter Execution of my custom Bytecode P U S H 5 | - - - - - - - - - - - | - - > P U S H 3 | 3 | P U S H 1 0 | - - - - - - - - - - - | A D D | 5 | A D D | - - - - - - - - - - - | P O P 38 / 49
40. CLI Parsing Compiler Interpreter Interpreter Execution of my custom Bytecode P U S H 5 | - - - - - - - - - - - | P U S H 3 | 1 0 | - - > P U S H 1 0 | - - - - - - - - - - - | A D D | 3 | A D D | - - - - - - - - - - - | P O P | 5 | | - - - - - - - - - - - | 39 / 49
41. CLI Parsing Compiler Interpreter Interpreter Execution of my custom Bytecode P U S H 5 | - - - - - - - - - - - | P U S H 3 | 1 3 | P U S H 1 0 | - - - - - - - - - - - | - - > A D D | 5 | A D D | - - - - - - - - - - - | P O P 40 / 49
42. CLI Parsing Compiler Interpreter Interpreter Execution of my custom Bytecode P U S H 5 | - - - - - - - - - - - | P U S H 3 | 1 8 | P U S H 1 0 | - - - - - - - - - - - | A D D - - > A D D P O P 41 / 49
43. CLI Parsing Compiler Interpreter Interpreter Execution of my custom Bytecode P U S H 5 | - - - - - - - - - - - | P U S H 3 | | P U S H 1 0 | - - - - - - - - - - - | A D D A D D - - > P O P 42 / 49
44. CLI Parsing Compiler Interpreter Interpreter Do you remember this function ? > > > d e f f u n c ( x ) : . . . x = x + 1 . . . r e t u r n x . . . And the Bytecode ? > > > d i s . d i s ( f u n c ) 2 0 L O A D _ F A S T 0 ( x ) 3 L O A D _ C O N S T 1 ( 1 ) 6 B I N A R Y _ A D D 7 S T O R E _ F A S T 0 ( x ) 3 1 0 L O A D _ F A S T 0 ( x ) 1 3 R E T U R N _ V A L U E 43 / 49
45. CLI Parsing Compiler Interpreter Interpreter 2 0 L O A D _ F A S T 0 ( x ) 3 L O A D _ C O N S T 1 ( 1 ) T A R G E T ( L O A D _ F A S T ) { P y O b j e c t * v a l u e = G E T L O C A L ( o p a r g ) ; i f ( v a l u e = = N U L L ) { f o r m a t _ e x c _ c h e c k _ a r g ( P y E x c _ U n b o u n d L o c a l E r r o r , U N B O U N D L O C A L _ E R R O R _ M S G , P y T u p l e _ G e t I t e m ( c o - > c o _ v a r n a m e s , o p a r g ) ) ; g o t o e r r o r ; } P y _ I N C R E F ( v a l u e ) ; P U S H ( v a l u e ) ; F A S T _ D I S P A T C H ( ) ; } T A R G E T ( L O A D _ C O N S T ) { P y O b j e c t * v a l u e = G E T I T E M ( c o n s t s , o p a r g ) ; P y _ I N C R E F ( v a l u e ) ; P U S H ( v a l u e ) ; F A S T _ D I S P A T C H ( ) ; } 44 / 49
46. CLI Parsing Compiler Interpreter Interpreter 6 B I N A R Y _ A D D T A R G E T ( B I N A R Y _ A D D ) { P y O b j e c t * r i g h t = P O P ( ) ; P y O b j e c t * l e f t = T O P ( ) ; P y O b j e c t * s u m ; i f ( P y U n i c o d e _ C h e c k E x a c t ( l e f t ) & & P y U n i c o d e _ C h e c k E x a c t ( r i g h t ) ) { s u m = u n i c o d e _ c o n c a t e n a t e ( l e f t , r i g h t , f , n e x t _ i n s t r ) ; } e l s e { s u m = P y N u m b e r _ A d d ( l e f t , r i g h t ) ; P y _ D E C R E F ( l e f t ) ; } P y _ D E C R E F ( r i g h t ) ; S E T _ T O P ( s u m ) ; i f ( s u m = = N U L L ) g o t o e r r o r ; D I S P A T C H ( ) ; } 45 / 49
47. CLI Parsing Compiler Interpreter Interpreter 7 S T O R E _ F A S T 0 ( x ) 1 3 R E T U R N _ V A L U E T A R G E T ( S T O R E _ F A S T ) { P y O b j e c t * v a l u e = P O P ( ) ; S E T L O C A L ( o p a r g , v a l u e ) ; F A S T _ D I S P A T C H ( ) ; } T A R G E T ( R E T U R N _ V A L U E ) { r e t v a l = P O P ( ) ; w h y = W H Y _ R E T U R N ; g o t o f a s t _ b l o c k _ e n d ; } 46 / 49
48. CLI Parsing Compiler Interpreter Interpreter And just for the fun, Everything is in P y t h o n / c e v a l . c but especially in one function P y E v a l _ E v a l F r a m e E x 7 9 6 P y O b j e c t * 7 9 7 P y E v a l _ E v a l F r a m e E x ( P y F r a m e O b j e c t * f , i n t t h r o w f l a g ) 7 9 8 { . . . 1 8 9 9 # i f d e f C A S E _ T O O _ B I G 1 9 0 0 d e f a u l t : s w i t c h ( o p c o d e ) { 1 9 0 1 # e n d i f . . . 3 6 4 4 } 47 / 49
49. Summary 48 / 49
50. Really Fun 49 / 49
51. 49 / 49
52. Thank you 49 / 49
53. Questions ? http://wirtel.be stephane@wirtel.be @matrixise github.com/matrixise 49 / 49
54. Credits https://igor.io/2013/08/28/stack-machines-fundamentals.html https://www.flickr.com/photos/djll/3006780451/ 49 / 49