Slide 1

Slide 1 text

Exploring our Python Interpreter with CPython 3.6.0a3+ Stephane Wirtel EuroPython 2016 - Bilbao - 07/19/2016 1 / 49

Slide 2

Slide 2 text

I am ... Stephane Wirtel a #python lover not a CPython Core Dev just a contributor of CPython, Gunicorn fellow @thePSF member of the EuroPython Society 1 / 49

Slide 3

Slide 3 text

Reminder It's just an introduction I am not a CPython Core Dev, just a contributor 2 / 49

Slide 4

Slide 4 text

Schedule How to start with CPython ? What's the result of python -c "2 + 2"? Summary 3 / 49

Slide 5

Slide 5 text

How to start ? good question... maybe with ... 4 / 49

Slide 6

Slide 6 text

Start with - DevGuide Developer's Guide Comprehensive resource for contributing to Python Quick Start Grammar Design of the CPython Compiler ... https://docs.python.org/devguide 5 / 49

Slide 7

Slide 7 text

Start with - DevGuide - Mentorship Mentors Provide an open and welcoming place to connect students, programmers and anyone interested in contributing to the Python Core development. Try to maintain contributor interest One mailing list Mentors: Guido Van Rossum Brett Cannon David R. Murray Victor Stinner ... myself ... ;-) ... http://www.pythonmentors.com 6 / 49

Slide 8

Slide 8 text

Start with - DevGuide - Mentorship - Mailing Lists The Mailing Lists Python Bugs Announce: ML/new-bugs-announce Python Bugs: ML/python-bugs-list Python Core Mentorship: ML/core-mentorship Python Dev: ML/python-dev Python Ideas: ML/python-ideas FAT Python Python Speed: ML/speed ML = https://mail.python.org/mailman/listinfo 7 / 49

Slide 9

Slide 9 text

Start with - DevGuide - Mentorship - Mailing Lists - Contributing Step 1 Create a user account on https://bugs.python.org Sign a Contributor Agreement after that, you can ... 8 / 49

Slide 10

Slide 10 text

Start with - DevGuide - Mentorship - Mailing Lists - Contributing Step 2 Improve the docs Fill Issues if you find a bug Test the proposed patches (Review) Write Patches (Fixes, features) be stressed and now, wait 6 months for a review ;-) BUT the process is really slow and laborious :/ We need reviewers and developers FYI: Migrating from hg.python.org to Github PEP512 9 / 49

Slide 11

Slide 11 text

Ok and now? 10 / 49

Slide 12

Slide 12 text

The directories of Python or where can we find the information? 11 / 49

Slide 13

Slide 13 text

Overview of the directories . ├── D o c = = > t h e F i n e M a n u a l . . . ├── G r a m m a r = = > W h e r e t h e g r a m m a r i s d e f i n e d ├── I n c l u d e = = > t h e C H e a d e r s ├── L i b = = > t h e . p y m o d u l e s ├── M o d u l e s = = > t h e . c m o d u l e s ├── O b j e c t s = = > t h e b u i l t i n o b j e c t s ( b o o l , d i c t , s e t , t u p l e , l i s t , u n i c o d e . . . ) ├── P a r s e r = = > g r a m m a r , l e x e r , p a r s e r , c o m p i l e r ├── P r o g r a m s = = > t h e p y t h o n e x e c u t a b l e └── P y t h o n = = > t h e v i r t u a l m a c h i n e 12 / 49

Slide 14

Slide 14 text

About the documentation Language Reference Library Reference Python/C API reference (PyObject, ...) 13 / 49

Slide 15

Slide 15 text

About the documentation Do YOU want to contribute ??? here is a small idea: We need a tutorial on AsyncIO and how to use it because the documentation of AsyncIO is useful but it's not a tutorial but a REFERENCE. 14 / 49

Slide 16

Slide 16 text

I have one question ! 15 / 49

Slide 17

Slide 17 text

yes, really, just one 16 / 49

Slide 18

Slide 18 text

Give me the result of python -c "x = 2 + 2" ? 17 / 49

Slide 19

Slide 19 text

x = 4 18 / 49

Slide 20

Slide 20 text

are you sure ? 19 / 49

Slide 21

Slide 21 text

Answer Command line (CLI) Parsing Compiler Interpreter 20 / 49

Slide 22

Slide 22 text

CLI Command Line # T h e P y t h o n e x e c u t a b l e P r o g r a m / p y t h o n . c : m a i n # T h e P y t h o n l i b r a r y - > M o d u l e s / m a i n . c : P y _ M a i n # I n i t i a l i z e C P y t h o n - > P y I n i t i a l i z e - > P y R u n _ S i m p l e S t r i n g F l a g s - > P y I m p o r t _ A d d M o d u l e - > P y R u n _ S t r i n g F l a g s - > P y P a r s e r _ A S T F r o m S t r i n g O b j e c t - > P y P a r s e r _ P a r s e S t r i n g O b j e c t - > P y A S T _ C o m p i l e O b j e c t - > P y E v a l _ E v a l C o d e 21 / 49

Slide 23

Slide 23 text

CLI Parsing - Lexer Lexer t o k e n s = l e x e r ( ' x = 2 + 2 ' ) > > > s t a t e m e n t = ' x = 2 + 2 ' > > > s t r I O = i o . S t r i n g I O ( s t a t e m e n t ) . r e a d l i n e > > > t o k e n s = t o k e n i z e . g e n e r a t e _ t o k e n s ( s t r I O ) > > > p p ( l i s t ( t o k e n i z e . g e n e r a t e _ t o k e n s ( t o k e n s ) ) ) [ T o k e n I n f o ( t y p e = N A M E , s t r i n g = ' x ' ) , T o k e n I n f o ( t y p e = O P , s t r i n g = ' = ' ) , T o k e n I n f o ( t y p e = N U M B E R , s t r i n g = ' 2 ' ) , T o k e n I n f o ( t y p e = O P , s t r i n g = ' + ' ) , T o k e n I n f o ( t y p e = N U M B E R , s t r i n g = ' 2 ' ) , T o k e n I n f o ( t y p e = E N D M A R K E R , s t r i n g = ' ' ) , ] in P a r s e r / t o k e n i z e r . c -> P y T o k e n i z e r _ F r o m S t r i n g in P a r s e r / p a r s e t o k . c -> p a r s e t o k in L i b / t o k e n i z e . p y 22 / 49

Slide 24

Slide 24 text

CLI Parsing - Lexer Lexer Mysterious async keyword in Python 3.5 > > > i m p o r t k e y w o r d > > > ' a s y n c ' i n k e y w o r d . k w l i s t F a l s e Proof > > > s t a t e m e n t = ' a s y n c = T r u e ' > > > s t r I O = i o . S t r i n g I O ( s t a t e m e n t ) . r e a d l i n e > > > t o k e n s = t o k e n i z e . g e n e r a t e _ t o k e n s ( s t r I O ) > > > p p ( l i s t ( t o k e n s ) ) [ T o k e n I n f o ( t y p e = N A M E , s t r i n g = ' a s y n c ' ) , T o k e n I n f o ( t y p e = O P , s t r i n g = ' = ' ) , T o k e n I n f o ( t y p e = N A M E , s t r i n g = ' T r u e ' ) , T o k e n I n f o ( t y p e = E N D M A R K E R , s t r i n g = ' ' ) ] 23 / 49

Slide 25

Slide 25 text

CLI Parsing - Lexer - Parser Parser a s t = p a r s e r ( t o k e n s ) > > > i m p o r t a s t > > > t r e e = a s t . p a r s e ( ' x = 2 + 2 ' ) > > > a s t . d u m p ( t r e e ) M o d u l e ( b o d y = [ A s s i g n ( t a r g e t s = [ N a m e ( i d = ' x ' ) ] , v a l u e = B i n O p ( l e f t = N u m ( n = 2 ) , o p = A d d ( ) , r i g h t = N u m ( n = 2 ) ) ) ] ) P y t h o n / p y t h o n r u n . c -> P y P a r s e r _ A S T F r o m S t r i n g O b j e c t 24 / 49

Slide 26

Slide 26 text

CLI Parsing Compiler - Compiler Compiler b y t e c o d e s = c o m p i l e r ( a s t ) > > > i m p o r t d i s > > > i m p o r t a s t > > > t r e e = a s t . p a r s e ( ' x = 2 + 2 ' ) > > > c o d e _ o b j e c t = c o m p i l e ( t r e e , ' d u m m y . p y ' , m o d e = ' e x e c ' ) > > > d i s . d i s ( c o d e _ o b j e c t ) 1 0 L O A D _ C O N S T 2 ( 4 ) 3 S T O R E _ N A M E 0 ( x ) 6 L O A D _ C O N S T 1 ( N o n e ) 9 R E T U R N _ V A L U E See PEP-0339 in P y t h o n / c o m p i l e . c -> P y A S T _ C o m p i l e O b j e c t 25 / 49

Slide 27

Slide 27 text

CLI Parsing Compiler - Compiler - Bytecodes Bytecodes compact numeric codes portable code one byte followed by optional parameters used by a software interpreter # d e f i n e P O P _ T O P 1 # d e f i n e R O T _ T W O 2 . . . # d e f i n e B I N A R Y _ A D D 2 3 # d e f i n e B I N A R Y _ S U B S T R A C T 2 4 . . . # d e f i n e S E T U P _ A S Y N C _ W I T H 1 5 4 # d e f i n e F O R M A T _ V A L U E 1 5 5 See in I n c l u d e / o p c o d e . h 26 / 49

Slide 28

Slide 28 text

CLI Parsing Compiler - Compiler - Bytecodes Bytecodes in / t m p / e m p t y . p y # ! / u s r / b i n / e n v p y t h o n # a s u p e r c o m m e n t Here is the bytecode 1 0 L O A D _ C O N S T 0 ( N o n e ) 3 R E T U R N _ V A L U E 27 / 49

Slide 29

Slide 29 text

CLI Parsing Compiler - Compiler - Bytecodes Bytecodes > > > d e f f u n c ( x ) : . . . x = x + 1 . . . r e t u r n x . . . > > > d i s . d i s ( f u n c ) 2 0 L O A D _ F A S T 0 ( x ) 3 L O A D _ C O N S T 1 ( 1 ) 6 B I N A R Y _ A D D 7 S T O R E _ F A S T 0 ( x ) 3 1 0 L O A D _ F A S T 0 ( x ) 1 3 R E T U R N _ V A L U E 28 / 49

Slide 30

Slide 30 text

CLI Parsing Compiler - Compiler - Bytecodes - Peepholer Peepholer b y t e c o d e s = o p t i m i z e r ( b y t e c o d e s ) performs basic peephole optimizations removes the dead branches in P y t h o n / p e e p h o l e . c -> P y C o d e _ O p t i m i z e 29 / 49

Slide 31

Slide 31 text

CLI Parsing Compiler - Compiler - Bytecodes - Peepholer Peepholer x = 2 + 2 1 0 L O A D _ C O N S T 2 ( 4 ) 3 S T O R E _ N A M E 0 ( x ) 6 L O A D _ C O N S T 1 ( N o n e ) 9 R E T U R N _ V A L U E 30 / 49

Slide 32

Slide 32 text

CLI Parsing Compiler - Compiler - Bytecodes - Peepholer Peepholer i f 1 : p r i n t ( ' h e l l o ' ) 1 0 L O A D _ N A M E 0 ( p r i n t ) 3 L O A D _ C O N S T 0 ( ' h e l l o ' ) 6 C A L L _ F U N C T I O N 1 ( 1 p o s i t i o n a l , 0 k e y w o r d p a i r ) 9 P O P _ T O P 1 0 L O A D _ C O N S T 1 ( N o n e ) 1 3 R E T U R N _ V A L U E 31 / 49

Slide 33

Slide 33 text

CLI Parsing Compiler - Compiler - Bytecodes - Peepholer Peepholer i f 0 : p r i n t ( ' h e l l o ' ) 1 0 L O A D _ C O N S T 0 ( N o n e ) 3 R E T U R N _ V A L U E 32 / 49

Slide 34

Slide 34 text

CLI Parsing Compiler Interpreter Interpreter e x e c u t i o n = i n t e r p r e t e r ( b y t e c o d e s ) Virtual Stack Machine Execute the ByteCode 33 / 49

Slide 35

Slide 35 text

CLI Parsing Compiler Interpreter Interpreter a Virtual Stack Machine c l a s s I n t e r p r e t e r : d e f _ _ i n i t _ _ ( s e l f , i n s t r u c t i o n s ) : s e l f . i n s t r u c t i o n s = i n s t r u c t i o n s s e l f . s t a c k = [ ] s e l f . i n s t _ p t r = 0 s e l f . r u n n i n g = T r u e d e f r u n ( s e l f ) : w h i l e s e l f . r u n n i n g : s e l f . e v a l ( s e l f . f e t c h ( ) ) s e l f . i n s t _ p t r + = 1 d e f f e t c h ( s e l f ) : r e t u r n s e l f . i n s t r u c t i o n s [ s e l f . i n s t _ p t r ] d e f e v a l ( s e l f , i n s t r u c t i o n ) : g e t a t t r ( s e l f , ' _ e v a l _ { } ' . f o r m a t ( i n s t r u c t i o n ) ) ( ) 34 / 49

Slide 36

Slide 36 text

CLI Parsing Compiler Interpreter Interpreter a Virtual Stack Machine d e f _ e v a l _ P U S H ( s e l f ) : s e l f . i n s t _ p t r + = 1 s e l f . s t a c k . a p p e n d ( s e l f . i n s t r u c t i o n s [ s e l f . i n s t _ p t r ] ) d e f _ e v a l _ A D D ( s e l f ) : l e f t = s e l f . s t a c k . p o p ( ) r i g h t = s e l f . s t a c k . p o p ( ) r e s u l t = l e f t + r i g h t s e l f . s t a c k . a p p e n d ( r e s u l t ) d e f _ e v a l _ P O P ( s e l f ) : v a l u e = s e l f . s t a c k . p o p ( ) p r i n t ( " P o p p e d % r " % v a l u e ) d e f _ e v a l _ H A L T ( s e l f ) : s e l f . r u n n i n g = F a l s e 35 / 49

Slide 37

Slide 37 text

CLI Parsing Compiler Interpreter Interpreter Execution of my custom Bytecode P U S H 5 | - - - - - - - - - - - | P U S H 3 | | P U S H 1 0 | - - - - - - - - - - - | A D D A D D P O P 36 / 49

Slide 38

Slide 38 text

CLI Parsing Compiler Interpreter Interpreter Execution of my custom Bytecode - - > P U S H 5 | - - - - - - - - - - - | P U S H 3 | 5 | P U S H 1 0 | - - - - - - - - - - - | A D D A D D P O P 37 / 49

Slide 39

Slide 39 text

CLI Parsing Compiler Interpreter Interpreter Execution of my custom Bytecode P U S H 5 | - - - - - - - - - - - | - - > P U S H 3 | 3 | P U S H 1 0 | - - - - - - - - - - - | A D D | 5 | A D D | - - - - - - - - - - - | P O P 38 / 49

Slide 40

Slide 40 text

CLI Parsing Compiler Interpreter Interpreter Execution of my custom Bytecode P U S H 5 | - - - - - - - - - - - | P U S H 3 | 1 0 | - - > P U S H 1 0 | - - - - - - - - - - - | A D D | 3 | A D D | - - - - - - - - - - - | P O P | 5 | | - - - - - - - - - - - | 39 / 49

Slide 41

Slide 41 text

CLI Parsing Compiler Interpreter Interpreter Execution of my custom Bytecode P U S H 5 | - - - - - - - - - - - | P U S H 3 | 1 3 | P U S H 1 0 | - - - - - - - - - - - | - - > A D D | 5 | A D D | - - - - - - - - - - - | P O P 40 / 49

Slide 42

Slide 42 text

CLI Parsing Compiler Interpreter Interpreter Execution of my custom Bytecode P U S H 5 | - - - - - - - - - - - | P U S H 3 | 1 8 | P U S H 1 0 | - - - - - - - - - - - | A D D - - > A D D P O P 41 / 49

Slide 43

Slide 43 text

CLI Parsing Compiler Interpreter Interpreter Execution of my custom Bytecode P U S H 5 | - - - - - - - - - - - | P U S H 3 | | P U S H 1 0 | - - - - - - - - - - - | A D D A D D - - > P O P 42 / 49

Slide 44

Slide 44 text

CLI Parsing Compiler Interpreter Interpreter Do you remember this function ? > > > d e f f u n c ( x ) : . . . x = x + 1 . . . r e t u r n x . . . And the Bytecode ? > > > d i s . d i s ( f u n c ) 2 0 L O A D _ F A S T 0 ( x ) 3 L O A D _ C O N S T 1 ( 1 ) 6 B I N A R Y _ A D D 7 S T O R E _ F A S T 0 ( x ) 3 1 0 L O A D _ F A S T 0 ( x ) 1 3 R E T U R N _ V A L U E 43 / 49

Slide 45

Slide 45 text

CLI Parsing Compiler Interpreter Interpreter 2 0 L O A D _ F A S T 0 ( x ) 3 L O A D _ C O N S T 1 ( 1 ) T A R G E T ( L O A D _ F A S T ) { P y O b j e c t * v a l u e = G E T L O C A L ( o p a r g ) ; i f ( v a l u e = = N U L L ) { f o r m a t _ e x c _ c h e c k _ a r g ( P y E x c _ U n b o u n d L o c a l E r r o r , U N B O U N D L O C A L _ E R R O R _ M S G , P y T u p l e _ G e t I t e m ( c o - > c o _ v a r n a m e s , o p a r g ) ) ; g o t o e r r o r ; } P y _ I N C R E F ( v a l u e ) ; P U S H ( v a l u e ) ; F A S T _ D I S P A T C H ( ) ; } T A R G E T ( L O A D _ C O N S T ) { P y O b j e c t * v a l u e = G E T I T E M ( c o n s t s , o p a r g ) ; P y _ I N C R E F ( v a l u e ) ; P U S H ( v a l u e ) ; F A S T _ D I S P A T C H ( ) ; } 44 / 49

Slide 46

Slide 46 text

CLI Parsing Compiler Interpreter Interpreter 6 B I N A R Y _ A D D T A R G E T ( B I N A R Y _ A D D ) { P y O b j e c t * r i g h t = P O P ( ) ; P y O b j e c t * l e f t = T O P ( ) ; P y O b j e c t * s u m ; i f ( P y U n i c o d e _ C h e c k E x a c t ( l e f t ) & & P y U n i c o d e _ C h e c k E x a c t ( r i g h t ) ) { s u m = u n i c o d e _ c o n c a t e n a t e ( l e f t , r i g h t , f , n e x t _ i n s t r ) ; } e l s e { s u m = P y N u m b e r _ A d d ( l e f t , r i g h t ) ; P y _ D E C R E F ( l e f t ) ; } P y _ D E C R E F ( r i g h t ) ; S E T _ T O P ( s u m ) ; i f ( s u m = = N U L L ) g o t o e r r o r ; D I S P A T C H ( ) ; } 45 / 49

Slide 47

Slide 47 text

CLI Parsing Compiler Interpreter Interpreter 7 S T O R E _ F A S T 0 ( x ) 1 3 R E T U R N _ V A L U E T A R G E T ( S T O R E _ F A S T ) { P y O b j e c t * v a l u e = P O P ( ) ; S E T L O C A L ( o p a r g , v a l u e ) ; F A S T _ D I S P A T C H ( ) ; } T A R G E T ( R E T U R N _ V A L U E ) { r e t v a l = P O P ( ) ; w h y = W H Y _ R E T U R N ; g o t o f a s t _ b l o c k _ e n d ; } 46 / 49

Slide 48

Slide 48 text

CLI Parsing Compiler Interpreter Interpreter And just for the fun, Everything is in P y t h o n / c e v a l . c but especially in one function P y E v a l _ E v a l F r a m e E x 7 9 6 P y O b j e c t * 7 9 7 P y E v a l _ E v a l F r a m e E x ( P y F r a m e O b j e c t * f , i n t t h r o w f l a g ) 7 9 8 { . . . 1 8 9 9 # i f d e f C A S E _ T O O _ B I G 1 9 0 0 d e f a u l t : s w i t c h ( o p c o d e ) { 1 9 0 1 # e n d i f . . . 3 6 4 4 } 47 / 49

Slide 49

Slide 49 text

Summary 48 / 49

Slide 50

Slide 50 text

Really Fun 49 / 49

Slide 51

Slide 51 text

49 / 49

Slide 52

Slide 52 text

Thank you 49 / 49

Slide 53

Slide 53 text

Questions ? http://wirtel.be [email protected] @matrixise github.com/matrixise 49 / 49

Slide 54

Slide 54 text

Credits https://igor.io/2013/08/28/stack-machines-fundamentals.html https://www.flickr.com/photos/djll/3006780451/ 49 / 49