I am ... Stephane Wirtel a #python lover not a CPython Core Dev just a contributor of CPython, Gunicorn fellow @thePSF member of the EuroPython Society 1 / 49
Start with - DevGuide Developer's Guide Comprehensive resource for contributing to Python Quick Start Grammar Design of the CPython Compiler ... https://docs.python.org/devguide 5 / 49
Start with - DevGuide - Mentorship Mentors Provide an open and welcoming place to connect students, programmers and anyone interested in contributing to the Python Core development. Try to maintain contributor interest One mailing list Mentors: Guido Van Rossum Brett Cannon David R. Murray Victor Stinner ... myself ... ;-) ... http://www.pythonmentors.com 6 / 49
Start with - DevGuide - Mentorship - Mailing Lists - Contributing Step 1 Create a user account on https://bugs.python.org Sign a Contributor Agreement after that, you can ... 8 / 49
Start with - DevGuide - Mentorship - Mailing Lists - Contributing Step 2 Improve the docs Fill Issues if you find a bug Test the proposed patches (Review) Write Patches (Fixes, features) be stressed and now, wait 6 months for a review ;-) BUT the process is really slow and laborious :/ We need reviewers and developers FYI: Migrating from hg.python.org to Github PEP512 9 / 49
Overview of the directories . ├── D o c = = > t h e F i n e M a n u a l . . . ├── G r a m m a r = = > W h e r e t h e g r a m m a r i s d e f i n e d ├── I n c l u d e = = > t h e C H e a d e r s ├── L i b = = > t h e . p y m o d u l e s ├── M o d u l e s = = > t h e . c m o d u l e s ├── O b j e c t s = = > t h e b u i l t i n o b j e c t s ( b o o l , d i c t , s e t , t u p l e , l i s t , u n i c o d e . . . ) ├── P a r s e r = = > g r a m m a r , l e x e r , p a r s e r , c o m p i l e r ├── P r o g r a m s = = > t h e p y t h o n e x e c u t a b l e └── P y t h o n = = > t h e v i r t u a l m a c h i n e 12 / 49
About the documentation Do YOU want to contribute ??? here is a small idea: We need a tutorial on AsyncIO and how to use it because the documentation of AsyncIO is useful but it's not a tutorial but a REFERENCE. 14 / 49
CLI Command Line # T h e P y t h o n e x e c u t a b l e P r o g r a m / p y t h o n . c : m a i n # T h e P y t h o n l i b r a r y - > M o d u l e s / m a i n . c : P y _ M a i n # I n i t i a l i z e C P y t h o n - > P y I n i t i a l i z e - > P y R u n _ S i m p l e S t r i n g F l a g s - > P y I m p o r t _ A d d M o d u l e - > P y R u n _ S t r i n g F l a g s - > P y P a r s e r _ A S T F r o m S t r i n g O b j e c t - > P y P a r s e r _ P a r s e S t r i n g O b j e c t - > P y A S T _ C o m p i l e O b j e c t - > P y E v a l _ E v a l C o d e 21 / 49
CLI Parsing - Lexer Lexer t o k e n s = l e x e r ( ' x = 2 + 2 ' ) > > > s t a t e m e n t = ' x = 2 + 2 ' > > > s t r I O = i o . S t r i n g I O ( s t a t e m e n t ) . r e a d l i n e > > > t o k e n s = t o k e n i z e . g e n e r a t e _ t o k e n s ( s t r I O ) > > > p p ( l i s t ( t o k e n i z e . g e n e r a t e _ t o k e n s ( t o k e n s ) ) ) [ T o k e n I n f o ( t y p e = N A M E , s t r i n g = ' x ' ) , T o k e n I n f o ( t y p e = O P , s t r i n g = ' = ' ) , T o k e n I n f o ( t y p e = N U M B E R , s t r i n g = ' 2 ' ) , T o k e n I n f o ( t y p e = O P , s t r i n g = ' + ' ) , T o k e n I n f o ( t y p e = N U M B E R , s t r i n g = ' 2 ' ) , T o k e n I n f o ( t y p e = E N D M A R K E R , s t r i n g = ' ' ) , ] in P a r s e r / t o k e n i z e r . c -> P y T o k e n i z e r _ F r o m S t r i n g in P a r s e r / p a r s e t o k . c -> p a r s e t o k in L i b / t o k e n i z e . p y 22 / 49
CLI Parsing - Lexer Lexer Mysterious async keyword in Python 3.5 > > > i m p o r t k e y w o r d > > > ' a s y n c ' i n k e y w o r d . k w l i s t F a l s e Proof > > > s t a t e m e n t = ' a s y n c = T r u e ' > > > s t r I O = i o . S t r i n g I O ( s t a t e m e n t ) . r e a d l i n e > > > t o k e n s = t o k e n i z e . g e n e r a t e _ t o k e n s ( s t r I O ) > > > p p ( l i s t ( t o k e n s ) ) [ T o k e n I n f o ( t y p e = N A M E , s t r i n g = ' a s y n c ' ) , T o k e n I n f o ( t y p e = O P , s t r i n g = ' = ' ) , T o k e n I n f o ( t y p e = N A M E , s t r i n g = ' T r u e ' ) , T o k e n I n f o ( t y p e = E N D M A R K E R , s t r i n g = ' ' ) ] 23 / 49
CLI Parsing - Lexer - Parser Parser a s t = p a r s e r ( t o k e n s ) > > > i m p o r t a s t > > > t r e e = a s t . p a r s e ( ' x = 2 + 2 ' ) > > > a s t . d u m p ( t r e e ) M o d u l e ( b o d y = [ A s s i g n ( t a r g e t s = [ N a m e ( i d = ' x ' ) ] , v a l u e = B i n O p ( l e f t = N u m ( n = 2 ) , o p = A d d ( ) , r i g h t = N u m ( n = 2 ) ) ) ] ) P y t h o n / p y t h o n r u n . c -> P y P a r s e r _ A S T F r o m S t r i n g O b j e c t 24 / 49
CLI Parsing Compiler - Compiler Compiler b y t e c o d e s = c o m p i l e r ( a s t ) > > > i m p o r t d i s > > > i m p o r t a s t > > > t r e e = a s t . p a r s e ( ' x = 2 + 2 ' ) > > > c o d e _ o b j e c t = c o m p i l e ( t r e e , ' d u m m y . p y ' , m o d e = ' e x e c ' ) > > > d i s . d i s ( c o d e _ o b j e c t ) 1 0 L O A D _ C O N S T 2 ( 4 ) 3 S T O R E _ N A M E 0 ( x ) 6 L O A D _ C O N S T 1 ( N o n e ) 9 R E T U R N _ V A L U E See PEP-0339 in P y t h o n / c o m p i l e . c -> P y A S T _ C o m p i l e O b j e c t 25 / 49
CLI Parsing Compiler - Compiler - Bytecodes Bytecodes compact numeric codes portable code one byte followed by optional parameters used by a software interpreter # d e f i n e P O P _ T O P 1 # d e f i n e R O T _ T W O 2 . . . # d e f i n e B I N A R Y _ A D D 2 3 # d e f i n e B I N A R Y _ S U B S T R A C T 2 4 . . . # d e f i n e S E T U P _ A S Y N C _ W I T H 1 5 4 # d e f i n e F O R M A T _ V A L U E 1 5 5 See in I n c l u d e / o p c o d e . h 26 / 49
CLI Parsing Compiler - Compiler - Bytecodes Bytecodes in / t m p / e m p t y . p y # ! / u s r / b i n / e n v p y t h o n # a s u p e r c o m m e n t Here is the bytecode 1 0 L O A D _ C O N S T 0 ( N o n e ) 3 R E T U R N _ V A L U E 27 / 49
CLI Parsing Compiler - Compiler - Bytecodes Bytecodes > > > d e f f u n c ( x ) : . . . x = x + 1 . . . r e t u r n x . . . > > > d i s . d i s ( f u n c ) 2 0 L O A D _ F A S T 0 ( x ) 3 L O A D _ C O N S T 1 ( 1 ) 6 B I N A R Y _ A D D 7 S T O R E _ F A S T 0 ( x ) 3 1 0 L O A D _ F A S T 0 ( x ) 1 3 R E T U R N _ V A L U E 28 / 49
CLI Parsing Compiler - Compiler - Bytecodes - Peepholer Peepholer b y t e c o d e s = o p t i m i z e r ( b y t e c o d e s ) performs basic peephole optimizations removes the dead branches in P y t h o n / p e e p h o l e . c -> P y C o d e _ O p t i m i z e 29 / 49
CLI Parsing Compiler - Compiler - Bytecodes - Peepholer Peepholer x = 2 + 2 1 0 L O A D _ C O N S T 2 ( 4 ) 3 S T O R E _ N A M E 0 ( x ) 6 L O A D _ C O N S T 1 ( N o n e ) 9 R E T U R N _ V A L U E 30 / 49
CLI Parsing Compiler - Compiler - Bytecodes - Peepholer Peepholer i f 1 : p r i n t ( ' h e l l o ' ) 1 0 L O A D _ N A M E 0 ( p r i n t ) 3 L O A D _ C O N S T 0 ( ' h e l l o ' ) 6 C A L L _ F U N C T I O N 1 ( 1 p o s i t i o n a l , 0 k e y w o r d p a i r ) 9 P O P _ T O P 1 0 L O A D _ C O N S T 1 ( N o n e ) 1 3 R E T U R N _ V A L U E 31 / 49
CLI Parsing Compiler - Compiler - Bytecodes - Peepholer Peepholer i f 0 : p r i n t ( ' h e l l o ' ) 1 0 L O A D _ C O N S T 0 ( N o n e ) 3 R E T U R N _ V A L U E 32 / 49
CLI Parsing Compiler Interpreter Interpreter e x e c u t i o n = i n t e r p r e t e r ( b y t e c o d e s ) Virtual Stack Machine Execute the ByteCode 33 / 49
CLI Parsing Compiler Interpreter Interpreter a Virtual Stack Machine c l a s s I n t e r p r e t e r : d e f _ _ i n i t _ _ ( s e l f , i n s t r u c t i o n s ) : s e l f . i n s t r u c t i o n s = i n s t r u c t i o n s s e l f . s t a c k = [ ] s e l f . i n s t _ p t r = 0 s e l f . r u n n i n g = T r u e d e f r u n ( s e l f ) : w h i l e s e l f . r u n n i n g : s e l f . e v a l ( s e l f . f e t c h ( ) ) s e l f . i n s t _ p t r + = 1 d e f f e t c h ( s e l f ) : r e t u r n s e l f . i n s t r u c t i o n s [ s e l f . i n s t _ p t r ] d e f e v a l ( s e l f , i n s t r u c t i o n ) : g e t a t t r ( s e l f , ' _ e v a l _ { } ' . f o r m a t ( i n s t r u c t i o n ) ) ( ) 34 / 49
CLI Parsing Compiler Interpreter Interpreter a Virtual Stack Machine d e f _ e v a l _ P U S H ( s e l f ) : s e l f . i n s t _ p t r + = 1 s e l f . s t a c k . a p p e n d ( s e l f . i n s t r u c t i o n s [ s e l f . i n s t _ p t r ] ) d e f _ e v a l _ A D D ( s e l f ) : l e f t = s e l f . s t a c k . p o p ( ) r i g h t = s e l f . s t a c k . p o p ( ) r e s u l t = l e f t + r i g h t s e l f . s t a c k . a p p e n d ( r e s u l t ) d e f _ e v a l _ P O P ( s e l f ) : v a l u e = s e l f . s t a c k . p o p ( ) p r i n t ( " P o p p e d % r " % v a l u e ) d e f _ e v a l _ H A L T ( s e l f ) : s e l f . r u n n i n g = F a l s e 35 / 49
CLI Parsing Compiler Interpreter Interpreter Execution of my custom Bytecode P U S H 5 | - - - - - - - - - - - | P U S H 3 | | P U S H 1 0 | - - - - - - - - - - - | A D D A D D P O P 36 / 49
CLI Parsing Compiler Interpreter Interpreter Execution of my custom Bytecode - - > P U S H 5 | - - - - - - - - - - - | P U S H 3 | 5 | P U S H 1 0 | - - - - - - - - - - - | A D D A D D P O P 37 / 49
CLI Parsing Compiler Interpreter Interpreter Execution of my custom Bytecode P U S H 5 | - - - - - - - - - - - | - - > P U S H 3 | 3 | P U S H 1 0 | - - - - - - - - - - - | A D D | 5 | A D D | - - - - - - - - - - - | P O P 38 / 49
CLI Parsing Compiler Interpreter Interpreter Execution of my custom Bytecode P U S H 5 | - - - - - - - - - - - | P U S H 3 | 1 0 | - - > P U S H 1 0 | - - - - - - - - - - - | A D D | 3 | A D D | - - - - - - - - - - - | P O P | 5 | | - - - - - - - - - - - | 39 / 49
CLI Parsing Compiler Interpreter Interpreter Execution of my custom Bytecode P U S H 5 | - - - - - - - - - - - | P U S H 3 | 1 3 | P U S H 1 0 | - - - - - - - - - - - | - - > A D D | 5 | A D D | - - - - - - - - - - - | P O P 40 / 49
CLI Parsing Compiler Interpreter Interpreter Execution of my custom Bytecode P U S H 5 | - - - - - - - - - - - | P U S H 3 | 1 8 | P U S H 1 0 | - - - - - - - - - - - | A D D - - > A D D P O P 41 / 49
CLI Parsing Compiler Interpreter Interpreter Execution of my custom Bytecode P U S H 5 | - - - - - - - - - - - | P U S H 3 | | P U S H 1 0 | - - - - - - - - - - - | A D D A D D - - > P O P 42 / 49
CLI Parsing Compiler Interpreter Interpreter Do you remember this function ? > > > d e f f u n c ( x ) : . . . x = x + 1 . . . r e t u r n x . . . And the Bytecode ? > > > d i s . d i s ( f u n c ) 2 0 L O A D _ F A S T 0 ( x ) 3 L O A D _ C O N S T 1 ( 1 ) 6 B I N A R Y _ A D D 7 S T O R E _ F A S T 0 ( x ) 3 1 0 L O A D _ F A S T 0 ( x ) 1 3 R E T U R N _ V A L U E 43 / 49
CLI Parsing Compiler Interpreter Interpreter 2 0 L O A D _ F A S T 0 ( x ) 3 L O A D _ C O N S T 1 ( 1 ) T A R G E T ( L O A D _ F A S T ) { P y O b j e c t * v a l u e = G E T L O C A L ( o p a r g ) ; i f ( v a l u e = = N U L L ) { f o r m a t _ e x c _ c h e c k _ a r g ( P y E x c _ U n b o u n d L o c a l E r r o r , U N B O U N D L O C A L _ E R R O R _ M S G , P y T u p l e _ G e t I t e m ( c o - > c o _ v a r n a m e s , o p a r g ) ) ; g o t o e r r o r ; } P y _ I N C R E F ( v a l u e ) ; P U S H ( v a l u e ) ; F A S T _ D I S P A T C H ( ) ; } T A R G E T ( L O A D _ C O N S T ) { P y O b j e c t * v a l u e = G E T I T E M ( c o n s t s , o p a r g ) ; P y _ I N C R E F ( v a l u e ) ; P U S H ( v a l u e ) ; F A S T _ D I S P A T C H ( ) ; } 44 / 49
CLI Parsing Compiler Interpreter Interpreter 6 B I N A R Y _ A D D T A R G E T ( B I N A R Y _ A D D ) { P y O b j e c t * r i g h t = P O P ( ) ; P y O b j e c t * l e f t = T O P ( ) ; P y O b j e c t * s u m ; i f ( P y U n i c o d e _ C h e c k E x a c t ( l e f t ) & & P y U n i c o d e _ C h e c k E x a c t ( r i g h t ) ) { s u m = u n i c o d e _ c o n c a t e n a t e ( l e f t , r i g h t , f , n e x t _ i n s t r ) ; } e l s e { s u m = P y N u m b e r _ A d d ( l e f t , r i g h t ) ; P y _ D E C R E F ( l e f t ) ; } P y _ D E C R E F ( r i g h t ) ; S E T _ T O P ( s u m ) ; i f ( s u m = = N U L L ) g o t o e r r o r ; D I S P A T C H ( ) ; } 45 / 49
CLI Parsing Compiler Interpreter Interpreter 7 S T O R E _ F A S T 0 ( x ) 1 3 R E T U R N _ V A L U E T A R G E T ( S T O R E _ F A S T ) { P y O b j e c t * v a l u e = P O P ( ) ; S E T L O C A L ( o p a r g , v a l u e ) ; F A S T _ D I S P A T C H ( ) ; } T A R G E T ( R E T U R N _ V A L U E ) { r e t v a l = P O P ( ) ; w h y = W H Y _ R E T U R N ; g o t o f a s t _ b l o c k _ e n d ; } 46 / 49
CLI Parsing Compiler Interpreter Interpreter And just for the fun, Everything is in P y t h o n / c e v a l . c but especially in one function P y E v a l _ E v a l F r a m e E x 7 9 6 P y O b j e c t * 7 9 7 P y E v a l _ E v a l F r a m e E x ( P y F r a m e O b j e c t * f , i n t t h r o w f l a g ) 7 9 8 { . . . 1 8 9 9 # i f d e f C A S E _ T O O _ B I G 1 9 0 0 d e f a u l t : s w i t c h ( o p c o d e ) { 1 9 0 1 # e n d i f . . . 3 6 4 4 } 47 / 49