Welcome to async/await era Igor Davydenko PyCon Finland 2015

I am... Igor Davydenko Python & React.js developer From Kyiv, Ukraine Works on Ezhome Inc. Primarly designs & develops backend API Personal Site @ GitHub @ Twitter

PEP 492 Coroutines with async and await syntax

Quick Summary Created by Yuri Selivanov in April, 2015 Included in Python 3.5 Added 4 new statements to the standard library: a s y n c d e f a w a i t a s y n c w i t h a s y n c f o r Received great response from the Python community PEP 492 @ Python.org

@ a s y n c i o . c o r o u t i n e i m p o r t a s y n c i o @ a s y n c i o . c o r o u t i n e d e f h e l l o ( ) : r e t u r n ' H e l l o , w o r l d ! ' l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) m e s s a g e = l o o p . r u n _ u n t i l _ c o m p l e t e ( h e l l o ( ) ) p r i n t ( m e s s a g e ) l o o p . c l o s e ( )

a s y n c d e f a s y n c d e f h e l l o ( ) : r e t u r n ' H e l l o , w o r l d ! ' i m p o r t a s y n c i o l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) m e s s a g e = l o o p . r u n _ u n t i l _ c o m p l e t e ( h e l l o ( ) ) p r i n t ( m e s s a g e ) l o o p . c l o s e ( )

@ a s y n c i o . c o r o u t i n e + y i e l d f r o m i m p o r t a s y n c i o f r o m a i o h t t p i m p o r t c l i e n t @ a s y n c i o . c o r o u t i n e d e f f e t c h _ p a g e ( u r l ) : r e s p o n s e = y i e l d f r o m c l i e n t . g e t ( u r l ) r e t u r n ( y i e l d f r o m r e s p o n s e . t e x t ( ) ) l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) c o n t e n t = l o o p . r u n _ u n t i l _ c o m p l e t e ( f e t c h _ p a g e ( ' h t t p : / / f i . p y c o n . o r g / ' ) ) p r i n t ( c o n t e n t ) l o o p . c l o s e ( )

a s y n c d e f + a w a i t a s y n c d e f f e t c h _ p a g e ( u r l ) : r e s p o n s e = a w a i t c l i e n t . g e t ( u r l ) r e t u r n a w a i t r e s p o n s e . t e x t ( ) i m p o r t a s y n c i o f r o m a i o h t t p i m p o r t c l i e n t l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) c o n t e n t = l o o p . r u n _ u n t i l _ c o m p l e t e ( f e t c h _ p a g e ( ' h t t p : / / f i . p y c o n . o r g / ' ) ) p r i n t ( c o n t e n t ) l o o p . c l o s e ( )

w i t h ( y i e l d f r o m . . . ) i m p o r t a s y n c i o i m p o r t s q l a l c h e m y a s s a f r o m a i o p g . s a i m p o r t c r e a t e _ e n g i n e @ a s y n c i o . c o r o u t i n e d e f c o u n t _ d a t a ( d s n ) : e n g i n e = y i e l d f r o m c r e a t e _ e n g i n e ( d s n ) w i t h ( y i e l d f r o m e n g i n e ) a s c o n n : q u e r y = s a . s e l e c t ( . . . ) . c o u n t ( ) r e t u r n ( y i e l d f r o m c o n n . s c a l a r ( q u e r y ) ) l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) c o u n t e r = l o o p . r u n _ u n t i l _ c o m p l e t e ( c o u n t _ d a t a ( ' p o s t g r e s q l : / / . . . ' ) ) p r i n t ( c o u n t e r ) l o o p . c l o s e ( )

a s y n c w i t h f r o m . u t i l s i m p o r t C o n n e c t i o n C o n t e x t M a n a g e r a s y n c d e f c o u n t _ d a t a ( d s n ) : e n g i n e = a w a i t c r e a t e _ e n g i n e ( d s n ) a s y n c w i t h C o n n e c t i o n C o n t e x t M a n a g e r ( e n g i n e ) a s c o n n : q u e r y = s a . s e l e c t ( . . . ) . c o u n t ( ) r e t u r n a w a i t c o n n . s c a l a r ( q u e r y ) i m p o r t a s y n c i o i m p o r t s q l a l c h e m y a s s a f r o m a i o p g . s a i m p o r t c r e a t e _ e n g i n e l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) c o u n t e r = l o o p . r u n _ u n t i l _ c o m p l e t e ( c o u n t _ d a t a ( ' p o s t g r e s q l : / / . . . ' ) ) p r i n t ( c o u n t e r ) l o o p . c l o s e ( )

a s y n c w i t h utils.py c l a s s C o n n e c t i o n C o n t e x t M a n a g e r ( o b j e c t ) : d e f _ _ i n i t _ _ ( s e l f , e n g i n e ) : s e l f . c o n n = N o n e s e l f . e n g i n e = e n g i n e a s y n c d e f _ _ a e n t e r _ _ ( s e l f ) : s e l f . c o n n = a w a i t s e l f . e n g i n e . a c q u i r e ( ) r e t u r n s e l f . c o n n a s y n c d e f _ _ a e x i t _ _ ( s e l f , e x c _ t y p e , e x c , t b ) : t r y : s e l f . e n g i n e . r e l e a s e ( s e l f . c o n n ) f i n a l l y : s e l f . c o n n = N o n e s e l f . e n g i n e = N o n e

f o r r o w i n ( y i e l d f r o m . . . ) : i m p o r t a s y n c i o i m p o r t s q l a l c h e m y a s s a f r o m a i o p g . s a i m p o r t c r e a t e _ e n g i n e @ a s y n c i o . c o r o u t i n e d e f f e t c h _ d a t a ( d s n ) : d a t a = [ ] e n g i n e = y i e l d f r o m c r e a t e _ e n g i n e ( d s n ) w i t h ( y i e l d f r o m e n g i n e ) a s c o n n : r e s u l t = y i e l d f r o m c o n n . e x e c u t e ( s a . s e l e c t ( . . . ) ) f o r r o w i n r e s u l t : d a t a . a p p e n d ( r o w ) r e t u r n d a t a l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) d a t a = l o o p . r u n _ u n t i l _ c o m p l e t e ( f e t c h _ d a t a ( ' p o s t g r e s q l : / / . . . ' ) ) l o o p . c l o s e ( )

a s y n c f o r , R e s u l t I t e r a s y n c d e f f e t c h _ d a t a ( d s n ) : d a t a = [ ] e n g i n e = a w a i t c r e a t e _ e n g i n e ( d s n ) a s y n c w i t h C o n n e c t i o n C o n t e x t M a n a g e r ( e n g i n e ) a s c o n n : a s y n c f o r r o w i n R e s u l t I t e r ( a w a i t c o n n . e x e c u t e ( s a . s e l e c t ( . . . ) ) ) : d a t a . a p p e n d ( r o w ) r e t u r n d a t a i m p o r t a s y n c i o i m p o r t s q l a l c h e m y a s s a f r o m a i o p g . s a i m p o r t c r e a t e _ e n g i n e f r o m . u t i l s i m p o r t C o n n e c t i o n C o n t e x t M a n a g e r l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) d a t a = l o o p . r u n _ u n t i l _ c o m p l e t e ( f e t c h _ d a t a ( ' p o s t g r e s q l : / / . . . ' ) ) l o o p . c l o s e ( )

a s y n c f o r utils.py f r o m a i o p g . s a . e x c i m p o r t R e s o u r c e C l o s e d E r r o r c l a s s R e s u l t I t e r ( o b j e c t ) : d e f _ _ i n i t _ _ ( s e l f , r e s u l t ) : s e l f . r e s u l t = r e s u l t a s y n c d e f _ _ a i t e r _ _ ( s e l f ) : r e t u r n s e l f a s y n c d e f _ _ a n e x t _ _ ( s e l f ) : t r y : d a t a = a w a i t s e l f . r e s u l t . f e t c h o n e ( ) e x c e p t R e s o u r c e C l o s e d E r r o r : d a t a = N o n e i f d a t a : r e t u r n d a t a r a i s e S t o p A s y n c I t e r a t i o n

Other additions to standard library @ t y p e s . c o r o u t i n e bridge between generator based and native coroutines New _ _ a w a i t _ _ magic method New functions in i n s p e c t library as i s c o r o u t i n e , i s a w a i t a b l e , etc New abstract base classes: a b c . A w a i t a b l e , a b c . C o r o u t i n e , a b c . A s y n c I t e r a b l e , a b c . A s y n c I t e r a t o r

Conclusion on functions and methods Method Can contain Can't contain a s y n c d e f f u n c a w a i t , r e t u r n v a l u e y i e l d , y i e l d f r o m a s y n c d e f _ _ a * _ _ a w a i t , r e t u r n v a l u e y i e l d , y i e l d f r o m d e f _ _ a * _ _ r e t u r n a w a i t a b l e a w a i t d e f _ _ a w a i t _ _ y i e l d , y i e l d f r o m , r e t u r n i t e r a b l e a w a i t g e n e r a t o r y i e l d , y i e l d f r o m , r e t u r n v a l u e a w a i t

async/await in real life Libraries

aiohttp HTTP client/server for asyncio Latest version: 0 . 1 7 . 4 http://aiohttp.readthedocs.org/ i m p o r t a s y n c i o i m p o r t u j s o n f r o m a i o h t t p i m p o r t c l i e n t a s y n c d e f g i t h u b _ s e a r c h ( q u e r y ) : r e s p o n s e = a w a i t c l i e n t . g e t ( ' h t t p s : / / a p i . g i t h u b . c o m / s e a r c h / r e p o s i t o r i e s ' , p a r a m s = { ' q ' : q u e r y } ) r e t u r n u j s o n . l o a d s ( a w a i t r e s p o n s e . r e a d ( ) ) l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) r e s p o n s e = l o o p . r u n _ u n t i l _ c o m p l e t e ( g i t h u b _ s e a r c h ( ' a s y n c i o ' ) ) p r i n t ( ' \ n ' . j o i n ( r e p o [ ' f u l l _ n a m e ' ] f o r r e p o i n r e s p o n s e [ ' i t e m s ' ] ) ) l o o p . c l o s e ( )

aiohttp.web Web framework for asyncio http://aiohttp.readthedocs.org/en/latest/web.html i m p o r t u j s o n f r o m a i o h t t p i m p o r t w e b a s y n c d e f a p i _ i n d e x ( r e q u e s t ) : o u t p u t = u j s o n . d u m p s ( { . . . } ) r e t u r n w e b . R e s p o n s e ( b o d y = o u t p u t , c o n t e n t _ t y p e = ' a p p l i c a t i o n / j s o n ' ) a p p = w e b . A p p l i c a t i o n ( ) a p p . r o u t e r . a d d _ r o u t e ( ' G E T ' , ' / a p i / ' , a p i _ i n d e x ) $ g u n i c o r n - k a i o h t t p . w o r k e r . G u n i c o r n W e b W o r k e r - w 9 - t 6 0 a p p : a p p

aiopg Accessing PostgreSQL database from the asyncio Latest version: 0 . 7 . 0 http://aiopg.readthedocs.org/ i m p o r t a s y n c i o f r o m a i o p g i m p o r t c r e a t e _ p o o l a s y n c d e f s e l e c t _ o n e ( d s n ) : p o o l = a w a i t c r e a t e _ p o o l ( d s n ) w i t h ( a w a i t p o o l . c u r s o r ( ) ) a s c u r s o r : a w a i t c u r s o r . e x e c u t e ( ' S E L E C T 1 ' ) s e l e c t e d = a w a i t c u r s o r . f e t c h o n e ( ) a s s e r t s e l e c t e d = = ( 1 , ) l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) l o o p . r u n _ u n t i l _ c o m p l e t e ( s e l e c t _ o n e ( ' d b n a m e = . . . u s e r = . . . p a s s w o r d = . . . h o s t = . . . ' ) ) l o o p . c l o s e ( )

aiopg.sa Layer for executing SQLAlchemy Core queries http://aiopg.readthedocs.org/en/stable/sa.html i m p o r t a s y n c i o i m p o r t s q l a l c h e m y a s s a f r o m a i o p g . s a i m p o r t c r e a t e _ e n g i n e f r o m . u t i l s i m p o r t C o n n e c t i o n C o n t e x t M a n a g e r a s y n c d e f s e l e c t _ o n e ( d s n ) : e n g i n e = a w a i t c r e a t e _ e n g i n e ( d s n ) a s y n c w i t h C o n n e c t i o n C o n t e x t M a n a g e r ( e n g i n e ) a s c o n n : a w a i t c o n n . e x e c u t e ( s a . s e l e c t ( [ s a . t e x t ( ' 1 ' ) ] ) ) s e l e c t e d = a w a i t c o n n . f e t c h o n e ( ) a s s e r t s e l e c t e d = = ( 1 , ) l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) l o o p . r u n _ u n t i l _ c o m p l e t e ( s e l e c t _ o n e ( ' p o s t g r e s q l : / / . . . ' ) ) l o o p . c l o s e ( )

aioredis Redis client library Latest version: 0 . 2 . 4 http://aioredis.readthedocs.org/ i m p o r t a s y n c i o f r o m a i o r e d i s i m p o r t c r e a t e _ r e d i s a s y n c d e f r e d i s _ s e t _ g e t _ d e l e t e ( a d d r e s s , * * o p t i o n s ) : o p t i o n s . s e t d e f a u l t ( ' e n c o d i n g ' , ' u t f - 8 ' ) r e d i s = a w a i t c r e a t e _ r e d i s ( a d d r e s s , * * o p t i o n s ) a s s e r t a w a i t r e d i s . s e t ( ' k e y ' , ' v a l u e ' ) i s T r u e a s s e r t a w a i t r e d i s . g e t ( ' k e y ' ) = = ' v a l u e ' a s s e r t a w a i t r e d i s . d e l e t e ( ' k e y ' ) = = 1 l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) l o o p . r u n _ u n t i l _ c o m p l e t e ( r e d i s _ s e t _ g e t _ d e l e t e ( ( ' l o c a l h o s t ' , 6 3 7 9 ) ) ) l o o p . c l o s e ( )

And others Python Asyncio Resources MySQL: aiomysql Mongo: asyncio_mongo CouchDB: aiocouchdb ElasticSearch: aioes Memcached: aiomcache AMQP: aioamqp ØMQ: aiozmq

async/await in real life Fetching data from remote API

Task. Fetch data for the NFL Season NFL Season lasts 5 preseason and 17 regular season weeks This totals 22 requests to NFL.com endpoint Each request can be processed asyncronously After all requests are done we need to call extra function

Step 1. Sync fetch week data i m p o r t r e q u e s t s f r o m l x m l i m p o r t e t r e e N F L _ U R L = ' h t t p : / / w w w . n f l . c o m / a j a x / s c o r e s t r i p ' d e f f e t c h _ w e e k ( s e a s o n , w e e k , i s _ p r e s e a s o n = F a l s e ) : r e s p o n s e = r e q u e s t s . g e t ( N F L _ U R L , p a r a m s = { ' s e a s o n ' : s e a s o n , ' s e a s o n T y p e ' : ' P R E ' i f i s _ p r e s e a s o n e l s e ' R E G ' , ' w e e k ' : w e e k , } ) r e t u r n e t r e e . f r o m s t r i n g ( r e s p o n s e . c o n t e n t )

Step 2. Sync fetch season data d e f f e t c h _ s e a s o n ( s e a s o n ) : p r e _ c a l l ( . . . ) f o r i s _ p r e s e a s o n , w e e k s i n ( ( F a l s e , r a n g e ( 5 ) ) , ( T r u e , r a n g e ( 1 , 1 8 ) ) ) : f o r w e e k i n w e e k s : f e t c h _ w e e k ( s e a s o n , w e e k , i s _ p r e s e a s o n ) . . . p o s t _ c a l l ( . . . )

Step 3. Async fetch week data f r o m a i o h t t p i m p o r t c l i e n t f r o m l x m l i m p o r t e t r e e a s y n c d e f a i o _ f e t c h _ w e e k ( s e a s o n , w e e k , i s _ p r e s e a s o n = F a l s e ) : r e s p o n s e = a w a i t c l i e n t . g e t ( N F L _ U R L , p a r a m s = { ' s e a s o n ' : s e a s o n , ' s e a s o n T y p e ' : ' P R E ' i f i s _ p r e s e a s o n e l s e ' R E G ' , ' w e e k ' : w e e k , } ) r e t u r n e t r e e . f r o m s t r i n g ( a w a i t r e s p o n s e . r e a d ( ) )

Step 4. Async fetch season data i m p o r t a s y n c i o d e f f e t c h _ s e a s o n ( s e a s o n ) : l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) l o o p . r u n _ u n t i l _ c o m p l e t e ( p r e _ c a l l ( . . . ) ) t a s k s = [ a i o _ f e t c h _ w e e k ( s e a s o n , y e a r , i s _ p r e s e a s o n ) f o r i s _ p r e s e a s o n , w e e k s i n ( ( F a l s e , r a n g e ( 5 ) ) , ( T r u e , r a n g e ( 1 , 1 8 ) ) ) f o r w e e k i n w e e k s ] l o o p . r u n _ u n t i l _ c o m p l e t e ( a s y n c i o . w a i t ( t a s k s , l o o p = l o o p ) ) l o o p . r u n _ u n t i l _ c o m p l e t e ( p o s t _ c a l l ( . . . ) ) l o o p . c l o s e ( )

Step 5. Running fetching season data scripts/fetch_season_data.py i m p o r t s y s d e f m a i n ( * a r g s ) : f e t c h _ s e a s o n ( i n t ( a r g s [ 0 ] ) ) r e t u r n F a l s e i f _ _ n a m e _ _ = = ' _ _ m a i n _ _ ' : s y s . e x i t ( i n t ( m a i n ( * s y s . a r g v [ 1 : ] ) ) ) $ p y t h o n s c r i p t s / f e t c h _ s e a s o n _ d a t a . p y 2 0 1 5

Step 6. See in action . . . A l l g a m e s f o r S e a s o n 2 0 1 5 / R E G . W e e k 1 7 p r o c e s s e d ! S e a s o n 2 0 1 5 / R E G . W e e k 1 2 . G a m e I D 2 0 1 5 1 1 3 0 0 0 , C L E @ B A L , s c h e d u l e d a t 2 0 1 5 - 1 1 - 3 0 T 2 0 : 3 0 : 0 0 - 0 5 : 0 0 u p d a t e d i n d a t a b a s e A l l g a m e s f o r S e a s o n 2 0 1 5 / R E G . W e e k 1 2 p r o c e s s e d ! S e a s o n 2 0 1 5 / P R E . W e e k 4 . G a m e I D 2 0 1 5 0 9 0 3 6 1 , D A L @ H O U , s c h e d u l e d a t 2 0 1 5 - 0 9 - 0 3 T 2 0 : 0 0 : 0 0 - 0 4 : 0 0 u p d a t e d i n d a t a b a s e S e a s o n 2 0 1 5 / R E G . W e e k 1 6 . G a m e I D 2 0 1 5 1 2 2 7 1 1 , S E A @ S T L , s c h e d u l e d a t 2 0 1 5 - 1 2 - 2 7 T 1 6 : 2 5 : 0 0 - 0 5 : 0 0 u p d a t e d i n d a t a b a s e S e a s o n 2 0 1 5 / P R E . W e e k 4 . G a m e I D 2 0 1 5 0 9 0 3 6 2 , S T L @ K C , s c h e d u l e d a t 2 0 1 5 - 0 9 - 0 3 T 2 0 : 0 0 : 0 0 - 0 4 : 0 0 u p d a t e d i n d a t a b a s e S e a s o n 2 0 1 5 / R E G . W e e k 1 6 . G a m e I D 2 0 1 5 1 2 2 7 1 2 , B A L @ P I T , s c h e d u l e d a t 2 0 1 5 - 1 2 - 2 7 T 2 0 : 3 0 : 0 0 - 0 5 : 0 0 u p d a t e d i n d a t a b a s e S e a s o n 2 0 1 5 / P R E . W e e k 4 . G a m e I D 2 0 1 5 0 9 0 3 6 3 , T E N @ M I N , s c h e d u l e d a t 2 0 1 5 - 0 9 - 0 3 T 2 0 : 0 0 : 0 0 - 0 4 : 0 0 u p d a t e d i n d a t a b a s e S e a s o n 2 0 1 5 / R E G . W e e k 1 6 . G a m e I D 2 0 1 5 1 2 2 8 0 0 , D E N @ C I N , s c h e d u l e d a t 2 0 1 5 - 1 2 - 2 8 T 2 0 : 3 0 : 0 0 - 0 5 : 0 0 u p d a t e d i n d a t a b a s e A l l g a m e s f o r S e a s o n 2 0 1 5 / R E G . W e e k 1 6 p r o c e s s e d ! S e a s o n 2 0 1 5 / P R E . W e e k 4 . G a m e I D 2 0 1 5 0 9 0 3 6 4 , D E N @ A R I , s c h e d u l e d a t 2 0 1 5 - 0 9 - 0 3 T 2 1 : 0 0 : 0 0 - 0 4 : 0 0 u p d a t e d i n d a t a b a s e S e a s o n 2 0 1 5 / P R E . W e e k 4 . G a m e I D 2 0 1 5 0 9 0 3 6 5 , S F @ S D , s c h e d u l e d a t 2 0 1 5 - 0 9 - 0 3 T 2 2 : 0 0 : 0 0 - 0 4 : 0 0 u p d a t e d i n d a t a b a s e S e a s o n 2 0 1 5 / P R E . W e e k 4 . G a m e I D 2 0 1 5 0 9 0 3 6 6 , S E A @ O A K , s c h e d u l e d a t 2 0 1 5 - 0 9 - 0 3 T 2 2 : 0 0 : 0 0 - 0 4 : 0 0 u p d a t e d i n d a t a b a s e A l l g a m e s f o r S e a s o n 2 0 1 5 / P R E . W e e k 4 p r o c e s s e d ! . . .

async/await in real life Backend API application

Task. Provide an API for week data NFL.com still returns data in X M L We need this data in G r a p h Q L Okay, okay, actually in J S O N And if there are no live games, we don't need to fetch data directly from NFL

Step 0. Original NFL.com data

Step 1. Flask base application f r o m f l a s k i m p o r t F l a s k f r o m . i m p o r t c a c h e f r o m . d a t a i m p o r t t o _ g a m e _ d a t a a p p = F l a s k ( _ _ n a m e _ _ ) @ a p p . r o u t e ( ' / a p i / s e a s o n / < i n t : s e a s o n _ y e a r > / < w e e k _ s l u g > ' ) d e f r e t r i e v e _ w e e k ( s e a s o n _ y e a r , w e e k _ s l u g ) : i s _ p r e s e a s o n = w e e k _ s l u g [ : 4 ] = = ' p r e - ' w e e k = i n t ( w e e k _ s l u g [ 4 : ] i f i s _ p r e s e a s o n e l s e w e e k _ s l u g ) c a c h e _ k e y = c a c h e . b u i l d _ k e y ( s e a s o n _ y e a r , w e e k _ s l u g ) w e e k _ d a t a = c a c h e . e n s u r e _ d a t a ( c a c h e _ k e y ) i f w e e k _ d a t a i s N o n e : w e e k _ o b j = f e t c h _ w e e k ( s e a s o n _ y e a r , w e e k , i s _ p r e s e a s o n ) w e e k _ d a t a = [ t o _ g a m e _ d a t a ( g a m e ) f o r g a m e i n w e e k _ o b j . i t e r f i n d ( ' g m s / g ' ) ] c a c h e . s t o r e _ d a t a ( c a c h e _ k e y , w e e k _ d a t a ) r e t u r n j s o n i f y ( w e e k _ d a t a )

Step 2. Running Flask application $ g u n i c o r n - k s y n c - w 5 - t 6 0 a p p : a p p $ g u n i c o r n - k e v e n t l e t - w 5 - t 6 0 a p p : a p p

Step 3. aiohttp.web base application i m p o r t u j s o n f r o m a i o h t t p i m p o r t w e b a s y n c d e f a i o _ r e t r i e v e _ w e e k ( r e q u e s t ) : s e a s o n _ y e a r = i n t ( r e q u e s t . m a t c h _ i n f o [ ' s e a s o n _ y e a r ' ] ) w e e k _ s l u g = r e q u e s t . m a t c h _ i n f o [ ' w e e k _ s l u g ' ] i s _ p r e s e a s o n = w e e k _ s l u g [ : 4 ] = = ' p r e - ' w e e k = i n t ( w e e k _ s l u g [ 4 : ] i f i s _ p r e s e a s o n e l s e w e e k _ s l u g ) c a c h e _ k e y = c a c h e . b u i l d _ k e y ( s e a s o n _ y e a r , w e e k , i s _ p r e s e a s o n ) w e e k _ d a t a = a w a i t c a c h e . a i o _ e n s u r e _ d a t a ( c a c h e _ k e y ) i f w e e k _ d a t a i s N o n e : w e e k _ o b j = a w a i t a i o _ f e t c h _ w e e k ( s e a s o n _ y e a r , w e e k , i s _ p r e s e a s o n ) w e e k _ d a t a = [ t o _ g a m e _ d a t a ( g a m e ) f o r g a m e i n w e e k _ o b j . i t e r f i n d ( ' g m s / g ' ) ] a w a i t c a c h e . s t o r e _ d a t a ( c a c h e _ k e y , w e e k _ d a t a ) r e t u r n w e b . R e s p o n s e ( u j s o n . d u m p s ( w e e k _ d a t a ) , c o n t e n t _ t y p e = ' a p p l i c a t i o n / j s o n ' )

Step 3. aiohttp.web base application (continue) a i o _ a p p = w e b . A p p l i c a t i o n ( ) a i o _ a p p . r o u t e r . a d d _ r o u t e ( ' G E T ' , ' / a p i / s e a s o n / { s e a s o n _ y e a r : \ d { 4 } } / { w e e k _ s l u g } ' , a i o _ r e t r i e v e _ w e e k )

aiohttp.web basics Each view function should be a c o r o u t i n e /a s y n c d e f and return response or raise an exc Each view function receives only one arg: r e q u e s t Query params data contains in r e q u e s t . G E T multidict Body data contains in r e q u e s t . P O S T multidict and should be awaited with a w a i t r e q u e s t . p o s t ( ) Matched data contains in r e q u e s t . m a t c h _ i n f o dict r e q u e s t also contains an a p p instance

Step 4. Runinng aiohttp.web application $ g u n i c o r n - k a i o h t t p . w o r k e r . G u n i c o r n W e b W o r k e r - w 5 - t 6 0 a p p : a i o _ a p p

Step 5. Resulted JSON

async/await in real life Server-Sent Events

Task. Livescore stream When there are live games, update the scores automagically for the user Do not reconnect each X seconds for new scores When new scores available - show them to user This is the case for Server-Sent Events (SSE)

Server-Sent Events. Message Format d a t a : H e l l o , P y C o n F i n l a n d ! < b l a n k l i n e > e v e n t : p y c o n f i d a t a : H e l l o , P y c o n F i n l a n d ! d a t a : I ' m g l a d t o b e h e r e < b l a n k l i n k > i d : 4 2 e v e n t : u a p y c o n d a t a : P y C o n U k r a i n e 2 0 1 6 w i l l h e l d a p l a c e i n L v i v a t A p r i l 2 0 1 6 < b l a n k l i n k >

Server-Sent Events. EventSource c o n s t e l e m e n t = d o c u m e n t . g e t E l e m e n t B y I d ( " n o t i f i c a t i o n s - c o n t a i n e r " ) ; c o n s t s o u r c e = n e w E v e n t S o u r c e ( " / a p i / n o t i f i c a t i o n s " ) ; s o u r c e . o n e r r o r = e r r = > { . . . } ; s o u r c e . a d d E v e n t L i s t e n e r ( " p y c o n f i " , e v t = > { e l e m e n t . i n n e r H T M L + = ` < c o d e > # p y c o n f i < / c o d e > $ { e v t . d a t a } < b r > ` ; } ) ; s o u r c e . a d d E v e n t L i s t e n e r ( " u a p y c o n " , e v t = > { e l e m e n t . i n n e r H T M L + = ` < c o d e > # u a p y c o n < / c o d e > $ { e v t . d a t a } < b r > ` ; } ) ; . . . s o u r c e . c l o s e ( ) ;

How to implement? 1. Run script to fetch live scores and publish new scores to Redis PUBSUB 2. Subscribe to this channel in livescore stream view, receive message and push it to client 3. Subscribe to livescore stream in client and update scores when new message received 4. ... 5. PROFIT

Step 1. Fetching & publishing scores change d e f m a i n ( * a r g s ) : s e a s o n , w e e k = a r g s l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) w e e k _ o b j = l o o p . r u n _ u n t i l _ c o m p l e t e ( a i o _ f e t c h _ w e e k ( s e a s o n , w e e k , F a l s e ) ) w e e k _ d a t a = [ t o _ g a m e _ d a t a ( g a m e ) f o r g a m e i n w e e k _ o b j . i t e r f i n d ( ' g m s / g ' ) ] r e d i s = l o o p . r u n _ u n t i l _ c o m p l e t e ( c r e a t e _ r e d i s ( ( ' l o c a l h o s t ' , 6 3 7 9 ) ) ) l o o p . r u n _ u n t i l _ c o m p l e t e ( r e d i s . p u b l i s h ( ' C H A N N E L ' , u j s o n . d u m p s ( w e e k _ d a t a ) ) ) l o o p . c l o s e ( ) r e t u r n F a l s e i f _ _ n a m e _ _ = = ' _ _ m a i n _ _ ' : s y s . e x i t ( i n t ( m a i n ( * s y s . a r g v [ 1 : ] ) ) )

Step 2. Implementing livescore stream a s y n c d e f l i v e s c o r e ( r e q u e s t ) : i f r e q u e s t . h e a d e r s [ ' A c c e p t ' ] ! = ' t e x t / e v e n t - s t r e a m ' : r a i s e w e b . H T T P F o u n d ( ' / a p i / ' ) r e s p o n s e = w e b . S t r e a m R e s p o n s e ( s t a t u s = 2 0 0 , h e a d e r s = { ' C o n t e n t - T y p e ' : ' t e x t / e v e n t - s t r e a m ' , ' C a c h e - C o n t r o l ' : ' n o - c a c h e ' , ' C o n n e c t i o n ' : ' k e e p - a l i v e ' , } ) r e s p o n s e . s t a r t ( r e q u e s t ) # T h i s w i l l b e c h a n g e d i n a s y n c i o = = 0 . 1 8 . 0 . . . r e t u r n r e s p o n s e

Step 2. Subscribing to scores update . . . r e d i s = a w a i t c r e a t e _ r e d i s ( ( ' l o c a l h o s t ' , 6 3 7 9 ) ) c h a n n e l = ( a w a i t r e d i s . s u b s c r i b e ( ' C H A N N E L ' ) ) [ 0 ] w h i l e ( a w a i t c h a n n e l . w a i t _ m e s s a g e ( ) ) : m e s s a g e = a w a i t c h a n n e l . g e t ( ) r e s p o n s e . w r i t e ( b ' e v e n t : u p d a t e \ r \ n ' ) r e s p o n s e . w r i t e ( b ' d a t a : ' + m e s s a g e + b ' \ r \ n \ r \ n ' ) . . .

Step 2. Registering stream to the app a i o _ a p p . r o u t e r . a d d _ r o u t e ( ' G E T ' , ' / a p i / l i v e s c o r e ' , l i v e s c o r e )

Step 3. Subscribing to livescore stream on client i m p o r t R e a c t , { C o m p o n e n t } f r o m " r e a c t " ; e x p o r t d e f a u l t A p p e x t e n d s C o m p o n e n t { l i v e S c o r e = n u l l ; c o n s t r u c t o r ( p r o p s ) { s u p e r ( p r o p s ) ; t h i s . s t a t e = { g a m e s : p r o p s . g a m e s } ; } c o m p o n e n t D i d M o u n t ( ) { t h i s . l i v e S c o r e = n e w E v e n t S o u r c e ( " / a p i / " ) ; t h i s . l i v e S c o r e . a d d E v e n t L i s t e n e r ( " u p d a t e " , e v t = > { t h i s . s e t S t a t e ( { g a m e s : J S O N . p a r s e ( e v t . d a t a ) } ) ; } ) ; } c o m p o n e n t W i l l U n m o u n t ( ) { t h i s . l i v e S c o r e . c l o s e ( ) ; } r e n d e r ( ) { r e t u r n } } < G a m e s d a t a = { t h i s . s t a t e . g a m e s } / > ;

Step 4. Setting up the nginx l o c a t i o n / a p i / l i v e s c o r e { c h u n k e d _ t r a n s f e r _ e n c o d i n g o f f ; p r o x y _ b u f f e r i n g o f f ; p r o x y _ c a c h e o f f ; p r o x y _ h t t p _ v e r s i o n 1 . 1 ; p r o x y _ p a s s h t t p : / / 1 2 7 . 0 . 0 . 1 : 8 0 0 0 ; p r o x y _ s e t _ h e a d e r C o n n e c t i o n ' ' ; p r o x y _ s e t _ h e a d e r H o s t $ h o s t ; p r o x y _ s e t _ h e a d e r X - R e a l - I P $ r e m o t e _ a d d r ; p r o x y _ s e t _ h e a d e r X - F o r w a r d e d - F o r $ p r o x y _ a d d _ x _ f o r w a r d e d _ f o r ; }

Step 5. It works

async/await era Conclusion

The Future is Here Python 3.5 released and ready to be used in production Python 3.5 contains new a s y n c /a w a i t statements a s y n c i o stack got great addition, code now is simple to read and understand asyncio stack (for web applications): a i o h t t p – interact with remote API a i o h t t p . w e b – create API backends a i o p g – interact with PostgreSQL (DB) a i o r e d i s – interact with Redis (cache)

The Future is Here a s y n c i o stack is a great reason why you'll finally need to switch to Python 3 a s y n c i o is good fit for: Database-less API Requesting remote API Async code execution Predictable async I/O And I'm not telling you about WebSockets, which supported by a i o h t t p out of the box

And the real reason of using asyncio

Cause it cool and trendy!

Questions?

Bonus. PyCon Ukraine 2016 April 2016 Lviv, Ukraine Related activities Looking for speakers: @hotsyk