Welcome to async/await era

PEP492 accepted and Python 3.5 will have new async/await statements. Talk will introduce them to the audience and shows examples on how to and when to use.

Talk also covers how async/await helps to write better and clear asyncio web applications on top of aiohttp library and shows common patterns to use, while making backend applications on Python 3.5.

Transcript

1. Welcome to async/await era Igor Davydenko PyCon Finland 2015
2. I am... Igor Davydenko Python & React.js developer From Kyiv, Ukraine Works on Ezhome Inc. Primarly designs & develops backend API Personal Site @ GitHub @ Twitter
3. PEP 492 Coroutines with async and await syntax
4. Quick Summary Created by Yuri Selivanov in April, 2015 Included in Python 3.5 Added 4 new statements to the standard library: a s y n c d e f a w a i t a s y n c w i t h a s y n c f o r Received great response from the Python community PEP 492 @ Python.org
5. @ a s y n c i o . c o r o u t i n e i m p o r t a s y n c i o @ a s y n c i o . c o r o u t i n e d e f h e l l o ( ) : r e t u r n ' H e l l o , w o r l d ! ' l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) m e s s a g e = l o o p . r u n _ u n t i l _ c o m p l e t e ( h e l l o ( ) ) p r i n t ( m e s s a g e ) l o o p . c l o s e ( )
6. a s y n c d e f a s y n c d e f h e l l o ( ) : r e t u r n ' H e l l o , w o r l d ! ' i m p o r t a s y n c i o l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) m e s s a g e = l o o p . r u n _ u n t i l _ c o m p l e t e ( h e l l o ( ) ) p r i n t ( m e s s a g e ) l o o p . c l o s e ( )
7. @ a s y n c i o . c o r o u t i n e + y i e l d f r o m i m p o r t a s y n c i o f r o m a i o h t t p i m p o r t c l i e n t @ a s y n c i o . c o r o u t i n e d e f f e t c h _ p a g e ( u r l ) : r e s p o n s e = y i e l d f r o m c l i e n t . g e t ( u r l ) r e t u r n ( y i e l d f r o m r e s p o n s e . t e x t ( ) ) l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) c o n t e n t = l o o p . r u n _ u n t i l _ c o m p l e t e ( f e t c h _ p a g e ( ' h t t p : / / f i . p y c o n . o r g / ' ) ) p r i n t ( c o n t e n t ) l o o p . c l o s e ( )
8. a s y n c d e f + a w a i t a s y n c d e f f e t c h _ p a g e ( u r l ) : r e s p o n s e = a w a i t c l i e n t . g e t ( u r l ) r e t u r n a w a i t r e s p o n s e . t e x t ( ) i m p o r t a s y n c i o f r o m a i o h t t p i m p o r t c l i e n t l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) c o n t e n t = l o o p . r u n _ u n t i l _ c o m p l e t e ( f e t c h _ p a g e ( ' h t t p : / / f i . p y c o n . o r g / ' ) ) p r i n t ( c o n t e n t ) l o o p . c l o s e ( )
9. w i t h ( y i e l d f r o m . . . ) i m p o r t a s y n c i o i m p o r t s q l a l c h e m y a s s a f r o m a i o p g . s a i m p o r t c r e a t e _ e n g i n e @ a s y n c i o . c o r o u t i n e d e f c o u n t _ d a t a ( d s n ) : e n g i n e = y i e l d f r o m c r e a t e _ e n g i n e ( d s n ) w i t h ( y i e l d f r o m e n g i n e ) a s c o n n : q u e r y = s a . s e l e c t ( . . . ) . c o u n t ( ) r e t u r n ( y i e l d f r o m c o n n . s c a l a r ( q u e r y ) ) l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) c o u n t e r = l o o p . r u n _ u n t i l _ c o m p l e t e ( c o u n t _ d a t a ( ' p o s t g r e s q l : / / . . . ' ) ) p r i n t ( c o u n t e r ) l o o p . c l o s e ( )
10. a s y n c w i t h f r o m . u t i l s i m p o r t C o n n e c t i o n C o n t e x t M a n a g e r a s y n c d e f c o u n t _ d a t a ( d s n ) : e n g i n e = a w a i t c r e a t e _ e n g i n e ( d s n ) a s y n c w i t h C o n n e c t i o n C o n t e x t M a n a g e r ( e n g i n e ) a s c o n n : q u e r y = s a . s e l e c t ( . . . ) . c o u n t ( ) r e t u r n a w a i t c o n n . s c a l a r ( q u e r y ) i m p o r t a s y n c i o i m p o r t s q l a l c h e m y a s s a f r o m a i o p g . s a i m p o r t c r e a t e _ e n g i n e l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) c o u n t e r = l o o p . r u n _ u n t i l _ c o m p l e t e ( c o u n t _ d a t a ( ' p o s t g r e s q l : / / . . . ' ) ) p r i n t ( c o u n t e r ) l o o p . c l o s e ( )
11. a s y n c w i t h utils.py c l a s s C o n n e c t i o n C o n t e x t M a n a g e r ( o b j e c t ) : d e f _ _ i n i t _ _ ( s e l f , e n g i n e ) : s e l f . c o n n = N o n e s e l f . e n g i n e = e n g i n e a s y n c d e f _ _ a e n t e r _ _ ( s e l f ) : s e l f . c o n n = a w a i t s e l f . e n g i n e . a c q u i r e ( ) r e t u r n s e l f . c o n n a s y n c d e f _ _ a e x i t _ _ ( s e l f , e x c _ t y p e , e x c , t b ) : t r y : s e l f . e n g i n e . r e l e a s e ( s e l f . c o n n ) f i n a l l y : s e l f . c o n n = N o n e s e l f . e n g i n e = N o n e
12. f o r r o w i n ( y i e l d f r o m . . . ) : i m p o r t a s y n c i o i m p o r t s q l a l c h e m y a s s a f r o m a i o p g . s a i m p o r t c r e a t e _ e n g i n e @ a s y n c i o . c o r o u t i n e d e f f e t c h _ d a t a ( d s n ) : d a t a = [ ] e n g i n e = y i e l d f r o m c r e a t e _ e n g i n e ( d s n ) w i t h ( y i e l d f r o m e n g i n e ) a s c o n n : r e s u l t = y i e l d f r o m c o n n . e x e c u t e ( s a . s e l e c t ( . . . ) ) f o r r o w i n r e s u l t : d a t a . a p p e n d ( r o w ) r e t u r n d a t a l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) d a t a = l o o p . r u n _ u n t i l _ c o m p l e t e ( f e t c h _ d a t a ( ' p o s t g r e s q l : / / . . . ' ) ) l o o p . c l o s e ( )
13. a s y n c f o r , R e s u l t I t e r a s y n c d e f f e t c h _ d a t a ( d s n ) : d a t a = [ ] e n g i n e = a w a i t c r e a t e _ e n g i n e ( d s n ) a s y n c w i t h C o n n e c t i o n C o n t e x t M a n a g e r ( e n g i n e ) a s c o n n : a s y n c f o r r o w i n R e s u l t I t e r ( a w a i t c o n n . e x e c u t e ( s a . s e l e c t ( . . . ) ) ) : d a t a . a p p e n d ( r o w ) r e t u r n d a t a i m p o r t a s y n c i o i m p o r t s q l a l c h e m y a s s a f r o m a i o p g . s a i m p o r t c r e a t e _ e n g i n e f r o m . u t i l s i m p o r t C o n n e c t i o n C o n t e x t M a n a g e r l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) d a t a = l o o p . r u n _ u n t i l _ c o m p l e t e ( f e t c h _ d a t a ( ' p o s t g r e s q l : / / . . . ' ) ) l o o p . c l o s e ( )
14. a s y n c f o r utils.py f r o m a i o p g . s a . e x c i m p o r t R e s o u r c e C l o s e d E r r o r c l a s s R e s u l t I t e r ( o b j e c t ) : d e f _ _ i n i t _ _ ( s e l f , r e s u l t ) : s e l f . r e s u l t = r e s u l t a s y n c d e f _ _ a i t e r _ _ ( s e l f ) : r e t u r n s e l f a s y n c d e f _ _ a n e x t _ _ ( s e l f ) : t r y : d a t a = a w a i t s e l f . r e s u l t . f e t c h o n e ( ) e x c e p t R e s o u r c e C l o s e d E r r o r : d a t a = N o n e i f d a t a : r e t u r n d a t a r a i s e S t o p A s y n c I t e r a t i o n
15. Other additions to standard library @ t y p e s . c o r o u t i n e bridge between generator based and native coroutines New _ _ a w a i t _ _ magic method New functions in i n s p e c t library as i s c o r o u t i n e , i s a w a i t a b l e , etc New abstract base classes: a b c . A w a i t a b l e , a b c . C o r o u t i n e , a b c . A s y n c I t e r a b l e , a b c . A s y n c I t e r a t o r
16. Conclusion on functions and methods Method Can contain Can't contain a s y n c d e f f u n c a w a i t , r e t u r n v a l u e y i e l d , y i e l d f r o m a s y n c d e f _ _ a * _ _ a w a i t , r e t u r n v a l u e y i e l d , y i e l d f r o m d e f _ _ a * _ _ r e t u r n a w a i t a b l e a w a i t d e f _ _ a w a i t _ _ y i e l d , y i e l d f r o m , r e t u r n i t e r a b l e a w a i t g e n e r a t o r y i e l d , y i e l d f r o m , r e t u r n v a l u e a w a i t
17. async/await in real life Libraries
18. aiohttp HTTP client/server for asyncio Latest version: 0 . 1 7 . 4 http://aiohttp.readthedocs.org/ i m p o r t a s y n c i o i m p o r t u j s o n f r o m a i o h t t p i m p o r t c l i e n t a s y n c d e f g i t h u b _ s e a r c h ( q u e r y ) : r e s p o n s e = a w a i t c l i e n t . g e t ( ' h t t p s : / / a p i . g i t h u b . c o m / s e a r c h / r e p o s i t o r i e s ' , p a r a m s = { ' q ' : q u e r y } ) r e t u r n u j s o n . l o a d s ( a w a i t r e s p o n s e . r e a d ( ) ) l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) r e s p o n s e = l o o p . r u n _ u n t i l _ c o m p l e t e ( g i t h u b _ s e a r c h ( ' a s y n c i o ' ) ) p r i n t ( ' \ n ' . j o i n ( r e p o [ ' f u l l _ n a m e ' ] f o r r e p o i n r e s p o n s e [ ' i t e m s ' ] ) ) l o o p . c l o s e ( )
19. aiohttp.web Web framework for asyncio http://aiohttp.readthedocs.org/en/latest/web.html i m p o r t u j s o n f r o m a i o h t t p i m p o r t w e b a s y n c d e f a p i _ i n d e x ( r e q u e s t ) : o u t p u t = u j s o n . d u m p s ( { . . . } ) r e t u r n w e b . R e s p o n s e ( b o d y = o u t p u t , c o n t e n t _ t y p e = ' a p p l i c a t i o n / j s o n ' ) a p p = w e b . A p p l i c a t i o n ( ) a p p . r o u t e r . a d d _ r o u t e ( ' G E T ' , ' / a p i / ' , a p i _ i n d e x ) $ g u n i c o r n - k a i o h t t p . w o r k e r . G u n i c o r n W e b W o r k e r - w 9 - t 6 0 a p p : a p p
20. aiopg Accessing PostgreSQL database from the asyncio Latest version: 0 . 7 . 0 http://aiopg.readthedocs.org/ i m p o r t a s y n c i o f r o m a i o p g i m p o r t c r e a t e _ p o o l a s y n c d e f s e l e c t _ o n e ( d s n ) : p o o l = a w a i t c r e a t e _ p o o l ( d s n ) w i t h ( a w a i t p o o l . c u r s o r ( ) ) a s c u r s o r : a w a i t c u r s o r . e x e c u t e ( ' S E L E C T 1 ' ) s e l e c t e d = a w a i t c u r s o r . f e t c h o n e ( ) a s s e r t s e l e c t e d = = ( 1 , ) l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) l o o p . r u n _ u n t i l _ c o m p l e t e ( s e l e c t _ o n e ( ' d b n a m e = . . . u s e r = . . . p a s s w o r d = . . . h o s t = . . . ' ) ) l o o p . c l o s e ( )
21. aiopg.sa Layer for executing SQLAlchemy Core queries http://aiopg.readthedocs.org/en/stable/sa.html i m p o r t a s y n c i o i m p o r t s q l a l c h e m y a s s a f r o m a i o p g . s a i m p o r t c r e a t e _ e n g i n e f r o m . u t i l s i m p o r t C o n n e c t i o n C o n t e x t M a n a g e r a s y n c d e f s e l e c t _ o n e ( d s n ) : e n g i n e = a w a i t c r e a t e _ e n g i n e ( d s n ) a s y n c w i t h C o n n e c t i o n C o n t e x t M a n a g e r ( e n g i n e ) a s c o n n : a w a i t c o n n . e x e c u t e ( s a . s e l e c t ( [ s a . t e x t ( ' 1 ' ) ] ) ) s e l e c t e d = a w a i t c o n n . f e t c h o n e ( ) a s s e r t s e l e c t e d = = ( 1 , ) l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) l o o p . r u n _ u n t i l _ c o m p l e t e ( s e l e c t _ o n e ( ' p o s t g r e s q l : / / . . . ' ) ) l o o p . c l o s e ( )
22. aioredis Redis client library Latest version: 0 . 2 . 4 http://aioredis.readthedocs.org/ i m p o r t a s y n c i o f r o m a i o r e d i s i m p o r t c r e a t e _ r e d i s a s y n c d e f r e d i s _ s e t _ g e t _ d e l e t e ( a d d r e s s , * * o p t i o n s ) : o p t i o n s . s e t d e f a u l t ( ' e n c o d i n g ' , ' u t f - 8 ' ) r e d i s = a w a i t c r e a t e _ r e d i s ( a d d r e s s , * * o p t i o n s ) a s s e r t a w a i t r e d i s . s e t ( ' k e y ' , ' v a l u e ' ) i s T r u e a s s e r t a w a i t r e d i s . g e t ( ' k e y ' ) = = ' v a l u e ' a s s e r t a w a i t r e d i s . d e l e t e ( ' k e y ' ) = = 1 l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) l o o p . r u n _ u n t i l _ c o m p l e t e ( r e d i s _ s e t _ g e t _ d e l e t e ( ( ' l o c a l h o s t ' , 6 3 7 9 ) ) ) l o o p . c l o s e ( )
23. And others Python Asyncio Resources MySQL: aiomysql Mongo: asyncio_mongo CouchDB: aiocouchdb ElasticSearch: aioes Memcached: aiomcache AMQP: aioamqp ØMQ: aiozmq
24. async/await in real life Fetching data from remote API
25. Task. Fetch data for the NFL Season NFL Season lasts 5 preseason and 17 regular season weeks This totals 22 requests to NFL.com endpoint Each request can be processed asyncronously After all requests are done we need to call extra function
26. Step 1. Sync fetch week data i m p o r t r e q u e s t s f r o m l x m l i m p o r t e t r e e N F L _ U R L = ' h t t p : / / w w w . n f l . c o m / a j a x / s c o r e s t r i p ' d e f f e t c h _ w e e k ( s e a s o n , w e e k , i s _ p r e s e a s o n = F a l s e ) : r e s p o n s e = r e q u e s t s . g e t ( N F L _ U R L , p a r a m s = { ' s e a s o n ' : s e a s o n , ' s e a s o n T y p e ' : ' P R E ' i f i s _ p r e s e a s o n e l s e ' R E G ' , ' w e e k ' : w e e k , } ) r e t u r n e t r e e . f r o m s t r i n g ( r e s p o n s e . c o n t e n t )
27. Step 2. Sync fetch season data d e f f e t c h _ s e a s o n ( s e a s o n ) : p r e _ c a l l ( . . . ) f o r i s _ p r e s e a s o n , w e e k s i n ( ( F a l s e , r a n g e ( 5 ) ) , ( T r u e , r a n g e ( 1 , 1 8 ) ) ) : f o r w e e k i n w e e k s : f e t c h _ w e e k ( s e a s o n , w e e k , i s _ p r e s e a s o n ) . . . p o s t _ c a l l ( . . . )
28. Step 3. Async fetch week data f r o m a i o h t t p i m p o r t c l i e n t f r o m l x m l i m p o r t e t r e e a s y n c d e f a i o _ f e t c h _ w e e k ( s e a s o n , w e e k , i s _ p r e s e a s o n = F a l s e ) : r e s p o n s e = a w a i t c l i e n t . g e t ( N F L _ U R L , p a r a m s = { ' s e a s o n ' : s e a s o n , ' s e a s o n T y p e ' : ' P R E ' i f i s _ p r e s e a s o n e l s e ' R E G ' , ' w e e k ' : w e e k , } ) r e t u r n e t r e e . f r o m s t r i n g ( a w a i t r e s p o n s e . r e a d ( ) )
29. Step 4. Async fetch season data i m p o r t a s y n c i o d e f f e t c h _ s e a s o n ( s e a s o n ) : l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) l o o p . r u n _ u n t i l _ c o m p l e t e ( p r e _ c a l l ( . . . ) ) t a s k s = [ a i o _ f e t c h _ w e e k ( s e a s o n , y e a r , i s _ p r e s e a s o n ) f o r i s _ p r e s e a s o n , w e e k s i n ( ( F a l s e , r a n g e ( 5 ) ) , ( T r u e , r a n g e ( 1 , 1 8 ) ) ) f o r w e e k i n w e e k s ] l o o p . r u n _ u n t i l _ c o m p l e t e ( a s y n c i o . w a i t ( t a s k s , l o o p = l o o p ) ) l o o p . r u n _ u n t i l _ c o m p l e t e ( p o s t _ c a l l ( . . . ) ) l o o p . c l o s e ( )
30. Step 5. Running fetching season data scripts/fetch_season_data.py i m p o r t s y s d e f m a i n ( * a r g s ) : f e t c h _ s e a s o n ( i n t ( a r g s [ 0 ] ) ) r e t u r n F a l s e i f _ _ n a m e _ _ = = ' _ _ m a i n _ _ ' : s y s . e x i t ( i n t ( m a i n ( * s y s . a r g v [ 1 : ] ) ) ) $ p y t h o n s c r i p t s / f e t c h _ s e a s o n _ d a t a . p y 2 0 1 5
31. Step 6. See in action . . . A l l g a m e s f o r S e a s o n 2 0 1 5 / R E G . W e e k 1 7 p r o c e s s e d ! S e a s o n 2 0 1 5 / R E G . W e e k 1 2 . G a m e I D 2 0 1 5 1 1 3 0 0 0 , C L E @ B A L , s c h e d u l e d a t 2 0 1 5 - 1 1 - 3 0 T 2 0 : 3 0 : 0 0 - 0 5 : 0 0 u p d a t e d i n d a t a b a s e A l l g a m e s f o r S e a s o n 2 0 1 5 / R E G . W e e k 1 2 p r o c e s s e d ! S e a s o n 2 0 1 5 / P R E . W e e k 4 . G a m e I D 2 0 1 5 0 9 0 3 6 1 , D A L @ H O U , s c h e d u l e d a t 2 0 1 5 - 0 9 - 0 3 T 2 0 : 0 0 : 0 0 - 0 4 : 0 0 u p d a t e d i n d a t a b a s e S e a s o n 2 0 1 5 / R E G . W e e k 1 6 . G a m e I D 2 0 1 5 1 2 2 7 1 1 , S E A @ S T L , s c h e d u l e d a t 2 0 1 5 - 1 2 - 2 7 T 1 6 : 2 5 : 0 0 - 0 5 : 0 0 u p d a t e d i n d a t a b a s e S e a s o n 2 0 1 5 / P R E . W e e k 4 . G a m e I D 2 0 1 5 0 9 0 3 6 2 , S T L @ K C , s c h e d u l e d a t 2 0 1 5 - 0 9 - 0 3 T 2 0 : 0 0 : 0 0 - 0 4 : 0 0 u p d a t e d i n d a t a b a s e S e a s o n 2 0 1 5 / R E G . W e e k 1 6 . G a m e I D 2 0 1 5 1 2 2 7 1 2 , B A L @ P I T , s c h e d u l e d a t 2 0 1 5 - 1 2 - 2 7 T 2 0 : 3 0 : 0 0 - 0 5 : 0 0 u p d a t e d i n d a t a b a s e S e a s o n 2 0 1 5 / P R E . W e e k 4 . G a m e I D 2 0 1 5 0 9 0 3 6 3 , T E N @ M I N , s c h e d u l e d a t 2 0 1 5 - 0 9 - 0 3 T 2 0 : 0 0 : 0 0 - 0 4 : 0 0 u p d a t e d i n d a t a b a s e S e a s o n 2 0 1 5 / R E G . W e e k 1 6 . G a m e I D 2 0 1 5 1 2 2 8 0 0 , D E N @ C I N , s c h e d u l e d a t 2 0 1 5 - 1 2 - 2 8 T 2 0 : 3 0 : 0 0 - 0 5 : 0 0 u p d a t e d i n d a t a b a s e A l l g a m e s f o r S e a s o n 2 0 1 5 / R E G . W e e k 1 6 p r o c e s s e d ! S e a s o n 2 0 1 5 / P R E . W e e k 4 . G a m e I D 2 0 1 5 0 9 0 3 6 4 , D E N @ A R I , s c h e d u l e d a t 2 0 1 5 - 0 9 - 0 3 T 2 1 : 0 0 : 0 0 - 0 4 : 0 0 u p d a t e d i n d a t a b a s e S e a s o n 2 0 1 5 / P R E . W e e k 4 . G a m e I D 2 0 1 5 0 9 0 3 6 5 , S F @ S D , s c h e d u l e d a t 2 0 1 5 - 0 9 - 0 3 T 2 2 : 0 0 : 0 0 - 0 4 : 0 0 u p d a t e d i n d a t a b a s e S e a s o n 2 0 1 5 / P R E . W e e k 4 . G a m e I D 2 0 1 5 0 9 0 3 6 6 , S E A @ O A K , s c h e d u l e d a t 2 0 1 5 - 0 9 - 0 3 T 2 2 : 0 0 : 0 0 - 0 4 : 0 0 u p d a t e d i n d a t a b a s e A l l g a m e s f o r S e a s o n 2 0 1 5 / P R E . W e e k 4 p r o c e s s e d ! . . .
32. async/await in real life Backend API application
33. Task. Provide an API for week data NFL.com still returns data in X M L We need this data in G r a p h Q L Okay, okay, actually in J S O N And if there are no live games, we don't need to fetch data directly from NFL
34. Step 0. Original NFL.com data
35. Step 1. Flask base application f r o m f l a s k i m p o r t F l a s k f r o m . i m p o r t c a c h e f r o m . d a t a i m p o r t t o _ g a m e _ d a t a a p p = F l a s k ( _ _ n a m e _ _ ) @ a p p . r o u t e ( ' / a p i / s e a s o n / < i n t : s e a s o n _ y e a r > / < w e e k _ s l u g > ' ) d e f r e t r i e v e _ w e e k ( s e a s o n _ y e a r , w e e k _ s l u g ) : i s _ p r e s e a s o n = w e e k _ s l u g [ : 4 ] = = ' p r e - ' w e e k = i n t ( w e e k _ s l u g [ 4 : ] i f i s _ p r e s e a s o n e l s e w e e k _ s l u g ) c a c h e _ k e y = c a c h e . b u i l d _ k e y ( s e a s o n _ y e a r , w e e k _ s l u g ) w e e k _ d a t a = c a c h e . e n s u r e _ d a t a ( c a c h e _ k e y ) i f w e e k _ d a t a i s N o n e : w e e k _ o b j = f e t c h _ w e e k ( s e a s o n _ y e a r , w e e k , i s _ p r e s e a s o n ) w e e k _ d a t a = [ t o _ g a m e _ d a t a ( g a m e ) f o r g a m e i n w e e k _ o b j . i t e r f i n d ( ' g m s / g ' ) ] c a c h e . s t o r e _ d a t a ( c a c h e _ k e y , w e e k _ d a t a ) r e t u r n j s o n i f y ( w e e k _ d a t a )
36. Step 2. Running Flask application $ g u n i c o r n - k s y n c - w 5 - t 6 0 a p p : a p p $ g u n i c o r n - k e v e n t l e t - w 5 - t 6 0 a p p : a p p
37. Step 3. aiohttp.web base application i m p o r t u j s o n f r o m a i o h t t p i m p o r t w e b a s y n c d e f a i o _ r e t r i e v e _ w e e k ( r e q u e s t ) : s e a s o n _ y e a r = i n t ( r e q u e s t . m a t c h _ i n f o [ ' s e a s o n _ y e a r ' ] ) w e e k _ s l u g = r e q u e s t . m a t c h _ i n f o [ ' w e e k _ s l u g ' ] i s _ p r e s e a s o n = w e e k _ s l u g [ : 4 ] = = ' p r e - ' w e e k = i n t ( w e e k _ s l u g [ 4 : ] i f i s _ p r e s e a s o n e l s e w e e k _ s l u g ) c a c h e _ k e y = c a c h e . b u i l d _ k e y ( s e a s o n _ y e a r , w e e k , i s _ p r e s e a s o n ) w e e k _ d a t a = a w a i t c a c h e . a i o _ e n s u r e _ d a t a ( c a c h e _ k e y ) i f w e e k _ d a t a i s N o n e : w e e k _ o b j = a w a i t a i o _ f e t c h _ w e e k ( s e a s o n _ y e a r , w e e k , i s _ p r e s e a s o n ) w e e k _ d a t a = [ t o _ g a m e _ d a t a ( g a m e ) f o r g a m e i n w e e k _ o b j . i t e r f i n d ( ' g m s / g ' ) ] a w a i t c a c h e . s t o r e _ d a t a ( c a c h e _ k e y , w e e k _ d a t a ) r e t u r n w e b . R e s p o n s e ( u j s o n . d u m p s ( w e e k _ d a t a ) , c o n t e n t _ t y p e = ' a p p l i c a t i o n / j s o n ' )
38. Step 3. aiohttp.web base application (continue) a i o _ a p p = w e b . A p p l i c a t i o n ( ) a i o _ a p p . r o u t e r . a d d _ r o u t e ( ' G E T ' , ' / a p i / s e a s o n / { s e a s o n _ y e a r : \ d { 4 } } / { w e e k _ s l u g } ' , a i o _ r e t r i e v e _ w e e k )
39. aiohttp.web basics Each view function should be a c o r o u t i n e /a s y n c d e f and return response or raise an exc Each view function receives only one arg: r e q u e s t Query params data contains in r e q u e s t . G E T multidict Body data contains in r e q u e s t . P O S T multidict and should be awaited with a w a i t r e q u e s t . p o s t ( ) Matched data contains in r e q u e s t . m a t c h _ i n f o dict r e q u e s t also contains an a p p instance
40. Step 4. Runinng aiohttp.web application $ g u n i c o r n - k a i o h t t p . w o r k e r . G u n i c o r n W e b W o r k e r - w 5 - t 6 0 a p p : a i o _ a p p
41. Step 5. Resulted JSON
42. async/await in real life Server-Sent Events
43. Task. Livescore stream When there are live games, update the scores automagically for the user Do not reconnect each X seconds for new scores When new scores available - show them to user This is the case for Server-Sent Events (SSE)
44. Server-Sent Events. Message Format d a t a : H e l l o , P y C o n F i n l a n d ! < b l a n k l i n e > e v e n t : p y c o n f i d a t a : H e l l o , P y c o n F i n l a n d ! d a t a : I ' m g l a d t o b e h e r e < b l a n k l i n k > i d : 4 2 e v e n t : u a p y c o n d a t a : P y C o n U k r a i n e 2 0 1 6 w i l l h e l d a p l a c e i n L v i v a t A p r i l 2 0 1 6 < b l a n k l i n k >
45. Server-Sent Events. EventSource c o n s t e l e m e n t = d o c u m e n t . g e t E l e m e n t B y I d ( " n o t i f i c a t i o n s - c o n t a i n e r " ) ; c o n s t s o u r c e = n e w E v e n t S o u r c e ( " / a p i / n o t i f i c a t i o n s " ) ; s o u r c e . o n e r r o r = e r r = > { . . . } ; s o u r c e . a d d E v e n t L i s t e n e r ( " p y c o n f i " , e v t = > { e l e m e n t . i n n e r H T M L + = ` < c o d e > # p y c o n f i < / c o d e > $ { e v t . d a t a } < b r > ` ; } ) ; s o u r c e . a d d E v e n t L i s t e n e r ( " u a p y c o n " , e v t = > { e l e m e n t . i n n e r H T M L + = ` < c o d e > # u a p y c o n < / c o d e > $ { e v t . d a t a } < b r > ` ; } ) ; . . . s o u r c e . c l o s e ( ) ;
46. How to implement? 1. Run script to fetch live scores and publish new scores to Redis PUBSUB 2. Subscribe to this channel in livescore stream view, receive message and push it to client 3. Subscribe to livescore stream in client and update scores when new message received 4. ... 5. PROFIT
47. Step 1. Fetching & publishing scores change d e f m a i n ( * a r g s ) : s e a s o n , w e e k = a r g s l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) w e e k _ o b j = l o o p . r u n _ u n t i l _ c o m p l e t e ( a i o _ f e t c h _ w e e k ( s e a s o n , w e e k , F a l s e ) ) w e e k _ d a t a = [ t o _ g a m e _ d a t a ( g a m e ) f o r g a m e i n w e e k _ o b j . i t e r f i n d ( ' g m s / g ' ) ] r e d i s = l o o p . r u n _ u n t i l _ c o m p l e t e ( c r e a t e _ r e d i s ( ( ' l o c a l h o s t ' , 6 3 7 9 ) ) ) l o o p . r u n _ u n t i l _ c o m p l e t e ( r e d i s . p u b l i s h ( ' C H A N N E L ' , u j s o n . d u m p s ( w e e k _ d a t a ) ) ) l o o p . c l o s e ( ) r e t u r n F a l s e i f _ _ n a m e _ _ = = ' _ _ m a i n _ _ ' : s y s . e x i t ( i n t ( m a i n ( * s y s . a r g v [ 1 : ] ) ) )
48. Step 2. Implementing livescore stream a s y n c d e f l i v e s c o r e ( r e q u e s t ) : i f r e q u e s t . h e a d e r s [ ' A c c e p t ' ] ! = ' t e x t / e v e n t - s t r e a m ' : r a i s e w e b . H T T P F o u n d ( ' / a p i / ' ) r e s p o n s e = w e b . S t r e a m R e s p o n s e ( s t a t u s = 2 0 0 , h e a d e r s = { ' C o n t e n t - T y p e ' : ' t e x t / e v e n t - s t r e a m ' , ' C a c h e - C o n t r o l ' : ' n o - c a c h e ' , ' C o n n e c t i o n ' : ' k e e p - a l i v e ' , } ) r e s p o n s e . s t a r t ( r e q u e s t ) # T h i s w i l l b e c h a n g e d i n a s y n c i o = = 0 . 1 8 . 0 . . . r e t u r n r e s p o n s e
49. Step 2. Subscribing to scores update . . . r e d i s = a w a i t c r e a t e _ r e d i s ( ( ' l o c a l h o s t ' , 6 3 7 9 ) ) c h a n n e l = ( a w a i t r e d i s . s u b s c r i b e ( ' C H A N N E L ' ) ) [ 0 ] w h i l e ( a w a i t c h a n n e l . w a i t _ m e s s a g e ( ) ) : m e s s a g e = a w a i t c h a n n e l . g e t ( ) r e s p o n s e . w r i t e ( b ' e v e n t : u p d a t e \ r \ n ' ) r e s p o n s e . w r i t e ( b ' d a t a : ' + m e s s a g e + b ' \ r \ n \ r \ n ' ) . . .
50. Step 2. Registering stream to the app a i o _ a p p . r o u t e r . a d d _ r o u t e ( ' G E T ' , ' / a p i / l i v e s c o r e ' , l i v e s c o r e )
51. Step 3. Subscribing to livescore stream on client i m p o r t R e a c t , { C o m p o n e n t } f r o m " r e a c t " ; e x p o r t d e f a u l t A p p e x t e n d s C o m p o n e n t { l i v e S c o r e = n u l l ; c o n s t r u c t o r ( p r o p s ) { s u p e r ( p r o p s ) ; t h i s . s t a t e = { g a m e s : p r o p s . g a m e s } ; } c o m p o n e n t D i d M o u n t ( ) { t h i s . l i v e S c o r e = n e w E v e n t S o u r c e ( " / a p i / " ) ; t h i s . l i v e S c o r e . a d d E v e n t L i s t e n e r ( " u p d a t e " , e v t = > { t h i s . s e t S t a t e ( { g a m e s : J S O N . p a r s e ( e v t . d a t a ) } ) ; } ) ; } c o m p o n e n t W i l l U n m o u n t ( ) { t h i s . l i v e S c o r e . c l o s e ( ) ; } r e n d e r ( ) { r e t u r n } } < G a m e s d a t a = { t h i s . s t a t e . g a m e s } / > ;
52. Step 4. Setting up the nginx l o c a t i o n / a p i / l i v e s c o r e { c h u n k e d _ t r a n s f e r _ e n c o d i n g o f f ; p r o x y _ b u f f e r i n g o f f ; p r o x y _ c a c h e o f f ; p r o x y _ h t t p _ v e r s i o n 1 . 1 ; p r o x y _ p a s s h t t p : / / 1 2 7 . 0 . 0 . 1 : 8 0 0 0 ; p r o x y _ s e t _ h e a d e r C o n n e c t i o n ' ' ; p r o x y _ s e t _ h e a d e r H o s t $ h o s t ; p r o x y _ s e t _ h e a d e r X - R e a l - I P $ r e m o t e _ a d d r ; p r o x y _ s e t _ h e a d e r X - F o r w a r d e d - F o r $ p r o x y _ a d d _ x _ f o r w a r d e d _ f o r ; }
53. Step 5. It works
54. async/await era Conclusion
55. The Future is Here Python 3.5 released and ready to be used in production Python 3.5 contains new a s y n c /a w a i t statements a s y n c i o stack got great addition, code now is simple to read and understand asyncio stack (for web applications): a i o h t t p – interact with remote API a i o h t t p . w e b – create API backends a i o p g – interact with PostgreSQL (DB) a i o r e d i s – interact with Redis (cache)
56. The Future is Here a s y n c i o stack is a great reason why you'll finally need to switch to Python 3 a s y n c i o is good fit for: Database-less API Requesting remote API Async code execution Predictable async I/O And I'm not telling you about WebSockets, which supported by a i o h t t p out of the box
57. And the real reason of using asyncio
58. Cause it cool and trendy!
59. Questions?
60. Bonus. PyCon Ukraine 2016 April 2016 Lviv, Ukraine Related activities Looking for speakers: @hotsyk