Pro Yearly is on sale from $80 to $50! »

Welcome to async/await era

Welcome to async/await era

PEP492 accepted and Python 3.5 will have new async/await statements. Talk will introduce them to the audience and shows examples on how to and when to use.

Talk also covers how async/await helps to write better and clear asyncio web applications on top of aiohttp library and shows common patterns to use, while making backend applications on Python 3.5.

D809571c81f596ccc197f90142c966ab?s=128

Igor Davydenko

October 19, 2015
Tweet

Transcript

  1. Welcome to async/await era Igor Davydenko PyCon Finland 2015

  2. I am... Igor Davydenko Python & React.js developer From Kyiv,

    Ukraine Works on Ezhome Inc. Primarly designs & develops backend API Personal Site @ GitHub @ Twitter
  3. PEP 492 Coroutines with async and await syntax

  4. Quick Summary Created by Yuri Selivanov in April, 2015 Included

    in Python 3.5 Added 4 new statements to the standard library: a s y n c d e f a w a i t a s y n c w i t h a s y n c f o r Received great response from the Python community PEP 492 @ Python.org
  5. @ a s y n c i o . c

    o r o u t i n e i m p o r t a s y n c i o @ a s y n c i o . c o r o u t i n e d e f h e l l o ( ) : r e t u r n ' H e l l o , w o r l d ! ' l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) m e s s a g e = l o o p . r u n _ u n t i l _ c o m p l e t e ( h e l l o ( ) ) p r i n t ( m e s s a g e ) l o o p . c l o s e ( )
  6. a s y n c d e f a s

    y n c d e f h e l l o ( ) : r e t u r n ' H e l l o , w o r l d ! ' i m p o r t a s y n c i o l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) m e s s a g e = l o o p . r u n _ u n t i l _ c o m p l e t e ( h e l l o ( ) ) p r i n t ( m e s s a g e ) l o o p . c l o s e ( )
  7. @ a s y n c i o . c

    o r o u t i n e + y i e l d f r o m i m p o r t a s y n c i o f r o m a i o h t t p i m p o r t c l i e n t @ a s y n c i o . c o r o u t i n e d e f f e t c h _ p a g e ( u r l ) : r e s p o n s e = y i e l d f r o m c l i e n t . g e t ( u r l ) r e t u r n ( y i e l d f r o m r e s p o n s e . t e x t ( ) ) l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) c o n t e n t = l o o p . r u n _ u n t i l _ c o m p l e t e ( f e t c h _ p a g e ( ' h t t p : / / f i . p y c o n . o r g / ' ) ) p r i n t ( c o n t e n t ) l o o p . c l o s e ( )
  8. a s y n c d e f + a

    w a i t a s y n c d e f f e t c h _ p a g e ( u r l ) : r e s p o n s e = a w a i t c l i e n t . g e t ( u r l ) r e t u r n a w a i t r e s p o n s e . t e x t ( ) i m p o r t a s y n c i o f r o m a i o h t t p i m p o r t c l i e n t l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) c o n t e n t = l o o p . r u n _ u n t i l _ c o m p l e t e ( f e t c h _ p a g e ( ' h t t p : / / f i . p y c o n . o r g / ' ) ) p r i n t ( c o n t e n t ) l o o p . c l o s e ( )
  9. w i t h ( y i e l d

    f r o m . . . ) i m p o r t a s y n c i o i m p o r t s q l a l c h e m y a s s a f r o m a i o p g . s a i m p o r t c r e a t e _ e n g i n e @ a s y n c i o . c o r o u t i n e d e f c o u n t _ d a t a ( d s n ) : e n g i n e = y i e l d f r o m c r e a t e _ e n g i n e ( d s n ) w i t h ( y i e l d f r o m e n g i n e ) a s c o n n : q u e r y = s a . s e l e c t ( . . . ) . c o u n t ( ) r e t u r n ( y i e l d f r o m c o n n . s c a l a r ( q u e r y ) ) l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) c o u n t e r = l o o p . r u n _ u n t i l _ c o m p l e t e ( c o u n t _ d a t a ( ' p o s t g r e s q l : / / . . . ' ) ) p r i n t ( c o u n t e r ) l o o p . c l o s e ( )
  10. a s y n c w i t h f

    r o m . u t i l s i m p o r t C o n n e c t i o n C o n t e x t M a n a g e r a s y n c d e f c o u n t _ d a t a ( d s n ) : e n g i n e = a w a i t c r e a t e _ e n g i n e ( d s n ) a s y n c w i t h C o n n e c t i o n C o n t e x t M a n a g e r ( e n g i n e ) a s c o n n : q u e r y = s a . s e l e c t ( . . . ) . c o u n t ( ) r e t u r n a w a i t c o n n . s c a l a r ( q u e r y ) i m p o r t a s y n c i o i m p o r t s q l a l c h e m y a s s a f r o m a i o p g . s a i m p o r t c r e a t e _ e n g i n e l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) c o u n t e r = l o o p . r u n _ u n t i l _ c o m p l e t e ( c o u n t _ d a t a ( ' p o s t g r e s q l : / / . . . ' ) ) p r i n t ( c o u n t e r ) l o o p . c l o s e ( )
  11. a s y n c w i t h utils.py

    c l a s s C o n n e c t i o n C o n t e x t M a n a g e r ( o b j e c t ) : d e f _ _ i n i t _ _ ( s e l f , e n g i n e ) : s e l f . c o n n = N o n e s e l f . e n g i n e = e n g i n e a s y n c d e f _ _ a e n t e r _ _ ( s e l f ) : s e l f . c o n n = a w a i t s e l f . e n g i n e . a c q u i r e ( ) r e t u r n s e l f . c o n n a s y n c d e f _ _ a e x i t _ _ ( s e l f , e x c _ t y p e , e x c , t b ) : t r y : s e l f . e n g i n e . r e l e a s e ( s e l f . c o n n ) f i n a l l y : s e l f . c o n n = N o n e s e l f . e n g i n e = N o n e
  12. f o r r o w i n ( y

    i e l d f r o m . . . ) : i m p o r t a s y n c i o i m p o r t s q l a l c h e m y a s s a f r o m a i o p g . s a i m p o r t c r e a t e _ e n g i n e @ a s y n c i o . c o r o u t i n e d e f f e t c h _ d a t a ( d s n ) : d a t a = [ ] e n g i n e = y i e l d f r o m c r e a t e _ e n g i n e ( d s n ) w i t h ( y i e l d f r o m e n g i n e ) a s c o n n : r e s u l t = y i e l d f r o m c o n n . e x e c u t e ( s a . s e l e c t ( . . . ) ) f o r r o w i n r e s u l t : d a t a . a p p e n d ( r o w ) r e t u r n d a t a l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) d a t a = l o o p . r u n _ u n t i l _ c o m p l e t e ( f e t c h _ d a t a ( ' p o s t g r e s q l : / / . . . ' ) ) l o o p . c l o s e ( )
  13. a s y n c f o r , R

    e s u l t I t e r a s y n c d e f f e t c h _ d a t a ( d s n ) : d a t a = [ ] e n g i n e = a w a i t c r e a t e _ e n g i n e ( d s n ) a s y n c w i t h C o n n e c t i o n C o n t e x t M a n a g e r ( e n g i n e ) a s c o n n : a s y n c f o r r o w i n R e s u l t I t e r ( a w a i t c o n n . e x e c u t e ( s a . s e l e c t ( . . . ) ) ) : d a t a . a p p e n d ( r o w ) r e t u r n d a t a i m p o r t a s y n c i o i m p o r t s q l a l c h e m y a s s a f r o m a i o p g . s a i m p o r t c r e a t e _ e n g i n e f r o m . u t i l s i m p o r t C o n n e c t i o n C o n t e x t M a n a g e r l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) d a t a = l o o p . r u n _ u n t i l _ c o m p l e t e ( f e t c h _ d a t a ( ' p o s t g r e s q l : / / . . . ' ) ) l o o p . c l o s e ( )
  14. a s y n c f o r utils.py f

    r o m a i o p g . s a . e x c i m p o r t R e s o u r c e C l o s e d E r r o r c l a s s R e s u l t I t e r ( o b j e c t ) : d e f _ _ i n i t _ _ ( s e l f , r e s u l t ) : s e l f . r e s u l t = r e s u l t a s y n c d e f _ _ a i t e r _ _ ( s e l f ) : r e t u r n s e l f a s y n c d e f _ _ a n e x t _ _ ( s e l f ) : t r y : d a t a = a w a i t s e l f . r e s u l t . f e t c h o n e ( ) e x c e p t R e s o u r c e C l o s e d E r r o r : d a t a = N o n e i f d a t a : r e t u r n d a t a r a i s e S t o p A s y n c I t e r a t i o n
  15. Other additions to standard library @ t y p e

    s . c o r o u t i n e bridge between generator based and native coroutines New _ _ a w a i t _ _ magic method New functions in i n s p e c t library as i s c o r o u t i n e , i s a w a i t a b l e , etc New abstract base classes: a b c . A w a i t a b l e , a b c . C o r o u t i n e , a b c . A s y n c I t e r a b l e , a b c . A s y n c I t e r a t o r
  16. Conclusion on functions and methods Method Can contain Can't contain

    a s y n c d e f f u n c a w a i t , r e t u r n v a l u e y i e l d , y i e l d f r o m a s y n c d e f _ _ a * _ _ a w a i t , r e t u r n v a l u e y i e l d , y i e l d f r o m d e f _ _ a * _ _ r e t u r n a w a i t a b l e a w a i t d e f _ _ a w a i t _ _ y i e l d , y i e l d f r o m , r e t u r n i t e r a b l e a w a i t g e n e r a t o r y i e l d , y i e l d f r o m , r e t u r n v a l u e a w a i t
  17. async/await in real life Libraries

  18. aiohttp HTTP client/server for asyncio Latest version: 0 . 1

    7 . 4 http://aiohttp.readthedocs.org/ i m p o r t a s y n c i o i m p o r t u j s o n f r o m a i o h t t p i m p o r t c l i e n t a s y n c d e f g i t h u b _ s e a r c h ( q u e r y ) : r e s p o n s e = a w a i t c l i e n t . g e t ( ' h t t p s : / / a p i . g i t h u b . c o m / s e a r c h / r e p o s i t o r i e s ' , p a r a m s = { ' q ' : q u e r y } ) r e t u r n u j s o n . l o a d s ( a w a i t r e s p o n s e . r e a d ( ) ) l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) r e s p o n s e = l o o p . r u n _ u n t i l _ c o m p l e t e ( g i t h u b _ s e a r c h ( ' a s y n c i o ' ) ) p r i n t ( ' \ n ' . j o i n ( r e p o [ ' f u l l _ n a m e ' ] f o r r e p o i n r e s p o n s e [ ' i t e m s ' ] ) ) l o o p . c l o s e ( )
  19. aiohttp.web Web framework for asyncio http://aiohttp.readthedocs.org/en/latest/web.html i m p o

    r t u j s o n f r o m a i o h t t p i m p o r t w e b a s y n c d e f a p i _ i n d e x ( r e q u e s t ) : o u t p u t = u j s o n . d u m p s ( { . . . } ) r e t u r n w e b . R e s p o n s e ( b o d y = o u t p u t , c o n t e n t _ t y p e = ' a p p l i c a t i o n / j s o n ' ) a p p = w e b . A p p l i c a t i o n ( ) a p p . r o u t e r . a d d _ r o u t e ( ' G E T ' , ' / a p i / ' , a p i _ i n d e x ) $ g u n i c o r n - k a i o h t t p . w o r k e r . G u n i c o r n W e b W o r k e r - w 9 - t 6 0 a p p : a p p
  20. aiopg Accessing PostgreSQL database from the asyncio Latest version: 0

    . 7 . 0 http://aiopg.readthedocs.org/ i m p o r t a s y n c i o f r o m a i o p g i m p o r t c r e a t e _ p o o l a s y n c d e f s e l e c t _ o n e ( d s n ) : p o o l = a w a i t c r e a t e _ p o o l ( d s n ) w i t h ( a w a i t p o o l . c u r s o r ( ) ) a s c u r s o r : a w a i t c u r s o r . e x e c u t e ( ' S E L E C T 1 ' ) s e l e c t e d = a w a i t c u r s o r . f e t c h o n e ( ) a s s e r t s e l e c t e d = = ( 1 , ) l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) l o o p . r u n _ u n t i l _ c o m p l e t e ( s e l e c t _ o n e ( ' d b n a m e = . . . u s e r = . . . p a s s w o r d = . . . h o s t = . . . ' ) ) l o o p . c l o s e ( )
  21. aiopg.sa Layer for executing SQLAlchemy Core queries http://aiopg.readthedocs.org/en/stable/sa.html i m

    p o r t a s y n c i o i m p o r t s q l a l c h e m y a s s a f r o m a i o p g . s a i m p o r t c r e a t e _ e n g i n e f r o m . u t i l s i m p o r t C o n n e c t i o n C o n t e x t M a n a g e r a s y n c d e f s e l e c t _ o n e ( d s n ) : e n g i n e = a w a i t c r e a t e _ e n g i n e ( d s n ) a s y n c w i t h C o n n e c t i o n C o n t e x t M a n a g e r ( e n g i n e ) a s c o n n : a w a i t c o n n . e x e c u t e ( s a . s e l e c t ( [ s a . t e x t ( ' 1 ' ) ] ) ) s e l e c t e d = a w a i t c o n n . f e t c h o n e ( ) a s s e r t s e l e c t e d = = ( 1 , ) l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) l o o p . r u n _ u n t i l _ c o m p l e t e ( s e l e c t _ o n e ( ' p o s t g r e s q l : / / . . . ' ) ) l o o p . c l o s e ( )
  22. aioredis Redis client library Latest version: 0 . 2 .

    4 http://aioredis.readthedocs.org/ i m p o r t a s y n c i o f r o m a i o r e d i s i m p o r t c r e a t e _ r e d i s a s y n c d e f r e d i s _ s e t _ g e t _ d e l e t e ( a d d r e s s , * * o p t i o n s ) : o p t i o n s . s e t d e f a u l t ( ' e n c o d i n g ' , ' u t f - 8 ' ) r e d i s = a w a i t c r e a t e _ r e d i s ( a d d r e s s , * * o p t i o n s ) a s s e r t a w a i t r e d i s . s e t ( ' k e y ' , ' v a l u e ' ) i s T r u e a s s e r t a w a i t r e d i s . g e t ( ' k e y ' ) = = ' v a l u e ' a s s e r t a w a i t r e d i s . d e l e t e ( ' k e y ' ) = = 1 l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) l o o p . r u n _ u n t i l _ c o m p l e t e ( r e d i s _ s e t _ g e t _ d e l e t e ( ( ' l o c a l h o s t ' , 6 3 7 9 ) ) ) l o o p . c l o s e ( )
  23. And others Python Asyncio Resources MySQL: aiomysql Mongo: asyncio_mongo CouchDB:

    aiocouchdb ElasticSearch: aioes Memcached: aiomcache AMQP: aioamqp ØMQ: aiozmq
  24. async/await in real life Fetching data from remote API

  25. Task. Fetch data for the NFL Season NFL Season lasts

    5 preseason and 17 regular season weeks This totals 22 requests to NFL.com endpoint Each request can be processed asyncronously After all requests are done we need to call extra function
  26. Step 1. Sync fetch week data i m p o

    r t r e q u e s t s f r o m l x m l i m p o r t e t r e e N F L _ U R L = ' h t t p : / / w w w . n f l . c o m / a j a x / s c o r e s t r i p ' d e f f e t c h _ w e e k ( s e a s o n , w e e k , i s _ p r e s e a s o n = F a l s e ) : r e s p o n s e = r e q u e s t s . g e t ( N F L _ U R L , p a r a m s = { ' s e a s o n ' : s e a s o n , ' s e a s o n T y p e ' : ' P R E ' i f i s _ p r e s e a s o n e l s e ' R E G ' , ' w e e k ' : w e e k , } ) r e t u r n e t r e e . f r o m s t r i n g ( r e s p o n s e . c o n t e n t )
  27. Step 2. Sync fetch season data d e f f

    e t c h _ s e a s o n ( s e a s o n ) : p r e _ c a l l ( . . . ) f o r i s _ p r e s e a s o n , w e e k s i n ( ( F a l s e , r a n g e ( 5 ) ) , ( T r u e , r a n g e ( 1 , 1 8 ) ) ) : f o r w e e k i n w e e k s : f e t c h _ w e e k ( s e a s o n , w e e k , i s _ p r e s e a s o n ) . . . p o s t _ c a l l ( . . . )
  28. Step 3. Async fetch week data f r o m

    a i o h t t p i m p o r t c l i e n t f r o m l x m l i m p o r t e t r e e a s y n c d e f a i o _ f e t c h _ w e e k ( s e a s o n , w e e k , i s _ p r e s e a s o n = F a l s e ) : r e s p o n s e = a w a i t c l i e n t . g e t ( N F L _ U R L , p a r a m s = { ' s e a s o n ' : s e a s o n , ' s e a s o n T y p e ' : ' P R E ' i f i s _ p r e s e a s o n e l s e ' R E G ' , ' w e e k ' : w e e k , } ) r e t u r n e t r e e . f r o m s t r i n g ( a w a i t r e s p o n s e . r e a d ( ) )
  29. Step 4. Async fetch season data i m p o

    r t a s y n c i o d e f f e t c h _ s e a s o n ( s e a s o n ) : l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) l o o p . r u n _ u n t i l _ c o m p l e t e ( p r e _ c a l l ( . . . ) ) t a s k s = [ a i o _ f e t c h _ w e e k ( s e a s o n , y e a r , i s _ p r e s e a s o n ) f o r i s _ p r e s e a s o n , w e e k s i n ( ( F a l s e , r a n g e ( 5 ) ) , ( T r u e , r a n g e ( 1 , 1 8 ) ) ) f o r w e e k i n w e e k s ] l o o p . r u n _ u n t i l _ c o m p l e t e ( a s y n c i o . w a i t ( t a s k s , l o o p = l o o p ) ) l o o p . r u n _ u n t i l _ c o m p l e t e ( p o s t _ c a l l ( . . . ) ) l o o p . c l o s e ( )
  30. Step 5. Running fetching season data scripts/fetch_season_data.py i m p

    o r t s y s d e f m a i n ( * a r g s ) : f e t c h _ s e a s o n ( i n t ( a r g s [ 0 ] ) ) r e t u r n F a l s e i f _ _ n a m e _ _ = = ' _ _ m a i n _ _ ' : s y s . e x i t ( i n t ( m a i n ( * s y s . a r g v [ 1 : ] ) ) ) $ p y t h o n s c r i p t s / f e t c h _ s e a s o n _ d a t a . p y 2 0 1 5
  31. Step 6. See in action . . . A l

    l g a m e s f o r S e a s o n 2 0 1 5 / R E G . W e e k 1 7 p r o c e s s e d ! S e a s o n 2 0 1 5 / R E G . W e e k 1 2 . G a m e I D 2 0 1 5 1 1 3 0 0 0 , C L E @ B A L , s c h e d u l e d a t 2 0 1 5 - 1 1 - 3 0 T 2 0 : 3 0 : 0 0 - 0 5 : 0 0 u p d a t e d i n d a t a b a s e A l l g a m e s f o r S e a s o n 2 0 1 5 / R E G . W e e k 1 2 p r o c e s s e d ! S e a s o n 2 0 1 5 / P R E . W e e k 4 . G a m e I D 2 0 1 5 0 9 0 3 6 1 , D A L @ H O U , s c h e d u l e d a t 2 0 1 5 - 0 9 - 0 3 T 2 0 : 0 0 : 0 0 - 0 4 : 0 0 u p d a t e d i n d a t a b a s e S e a s o n 2 0 1 5 / R E G . W e e k 1 6 . G a m e I D 2 0 1 5 1 2 2 7 1 1 , S E A @ S T L , s c h e d u l e d a t 2 0 1 5 - 1 2 - 2 7 T 1 6 : 2 5 : 0 0 - 0 5 : 0 0 u p d a t e d i n d a t a b a s e S e a s o n 2 0 1 5 / P R E . W e e k 4 . G a m e I D 2 0 1 5 0 9 0 3 6 2 , S T L @ K C , s c h e d u l e d a t 2 0 1 5 - 0 9 - 0 3 T 2 0 : 0 0 : 0 0 - 0 4 : 0 0 u p d a t e d i n d a t a b a s e S e a s o n 2 0 1 5 / R E G . W e e k 1 6 . G a m e I D 2 0 1 5 1 2 2 7 1 2 , B A L @ P I T , s c h e d u l e d a t 2 0 1 5 - 1 2 - 2 7 T 2 0 : 3 0 : 0 0 - 0 5 : 0 0 u p d a t e d i n d a t a b a s e S e a s o n 2 0 1 5 / P R E . W e e k 4 . G a m e I D 2 0 1 5 0 9 0 3 6 3 , T E N @ M I N , s c h e d u l e d a t 2 0 1 5 - 0 9 - 0 3 T 2 0 : 0 0 : 0 0 - 0 4 : 0 0 u p d a t e d i n d a t a b a s e S e a s o n 2 0 1 5 / R E G . W e e k 1 6 . G a m e I D 2 0 1 5 1 2 2 8 0 0 , D E N @ C I N , s c h e d u l e d a t 2 0 1 5 - 1 2 - 2 8 T 2 0 : 3 0 : 0 0 - 0 5 : 0 0 u p d a t e d i n d a t a b a s e A l l g a m e s f o r S e a s o n 2 0 1 5 / R E G . W e e k 1 6 p r o c e s s e d ! S e a s o n 2 0 1 5 / P R E . W e e k 4 . G a m e I D 2 0 1 5 0 9 0 3 6 4 , D E N @ A R I , s c h e d u l e d a t 2 0 1 5 - 0 9 - 0 3 T 2 1 : 0 0 : 0 0 - 0 4 : 0 0 u p d a t e d i n d a t a b a s e S e a s o n 2 0 1 5 / P R E . W e e k 4 . G a m e I D 2 0 1 5 0 9 0 3 6 5 , S F @ S D , s c h e d u l e d a t 2 0 1 5 - 0 9 - 0 3 T 2 2 : 0 0 : 0 0 - 0 4 : 0 0 u p d a t e d i n d a t a b a s e S e a s o n 2 0 1 5 / P R E . W e e k 4 . G a m e I D 2 0 1 5 0 9 0 3 6 6 , S E A @ O A K , s c h e d u l e d a t 2 0 1 5 - 0 9 - 0 3 T 2 2 : 0 0 : 0 0 - 0 4 : 0 0 u p d a t e d i n d a t a b a s e A l l g a m e s f o r S e a s o n 2 0 1 5 / P R E . W e e k 4 p r o c e s s e d ! . . .
  32. async/await in real life Backend API application

  33. Task. Provide an API for week data NFL.com still returns

    data in X M L We need this data in G r a p h Q L Okay, okay, actually in J S O N And if there are no live games, we don't need to fetch data directly from NFL
  34. Step 0. Original NFL.com data

  35. Step 1. Flask base application f r o m f

    l a s k i m p o r t F l a s k f r o m . i m p o r t c a c h e f r o m . d a t a i m p o r t t o _ g a m e _ d a t a a p p = F l a s k ( _ _ n a m e _ _ ) @ a p p . r o u t e ( ' / a p i / s e a s o n / < i n t : s e a s o n _ y e a r > / < w e e k _ s l u g > ' ) d e f r e t r i e v e _ w e e k ( s e a s o n _ y e a r , w e e k _ s l u g ) : i s _ p r e s e a s o n = w e e k _ s l u g [ : 4 ] = = ' p r e - ' w e e k = i n t ( w e e k _ s l u g [ 4 : ] i f i s _ p r e s e a s o n e l s e w e e k _ s l u g ) c a c h e _ k e y = c a c h e . b u i l d _ k e y ( s e a s o n _ y e a r , w e e k _ s l u g ) w e e k _ d a t a = c a c h e . e n s u r e _ d a t a ( c a c h e _ k e y ) i f w e e k _ d a t a i s N o n e : w e e k _ o b j = f e t c h _ w e e k ( s e a s o n _ y e a r , w e e k , i s _ p r e s e a s o n ) w e e k _ d a t a = [ t o _ g a m e _ d a t a ( g a m e ) f o r g a m e i n w e e k _ o b j . i t e r f i n d ( ' g m s / g ' ) ] c a c h e . s t o r e _ d a t a ( c a c h e _ k e y , w e e k _ d a t a ) r e t u r n j s o n i f y ( w e e k _ d a t a )
  36. Step 2. Running Flask application $ g u n i

    c o r n - k s y n c - w 5 - t 6 0 a p p : a p p $ g u n i c o r n - k e v e n t l e t - w 5 - t 6 0 a p p : a p p
  37. Step 3. aiohttp.web base application i m p o r

    t u j s o n f r o m a i o h t t p i m p o r t w e b a s y n c d e f a i o _ r e t r i e v e _ w e e k ( r e q u e s t ) : s e a s o n _ y e a r = i n t ( r e q u e s t . m a t c h _ i n f o [ ' s e a s o n _ y e a r ' ] ) w e e k _ s l u g = r e q u e s t . m a t c h _ i n f o [ ' w e e k _ s l u g ' ] i s _ p r e s e a s o n = w e e k _ s l u g [ : 4 ] = = ' p r e - ' w e e k = i n t ( w e e k _ s l u g [ 4 : ] i f i s _ p r e s e a s o n e l s e w e e k _ s l u g ) c a c h e _ k e y = c a c h e . b u i l d _ k e y ( s e a s o n _ y e a r , w e e k , i s _ p r e s e a s o n ) w e e k _ d a t a = a w a i t c a c h e . a i o _ e n s u r e _ d a t a ( c a c h e _ k e y ) i f w e e k _ d a t a i s N o n e : w e e k _ o b j = a w a i t a i o _ f e t c h _ w e e k ( s e a s o n _ y e a r , w e e k , i s _ p r e s e a s o n ) w e e k _ d a t a = [ t o _ g a m e _ d a t a ( g a m e ) f o r g a m e i n w e e k _ o b j . i t e r f i n d ( ' g m s / g ' ) ] a w a i t c a c h e . s t o r e _ d a t a ( c a c h e _ k e y , w e e k _ d a t a ) r e t u r n w e b . R e s p o n s e ( u j s o n . d u m p s ( w e e k _ d a t a ) , c o n t e n t _ t y p e = ' a p p l i c a t i o n / j s o n ' )
  38. Step 3. aiohttp.web base application (continue) a i o _

    a p p = w e b . A p p l i c a t i o n ( ) a i o _ a p p . r o u t e r . a d d _ r o u t e ( ' G E T ' , ' / a p i / s e a s o n / { s e a s o n _ y e a r : \ d { 4 } } / { w e e k _ s l u g } ' , a i o _ r e t r i e v e _ w e e k )
  39. aiohttp.web basics Each view function should be a c o

    r o u t i n e /a s y n c d e f and return response or raise an exc Each view function receives only one arg: r e q u e s t Query params data contains in r e q u e s t . G E T multidict Body data contains in r e q u e s t . P O S T multidict and should be awaited with a w a i t r e q u e s t . p o s t ( ) Matched data contains in r e q u e s t . m a t c h _ i n f o dict r e q u e s t also contains an a p p instance
  40. Step 4. Runinng aiohttp.web application $ g u n i

    c o r n - k a i o h t t p . w o r k e r . G u n i c o r n W e b W o r k e r - w 5 - t 6 0 a p p : a i o _ a p p
  41. Step 5. Resulted JSON

  42. async/await in real life Server-Sent Events

  43. Task. Livescore stream When there are live games, update the

    scores automagically for the user Do not reconnect each X seconds for new scores When new scores available - show them to user This is the case for Server-Sent Events (SSE)
  44. Server-Sent Events. Message Format d a t a : H

    e l l o , P y C o n F i n l a n d ! < b l a n k l i n e > e v e n t : p y c o n f i d a t a : H e l l o , P y c o n F i n l a n d ! d a t a : I ' m g l a d t o b e h e r e < b l a n k l i n k > i d : 4 2 e v e n t : u a p y c o n d a t a : P y C o n U k r a i n e 2 0 1 6 w i l l h e l d a p l a c e i n L v i v a t A p r i l 2 0 1 6 < b l a n k l i n k >
  45. Server-Sent Events. EventSource c o n s t e l

    e m e n t = d o c u m e n t . g e t E l e m e n t B y I d ( " n o t i f i c a t i o n s - c o n t a i n e r " ) ; c o n s t s o u r c e = n e w E v e n t S o u r c e ( " / a p i / n o t i f i c a t i o n s " ) ; s o u r c e . o n e r r o r = e r r = > { . . . } ; s o u r c e . a d d E v e n t L i s t e n e r ( " p y c o n f i " , e v t = > { e l e m e n t . i n n e r H T M L + = ` < c o d e > # p y c o n f i < / c o d e > $ { e v t . d a t a } < b r > ` ; } ) ; s o u r c e . a d d E v e n t L i s t e n e r ( " u a p y c o n " , e v t = > { e l e m e n t . i n n e r H T M L + = ` < c o d e > # u a p y c o n < / c o d e > $ { e v t . d a t a } < b r > ` ; } ) ; . . . s o u r c e . c l o s e ( ) ;
  46. How to implement? 1. Run script to fetch live scores

    and publish new scores to Redis PUBSUB 2. Subscribe to this channel in livescore stream view, receive message and push it to client 3. Subscribe to livescore stream in client and update scores when new message received 4. ... 5. PROFIT
  47. Step 1. Fetching & publishing scores change d e f

    m a i n ( * a r g s ) : s e a s o n , w e e k = a r g s l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) w e e k _ o b j = l o o p . r u n _ u n t i l _ c o m p l e t e ( a i o _ f e t c h _ w e e k ( s e a s o n , w e e k , F a l s e ) ) w e e k _ d a t a = [ t o _ g a m e _ d a t a ( g a m e ) f o r g a m e i n w e e k _ o b j . i t e r f i n d ( ' g m s / g ' ) ] r e d i s = l o o p . r u n _ u n t i l _ c o m p l e t e ( c r e a t e _ r e d i s ( ( ' l o c a l h o s t ' , 6 3 7 9 ) ) ) l o o p . r u n _ u n t i l _ c o m p l e t e ( r e d i s . p u b l i s h ( ' C H A N N E L ' , u j s o n . d u m p s ( w e e k _ d a t a ) ) ) l o o p . c l o s e ( ) r e t u r n F a l s e i f _ _ n a m e _ _ = = ' _ _ m a i n _ _ ' : s y s . e x i t ( i n t ( m a i n ( * s y s . a r g v [ 1 : ] ) ) )
  48. Step 2. Implementing livescore stream a s y n c

    d e f l i v e s c o r e ( r e q u e s t ) : i f r e q u e s t . h e a d e r s [ ' A c c e p t ' ] ! = ' t e x t / e v e n t - s t r e a m ' : r a i s e w e b . H T T P F o u n d ( ' / a p i / ' ) r e s p o n s e = w e b . S t r e a m R e s p o n s e ( s t a t u s = 2 0 0 , h e a d e r s = { ' C o n t e n t - T y p e ' : ' t e x t / e v e n t - s t r e a m ' , ' C a c h e - C o n t r o l ' : ' n o - c a c h e ' , ' C o n n e c t i o n ' : ' k e e p - a l i v e ' , } ) r e s p o n s e . s t a r t ( r e q u e s t ) # T h i s w i l l b e c h a n g e d i n a s y n c i o = = 0 . 1 8 . 0 . . . r e t u r n r e s p o n s e
  49. Step 2. Subscribing to scores update . . . r

    e d i s = a w a i t c r e a t e _ r e d i s ( ( ' l o c a l h o s t ' , 6 3 7 9 ) ) c h a n n e l = ( a w a i t r e d i s . s u b s c r i b e ( ' C H A N N E L ' ) ) [ 0 ] w h i l e ( a w a i t c h a n n e l . w a i t _ m e s s a g e ( ) ) : m e s s a g e = a w a i t c h a n n e l . g e t ( ) r e s p o n s e . w r i t e ( b ' e v e n t : u p d a t e \ r \ n ' ) r e s p o n s e . w r i t e ( b ' d a t a : ' + m e s s a g e + b ' \ r \ n \ r \ n ' ) . . .
  50. Step 2. Registering stream to the app a i o

    _ a p p . r o u t e r . a d d _ r o u t e ( ' G E T ' , ' / a p i / l i v e s c o r e ' , l i v e s c o r e )
  51. Step 3. Subscribing to livescore stream on client i m

    p o r t R e a c t , { C o m p o n e n t } f r o m " r e a c t " ; e x p o r t d e f a u l t A p p e x t e n d s C o m p o n e n t { l i v e S c o r e = n u l l ; c o n s t r u c t o r ( p r o p s ) { s u p e r ( p r o p s ) ; t h i s . s t a t e = { g a m e s : p r o p s . g a m e s } ; } c o m p o n e n t D i d M o u n t ( ) { t h i s . l i v e S c o r e = n e w E v e n t S o u r c e ( " / a p i / " ) ; t h i s . l i v e S c o r e . a d d E v e n t L i s t e n e r ( " u p d a t e " , e v t = > { t h i s . s e t S t a t e ( { g a m e s : J S O N . p a r s e ( e v t . d a t a ) } ) ; } ) ; } c o m p o n e n t W i l l U n m o u n t ( ) { t h i s . l i v e S c o r e . c l o s e ( ) ; } r e n d e r ( ) { r e t u r n } } < G a m e s d a t a = { t h i s . s t a t e . g a m e s } / > ;
  52. Step 4. Setting up the nginx l o c a

    t i o n / a p i / l i v e s c o r e { c h u n k e d _ t r a n s f e r _ e n c o d i n g o f f ; p r o x y _ b u f f e r i n g o f f ; p r o x y _ c a c h e o f f ; p r o x y _ h t t p _ v e r s i o n 1 . 1 ; p r o x y _ p a s s h t t p : / / 1 2 7 . 0 . 0 . 1 : 8 0 0 0 ; p r o x y _ s e t _ h e a d e r C o n n e c t i o n ' ' ; p r o x y _ s e t _ h e a d e r H o s t $ h o s t ; p r o x y _ s e t _ h e a d e r X - R e a l - I P $ r e m o t e _ a d d r ; p r o x y _ s e t _ h e a d e r X - F o r w a r d e d - F o r $ p r o x y _ a d d _ x _ f o r w a r d e d _ f o r ; }
  53. Step 5. It works

  54. async/await era Conclusion

  55. The Future is Here Python 3.5 released and ready to

    be used in production Python 3.5 contains new a s y n c /a w a i t statements a s y n c i o stack got great addition, code now is simple to read and understand asyncio stack (for web applications): a i o h t t p – interact with remote API a i o h t t p . w e b – create API backends a i o p g – interact with PostgreSQL (DB) a i o r e d i s – interact with Redis (cache)
  56. The Future is Here a s y n c i

    o stack is a great reason why you'll finally need to switch to Python 3 a s y n c i o is good fit for: Database-less API Requesting remote API Async code execution Predictable async I/O And I'm not telling you about WebSockets, which supported by a i o h t t p out of the box
  57. And the real reason of using asyncio

  58. Cause it cool and trendy!

  59. Questions?

  60. Bonus. PyCon Ukraine 2016 April 2016 Lviv, Ukraine Related activities

    Looking for speakers: @hotsyk