Welcome to async/await era

Welcome to async/await era

PEP492 accepted and Python 3.5 will have new async/await statements. Talk will introduce them to the audience and shows examples on how to and when to use.

Talk also covers how async/await helps to write better and clear asyncio web applications on top of aiohttp library and shows common patterns to use, while making backend applications on Python 3.5.

D809571c81f596ccc197f90142c966ab?s=128

Igor Davydenko

October 19, 2015
Tweet

Transcript

  1. 2.

    I am... Igor Davydenko Python & React.js developer From Kyiv,

    Ukraine Works on Ezhome Inc. Primarly designs & develops backend API Personal Site @ GitHub @ Twitter
  2. 4.

    Quick Summary Created by Yuri Selivanov in April, 2015 Included

    in Python 3.5 Added 4 new statements to the standard library: a s y n c d e f a w a i t a s y n c w i t h a s y n c f o r Received great response from the Python community PEP 492 @ Python.org
  3. 5.

    @ a s y n c i o . c

    o r o u t i n e i m p o r t a s y n c i o @ a s y n c i o . c o r o u t i n e d e f h e l l o ( ) : r e t u r n ' H e l l o , w o r l d ! ' l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) m e s s a g e = l o o p . r u n _ u n t i l _ c o m p l e t e ( h e l l o ( ) ) p r i n t ( m e s s a g e ) l o o p . c l o s e ( )
  4. 6.

    a s y n c d e f a s

    y n c d e f h e l l o ( ) : r e t u r n ' H e l l o , w o r l d ! ' i m p o r t a s y n c i o l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) m e s s a g e = l o o p . r u n _ u n t i l _ c o m p l e t e ( h e l l o ( ) ) p r i n t ( m e s s a g e ) l o o p . c l o s e ( )
  5. 7.

    @ a s y n c i o . c

    o r o u t i n e + y i e l d f r o m i m p o r t a s y n c i o f r o m a i o h t t p i m p o r t c l i e n t @ a s y n c i o . c o r o u t i n e d e f f e t c h _ p a g e ( u r l ) : r e s p o n s e = y i e l d f r o m c l i e n t . g e t ( u r l ) r e t u r n ( y i e l d f r o m r e s p o n s e . t e x t ( ) ) l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) c o n t e n t = l o o p . r u n _ u n t i l _ c o m p l e t e ( f e t c h _ p a g e ( ' h t t p : / / f i . p y c o n . o r g / ' ) ) p r i n t ( c o n t e n t ) l o o p . c l o s e ( )
  6. 8.

    a s y n c d e f + a

    w a i t a s y n c d e f f e t c h _ p a g e ( u r l ) : r e s p o n s e = a w a i t c l i e n t . g e t ( u r l ) r e t u r n a w a i t r e s p o n s e . t e x t ( ) i m p o r t a s y n c i o f r o m a i o h t t p i m p o r t c l i e n t l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) c o n t e n t = l o o p . r u n _ u n t i l _ c o m p l e t e ( f e t c h _ p a g e ( ' h t t p : / / f i . p y c o n . o r g / ' ) ) p r i n t ( c o n t e n t ) l o o p . c l o s e ( )
  7. 9.

    w i t h ( y i e l d

    f r o m . . . ) i m p o r t a s y n c i o i m p o r t s q l a l c h e m y a s s a f r o m a i o p g . s a i m p o r t c r e a t e _ e n g i n e @ a s y n c i o . c o r o u t i n e d e f c o u n t _ d a t a ( d s n ) : e n g i n e = y i e l d f r o m c r e a t e _ e n g i n e ( d s n ) w i t h ( y i e l d f r o m e n g i n e ) a s c o n n : q u e r y = s a . s e l e c t ( . . . ) . c o u n t ( ) r e t u r n ( y i e l d f r o m c o n n . s c a l a r ( q u e r y ) ) l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) c o u n t e r = l o o p . r u n _ u n t i l _ c o m p l e t e ( c o u n t _ d a t a ( ' p o s t g r e s q l : / / . . . ' ) ) p r i n t ( c o u n t e r ) l o o p . c l o s e ( )
  8. 10.

    a s y n c w i t h f

    r o m . u t i l s i m p o r t C o n n e c t i o n C o n t e x t M a n a g e r a s y n c d e f c o u n t _ d a t a ( d s n ) : e n g i n e = a w a i t c r e a t e _ e n g i n e ( d s n ) a s y n c w i t h C o n n e c t i o n C o n t e x t M a n a g e r ( e n g i n e ) a s c o n n : q u e r y = s a . s e l e c t ( . . . ) . c o u n t ( ) r e t u r n a w a i t c o n n . s c a l a r ( q u e r y ) i m p o r t a s y n c i o i m p o r t s q l a l c h e m y a s s a f r o m a i o p g . s a i m p o r t c r e a t e _ e n g i n e l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) c o u n t e r = l o o p . r u n _ u n t i l _ c o m p l e t e ( c o u n t _ d a t a ( ' p o s t g r e s q l : / / . . . ' ) ) p r i n t ( c o u n t e r ) l o o p . c l o s e ( )
  9. 11.

    a s y n c w i t h utils.py

    c l a s s C o n n e c t i o n C o n t e x t M a n a g e r ( o b j e c t ) : d e f _ _ i n i t _ _ ( s e l f , e n g i n e ) : s e l f . c o n n = N o n e s e l f . e n g i n e = e n g i n e a s y n c d e f _ _ a e n t e r _ _ ( s e l f ) : s e l f . c o n n = a w a i t s e l f . e n g i n e . a c q u i r e ( ) r e t u r n s e l f . c o n n a s y n c d e f _ _ a e x i t _ _ ( s e l f , e x c _ t y p e , e x c , t b ) : t r y : s e l f . e n g i n e . r e l e a s e ( s e l f . c o n n ) f i n a l l y : s e l f . c o n n = N o n e s e l f . e n g i n e = N o n e
  10. 12.

    f o r r o w i n ( y

    i e l d f r o m . . . ) : i m p o r t a s y n c i o i m p o r t s q l a l c h e m y a s s a f r o m a i o p g . s a i m p o r t c r e a t e _ e n g i n e @ a s y n c i o . c o r o u t i n e d e f f e t c h _ d a t a ( d s n ) : d a t a = [ ] e n g i n e = y i e l d f r o m c r e a t e _ e n g i n e ( d s n ) w i t h ( y i e l d f r o m e n g i n e ) a s c o n n : r e s u l t = y i e l d f r o m c o n n . e x e c u t e ( s a . s e l e c t ( . . . ) ) f o r r o w i n r e s u l t : d a t a . a p p e n d ( r o w ) r e t u r n d a t a l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) d a t a = l o o p . r u n _ u n t i l _ c o m p l e t e ( f e t c h _ d a t a ( ' p o s t g r e s q l : / / . . . ' ) ) l o o p . c l o s e ( )
  11. 13.

    a s y n c f o r , R

    e s u l t I t e r a s y n c d e f f e t c h _ d a t a ( d s n ) : d a t a = [ ] e n g i n e = a w a i t c r e a t e _ e n g i n e ( d s n ) a s y n c w i t h C o n n e c t i o n C o n t e x t M a n a g e r ( e n g i n e ) a s c o n n : a s y n c f o r r o w i n R e s u l t I t e r ( a w a i t c o n n . e x e c u t e ( s a . s e l e c t ( . . . ) ) ) : d a t a . a p p e n d ( r o w ) r e t u r n d a t a i m p o r t a s y n c i o i m p o r t s q l a l c h e m y a s s a f r o m a i o p g . s a i m p o r t c r e a t e _ e n g i n e f r o m . u t i l s i m p o r t C o n n e c t i o n C o n t e x t M a n a g e r l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) d a t a = l o o p . r u n _ u n t i l _ c o m p l e t e ( f e t c h _ d a t a ( ' p o s t g r e s q l : / / . . . ' ) ) l o o p . c l o s e ( )
  12. 14.

    a s y n c f o r utils.py f

    r o m a i o p g . s a . e x c i m p o r t R e s o u r c e C l o s e d E r r o r c l a s s R e s u l t I t e r ( o b j e c t ) : d e f _ _ i n i t _ _ ( s e l f , r e s u l t ) : s e l f . r e s u l t = r e s u l t a s y n c d e f _ _ a i t e r _ _ ( s e l f ) : r e t u r n s e l f a s y n c d e f _ _ a n e x t _ _ ( s e l f ) : t r y : d a t a = a w a i t s e l f . r e s u l t . f e t c h o n e ( ) e x c e p t R e s o u r c e C l o s e d E r r o r : d a t a = N o n e i f d a t a : r e t u r n d a t a r a i s e S t o p A s y n c I t e r a t i o n
  13. 15.

    Other additions to standard library @ t y p e

    s . c o r o u t i n e bridge between generator based and native coroutines New _ _ a w a i t _ _ magic method New functions in i n s p e c t library as i s c o r o u t i n e , i s a w a i t a b l e , etc New abstract base classes: a b c . A w a i t a b l e , a b c . C o r o u t i n e , a b c . A s y n c I t e r a b l e , a b c . A s y n c I t e r a t o r
  14. 16.

    Conclusion on functions and methods Method Can contain Can't contain

    a s y n c d e f f u n c a w a i t , r e t u r n v a l u e y i e l d , y i e l d f r o m a s y n c d e f _ _ a * _ _ a w a i t , r e t u r n v a l u e y i e l d , y i e l d f r o m d e f _ _ a * _ _ r e t u r n a w a i t a b l e a w a i t d e f _ _ a w a i t _ _ y i e l d , y i e l d f r o m , r e t u r n i t e r a b l e a w a i t g e n e r a t o r y i e l d , y i e l d f r o m , r e t u r n v a l u e a w a i t
  15. 18.

    aiohttp HTTP client/server for asyncio Latest version: 0 . 1

    7 . 4 http://aiohttp.readthedocs.org/ i m p o r t a s y n c i o i m p o r t u j s o n f r o m a i o h t t p i m p o r t c l i e n t a s y n c d e f g i t h u b _ s e a r c h ( q u e r y ) : r e s p o n s e = a w a i t c l i e n t . g e t ( ' h t t p s : / / a p i . g i t h u b . c o m / s e a r c h / r e p o s i t o r i e s ' , p a r a m s = { ' q ' : q u e r y } ) r e t u r n u j s o n . l o a d s ( a w a i t r e s p o n s e . r e a d ( ) ) l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) r e s p o n s e = l o o p . r u n _ u n t i l _ c o m p l e t e ( g i t h u b _ s e a r c h ( ' a s y n c i o ' ) ) p r i n t ( ' \ n ' . j o i n ( r e p o [ ' f u l l _ n a m e ' ] f o r r e p o i n r e s p o n s e [ ' i t e m s ' ] ) ) l o o p . c l o s e ( )
  16. 19.

    aiohttp.web Web framework for asyncio http://aiohttp.readthedocs.org/en/latest/web.html i m p o

    r t u j s o n f r o m a i o h t t p i m p o r t w e b a s y n c d e f a p i _ i n d e x ( r e q u e s t ) : o u t p u t = u j s o n . d u m p s ( { . . . } ) r e t u r n w e b . R e s p o n s e ( b o d y = o u t p u t , c o n t e n t _ t y p e = ' a p p l i c a t i o n / j s o n ' ) a p p = w e b . A p p l i c a t i o n ( ) a p p . r o u t e r . a d d _ r o u t e ( ' G E T ' , ' / a p i / ' , a p i _ i n d e x ) $ g u n i c o r n - k a i o h t t p . w o r k e r . G u n i c o r n W e b W o r k e r - w 9 - t 6 0 a p p : a p p
  17. 20.

    aiopg Accessing PostgreSQL database from the asyncio Latest version: 0

    . 7 . 0 http://aiopg.readthedocs.org/ i m p o r t a s y n c i o f r o m a i o p g i m p o r t c r e a t e _ p o o l a s y n c d e f s e l e c t _ o n e ( d s n ) : p o o l = a w a i t c r e a t e _ p o o l ( d s n ) w i t h ( a w a i t p o o l . c u r s o r ( ) ) a s c u r s o r : a w a i t c u r s o r . e x e c u t e ( ' S E L E C T 1 ' ) s e l e c t e d = a w a i t c u r s o r . f e t c h o n e ( ) a s s e r t s e l e c t e d = = ( 1 , ) l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) l o o p . r u n _ u n t i l _ c o m p l e t e ( s e l e c t _ o n e ( ' d b n a m e = . . . u s e r = . . . p a s s w o r d = . . . h o s t = . . . ' ) ) l o o p . c l o s e ( )
  18. 21.

    aiopg.sa Layer for executing SQLAlchemy Core queries http://aiopg.readthedocs.org/en/stable/sa.html i m

    p o r t a s y n c i o i m p o r t s q l a l c h e m y a s s a f r o m a i o p g . s a i m p o r t c r e a t e _ e n g i n e f r o m . u t i l s i m p o r t C o n n e c t i o n C o n t e x t M a n a g e r a s y n c d e f s e l e c t _ o n e ( d s n ) : e n g i n e = a w a i t c r e a t e _ e n g i n e ( d s n ) a s y n c w i t h C o n n e c t i o n C o n t e x t M a n a g e r ( e n g i n e ) a s c o n n : a w a i t c o n n . e x e c u t e ( s a . s e l e c t ( [ s a . t e x t ( ' 1 ' ) ] ) ) s e l e c t e d = a w a i t c o n n . f e t c h o n e ( ) a s s e r t s e l e c t e d = = ( 1 , ) l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) l o o p . r u n _ u n t i l _ c o m p l e t e ( s e l e c t _ o n e ( ' p o s t g r e s q l : / / . . . ' ) ) l o o p . c l o s e ( )
  19. 22.

    aioredis Redis client library Latest version: 0 . 2 .

    4 http://aioredis.readthedocs.org/ i m p o r t a s y n c i o f r o m a i o r e d i s i m p o r t c r e a t e _ r e d i s a s y n c d e f r e d i s _ s e t _ g e t _ d e l e t e ( a d d r e s s , * * o p t i o n s ) : o p t i o n s . s e t d e f a u l t ( ' e n c o d i n g ' , ' u t f - 8 ' ) r e d i s = a w a i t c r e a t e _ r e d i s ( a d d r e s s , * * o p t i o n s ) a s s e r t a w a i t r e d i s . s e t ( ' k e y ' , ' v a l u e ' ) i s T r u e a s s e r t a w a i t r e d i s . g e t ( ' k e y ' ) = = ' v a l u e ' a s s e r t a w a i t r e d i s . d e l e t e ( ' k e y ' ) = = 1 l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) l o o p . r u n _ u n t i l _ c o m p l e t e ( r e d i s _ s e t _ g e t _ d e l e t e ( ( ' l o c a l h o s t ' , 6 3 7 9 ) ) ) l o o p . c l o s e ( )
  20. 23.

    And others Python Asyncio Resources MySQL: aiomysql Mongo: asyncio_mongo CouchDB:

    aiocouchdb ElasticSearch: aioes Memcached: aiomcache AMQP: aioamqp ØMQ: aiozmq
  21. 25.

    Task. Fetch data for the NFL Season NFL Season lasts

    5 preseason and 17 regular season weeks This totals 22 requests to NFL.com endpoint Each request can be processed asyncronously After all requests are done we need to call extra function
  22. 26.

    Step 1. Sync fetch week data i m p o

    r t r e q u e s t s f r o m l x m l i m p o r t e t r e e N F L _ U R L = ' h t t p : / / w w w . n f l . c o m / a j a x / s c o r e s t r i p ' d e f f e t c h _ w e e k ( s e a s o n , w e e k , i s _ p r e s e a s o n = F a l s e ) : r e s p o n s e = r e q u e s t s . g e t ( N F L _ U R L , p a r a m s = { ' s e a s o n ' : s e a s o n , ' s e a s o n T y p e ' : ' P R E ' i f i s _ p r e s e a s o n e l s e ' R E G ' , ' w e e k ' : w e e k , } ) r e t u r n e t r e e . f r o m s t r i n g ( r e s p o n s e . c o n t e n t )
  23. 27.

    Step 2. Sync fetch season data d e f f

    e t c h _ s e a s o n ( s e a s o n ) : p r e _ c a l l ( . . . ) f o r i s _ p r e s e a s o n , w e e k s i n ( ( F a l s e , r a n g e ( 5 ) ) , ( T r u e , r a n g e ( 1 , 1 8 ) ) ) : f o r w e e k i n w e e k s : f e t c h _ w e e k ( s e a s o n , w e e k , i s _ p r e s e a s o n ) . . . p o s t _ c a l l ( . . . )
  24. 28.

    Step 3. Async fetch week data f r o m

    a i o h t t p i m p o r t c l i e n t f r o m l x m l i m p o r t e t r e e a s y n c d e f a i o _ f e t c h _ w e e k ( s e a s o n , w e e k , i s _ p r e s e a s o n = F a l s e ) : r e s p o n s e = a w a i t c l i e n t . g e t ( N F L _ U R L , p a r a m s = { ' s e a s o n ' : s e a s o n , ' s e a s o n T y p e ' : ' P R E ' i f i s _ p r e s e a s o n e l s e ' R E G ' , ' w e e k ' : w e e k , } ) r e t u r n e t r e e . f r o m s t r i n g ( a w a i t r e s p o n s e . r e a d ( ) )
  25. 29.

    Step 4. Async fetch season data i m p o

    r t a s y n c i o d e f f e t c h _ s e a s o n ( s e a s o n ) : l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) l o o p . r u n _ u n t i l _ c o m p l e t e ( p r e _ c a l l ( . . . ) ) t a s k s = [ a i o _ f e t c h _ w e e k ( s e a s o n , y e a r , i s _ p r e s e a s o n ) f o r i s _ p r e s e a s o n , w e e k s i n ( ( F a l s e , r a n g e ( 5 ) ) , ( T r u e , r a n g e ( 1 , 1 8 ) ) ) f o r w e e k i n w e e k s ] l o o p . r u n _ u n t i l _ c o m p l e t e ( a s y n c i o . w a i t ( t a s k s , l o o p = l o o p ) ) l o o p . r u n _ u n t i l _ c o m p l e t e ( p o s t _ c a l l ( . . . ) ) l o o p . c l o s e ( )
  26. 30.

    Step 5. Running fetching season data scripts/fetch_season_data.py i m p

    o r t s y s d e f m a i n ( * a r g s ) : f e t c h _ s e a s o n ( i n t ( a r g s [ 0 ] ) ) r e t u r n F a l s e i f _ _ n a m e _ _ = = ' _ _ m a i n _ _ ' : s y s . e x i t ( i n t ( m a i n ( * s y s . a r g v [ 1 : ] ) ) ) $ p y t h o n s c r i p t s / f e t c h _ s e a s o n _ d a t a . p y 2 0 1 5
  27. 31.

    Step 6. See in action . . . A l

    l g a m e s f o r S e a s o n 2 0 1 5 / R E G . W e e k 1 7 p r o c e s s e d ! S e a s o n 2 0 1 5 / R E G . W e e k 1 2 . G a m e I D 2 0 1 5 1 1 3 0 0 0 , C L E @ B A L , s c h e d u l e d a t 2 0 1 5 - 1 1 - 3 0 T 2 0 : 3 0 : 0 0 - 0 5 : 0 0 u p d a t e d i n d a t a b a s e A l l g a m e s f o r S e a s o n 2 0 1 5 / R E G . W e e k 1 2 p r o c e s s e d ! S e a s o n 2 0 1 5 / P R E . W e e k 4 . G a m e I D 2 0 1 5 0 9 0 3 6 1 , D A L @ H O U , s c h e d u l e d a t 2 0 1 5 - 0 9 - 0 3 T 2 0 : 0 0 : 0 0 - 0 4 : 0 0 u p d a t e d i n d a t a b a s e S e a s o n 2 0 1 5 / R E G . W e e k 1 6 . G a m e I D 2 0 1 5 1 2 2 7 1 1 , S E A @ S T L , s c h e d u l e d a t 2 0 1 5 - 1 2 - 2 7 T 1 6 : 2 5 : 0 0 - 0 5 : 0 0 u p d a t e d i n d a t a b a s e S e a s o n 2 0 1 5 / P R E . W e e k 4 . G a m e I D 2 0 1 5 0 9 0 3 6 2 , S T L @ K C , s c h e d u l e d a t 2 0 1 5 - 0 9 - 0 3 T 2 0 : 0 0 : 0 0 - 0 4 : 0 0 u p d a t e d i n d a t a b a s e S e a s o n 2 0 1 5 / R E G . W e e k 1 6 . G a m e I D 2 0 1 5 1 2 2 7 1 2 , B A L @ P I T , s c h e d u l e d a t 2 0 1 5 - 1 2 - 2 7 T 2 0 : 3 0 : 0 0 - 0 5 : 0 0 u p d a t e d i n d a t a b a s e S e a s o n 2 0 1 5 / P R E . W e e k 4 . G a m e I D 2 0 1 5 0 9 0 3 6 3 , T E N @ M I N , s c h e d u l e d a t 2 0 1 5 - 0 9 - 0 3 T 2 0 : 0 0 : 0 0 - 0 4 : 0 0 u p d a t e d i n d a t a b a s e S e a s o n 2 0 1 5 / R E G . W e e k 1 6 . G a m e I D 2 0 1 5 1 2 2 8 0 0 , D E N @ C I N , s c h e d u l e d a t 2 0 1 5 - 1 2 - 2 8 T 2 0 : 3 0 : 0 0 - 0 5 : 0 0 u p d a t e d i n d a t a b a s e A l l g a m e s f o r S e a s o n 2 0 1 5 / R E G . W e e k 1 6 p r o c e s s e d ! S e a s o n 2 0 1 5 / P R E . W e e k 4 . G a m e I D 2 0 1 5 0 9 0 3 6 4 , D E N @ A R I , s c h e d u l e d a t 2 0 1 5 - 0 9 - 0 3 T 2 1 : 0 0 : 0 0 - 0 4 : 0 0 u p d a t e d i n d a t a b a s e S e a s o n 2 0 1 5 / P R E . W e e k 4 . G a m e I D 2 0 1 5 0 9 0 3 6 5 , S F @ S D , s c h e d u l e d a t 2 0 1 5 - 0 9 - 0 3 T 2 2 : 0 0 : 0 0 - 0 4 : 0 0 u p d a t e d i n d a t a b a s e S e a s o n 2 0 1 5 / P R E . W e e k 4 . G a m e I D 2 0 1 5 0 9 0 3 6 6 , S E A @ O A K , s c h e d u l e d a t 2 0 1 5 - 0 9 - 0 3 T 2 2 : 0 0 : 0 0 - 0 4 : 0 0 u p d a t e d i n d a t a b a s e A l l g a m e s f o r S e a s o n 2 0 1 5 / P R E . W e e k 4 p r o c e s s e d ! . . .
  28. 33.

    Task. Provide an API for week data NFL.com still returns

    data in X M L We need this data in G r a p h Q L Okay, okay, actually in J S O N And if there are no live games, we don't need to fetch data directly from NFL
  29. 35.

    Step 1. Flask base application f r o m f

    l a s k i m p o r t F l a s k f r o m . i m p o r t c a c h e f r o m . d a t a i m p o r t t o _ g a m e _ d a t a a p p = F l a s k ( _ _ n a m e _ _ ) @ a p p . r o u t e ( ' / a p i / s e a s o n / < i n t : s e a s o n _ y e a r > / < w e e k _ s l u g > ' ) d e f r e t r i e v e _ w e e k ( s e a s o n _ y e a r , w e e k _ s l u g ) : i s _ p r e s e a s o n = w e e k _ s l u g [ : 4 ] = = ' p r e - ' w e e k = i n t ( w e e k _ s l u g [ 4 : ] i f i s _ p r e s e a s o n e l s e w e e k _ s l u g ) c a c h e _ k e y = c a c h e . b u i l d _ k e y ( s e a s o n _ y e a r , w e e k _ s l u g ) w e e k _ d a t a = c a c h e . e n s u r e _ d a t a ( c a c h e _ k e y ) i f w e e k _ d a t a i s N o n e : w e e k _ o b j = f e t c h _ w e e k ( s e a s o n _ y e a r , w e e k , i s _ p r e s e a s o n ) w e e k _ d a t a = [ t o _ g a m e _ d a t a ( g a m e ) f o r g a m e i n w e e k _ o b j . i t e r f i n d ( ' g m s / g ' ) ] c a c h e . s t o r e _ d a t a ( c a c h e _ k e y , w e e k _ d a t a ) r e t u r n j s o n i f y ( w e e k _ d a t a )
  30. 36.

    Step 2. Running Flask application $ g u n i

    c o r n - k s y n c - w 5 - t 6 0 a p p : a p p $ g u n i c o r n - k e v e n t l e t - w 5 - t 6 0 a p p : a p p
  31. 37.

    Step 3. aiohttp.web base application i m p o r

    t u j s o n f r o m a i o h t t p i m p o r t w e b a s y n c d e f a i o _ r e t r i e v e _ w e e k ( r e q u e s t ) : s e a s o n _ y e a r = i n t ( r e q u e s t . m a t c h _ i n f o [ ' s e a s o n _ y e a r ' ] ) w e e k _ s l u g = r e q u e s t . m a t c h _ i n f o [ ' w e e k _ s l u g ' ] i s _ p r e s e a s o n = w e e k _ s l u g [ : 4 ] = = ' p r e - ' w e e k = i n t ( w e e k _ s l u g [ 4 : ] i f i s _ p r e s e a s o n e l s e w e e k _ s l u g ) c a c h e _ k e y = c a c h e . b u i l d _ k e y ( s e a s o n _ y e a r , w e e k , i s _ p r e s e a s o n ) w e e k _ d a t a = a w a i t c a c h e . a i o _ e n s u r e _ d a t a ( c a c h e _ k e y ) i f w e e k _ d a t a i s N o n e : w e e k _ o b j = a w a i t a i o _ f e t c h _ w e e k ( s e a s o n _ y e a r , w e e k , i s _ p r e s e a s o n ) w e e k _ d a t a = [ t o _ g a m e _ d a t a ( g a m e ) f o r g a m e i n w e e k _ o b j . i t e r f i n d ( ' g m s / g ' ) ] a w a i t c a c h e . s t o r e _ d a t a ( c a c h e _ k e y , w e e k _ d a t a ) r e t u r n w e b . R e s p o n s e ( u j s o n . d u m p s ( w e e k _ d a t a ) , c o n t e n t _ t y p e = ' a p p l i c a t i o n / j s o n ' )
  32. 38.

    Step 3. aiohttp.web base application (continue) a i o _

    a p p = w e b . A p p l i c a t i o n ( ) a i o _ a p p . r o u t e r . a d d _ r o u t e ( ' G E T ' , ' / a p i / s e a s o n / { s e a s o n _ y e a r : \ d { 4 } } / { w e e k _ s l u g } ' , a i o _ r e t r i e v e _ w e e k )
  33. 39.

    aiohttp.web basics Each view function should be a c o

    r o u t i n e /a s y n c d e f and return response or raise an exc Each view function receives only one arg: r e q u e s t Query params data contains in r e q u e s t . G E T multidict Body data contains in r e q u e s t . P O S T multidict and should be awaited with a w a i t r e q u e s t . p o s t ( ) Matched data contains in r e q u e s t . m a t c h _ i n f o dict r e q u e s t also contains an a p p instance
  34. 40.

    Step 4. Runinng aiohttp.web application $ g u n i

    c o r n - k a i o h t t p . w o r k e r . G u n i c o r n W e b W o r k e r - w 5 - t 6 0 a p p : a i o _ a p p
  35. 43.

    Task. Livescore stream When there are live games, update the

    scores automagically for the user Do not reconnect each X seconds for new scores When new scores available - show them to user This is the case for Server-Sent Events (SSE)
  36. 44.

    Server-Sent Events. Message Format d a t a : H

    e l l o , P y C o n F i n l a n d ! < b l a n k l i n e > e v e n t : p y c o n f i d a t a : H e l l o , P y c o n F i n l a n d ! d a t a : I ' m g l a d t o b e h e r e < b l a n k l i n k > i d : 4 2 e v e n t : u a p y c o n d a t a : P y C o n U k r a i n e 2 0 1 6 w i l l h e l d a p l a c e i n L v i v a t A p r i l 2 0 1 6 < b l a n k l i n k >
  37. 45.

    Server-Sent Events. EventSource c o n s t e l

    e m e n t = d o c u m e n t . g e t E l e m e n t B y I d ( " n o t i f i c a t i o n s - c o n t a i n e r " ) ; c o n s t s o u r c e = n e w E v e n t S o u r c e ( " / a p i / n o t i f i c a t i o n s " ) ; s o u r c e . o n e r r o r = e r r = > { . . . } ; s o u r c e . a d d E v e n t L i s t e n e r ( " p y c o n f i " , e v t = > { e l e m e n t . i n n e r H T M L + = ` < c o d e > # p y c o n f i < / c o d e > $ { e v t . d a t a } < b r > ` ; } ) ; s o u r c e . a d d E v e n t L i s t e n e r ( " u a p y c o n " , e v t = > { e l e m e n t . i n n e r H T M L + = ` < c o d e > # u a p y c o n < / c o d e > $ { e v t . d a t a } < b r > ` ; } ) ; . . . s o u r c e . c l o s e ( ) ;
  38. 46.

    How to implement? 1. Run script to fetch live scores

    and publish new scores to Redis PUBSUB 2. Subscribe to this channel in livescore stream view, receive message and push it to client 3. Subscribe to livescore stream in client and update scores when new message received 4. ... 5. PROFIT
  39. 47.

    Step 1. Fetching & publishing scores change d e f

    m a i n ( * a r g s ) : s e a s o n , w e e k = a r g s l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) w e e k _ o b j = l o o p . r u n _ u n t i l _ c o m p l e t e ( a i o _ f e t c h _ w e e k ( s e a s o n , w e e k , F a l s e ) ) w e e k _ d a t a = [ t o _ g a m e _ d a t a ( g a m e ) f o r g a m e i n w e e k _ o b j . i t e r f i n d ( ' g m s / g ' ) ] r e d i s = l o o p . r u n _ u n t i l _ c o m p l e t e ( c r e a t e _ r e d i s ( ( ' l o c a l h o s t ' , 6 3 7 9 ) ) ) l o o p . r u n _ u n t i l _ c o m p l e t e ( r e d i s . p u b l i s h ( ' C H A N N E L ' , u j s o n . d u m p s ( w e e k _ d a t a ) ) ) l o o p . c l o s e ( ) r e t u r n F a l s e i f _ _ n a m e _ _ = = ' _ _ m a i n _ _ ' : s y s . e x i t ( i n t ( m a i n ( * s y s . a r g v [ 1 : ] ) ) )
  40. 48.

    Step 2. Implementing livescore stream a s y n c

    d e f l i v e s c o r e ( r e q u e s t ) : i f r e q u e s t . h e a d e r s [ ' A c c e p t ' ] ! = ' t e x t / e v e n t - s t r e a m ' : r a i s e w e b . H T T P F o u n d ( ' / a p i / ' ) r e s p o n s e = w e b . S t r e a m R e s p o n s e ( s t a t u s = 2 0 0 , h e a d e r s = { ' C o n t e n t - T y p e ' : ' t e x t / e v e n t - s t r e a m ' , ' C a c h e - C o n t r o l ' : ' n o - c a c h e ' , ' C o n n e c t i o n ' : ' k e e p - a l i v e ' , } ) r e s p o n s e . s t a r t ( r e q u e s t ) # T h i s w i l l b e c h a n g e d i n a s y n c i o = = 0 . 1 8 . 0 . . . r e t u r n r e s p o n s e
  41. 49.

    Step 2. Subscribing to scores update . . . r

    e d i s = a w a i t c r e a t e _ r e d i s ( ( ' l o c a l h o s t ' , 6 3 7 9 ) ) c h a n n e l = ( a w a i t r e d i s . s u b s c r i b e ( ' C H A N N E L ' ) ) [ 0 ] w h i l e ( a w a i t c h a n n e l . w a i t _ m e s s a g e ( ) ) : m e s s a g e = a w a i t c h a n n e l . g e t ( ) r e s p o n s e . w r i t e ( b ' e v e n t : u p d a t e \ r \ n ' ) r e s p o n s e . w r i t e ( b ' d a t a : ' + m e s s a g e + b ' \ r \ n \ r \ n ' ) . . .
  42. 50.

    Step 2. Registering stream to the app a i o

    _ a p p . r o u t e r . a d d _ r o u t e ( ' G E T ' , ' / a p i / l i v e s c o r e ' , l i v e s c o r e )
  43. 51.

    Step 3. Subscribing to livescore stream on client i m

    p o r t R e a c t , { C o m p o n e n t } f r o m " r e a c t " ; e x p o r t d e f a u l t A p p e x t e n d s C o m p o n e n t { l i v e S c o r e = n u l l ; c o n s t r u c t o r ( p r o p s ) { s u p e r ( p r o p s ) ; t h i s . s t a t e = { g a m e s : p r o p s . g a m e s } ; } c o m p o n e n t D i d M o u n t ( ) { t h i s . l i v e S c o r e = n e w E v e n t S o u r c e ( " / a p i / " ) ; t h i s . l i v e S c o r e . a d d E v e n t L i s t e n e r ( " u p d a t e " , e v t = > { t h i s . s e t S t a t e ( { g a m e s : J S O N . p a r s e ( e v t . d a t a ) } ) ; } ) ; } c o m p o n e n t W i l l U n m o u n t ( ) { t h i s . l i v e S c o r e . c l o s e ( ) ; } r e n d e r ( ) { r e t u r n } } < G a m e s d a t a = { t h i s . s t a t e . g a m e s } / > ;
  44. 52.

    Step 4. Setting up the nginx l o c a

    t i o n / a p i / l i v e s c o r e { c h u n k e d _ t r a n s f e r _ e n c o d i n g o f f ; p r o x y _ b u f f e r i n g o f f ; p r o x y _ c a c h e o f f ; p r o x y _ h t t p _ v e r s i o n 1 . 1 ; p r o x y _ p a s s h t t p : / / 1 2 7 . 0 . 0 . 1 : 8 0 0 0 ; p r o x y _ s e t _ h e a d e r C o n n e c t i o n ' ' ; p r o x y _ s e t _ h e a d e r H o s t $ h o s t ; p r o x y _ s e t _ h e a d e r X - R e a l - I P $ r e m o t e _ a d d r ; p r o x y _ s e t _ h e a d e r X - F o r w a r d e d - F o r $ p r o x y _ a d d _ x _ f o r w a r d e d _ f o r ; }
  45. 55.

    The Future is Here Python 3.5 released and ready to

    be used in production Python 3.5 contains new a s y n c /a w a i t statements a s y n c i o stack got great addition, code now is simple to read and understand asyncio stack (for web applications): a i o h t t p – interact with remote API a i o h t t p . w e b – create API backends a i o p g – interact with PostgreSQL (DB) a i o r e d i s – interact with Redis (cache)
  46. 56.

    The Future is Here a s y n c i

    o stack is a great reason why you'll finally need to switch to Python 3 a s y n c i o is good fit for: Database-less API Requesting remote API Async code execution Predictable async I/O And I'm not telling you about WebSockets, which supported by a i o h t t p out of the box