Slide 1

Slide 1 text

Welcome to async/await era Igor Davydenko PyCon Finland 2015

Slide 2

Slide 2 text

I am... Igor Davydenko Python & React.js developer From Kyiv, Ukraine Works on Ezhome Inc. Primarly designs & develops backend API Personal Site @ GitHub @ Twitter

Slide 3

Slide 3 text

PEP 492 Coroutines with async and await syntax

Slide 4

Slide 4 text

Quick Summary Created by Yuri Selivanov in April, 2015 Included in Python 3.5 Added 4 new statements to the standard library: a s y n c d e f a w a i t a s y n c w i t h a s y n c f o r Received great response from the Python community PEP 492 @ Python.org

Slide 5

Slide 5 text

@ a s y n c i o . c o r o u t i n e i m p o r t a s y n c i o @ a s y n c i o . c o r o u t i n e d e f h e l l o ( ) : r e t u r n ' H e l l o , w o r l d ! ' l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) m e s s a g e = l o o p . r u n _ u n t i l _ c o m p l e t e ( h e l l o ( ) ) p r i n t ( m e s s a g e ) l o o p . c l o s e ( )

Slide 6

Slide 6 text

a s y n c d e f a s y n c d e f h e l l o ( ) : r e t u r n ' H e l l o , w o r l d ! ' i m p o r t a s y n c i o l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) m e s s a g e = l o o p . r u n _ u n t i l _ c o m p l e t e ( h e l l o ( ) ) p r i n t ( m e s s a g e ) l o o p . c l o s e ( )

Slide 7

Slide 7 text

@ a s y n c i o . c o r o u t i n e + y i e l d f r o m i m p o r t a s y n c i o f r o m a i o h t t p i m p o r t c l i e n t @ a s y n c i o . c o r o u t i n e d e f f e t c h _ p a g e ( u r l ) : r e s p o n s e = y i e l d f r o m c l i e n t . g e t ( u r l ) r e t u r n ( y i e l d f r o m r e s p o n s e . t e x t ( ) ) l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) c o n t e n t = l o o p . r u n _ u n t i l _ c o m p l e t e ( f e t c h _ p a g e ( ' h t t p : / / f i . p y c o n . o r g / ' ) ) p r i n t ( c o n t e n t ) l o o p . c l o s e ( )

Slide 8

Slide 8 text

a s y n c d e f + a w a i t a s y n c d e f f e t c h _ p a g e ( u r l ) : r e s p o n s e = a w a i t c l i e n t . g e t ( u r l ) r e t u r n a w a i t r e s p o n s e . t e x t ( ) i m p o r t a s y n c i o f r o m a i o h t t p i m p o r t c l i e n t l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) c o n t e n t = l o o p . r u n _ u n t i l _ c o m p l e t e ( f e t c h _ p a g e ( ' h t t p : / / f i . p y c o n . o r g / ' ) ) p r i n t ( c o n t e n t ) l o o p . c l o s e ( )

Slide 9

Slide 9 text

w i t h ( y i e l d f r o m . . . ) i m p o r t a s y n c i o i m p o r t s q l a l c h e m y a s s a f r o m a i o p g . s a i m p o r t c r e a t e _ e n g i n e @ a s y n c i o . c o r o u t i n e d e f c o u n t _ d a t a ( d s n ) : e n g i n e = y i e l d f r o m c r e a t e _ e n g i n e ( d s n ) w i t h ( y i e l d f r o m e n g i n e ) a s c o n n : q u e r y = s a . s e l e c t ( . . . ) . c o u n t ( ) r e t u r n ( y i e l d f r o m c o n n . s c a l a r ( q u e r y ) ) l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) c o u n t e r = l o o p . r u n _ u n t i l _ c o m p l e t e ( c o u n t _ d a t a ( ' p o s t g r e s q l : / / . . . ' ) ) p r i n t ( c o u n t e r ) l o o p . c l o s e ( )

Slide 10

Slide 10 text

a s y n c w i t h f r o m . u t i l s i m p o r t C o n n e c t i o n C o n t e x t M a n a g e r a s y n c d e f c o u n t _ d a t a ( d s n ) : e n g i n e = a w a i t c r e a t e _ e n g i n e ( d s n ) a s y n c w i t h C o n n e c t i o n C o n t e x t M a n a g e r ( e n g i n e ) a s c o n n : q u e r y = s a . s e l e c t ( . . . ) . c o u n t ( ) r e t u r n a w a i t c o n n . s c a l a r ( q u e r y ) i m p o r t a s y n c i o i m p o r t s q l a l c h e m y a s s a f r o m a i o p g . s a i m p o r t c r e a t e _ e n g i n e l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) c o u n t e r = l o o p . r u n _ u n t i l _ c o m p l e t e ( c o u n t _ d a t a ( ' p o s t g r e s q l : / / . . . ' ) ) p r i n t ( c o u n t e r ) l o o p . c l o s e ( )

Slide 11

Slide 11 text

a s y n c w i t h utils.py c l a s s C o n n e c t i o n C o n t e x t M a n a g e r ( o b j e c t ) : d e f _ _ i n i t _ _ ( s e l f , e n g i n e ) : s e l f . c o n n = N o n e s e l f . e n g i n e = e n g i n e a s y n c d e f _ _ a e n t e r _ _ ( s e l f ) : s e l f . c o n n = a w a i t s e l f . e n g i n e . a c q u i r e ( ) r e t u r n s e l f . c o n n a s y n c d e f _ _ a e x i t _ _ ( s e l f , e x c _ t y p e , e x c , t b ) : t r y : s e l f . e n g i n e . r e l e a s e ( s e l f . c o n n ) f i n a l l y : s e l f . c o n n = N o n e s e l f . e n g i n e = N o n e

Slide 12

Slide 12 text

f o r r o w i n ( y i e l d f r o m . . . ) : i m p o r t a s y n c i o i m p o r t s q l a l c h e m y a s s a f r o m a i o p g . s a i m p o r t c r e a t e _ e n g i n e @ a s y n c i o . c o r o u t i n e d e f f e t c h _ d a t a ( d s n ) : d a t a = [ ] e n g i n e = y i e l d f r o m c r e a t e _ e n g i n e ( d s n ) w i t h ( y i e l d f r o m e n g i n e ) a s c o n n : r e s u l t = y i e l d f r o m c o n n . e x e c u t e ( s a . s e l e c t ( . . . ) ) f o r r o w i n r e s u l t : d a t a . a p p e n d ( r o w ) r e t u r n d a t a l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) d a t a = l o o p . r u n _ u n t i l _ c o m p l e t e ( f e t c h _ d a t a ( ' p o s t g r e s q l : / / . . . ' ) ) l o o p . c l o s e ( )

Slide 13

Slide 13 text

a s y n c f o r , R e s u l t I t e r a s y n c d e f f e t c h _ d a t a ( d s n ) : d a t a = [ ] e n g i n e = a w a i t c r e a t e _ e n g i n e ( d s n ) a s y n c w i t h C o n n e c t i o n C o n t e x t M a n a g e r ( e n g i n e ) a s c o n n : a s y n c f o r r o w i n R e s u l t I t e r ( a w a i t c o n n . e x e c u t e ( s a . s e l e c t ( . . . ) ) ) : d a t a . a p p e n d ( r o w ) r e t u r n d a t a i m p o r t a s y n c i o i m p o r t s q l a l c h e m y a s s a f r o m a i o p g . s a i m p o r t c r e a t e _ e n g i n e f r o m . u t i l s i m p o r t C o n n e c t i o n C o n t e x t M a n a g e r l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) d a t a = l o o p . r u n _ u n t i l _ c o m p l e t e ( f e t c h _ d a t a ( ' p o s t g r e s q l : / / . . . ' ) ) l o o p . c l o s e ( )

Slide 14

Slide 14 text

a s y n c f o r utils.py f r o m a i o p g . s a . e x c i m p o r t R e s o u r c e C l o s e d E r r o r c l a s s R e s u l t I t e r ( o b j e c t ) : d e f _ _ i n i t _ _ ( s e l f , r e s u l t ) : s e l f . r e s u l t = r e s u l t a s y n c d e f _ _ a i t e r _ _ ( s e l f ) : r e t u r n s e l f a s y n c d e f _ _ a n e x t _ _ ( s e l f ) : t r y : d a t a = a w a i t s e l f . r e s u l t . f e t c h o n e ( ) e x c e p t R e s o u r c e C l o s e d E r r o r : d a t a = N o n e i f d a t a : r e t u r n d a t a r a i s e S t o p A s y n c I t e r a t i o n

Slide 15

Slide 15 text

Other additions to standard library @ t y p e s . c o r o u t i n e bridge between generator based and native coroutines New _ _ a w a i t _ _ magic method New functions in i n s p e c t library as i s c o r o u t i n e , i s a w a i t a b l e , etc New abstract base classes: a b c . A w a i t a b l e , a b c . C o r o u t i n e , a b c . A s y n c I t e r a b l e , a b c . A s y n c I t e r a t o r

Slide 16

Slide 16 text

Conclusion on functions and methods Method Can contain Can't contain a s y n c d e f f u n c a w a i t , r e t u r n v a l u e y i e l d , y i e l d f r o m a s y n c d e f _ _ a * _ _ a w a i t , r e t u r n v a l u e y i e l d , y i e l d f r o m d e f _ _ a * _ _ r e t u r n a w a i t a b l e a w a i t d e f _ _ a w a i t _ _ y i e l d , y i e l d f r o m , r e t u r n i t e r a b l e a w a i t g e n e r a t o r y i e l d , y i e l d f r o m , r e t u r n v a l u e a w a i t

Slide 17

Slide 17 text

async/await in real life Libraries

Slide 18

Slide 18 text

aiohttp HTTP client/server for asyncio Latest version: 0 . 1 7 . 4 http://aiohttp.readthedocs.org/ i m p o r t a s y n c i o i m p o r t u j s o n f r o m a i o h t t p i m p o r t c l i e n t a s y n c d e f g i t h u b _ s e a r c h ( q u e r y ) : r e s p o n s e = a w a i t c l i e n t . g e t ( ' h t t p s : / / a p i . g i t h u b . c o m / s e a r c h / r e p o s i t o r i e s ' , p a r a m s = { ' q ' : q u e r y } ) r e t u r n u j s o n . l o a d s ( a w a i t r e s p o n s e . r e a d ( ) ) l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) r e s p o n s e = l o o p . r u n _ u n t i l _ c o m p l e t e ( g i t h u b _ s e a r c h ( ' a s y n c i o ' ) ) p r i n t ( ' \ n ' . j o i n ( r e p o [ ' f u l l _ n a m e ' ] f o r r e p o i n r e s p o n s e [ ' i t e m s ' ] ) ) l o o p . c l o s e ( )

Slide 19

Slide 19 text

aiohttp.web Web framework for asyncio http://aiohttp.readthedocs.org/en/latest/web.html i m p o r t u j s o n f r o m a i o h t t p i m p o r t w e b a s y n c d e f a p i _ i n d e x ( r e q u e s t ) : o u t p u t = u j s o n . d u m p s ( { . . . } ) r e t u r n w e b . R e s p o n s e ( b o d y = o u t p u t , c o n t e n t _ t y p e = ' a p p l i c a t i o n / j s o n ' ) a p p = w e b . A p p l i c a t i o n ( ) a p p . r o u t e r . a d d _ r o u t e ( ' G E T ' , ' / a p i / ' , a p i _ i n d e x ) $ g u n i c o r n - k a i o h t t p . w o r k e r . G u n i c o r n W e b W o r k e r - w 9 - t 6 0 a p p : a p p

Slide 20

Slide 20 text

aiopg Accessing PostgreSQL database from the asyncio Latest version: 0 . 7 . 0 http://aiopg.readthedocs.org/ i m p o r t a s y n c i o f r o m a i o p g i m p o r t c r e a t e _ p o o l a s y n c d e f s e l e c t _ o n e ( d s n ) : p o o l = a w a i t c r e a t e _ p o o l ( d s n ) w i t h ( a w a i t p o o l . c u r s o r ( ) ) a s c u r s o r : a w a i t c u r s o r . e x e c u t e ( ' S E L E C T 1 ' ) s e l e c t e d = a w a i t c u r s o r . f e t c h o n e ( ) a s s e r t s e l e c t e d = = ( 1 , ) l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) l o o p . r u n _ u n t i l _ c o m p l e t e ( s e l e c t _ o n e ( ' d b n a m e = . . . u s e r = . . . p a s s w o r d = . . . h o s t = . . . ' ) ) l o o p . c l o s e ( )

Slide 21

Slide 21 text

aiopg.sa Layer for executing SQLAlchemy Core queries http://aiopg.readthedocs.org/en/stable/sa.html i m p o r t a s y n c i o i m p o r t s q l a l c h e m y a s s a f r o m a i o p g . s a i m p o r t c r e a t e _ e n g i n e f r o m . u t i l s i m p o r t C o n n e c t i o n C o n t e x t M a n a g e r a s y n c d e f s e l e c t _ o n e ( d s n ) : e n g i n e = a w a i t c r e a t e _ e n g i n e ( d s n ) a s y n c w i t h C o n n e c t i o n C o n t e x t M a n a g e r ( e n g i n e ) a s c o n n : a w a i t c o n n . e x e c u t e ( s a . s e l e c t ( [ s a . t e x t ( ' 1 ' ) ] ) ) s e l e c t e d = a w a i t c o n n . f e t c h o n e ( ) a s s e r t s e l e c t e d = = ( 1 , ) l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) l o o p . r u n _ u n t i l _ c o m p l e t e ( s e l e c t _ o n e ( ' p o s t g r e s q l : / / . . . ' ) ) l o o p . c l o s e ( )

Slide 22

Slide 22 text

aioredis Redis client library Latest version: 0 . 2 . 4 http://aioredis.readthedocs.org/ i m p o r t a s y n c i o f r o m a i o r e d i s i m p o r t c r e a t e _ r e d i s a s y n c d e f r e d i s _ s e t _ g e t _ d e l e t e ( a d d r e s s , * * o p t i o n s ) : o p t i o n s . s e t d e f a u l t ( ' e n c o d i n g ' , ' u t f - 8 ' ) r e d i s = a w a i t c r e a t e _ r e d i s ( a d d r e s s , * * o p t i o n s ) a s s e r t a w a i t r e d i s . s e t ( ' k e y ' , ' v a l u e ' ) i s T r u e a s s e r t a w a i t r e d i s . g e t ( ' k e y ' ) = = ' v a l u e ' a s s e r t a w a i t r e d i s . d e l e t e ( ' k e y ' ) = = 1 l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) l o o p . r u n _ u n t i l _ c o m p l e t e ( r e d i s _ s e t _ g e t _ d e l e t e ( ( ' l o c a l h o s t ' , 6 3 7 9 ) ) ) l o o p . c l o s e ( )

Slide 23

Slide 23 text

And others Python Asyncio Resources MySQL: aiomysql Mongo: asyncio_mongo CouchDB: aiocouchdb ElasticSearch: aioes Memcached: aiomcache AMQP: aioamqp ØMQ: aiozmq

Slide 24

Slide 24 text

async/await in real life Fetching data from remote API

Slide 25

Slide 25 text

Task. Fetch data for the NFL Season NFL Season lasts 5 preseason and 17 regular season weeks This totals 22 requests to NFL.com endpoint Each request can be processed asyncronously After all requests are done we need to call extra function

Slide 26

Slide 26 text

Step 1. Sync fetch week data i m p o r t r e q u e s t s f r o m l x m l i m p o r t e t r e e N F L _ U R L = ' h t t p : / / w w w . n f l . c o m / a j a x / s c o r e s t r i p ' d e f f e t c h _ w e e k ( s e a s o n , w e e k , i s _ p r e s e a s o n = F a l s e ) : r e s p o n s e = r e q u e s t s . g e t ( N F L _ U R L , p a r a m s = { ' s e a s o n ' : s e a s o n , ' s e a s o n T y p e ' : ' P R E ' i f i s _ p r e s e a s o n e l s e ' R E G ' , ' w e e k ' : w e e k , } ) r e t u r n e t r e e . f r o m s t r i n g ( r e s p o n s e . c o n t e n t )

Slide 27

Slide 27 text

Step 2. Sync fetch season data d e f f e t c h _ s e a s o n ( s e a s o n ) : p r e _ c a l l ( . . . ) f o r i s _ p r e s e a s o n , w e e k s i n ( ( F a l s e , r a n g e ( 5 ) ) , ( T r u e , r a n g e ( 1 , 1 8 ) ) ) : f o r w e e k i n w e e k s : f e t c h _ w e e k ( s e a s o n , w e e k , i s _ p r e s e a s o n ) . . . p o s t _ c a l l ( . . . )

Slide 28

Slide 28 text

Step 3. Async fetch week data f r o m a i o h t t p i m p o r t c l i e n t f r o m l x m l i m p o r t e t r e e a s y n c d e f a i o _ f e t c h _ w e e k ( s e a s o n , w e e k , i s _ p r e s e a s o n = F a l s e ) : r e s p o n s e = a w a i t c l i e n t . g e t ( N F L _ U R L , p a r a m s = { ' s e a s o n ' : s e a s o n , ' s e a s o n T y p e ' : ' P R E ' i f i s _ p r e s e a s o n e l s e ' R E G ' , ' w e e k ' : w e e k , } ) r e t u r n e t r e e . f r o m s t r i n g ( a w a i t r e s p o n s e . r e a d ( ) )

Slide 29

Slide 29 text

Step 4. Async fetch season data i m p o r t a s y n c i o d e f f e t c h _ s e a s o n ( s e a s o n ) : l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) l o o p . r u n _ u n t i l _ c o m p l e t e ( p r e _ c a l l ( . . . ) ) t a s k s = [ a i o _ f e t c h _ w e e k ( s e a s o n , y e a r , i s _ p r e s e a s o n ) f o r i s _ p r e s e a s o n , w e e k s i n ( ( F a l s e , r a n g e ( 5 ) ) , ( T r u e , r a n g e ( 1 , 1 8 ) ) ) f o r w e e k i n w e e k s ] l o o p . r u n _ u n t i l _ c o m p l e t e ( a s y n c i o . w a i t ( t a s k s , l o o p = l o o p ) ) l o o p . r u n _ u n t i l _ c o m p l e t e ( p o s t _ c a l l ( . . . ) ) l o o p . c l o s e ( )

Slide 30

Slide 30 text

Step 5. Running fetching season data scripts/fetch_season_data.py i m p o r t s y s d e f m a i n ( * a r g s ) : f e t c h _ s e a s o n ( i n t ( a r g s [ 0 ] ) ) r e t u r n F a l s e i f _ _ n a m e _ _ = = ' _ _ m a i n _ _ ' : s y s . e x i t ( i n t ( m a i n ( * s y s . a r g v [ 1 : ] ) ) ) $ p y t h o n s c r i p t s / f e t c h _ s e a s o n _ d a t a . p y 2 0 1 5

Slide 31

Slide 31 text

Step 6. See in action . . . A l l g a m e s f o r S e a s o n 2 0 1 5 / R E G . W e e k 1 7 p r o c e s s e d ! S e a s o n 2 0 1 5 / R E G . W e e k 1 2 . G a m e I D 2 0 1 5 1 1 3 0 0 0 , C L E @ B A L , s c h e d u l e d a t 2 0 1 5 - 1 1 - 3 0 T 2 0 : 3 0 : 0 0 - 0 5 : 0 0 u p d a t e d i n d a t a b a s e A l l g a m e s f o r S e a s o n 2 0 1 5 / R E G . W e e k 1 2 p r o c e s s e d ! S e a s o n 2 0 1 5 / P R E . W e e k 4 . G a m e I D 2 0 1 5 0 9 0 3 6 1 , D A L @ H O U , s c h e d u l e d a t 2 0 1 5 - 0 9 - 0 3 T 2 0 : 0 0 : 0 0 - 0 4 : 0 0 u p d a t e d i n d a t a b a s e S e a s o n 2 0 1 5 / R E G . W e e k 1 6 . G a m e I D 2 0 1 5 1 2 2 7 1 1 , S E A @ S T L , s c h e d u l e d a t 2 0 1 5 - 1 2 - 2 7 T 1 6 : 2 5 : 0 0 - 0 5 : 0 0 u p d a t e d i n d a t a b a s e S e a s o n 2 0 1 5 / P R E . W e e k 4 . G a m e I D 2 0 1 5 0 9 0 3 6 2 , S T L @ K C , s c h e d u l e d a t 2 0 1 5 - 0 9 - 0 3 T 2 0 : 0 0 : 0 0 - 0 4 : 0 0 u p d a t e d i n d a t a b a s e S e a s o n 2 0 1 5 / R E G . W e e k 1 6 . G a m e I D 2 0 1 5 1 2 2 7 1 2 , B A L @ P I T , s c h e d u l e d a t 2 0 1 5 - 1 2 - 2 7 T 2 0 : 3 0 : 0 0 - 0 5 : 0 0 u p d a t e d i n d a t a b a s e S e a s o n 2 0 1 5 / P R E . W e e k 4 . G a m e I D 2 0 1 5 0 9 0 3 6 3 , T E N @ M I N , s c h e d u l e d a t 2 0 1 5 - 0 9 - 0 3 T 2 0 : 0 0 : 0 0 - 0 4 : 0 0 u p d a t e d i n d a t a b a s e S e a s o n 2 0 1 5 / R E G . W e e k 1 6 . G a m e I D 2 0 1 5 1 2 2 8 0 0 , D E N @ C I N , s c h e d u l e d a t 2 0 1 5 - 1 2 - 2 8 T 2 0 : 3 0 : 0 0 - 0 5 : 0 0 u p d a t e d i n d a t a b a s e A l l g a m e s f o r S e a s o n 2 0 1 5 / R E G . W e e k 1 6 p r o c e s s e d ! S e a s o n 2 0 1 5 / P R E . W e e k 4 . G a m e I D 2 0 1 5 0 9 0 3 6 4 , D E N @ A R I , s c h e d u l e d a t 2 0 1 5 - 0 9 - 0 3 T 2 1 : 0 0 : 0 0 - 0 4 : 0 0 u p d a t e d i n d a t a b a s e S e a s o n 2 0 1 5 / P R E . W e e k 4 . G a m e I D 2 0 1 5 0 9 0 3 6 5 , S F @ S D , s c h e d u l e d a t 2 0 1 5 - 0 9 - 0 3 T 2 2 : 0 0 : 0 0 - 0 4 : 0 0 u p d a t e d i n d a t a b a s e S e a s o n 2 0 1 5 / P R E . W e e k 4 . G a m e I D 2 0 1 5 0 9 0 3 6 6 , S E A @ O A K , s c h e d u l e d a t 2 0 1 5 - 0 9 - 0 3 T 2 2 : 0 0 : 0 0 - 0 4 : 0 0 u p d a t e d i n d a t a b a s e A l l g a m e s f o r S e a s o n 2 0 1 5 / P R E . W e e k 4 p r o c e s s e d ! . . .

Slide 32

Slide 32 text

async/await in real life Backend API application

Slide 33

Slide 33 text

Task. Provide an API for week data NFL.com still returns data in X M L We need this data in G r a p h Q L Okay, okay, actually in J S O N And if there are no live games, we don't need to fetch data directly from NFL

Slide 34

Slide 34 text

Step 0. Original NFL.com data

Slide 35

Slide 35 text

Step 1. Flask base application f r o m f l a s k i m p o r t F l a s k f r o m . i m p o r t c a c h e f r o m . d a t a i m p o r t t o _ g a m e _ d a t a a p p = F l a s k ( _ _ n a m e _ _ ) @ a p p . r o u t e ( ' / a p i / s e a s o n / < i n t : s e a s o n _ y e a r > / < w e e k _ s l u g > ' ) d e f r e t r i e v e _ w e e k ( s e a s o n _ y e a r , w e e k _ s l u g ) : i s _ p r e s e a s o n = w e e k _ s l u g [ : 4 ] = = ' p r e - ' w e e k = i n t ( w e e k _ s l u g [ 4 : ] i f i s _ p r e s e a s o n e l s e w e e k _ s l u g ) c a c h e _ k e y = c a c h e . b u i l d _ k e y ( s e a s o n _ y e a r , w e e k _ s l u g ) w e e k _ d a t a = c a c h e . e n s u r e _ d a t a ( c a c h e _ k e y ) i f w e e k _ d a t a i s N o n e : w e e k _ o b j = f e t c h _ w e e k ( s e a s o n _ y e a r , w e e k , i s _ p r e s e a s o n ) w e e k _ d a t a = [ t o _ g a m e _ d a t a ( g a m e ) f o r g a m e i n w e e k _ o b j . i t e r f i n d ( ' g m s / g ' ) ] c a c h e . s t o r e _ d a t a ( c a c h e _ k e y , w e e k _ d a t a ) r e t u r n j s o n i f y ( w e e k _ d a t a )

Slide 36

Slide 36 text

Step 2. Running Flask application $ g u n i c o r n - k s y n c - w 5 - t 6 0 a p p : a p p $ g u n i c o r n - k e v e n t l e t - w 5 - t 6 0 a p p : a p p

Slide 37

Slide 37 text

Step 3. aiohttp.web base application i m p o r t u j s o n f r o m a i o h t t p i m p o r t w e b a s y n c d e f a i o _ r e t r i e v e _ w e e k ( r e q u e s t ) : s e a s o n _ y e a r = i n t ( r e q u e s t . m a t c h _ i n f o [ ' s e a s o n _ y e a r ' ] ) w e e k _ s l u g = r e q u e s t . m a t c h _ i n f o [ ' w e e k _ s l u g ' ] i s _ p r e s e a s o n = w e e k _ s l u g [ : 4 ] = = ' p r e - ' w e e k = i n t ( w e e k _ s l u g [ 4 : ] i f i s _ p r e s e a s o n e l s e w e e k _ s l u g ) c a c h e _ k e y = c a c h e . b u i l d _ k e y ( s e a s o n _ y e a r , w e e k , i s _ p r e s e a s o n ) w e e k _ d a t a = a w a i t c a c h e . a i o _ e n s u r e _ d a t a ( c a c h e _ k e y ) i f w e e k _ d a t a i s N o n e : w e e k _ o b j = a w a i t a i o _ f e t c h _ w e e k ( s e a s o n _ y e a r , w e e k , i s _ p r e s e a s o n ) w e e k _ d a t a = [ t o _ g a m e _ d a t a ( g a m e ) f o r g a m e i n w e e k _ o b j . i t e r f i n d ( ' g m s / g ' ) ] a w a i t c a c h e . s t o r e _ d a t a ( c a c h e _ k e y , w e e k _ d a t a ) r e t u r n w e b . R e s p o n s e ( u j s o n . d u m p s ( w e e k _ d a t a ) , c o n t e n t _ t y p e = ' a p p l i c a t i o n / j s o n ' )

Slide 38

Slide 38 text

Step 3. aiohttp.web base application (continue) a i o _ a p p = w e b . A p p l i c a t i o n ( ) a i o _ a p p . r o u t e r . a d d _ r o u t e ( ' G E T ' , ' / a p i / s e a s o n / { s e a s o n _ y e a r : \ d { 4 } } / { w e e k _ s l u g } ' , a i o _ r e t r i e v e _ w e e k )

Slide 39

Slide 39 text

aiohttp.web basics Each view function should be a c o r o u t i n e /a s y n c d e f and return response or raise an exc Each view function receives only one arg: r e q u e s t Query params data contains in r e q u e s t . G E T multidict Body data contains in r e q u e s t . P O S T multidict and should be awaited with a w a i t r e q u e s t . p o s t ( ) Matched data contains in r e q u e s t . m a t c h _ i n f o dict r e q u e s t also contains an a p p instance

Slide 40

Slide 40 text

Step 4. Runinng aiohttp.web application $ g u n i c o r n - k a i o h t t p . w o r k e r . G u n i c o r n W e b W o r k e r - w 5 - t 6 0 a p p : a i o _ a p p

Slide 41

Slide 41 text

Step 5. Resulted JSON

Slide 42

Slide 42 text

async/await in real life Server-Sent Events

Slide 43

Slide 43 text

Task. Livescore stream When there are live games, update the scores automagically for the user Do not reconnect each X seconds for new scores When new scores available - show them to user This is the case for Server-Sent Events (SSE)

Slide 44

Slide 44 text

Server-Sent Events. Message Format d a t a : H e l l o , P y C o n F i n l a n d ! < b l a n k l i n e > e v e n t : p y c o n f i d a t a : H e l l o , P y c o n F i n l a n d ! d a t a : I ' m g l a d t o b e h e r e < b l a n k l i n k > i d : 4 2 e v e n t : u a p y c o n d a t a : P y C o n U k r a i n e 2 0 1 6 w i l l h e l d a p l a c e i n L v i v a t A p r i l 2 0 1 6 < b l a n k l i n k >

Slide 45

Slide 45 text

Server-Sent Events. EventSource c o n s t e l e m e n t = d o c u m e n t . g e t E l e m e n t B y I d ( " n o t i f i c a t i o n s - c o n t a i n e r " ) ; c o n s t s o u r c e = n e w E v e n t S o u r c e ( " / a p i / n o t i f i c a t i o n s " ) ; s o u r c e . o n e r r o r = e r r = > { . . . } ; s o u r c e . a d d E v e n t L i s t e n e r ( " p y c o n f i " , e v t = > { e l e m e n t . i n n e r H T M L + = ` < c o d e > # p y c o n f i < / c o d e > $ { e v t . d a t a } < b r > ` ; } ) ; s o u r c e . a d d E v e n t L i s t e n e r ( " u a p y c o n " , e v t = > { e l e m e n t . i n n e r H T M L + = ` < c o d e > # u a p y c o n < / c o d e > $ { e v t . d a t a } < b r > ` ; } ) ; . . . s o u r c e . c l o s e ( ) ;

Slide 46

Slide 46 text

How to implement? 1. Run script to fetch live scores and publish new scores to Redis PUBSUB 2. Subscribe to this channel in livescore stream view, receive message and push it to client 3. Subscribe to livescore stream in client and update scores when new message received 4. ... 5. PROFIT

Slide 47

Slide 47 text

Step 1. Fetching & publishing scores change d e f m a i n ( * a r g s ) : s e a s o n , w e e k = a r g s l o o p = a s y n c i o . g e t _ e v e n t _ l o o p ( ) w e e k _ o b j = l o o p . r u n _ u n t i l _ c o m p l e t e ( a i o _ f e t c h _ w e e k ( s e a s o n , w e e k , F a l s e ) ) w e e k _ d a t a = [ t o _ g a m e _ d a t a ( g a m e ) f o r g a m e i n w e e k _ o b j . i t e r f i n d ( ' g m s / g ' ) ] r e d i s = l o o p . r u n _ u n t i l _ c o m p l e t e ( c r e a t e _ r e d i s ( ( ' l o c a l h o s t ' , 6 3 7 9 ) ) ) l o o p . r u n _ u n t i l _ c o m p l e t e ( r e d i s . p u b l i s h ( ' C H A N N E L ' , u j s o n . d u m p s ( w e e k _ d a t a ) ) ) l o o p . c l o s e ( ) r e t u r n F a l s e i f _ _ n a m e _ _ = = ' _ _ m a i n _ _ ' : s y s . e x i t ( i n t ( m a i n ( * s y s . a r g v [ 1 : ] ) ) )

Slide 48

Slide 48 text

Step 2. Implementing livescore stream a s y n c d e f l i v e s c o r e ( r e q u e s t ) : i f r e q u e s t . h e a d e r s [ ' A c c e p t ' ] ! = ' t e x t / e v e n t - s t r e a m ' : r a i s e w e b . H T T P F o u n d ( ' / a p i / ' ) r e s p o n s e = w e b . S t r e a m R e s p o n s e ( s t a t u s = 2 0 0 , h e a d e r s = { ' C o n t e n t - T y p e ' : ' t e x t / e v e n t - s t r e a m ' , ' C a c h e - C o n t r o l ' : ' n o - c a c h e ' , ' C o n n e c t i o n ' : ' k e e p - a l i v e ' , } ) r e s p o n s e . s t a r t ( r e q u e s t ) # T h i s w i l l b e c h a n g e d i n a s y n c i o = = 0 . 1 8 . 0 . . . r e t u r n r e s p o n s e

Slide 49

Slide 49 text

Step 2. Subscribing to scores update . . . r e d i s = a w a i t c r e a t e _ r e d i s ( ( ' l o c a l h o s t ' , 6 3 7 9 ) ) c h a n n e l = ( a w a i t r e d i s . s u b s c r i b e ( ' C H A N N E L ' ) ) [ 0 ] w h i l e ( a w a i t c h a n n e l . w a i t _ m e s s a g e ( ) ) : m e s s a g e = a w a i t c h a n n e l . g e t ( ) r e s p o n s e . w r i t e ( b ' e v e n t : u p d a t e \ r \ n ' ) r e s p o n s e . w r i t e ( b ' d a t a : ' + m e s s a g e + b ' \ r \ n \ r \ n ' ) . . .

Slide 50

Slide 50 text

Step 2. Registering stream to the app a i o _ a p p . r o u t e r . a d d _ r o u t e ( ' G E T ' , ' / a p i / l i v e s c o r e ' , l i v e s c o r e )

Slide 51

Slide 51 text

Step 3. Subscribing to livescore stream on client i m p o r t R e a c t , { C o m p o n e n t } f r o m " r e a c t " ; e x p o r t d e f a u l t A p p e x t e n d s C o m p o n e n t { l i v e S c o r e = n u l l ; c o n s t r u c t o r ( p r o p s ) { s u p e r ( p r o p s ) ; t h i s . s t a t e = { g a m e s : p r o p s . g a m e s } ; } c o m p o n e n t D i d M o u n t ( ) { t h i s . l i v e S c o r e = n e w E v e n t S o u r c e ( " / a p i / " ) ; t h i s . l i v e S c o r e . a d d E v e n t L i s t e n e r ( " u p d a t e " , e v t = > { t h i s . s e t S t a t e ( { g a m e s : J S O N . p a r s e ( e v t . d a t a ) } ) ; } ) ; } c o m p o n e n t W i l l U n m o u n t ( ) { t h i s . l i v e S c o r e . c l o s e ( ) ; } r e n d e r ( ) { r e t u r n } } < G a m e s d a t a = { t h i s . s t a t e . g a m e s } / > ;

Slide 52

Slide 52 text

Step 4. Setting up the nginx l o c a t i o n / a p i / l i v e s c o r e { c h u n k e d _ t r a n s f e r _ e n c o d i n g o f f ; p r o x y _ b u f f e r i n g o f f ; p r o x y _ c a c h e o f f ; p r o x y _ h t t p _ v e r s i o n 1 . 1 ; p r o x y _ p a s s h t t p : / / 1 2 7 . 0 . 0 . 1 : 8 0 0 0 ; p r o x y _ s e t _ h e a d e r C o n n e c t i o n ' ' ; p r o x y _ s e t _ h e a d e r H o s t $ h o s t ; p r o x y _ s e t _ h e a d e r X - R e a l - I P $ r e m o t e _ a d d r ; p r o x y _ s e t _ h e a d e r X - F o r w a r d e d - F o r $ p r o x y _ a d d _ x _ f o r w a r d e d _ f o r ; }

Slide 53

Slide 53 text

Step 5. It works

Slide 54

Slide 54 text

async/await era Conclusion

Slide 55

Slide 55 text

The Future is Here Python 3.5 released and ready to be used in production Python 3.5 contains new a s y n c /a w a i t statements a s y n c i o stack got great addition, code now is simple to read and understand asyncio stack (for web applications): a i o h t t p – interact with remote API a i o h t t p . w e b – create API backends a i o p g – interact with PostgreSQL (DB) a i o r e d i s – interact with Redis (cache)

Slide 56

Slide 56 text

The Future is Here a s y n c i o stack is a great reason why you'll finally need to switch to Python 3 a s y n c i o is good fit for: Database-less API Requesting remote API Async code execution Predictable async I/O And I'm not telling you about WebSockets, which supported by a i o h t t p out of the box

Slide 57

Slide 57 text

And the real reason of using asyncio

Slide 58

Slide 58 text

Cause it cool and trendy!

Slide 59

Slide 59 text

Questions?

Slide 60

Slide 60 text

Bonus. PyCon Ukraine 2016 April 2016 Lviv, Ukraine Related activities Looking for speakers: @hotsyk