Reactive Programming in Python

In relation to Internet of Things, micro services, and big data, a developer is easily being expected to handle the stream of data flow. A growing fantasy of Reactive programming is being told that a paradigm can help people to face these challenges in theory and practice and to make life easier. Is it True? Or, does it SCALE?

Join the quest to to discover reactive design and data workflow implemented in Python. We’ll inspect their features and use cases of reactive programming, to name a few, Python built-in, PyFunctional, RxPy, concurrent.futures, … etc., study their best practices, and discover the elegant part compared with commonly seen sequential chaining. We also want to know when it may complicate your code.

Keywords: functional, asynchronous, map, data flow, stream

Transcript

1. What is Reactive? 1
2. 2 Game is reactive.
3. 3 Presentation? not
4. ... if you fall into sleep 4
5. Reactive Programming in Python Keith Yang y a n g @ k e i t h e i s . o r g @ k e i t h e i s    P y C o n T a i w a n 2 0 1 6 5
7.  Keith Yang Sofeware Engineering Artisan • Coding • Agile • Web & Cloud & Virtualization @Taipeipy & @PyConTW 7
8. Outlines I. What & Why  II. Dive into  & play with  III. Thumbs up  Reactive Programming in Python 8
9. 9 I. Reactive Programming
10. My hand... tell me! 10
11. Seen on web application • The feel of interactive • For example: • Hackfoldr, Gitter, Slack, Web IRC, ... • Gmail, Facebook, Twitter, Pinterest, Plurk, ... • Online map, calendars, document, ... 11
12. 12 I mean... programming?
13. 13 Functional Python 101 
14. Map Map a iterable, i.e., a list, to a function. I n [ 1 ] : s t r _ m a p = m a p ( s t r , [ 1 , 2 , 3 ] ) I n [ 2 ] : l i s t ( s t r _ m a p ) O u t [ 2 ] : [ ' 1 ' , ' 2 ' , ' 3 ' ] 14
15. Lambda Create small anonymous function inline I n [ 1 ] : m u l t i p l e _ 7 = l a m b d a x : x * 7 I n [ 2 ] : t y p e ( m u l t i p l e _ 7 ) O u t [ 2 ] : f u n c t i o n I n [ 3 ] : m u l t i p l e _ 7 ( 6 ) O u t [ 3 ] : 4 2 15
16. Map & Lambda I n [ 1 ] : l i s t ( m a p ( l a m b d a x : x * 2 , [ 1 , 2 , 3 ] ) ) O u t [ 1 ] : [ 2 , 4 , 6 ] 16
17. Programming Paradigms • Declarative (i.e., Functional programming) Functional languages: Scheme, Clojure, Erlang, Haskell, OCaml • Imperative (i.e., Procedural programming) Procedural programming languages: C, Go, Fortran, Pascal, and BASIC 17
18. Declarative: Dataflow: Reactive Category of Reactive Programming Paradigm Relatives: • Declarative: Functional • Dataflow: Flow-based programming 18
19. 19 (Events) Stream
20. Stream: Core Spirit of Reactive “Everything can be a stream!” The mantra of Reactive Programming. In brief a sequence of events. 20
21. Stream 21
22. Stream in for in if ( e x p r e s s i o n f o r e x p r i n s e q u e n c e 1 i f c o n d i t i o n 1 f o r e x p r 2 i n s e q u e n c e 2 i f c o n d i t i o n 2 f o r e x p r 3 i n s e q u e n c e 3 . . . i f c o n d i t i o n 3 f o r e x p r N i n s e q u e n c e N i f c o n d i t i o n N ) visual messive! 22
23. f r o m p a t h l i b i m p o r t P a t h f r o m z i p f i l e i m p o r t Z i p F i l e f i l e s _ s t r e a m = ( Z i p F i l e ( f i l e . n a m e ) f o r f i l e i n P a t h ( " . " ) . i t e r d i r ( ) i f f i l e . n a m e . l o w e r ( ) . e n d s w i t h ( ' . z i p ' ) ) z i p p e d _ c o n t e n t _ l i s t = [ ] f o r a _ z i p f i l e i n f i l e s _ s t r e a m : f o r i t e m _ n a m e i n a _ z i p f i l e . n a m e l i s t ( ) : z i p p e d _ c o n t e n t _ l i s t . a p p e n d ( i t e m _ n a m e ) 23
24. 24 Rewrite in a reactive style
25. f r o m p a t h l i b i m p o r t P a t h f r o m z i p f i l e i m p o r t Z i p F i l e f r o m f u n c t i o n a l i m p o r t s e q z i p p e d _ c o n t e n t _ l i s t = ( s e q ( P a t h ( " . " ) . i t e r d i r ( ) ) . f i l t e r ( l a m b d a i t e m : i t e m . i s _ f i l e ( ) ) . f i l t e r ( l a m b d a f i l e : f i l e . n a m e . l o w e r ( ) . e n d s w i t h ( ' . z i p ' ) ) . m a p ( Z i p F i l e ) . f l a t _ m a p ( l a m b d a a _ z i p f i l e : a _ z i p f i l e . n a m e l i s t ( ) ) ) 25
26. Feel like SQL query d b . q u e r y ( u s e r s ) \ . f i l t e r ( r o l e = = C a f e . c u s t o m e r ) \ . f i l t e r ( c u p s _ b o u g h t > = 1 0 0 _ 0 0 0 _ 0 0 0 ) \ . u p d a t e ( s t a t e = " c a f f e i n a t e d " ) 26
27. 27 Better? It depends.
28. Why Reactive Programming • Abstraction • Focus on business logic • Concise code • Performance • Usually seen in asynchronous context • Parallel processing, smallest unit of change 28
29. 29 Aannndddd...
30. 30  Functional  潮到出水
31. Quicksort Example 31
32. Quicksort logic in Ocaml l e t r e c q s o r t = f u n c t i o n | [ ] - > [ ] | p i v o t : : r e s t - > l e t i s _ l e s s x = x < p i v o t i n l e t l e f t , r i g h t = L i s t . p a r t i t i o n i s _ l e s s r e s t i n q s o r t l e f t @ [ p i v o t ] @ q s o r t r i g h t ; ; 32
33. Code Smell n u m _ s t r e a m = [ 1 , 2 , 3 , 4 , 5 , 6 ] e v e n s 1 = [ ] f o r n i n n u m b e r s : i f ( n % 2 ) = = 0 : e v e n s 1 . a p p e n d ( n ) # I m p e r a c t i v e e v e n s 2 = [ n f o r n i n n u m _ s t r e a m i f ( n % 2 ) = = 0 ] # L i s t c o m p r e h e n s i o n r e s u l t : [ 2 , 4 , 6 ] 33
34. 34 Try PyFunctional
35. Code Smell (Cont.) Functional Reactive Programming (FRP) f r o m f u n c t i o n a l i m p o r t s e q d e f i s _ e v e n ( n u m ) : # B i g b u s i n e s s l o g i c ! r e t u r n ( n u m % 2 ) = = 0 e v e n s 3 = s e q ( n u m _ s t r e a m ) . f i l t e r ( i s _ e v e n ) # [ 2 , 4 , 6 ] f u n c t i o n a l module is provided by PyFunctional 35
36. Real life example: find_kernels() k e r n e l s = [ f . n a m e f o r f i n f o l d e r _ p a t h . i t e r d i r ( ) i f f . i s _ f i l e ( ) a n d f . n a m e . s t a r t s w i t h ( ' v m l i n u z ' ) ] # H o w t o a d d o n e o r t w o m o r e l e v e l s o f l o o p ? k e r n e l s = s e q ( f o l d e r _ p a t h . i t e r d i r ( ) ) \ . f i l t e r ( l a m b d a f : f . i s _ f i l e ( ) ) \ . f i l t e r ( l a m b d a f : f . n a m e . s t a r t s w i t h ( ' v m l i n u z ' ) ) \ . f i l t e r ( l a m b d a f : ' r e s c u e ' n o t i n k . n a m e ) \ . m a p ( l a m b d a f : f . n a m e ) 36
37. 37 A = B + C
38. Spreadsheet 38
39. Spreadsheet is “Reactive” 39
40. Recap: What Reactive Programming • A programming paradigm • Core spirit: Stream • Code look: sequence, map, and lambda 40
41. Recap: Why Reactive Programming • Functional style  focus on logic • Asynchronous candy  performance in a concise way 41
42. 42 II. Dive into Reactive 
43. Stream Fun 1. Alike workload 2. Merge 3. Re-use 43
44. A example powered by RxPy (rx) i m p o r t r x n u m _ s t r e a m = [ 1 , 2 , 3 , 4 ] n u m _ f l o w = r x . O b s e r v a b l e . f r o m _ ( n u m _ s t r e a m ) n u m _ f l o w . s u b s c r i b e ( p r i n t ) 44
45. I n [ 1 ] : i m p o r t r x I n [ 2 ] : n u m _ s t r e a m = [ 1 , 2 , 3 , 4 ] I n [ 3 ] : n u m _ f l o w = r x . O b s e r v a b l e . f r o m _ ( n u m _ s t r e a m ) I n [ 4 ] : n u m _ f l o w . s u b s c r i b e ( p r i n t ) 1 2 3 4 O u t [ 4 ] : < r x . d i s p o s a b l e s . A n o n y m o u s D i s p o s a b l e . A n o n y m o u s D i s p o s a b l e a t 0 x 1 0 4 1 f c a c 8 > 45
46. Merge c h a r s _ f l o w = r x . O b s e r v a b l e . f r o m _ ( [ " a " , " b " , " c " ] ) n u m b e r s _ f l o w = r x . O b s e r v a b l e . f r o m _ ( [ 1 , 2 , 3 , 4 ] ) # o r r x . O b s e r v a b l e . r a n g e ( 1 , 4 ) p r i n t a b l e _ f l o w = n u m b e r s _ f l o w . m a p ( l a m b d a n u m : n u m * 2 ) . m e r g e ( c h a r s _ f l o w ) p r i n t a b l e _ f l o w . s u b s c r i b e ( p r i n t ) # a 2 b 4 c 6 8 1 0 46
47. Most.js - Live Demo Mouse position X, Y = 166, 14 m o s t . f r o m E v e n t ( ' m o u s e m o v e ' , d o c u m e n t ) . m a p ( f u n c t i o n ( e v e n t ) { r e t u r n e v e n t . c l i e n t X + ' , ' + e v e n t . c l i e n t Y ; } ) . s t a r t W i t h ( ' m o v e t h e m o u s e , p l e a s e ' ) . o b s e r v e ( f u n c t i o n ( x y _ s t r ) { d o c u m e n t . b o d y . t e x t C o n t e n t = x y _ s t r ; } ) ; Most.js - Monadic reactive streams ” 47
48. ReactiveX • Microsoft's open-source Reactive Extensions library (Rx) • Compose programs of • Asynchronous • Event-based • By using observable sequences. 48
49. ReactiveX (Cont.) • RxPy • RxJava • RxJS • Rx.NET • RxSwift 49
50. Even in C++: RxCpp a u t o n u m _ f l o w = r x c p p : : o b s e r v a b l e < > : : c r e a t e ( [ ] ( r x c p p : : s u b s c r i b e r < i n t > s ) { f o r ( i = 0 ; i < = 5 ; i + + ) s . o n _ n e x t ( i ) ; s . o n _ c o m p l e t e d ( ) ; } ) ; n u m _ f l o w . s u b s c r i b e ( [ ] ( i n t v ) { p r i n t f ( " O n N e x t : % d \ n " , v ) ; } , [ ] ( ) { p r i n t f ( " O n C o m p l e t e d \ n " ) ; } ) ; 50
51. Abstract away • Low-level threading • Synchronization • Thread-safety • Concurrent data structures • Non-blocking I/O 51
52. 52 Focus  on biz logic
53. 53 Asynchronous Programming
54. i m p o r t t i m e i m p o r t c o n c u r r e n t . f u t u r e s i m p o r t r x n u m _ s t r e a m = [ 1 , 2 , 3 , 4 , 5 ] d e f w o r k _ s l o w l y ( d a t a ) : t i m e . s l e e p ( 1 ) r e t u r n d a t a * 2 w i t h c o n c u r r e n t . f u t u r e s . P r o c e s s P o o l E x e c u t o r ( 5 ) a s w o r k e r : r x . O b s e r v a b l e . f r o m _ ( n u m _ s t r e a m ) \ . f l a t _ m a p ( l a m b d a n u m : w o r k e r . s u b m i t ( w o r k _ s l o w l y , n u m ) ) . s u b s c r i b e ( p r i n t ) 54
55. Observable, flat_map, subscribe w i t h c o n c u r r e n t . f u t u r e s . P r o c e s s P o o l E x e c u t o r ( 5 ) a s w o r k e r : r x . O b s e r v a b l e . f r o m _ ( n u m _ s t r e a m ) \ . f l a t _ m a p ( l a m b d a n u m : w o r k e r . s u b m i t ( w o r k _ s l o w l y , n u m ) ) . s u b s c r i b e ( p r i n t ) 55
56. 56 Why Observables?
57. Observables “Observable model allows you to treat streams of Asynchronous events with the same sort of operations that you use for collections of data items like arrays. It frees you from callbacks, and thereby makes your code more readable and less prone to bugs.” 57
58. Observables (cont.) In brief: • Use same interface to treat streams • More readable code • ease the pain of callback • Less bugs 58
59. flap_map I want values,  inside each event. ” 59
60. Demo the behavior of flap_map() s t r _ l i s t s = [ [ " 6 " , " 0 3 " ] , [ " 4 2 " ] ] l i s t 1 = [ ] f o r s t r _ l i s t i n s t r _ l i s t s : f o r v a r i n s t r _ l i s t : l i s t 1 . a p p e n d ( i n t ( v a r ) ) # [ 6 , 3 , 4 2 ] 60
61. flat_map in PyFunctional f r o m f u n c t i o n a l i m p o r t s e q d e f i n t _ l i s t ( i t e r a b l e ) : r e t u r n [ i n t ( v a r ) f o r v a r i n i t e r a b l e ] l i s t 2 = s e q ( s t r _ l i s t s ) . f l a t _ m a p ( i n t _ l i s t ) # [ 6 , 3 , 4 2 ] 61
62. Parallel with Rx Schedulers You can use every scheduler you like to decide where to execute the work. 62
63. A RxJava example O b s e r v a b l e < I n t e g e r > n u m _ f l o w = O b s e r v a b l e . r a n g e ( 1 , 6 ) . d o O n E a c h ( d e b u g ( " D i d " ) ) ; n u m _ f l o w . s u b s c r i b e ( n u m - > S y s t e m . o u t . p r i n t l n ( n u m ) ) ; 63
64. Parallel with RxJava O b s e r v a b l e < I n t e g e r > n u m _ f l o w = O b s e r v a b l e . r a n g e ( 1 , 6 ) . f l a t M a p ( n - > O b s e r v a b l e . r a n g e ( n , 3 ) . s u b s c r i b e O n ( S c h e d u l e r s . c o m p u t a t i o n ( ) ) . d o O n E a c h ( d e b u g ( " D i d " ) ) ) ; n u m _ f l o w . s u b s c r i b e ( n u m - > S y s t e m . o u t . p r i n t l n ( n u m ) ) ; 64
65. The look of parallel output R x C o m p u t a t i o n T h r e a d P o o l - 3 | D i d : > 3 R x C o m p u t a t i o n T h r e a d P o o l - 1 | D i d : > 1 R x C o m p u t a t i o n T h r e a d P o o l - 2 | D i d : > 4 R x C o m p u t a t i o n T h r e a d P o o l - 3 | D i d : - > 2 R x C o m p u t a t i o n T h r e a d P o o l - 1 | D i d : > 7 R x C o m p u t a t i o n T h r e a d P o o l - 2 | D i d : - > 6 R x C o m p u t a t i o n T h r e a d P o o l - 3 | D i d : - - > 5 R x C o m p u t a t i o n T h r e a d P o o l - 3 | D i d : - - - > | 65
66. 66 III. Summary 
67. 67 Try functional & async with Rx
68. Try functional & async with Rx • Win the stream of repeated and alike events • Focus on the business logic • powered by functional programming. • Drive asynchronous programming 68
69. References • Book: Python Reactive Programming, November 2016 • Book: Reactive Programming with Scala and Akka, February 2016 • Book: Reactive Programming with RxJava, August 2016 69
70. References (Cont.) • On Github  : PyFunctional, RxPy • Gist: The introduction to Reactive Programming you've been missing by @andrestaltz, including the use case of Promise • Slide:Functional Reactive Python On Introducing CS to High School Students by John Peterson 70
71. 71 Q&A 
72. 72 Thank You! 