Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Reactive Programming in Python

Reactive Programming in Python

In relation to Internet of Things, micro services, and big data, a developer is easily being expected to handle the stream of data flow. A growing fantasy of Reactive programming is being told that a paradigm can help people to face these challenges in theory and practice and to make life easier. Is it True? Or, does it SCALE?

Join the quest to to discover reactive design and data workflow implemented in Python. We’ll inspect their features and use cases of reactive programming, to name a few, Python built-in, PyFunctional, RxPy, concurrent.futures, … etc., study their best practices, and discover the elegant part compared with commonly seen sequential chaining. We also want to know when it may complicate your code.

Keywords: functional, asynchronous, map, data flow, stream

D1804d3c315cbb850aaa88d2fbadc3cf?s=128

Keith Yang

June 03, 2016
Tweet

Transcript

  1. What is Reactive? 1

  2. 2 Game is reactive.

  3. 3 Presentation? not

  4. ... if you fall into sleep 4

  5. Reactive Programming in Python Keith Yang y a n g

    @ k e i t h e i s . o r g @ k e i t h e i s    P y C o n T a i w a n 2 0 1 6 5
  6. 6

  7.  Keith Yang Sofeware Engineering Artisan • Coding • Agile

    • Web & Cloud & Virtualization @Taipeipy & @PyConTW 7
  8. Outlines I. What & Why  II. Dive into 

    & play with  III. Thumbs up  Reactive Programming in Python 8
  9. 9 I. Reactive Programming

  10. My hand... tell me! 10

  11. Seen on web application • The feel of interactive •

    For example: • Hackfoldr, Gitter, Slack, Web IRC, ... • Gmail, Facebook, Twitter, Pinterest, Plurk, ... • Online map, calendars, document, ... 11
  12. 12 I mean... programming?

  13. 13 Functional Python 101 

  14. Map Map a iterable, i.e., a list, to a function.

    I n [ 1 ] : s t r _ m a p = m a p ( s t r , [ 1 , 2 , 3 ] ) I n [ 2 ] : l i s t ( s t r _ m a p ) O u t [ 2 ] : [ ' 1 ' , ' 2 ' , ' 3 ' ] 14
  15. Lambda Create small anonymous function inline I n [ 1

    ] : m u l t i p l e _ 7 = l a m b d a x : x * 7 I n [ 2 ] : t y p e ( m u l t i p l e _ 7 ) O u t [ 2 ] : f u n c t i o n I n [ 3 ] : m u l t i p l e _ 7 ( 6 ) O u t [ 3 ] : 4 2 15
  16. Map & Lambda I n [ 1 ] : l

    i s t ( m a p ( l a m b d a x : x * 2 , [ 1 , 2 , 3 ] ) ) O u t [ 1 ] : [ 2 , 4 , 6 ] 16
  17. Programming Paradigms • Declarative (i.e., Functional programming) Functional languages: Scheme,

    Clojure, Erlang, Haskell, OCaml • Imperative (i.e., Procedural programming) Procedural programming languages: C, Go, Fortran, Pascal, and BASIC 17
  18. Declarative: Dataflow: Reactive Category of Reactive Programming Paradigm Relatives: •

    Declarative: Functional • Dataflow: Flow-based programming 18
  19. 19 (Events) Stream

  20. Stream: Core Spirit of Reactive “Everything can be a stream!”

    The mantra of Reactive Programming. In brief a sequence of events. 20
  21. Stream 21

  22. Stream in for in if ( e x p r

    e s s i o n f o r e x p r i n s e q u e n c e 1 i f c o n d i t i o n 1 f o r e x p r 2 i n s e q u e n c e 2 i f c o n d i t i o n 2 f o r e x p r 3 i n s e q u e n c e 3 . . . i f c o n d i t i o n 3 f o r e x p r N i n s e q u e n c e N i f c o n d i t i o n N ) visual messive! 22
  23. f r o m p a t h l i

    b i m p o r t P a t h f r o m z i p f i l e i m p o r t Z i p F i l e f i l e s _ s t r e a m = ( Z i p F i l e ( f i l e . n a m e ) f o r f i l e i n P a t h ( " . " ) . i t e r d i r ( ) i f f i l e . n a m e . l o w e r ( ) . e n d s w i t h ( ' . z i p ' ) ) z i p p e d _ c o n t e n t _ l i s t = [ ] f o r a _ z i p f i l e i n f i l e s _ s t r e a m : f o r i t e m _ n a m e i n a _ z i p f i l e . n a m e l i s t ( ) : z i p p e d _ c o n t e n t _ l i s t . a p p e n d ( i t e m _ n a m e ) 23
  24. 24 Rewrite in a reactive style

  25. f r o m p a t h l i

    b i m p o r t P a t h f r o m z i p f i l e i m p o r t Z i p F i l e f r o m f u n c t i o n a l i m p o r t s e q z i p p e d _ c o n t e n t _ l i s t = ( s e q ( P a t h ( " . " ) . i t e r d i r ( ) ) . f i l t e r ( l a m b d a i t e m : i t e m . i s _ f i l e ( ) ) . f i l t e r ( l a m b d a f i l e : f i l e . n a m e . l o w e r ( ) . e n d s w i t h ( ' . z i p ' ) ) . m a p ( Z i p F i l e ) . f l a t _ m a p ( l a m b d a a _ z i p f i l e : a _ z i p f i l e . n a m e l i s t ( ) ) ) 25
  26. Feel like SQL query d b . q u e

    r y ( u s e r s ) \ . f i l t e r ( r o l e = = C a f e . c u s t o m e r ) \ . f i l t e r ( c u p s _ b o u g h t > = 1 0 0 _ 0 0 0 _ 0 0 0 ) \ . u p d a t e ( s t a t e = " c a f f e i n a t e d " ) 26
  27. 27 Better? It depends.

  28. Why Reactive Programming • Abstraction • Focus on business logic

    • Concise code • Performance • Usually seen in asynchronous context • Parallel processing, smallest unit of change 28
  29. 29 Aannndddd...

  30. 30  Functional  潮到出水

  31. Quicksort Example 31

  32. Quicksort logic in Ocaml l e t r e c

    q s o r t = f u n c t i o n | [ ] - > [ ] | p i v o t : : r e s t - > l e t i s _ l e s s x = x < p i v o t i n l e t l e f t , r i g h t = L i s t . p a r t i t i o n i s _ l e s s r e s t i n q s o r t l e f t @ [ p i v o t ] @ q s o r t r i g h t ; ; 32
  33. Code Smell n u m _ s t r e

    a m = [ 1 , 2 , 3 , 4 , 5 , 6 ] e v e n s 1 = [ ] f o r n i n n u m b e r s : i f ( n % 2 ) = = 0 : e v e n s 1 . a p p e n d ( n ) # I m p e r a c t i v e e v e n s 2 = [ n f o r n i n n u m _ s t r e a m i f ( n % 2 ) = = 0 ] # L i s t c o m p r e h e n s i o n r e s u l t : [ 2 , 4 , 6 ] 33
  34. 34 Try PyFunctional

  35. Code Smell (Cont.) Functional Reactive Programming (FRP) f r o

    m f u n c t i o n a l i m p o r t s e q d e f i s _ e v e n ( n u m ) : # B i g b u s i n e s s l o g i c ! r e t u r n ( n u m % 2 ) = = 0 e v e n s 3 = s e q ( n u m _ s t r e a m ) . f i l t e r ( i s _ e v e n ) # [ 2 , 4 , 6 ] f u n c t i o n a l module is provided by PyFunctional 35
  36. Real life example: find_kernels() k e r n e l

    s = [ f . n a m e f o r f i n f o l d e r _ p a t h . i t e r d i r ( ) i f f . i s _ f i l e ( ) a n d f . n a m e . s t a r t s w i t h ( ' v m l i n u z ' ) ] # H o w t o a d d o n e o r t w o m o r e l e v e l s o f l o o p ? k e r n e l s = s e q ( f o l d e r _ p a t h . i t e r d i r ( ) ) \ . f i l t e r ( l a m b d a f : f . i s _ f i l e ( ) ) \ . f i l t e r ( l a m b d a f : f . n a m e . s t a r t s w i t h ( ' v m l i n u z ' ) ) \ . f i l t e r ( l a m b d a f : ' r e s c u e ' n o t i n k . n a m e ) \ . m a p ( l a m b d a f : f . n a m e ) 36
  37. 37 A = B + C

  38. Spreadsheet 38

  39. Spreadsheet is “Reactive” 39

  40. Recap: What Reactive Programming • A programming paradigm • Core

    spirit: Stream • Code look: sequence, map, and lambda 40
  41. Recap: Why Reactive Programming • Functional style  focus on

    logic • Asynchronous candy  performance in a concise way 41
  42. 42 II. Dive into Reactive 

  43. Stream Fun 1. Alike workload 2. Merge 3. Re-use 43

  44. A example powered by RxPy (rx) i m p o

    r t r x n u m _ s t r e a m = [ 1 , 2 , 3 , 4 ] n u m _ f l o w = r x . O b s e r v a b l e . f r o m _ ( n u m _ s t r e a m ) n u m _ f l o w . s u b s c r i b e ( p r i n t ) 44
  45. I n [ 1 ] : i m p o

    r t r x I n [ 2 ] : n u m _ s t r e a m = [ 1 , 2 , 3 , 4 ] I n [ 3 ] : n u m _ f l o w = r x . O b s e r v a b l e . f r o m _ ( n u m _ s t r e a m ) I n [ 4 ] : n u m _ f l o w . s u b s c r i b e ( p r i n t ) 1 2 3 4 O u t [ 4 ] : < r x . d i s p o s a b l e s . A n o n y m o u s D i s p o s a b l e . A n o n y m o u s D i s p o s a b l e a t 0 x 1 0 4 1 f c a c 8 > 45
  46. Merge c h a r s _ f l o

    w = r x . O b s e r v a b l e . f r o m _ ( [ " a " , " b " , " c " ] ) n u m b e r s _ f l o w = r x . O b s e r v a b l e . f r o m _ ( [ 1 , 2 , 3 , 4 ] ) # o r r x . O b s e r v a b l e . r a n g e ( 1 , 4 ) p r i n t a b l e _ f l o w = n u m b e r s _ f l o w . m a p ( l a m b d a n u m : n u m * 2 ) . m e r g e ( c h a r s _ f l o w ) p r i n t a b l e _ f l o w . s u b s c r i b e ( p r i n t ) # a 2 b 4 c 6 8 1 0 46
  47. Most.js - Live Demo Mouse position X, Y = 166,

    14 m o s t . f r o m E v e n t ( ' m o u s e m o v e ' , d o c u m e n t ) . m a p ( f u n c t i o n ( e v e n t ) { r e t u r n e v e n t . c l i e n t X + ' , ' + e v e n t . c l i e n t Y ; } ) . s t a r t W i t h ( ' m o v e t h e m o u s e , p l e a s e ' ) . o b s e r v e ( f u n c t i o n ( x y _ s t r ) { d o c u m e n t . b o d y . t e x t C o n t e n t = x y _ s t r ; } ) ; Most.js - Monadic reactive streams ” 47
  48. ReactiveX • Microsoft's open-source Reactive Extensions library (Rx) • Compose

    programs of • Asynchronous • Event-based • By using observable sequences. 48
  49. ReactiveX (Cont.) • RxPy • RxJava • RxJS • Rx.NET

    • RxSwift 49
  50. Even in C++: RxCpp a u t o n u

    m _ f l o w = r x c p p : : o b s e r v a b l e < > : : c r e a t e ( [ ] ( r x c p p : : s u b s c r i b e r < i n t > s ) { f o r ( i = 0 ; i < = 5 ; i + + ) s . o n _ n e x t ( i ) ; s . o n _ c o m p l e t e d ( ) ; } ) ; n u m _ f l o w . s u b s c r i b e ( [ ] ( i n t v ) { p r i n t f ( " O n N e x t : % d \ n " , v ) ; } , [ ] ( ) { p r i n t f ( " O n C o m p l e t e d \ n " ) ; } ) ; 50
  51. Abstract away • Low-level threading • Synchronization • Thread-safety •

    Concurrent data structures • Non-blocking I/O 51
  52. 52 Focus  on biz logic

  53. 53 Asynchronous Programming

  54. i m p o r t t i m e

    i m p o r t c o n c u r r e n t . f u t u r e s i m p o r t r x n u m _ s t r e a m = [ 1 , 2 , 3 , 4 , 5 ] d e f w o r k _ s l o w l y ( d a t a ) : t i m e . s l e e p ( 1 ) r e t u r n d a t a * 2 w i t h c o n c u r r e n t . f u t u r e s . P r o c e s s P o o l E x e c u t o r ( 5 ) a s w o r k e r : r x . O b s e r v a b l e . f r o m _ ( n u m _ s t r e a m ) \ . f l a t _ m a p ( l a m b d a n u m : w o r k e r . s u b m i t ( w o r k _ s l o w l y , n u m ) ) . s u b s c r i b e ( p r i n t ) 54
  55. Observable, flat_map, subscribe w i t h c o n

    c u r r e n t . f u t u r e s . P r o c e s s P o o l E x e c u t o r ( 5 ) a s w o r k e r : r x . O b s e r v a b l e . f r o m _ ( n u m _ s t r e a m ) \ . f l a t _ m a p ( l a m b d a n u m : w o r k e r . s u b m i t ( w o r k _ s l o w l y , n u m ) ) . s u b s c r i b e ( p r i n t ) 55
  56. 56 Why Observables?

  57. Observables “Observable model allows you to treat streams of Asynchronous

    events with the same sort of operations that you use for collections of data items like arrays. It frees you from callbacks, and thereby makes your code more readable and less prone to bugs.” 57
  58. Observables (cont.) In brief: • Use same interface to treat

    streams • More readable code • ease the pain of callback • Less bugs 58
  59. flap_map I want values,  inside each event. ” 59

  60. Demo the behavior of flap_map() s t r _ l

    i s t s = [ [ " 6 " , " 0 3 " ] , [ " 4 2 " ] ] l i s t 1 = [ ] f o r s t r _ l i s t i n s t r _ l i s t s : f o r v a r i n s t r _ l i s t : l i s t 1 . a p p e n d ( i n t ( v a r ) ) # [ 6 , 3 , 4 2 ] 60
  61. flat_map in PyFunctional f r o m f u n

    c t i o n a l i m p o r t s e q d e f i n t _ l i s t ( i t e r a b l e ) : r e t u r n [ i n t ( v a r ) f o r v a r i n i t e r a b l e ] l i s t 2 = s e q ( s t r _ l i s t s ) . f l a t _ m a p ( i n t _ l i s t ) # [ 6 , 3 , 4 2 ] 61
  62. Parallel with Rx Schedulers You can use every scheduler you

    like to decide where to execute the work. 62
  63. A RxJava example O b s e r v a

    b l e < I n t e g e r > n u m _ f l o w = O b s e r v a b l e . r a n g e ( 1 , 6 ) . d o O n E a c h ( d e b u g ( " D i d " ) ) ; n u m _ f l o w . s u b s c r i b e ( n u m - > S y s t e m . o u t . p r i n t l n ( n u m ) ) ; 63
  64. Parallel with RxJava O b s e r v a

    b l e < I n t e g e r > n u m _ f l o w = O b s e r v a b l e . r a n g e ( 1 , 6 ) . f l a t M a p ( n - > O b s e r v a b l e . r a n g e ( n , 3 ) . s u b s c r i b e O n ( S c h e d u l e r s . c o m p u t a t i o n ( ) ) . d o O n E a c h ( d e b u g ( " D i d " ) ) ) ; n u m _ f l o w . s u b s c r i b e ( n u m - > S y s t e m . o u t . p r i n t l n ( n u m ) ) ; 64
  65. The look of parallel output R x C o m

    p u t a t i o n T h r e a d P o o l - 3 | D i d : > 3 R x C o m p u t a t i o n T h r e a d P o o l - 1 | D i d : > 1 R x C o m p u t a t i o n T h r e a d P o o l - 2 | D i d : > 4 R x C o m p u t a t i o n T h r e a d P o o l - 3 | D i d : - > 2 R x C o m p u t a t i o n T h r e a d P o o l - 1 | D i d : > 7 R x C o m p u t a t i o n T h r e a d P o o l - 2 | D i d : - > 6 R x C o m p u t a t i o n T h r e a d P o o l - 3 | D i d : - - > 5 R x C o m p u t a t i o n T h r e a d P o o l - 3 | D i d : - - - > | 65
  66. 66 III. Summary 

  67. 67 Try functional & async with Rx

  68. Try functional & async with Rx • Win the stream

    of repeated and alike events • Focus on the business logic • powered by functional programming. • Drive asynchronous programming 68
  69. References • Book: Python Reactive Programming, November 2016 • Book:

    Reactive Programming with Scala and Akka, February 2016 • Book: Reactive Programming with RxJava, August 2016 69
  70. References (Cont.) • On Github  : PyFunctional, RxPy •

    Gist: The introduction to Reactive Programming you've been missing by @andrestaltz, including the use case of Promise • Slide:Functional Reactive Python On Introducing CS to High School Students by John Peterson 70
  71. 71 Q&A 

  72. 72 Thank You! 