Slide 1

Slide 1 text

What Python can learn from Haskell Saturday, May 11, 13

Slide 2

Slide 2 text

Daniël @danieldekok Developer Lan ua e Technolo y at Gridline Workin on NLP based technolo y on a day to day basis (Java, C++, Haskell) Saturday, May 11, 13

Slide 3

Slide 3 text

Gideon @ ideondk Tech-lead at SpotDo Focusin on functional and actor based pro rammin throu h Scala and native application development Saturday, May 11, 13

Slide 4

Slide 4 text

Saturday, May 11, 13

Slide 5

Slide 5 text

MapReduce wasn’t invented by Goo le Saturday, May 11, 13

Slide 6

Slide 6 text

Fundamentals based on the principles of functional pro rammin Saturday, May 11, 13

Slide 7

Slide 7 text

So why bother about functional pro rammin ? Saturday, May 11, 13

Slide 8

Slide 8 text

New challen es in computin Saturday, May 11, 13

Slide 9

Slide 9 text

Bi data Saturday, May 11, 13

Slide 10

Slide 10 text

Bi data applyin filterin , transformations, and reductions to hu e streams of data Saturday, May 11, 13

Slide 11

Slide 11 text

The real-time web Saturday, May 11, 13

Slide 12

Slide 12 text

The real-time web constant exchan e of data, both in lar e and small packa es Saturday, May 11, 13

Slide 13

Slide 13 text

Bi data implications Saturday, May 11, 13

Slide 14

Slide 14 text

OO isn’t that well suited transformations are not inherent properties of objects Saturday, May 11, 13

Slide 15

Slide 15 text

A flat imperative model neither imperative functions are not easily composable Saturday, May 11, 13

Slide 16

Slide 16 text

Real-time web implications Saturday, May 11, 13

Slide 17

Slide 17 text

Traditional stacks aren’t suited Saturday, May 11, 13

Slide 18

Slide 18 text

Modern stacks neither ;-) Saturday, May 11, 13

Slide 19

Slide 19 text

Callback hell anyone? fs.readdir(source, function(err, files) { if (err) { console.log('Error finding files: ' + err) } else { files.forEach(function(filename, fileIndex) { console.log(filename) gm(source + filename).size(function(err, values) { if (err) { console.log('Error identifying file size: ' + err) } else { console.log(filename + ' : ' + values) aspect = (values.width / values.height) widths.forEach(function(width, widthIndex) { height = Math.round(width / aspect) console.log('resizing ' + filename + 'to ' + height + 'x' + height) this.resize(width, height).write(destination + 'w' + width + '_' + filename, function(err) { if (err) console.log('Error writing file: ' + err) }) }.bind(this)) } }) }) } }) Saturday, May 11, 13

Slide 20

Slide 20 text

Bi data & real-time requirements process lots of data, with as few resources as possible Saturday, May 11, 13

Slide 21

Slide 21 text

Developer requirements composable transformations, to work on lar e streams of data as easily as possible ( while keepin yourself DRY ) Saturday, May 11, 13

Slide 22

Slide 22 text

Goin functional! Saturday, May 11, 13

Slide 23

Slide 23 text

Stream processin was born in functional lan ua es Saturday, May 11, 13

Slide 24

Slide 24 text

Mappin map :: (a -> b) -> [a] -> [b] map f xs = [f x | x <- xs] Saturday, May 11, 13

Slide 25

Slide 25 text

Mappin example Prelude> map (\x -> x + 4) [1,2,3,4,5] [5,6,7,8,9] Saturday, May 11, 13

Slide 26

Slide 26 text

A Haskell example obtainKeywords :: [HTMLPage] -> [KeywordSet] obtainKeywords = map extractPlainText . map tokenize . map toTokenSet Saturday, May 11, 13

Slide 27

Slide 27 text

Desirable properties obtainKeywords = map extractPlainText . map tokenize . map toTokenSet Saturday, May 11, 13

Slide 28

Slide 28 text

More transformations obtainImportantKeywords :: [HTMLPage] -> KeywordSet obtainImportantKeywords = filter importantDocument . map extractPlainText . map tokenize . map toTokenSet . foldl' Set.union Set.empty Saturday, May 11, 13

Slide 29

Slide 29 text

Downsides of list-based streams Saturday, May 11, 13

Slide 30

Slide 30 text

Lazy I/O is ross :-( Saturday, May 11, 13

Slide 31

Slide 31 text

Like in fancy restaurants no do y ba s allowed! Saturday, May 11, 13

Slide 32

Slide 32 text

Not so fancy error handlin ‘either’ Saturday, May 11, 13

Slide 33

Slide 33 text

Not so fancy error handlin ‘either’ obtainImportantKeywords :: [HTMLPage] -> KeywordSet obtainImportantKeywords = filter importantDocument . map extractPlainText . map tokenize . map toTokenSet . foldl' Set.union Set.empty Saturday, May 11, 13

Slide 34

Slide 34 text

Iteratee-based pipelines Saturday, May 11, 13

Slide 35

Slide 35 text

Packa es: one too many enumerator, conduit, pipes, io-streams Saturday, May 11, 13

Slide 36

Slide 36 text

Teachin an old fold new tricks obtainImportantKeywords :: Iteratee KeywordSet m KeywordSet obtainImportantKeywords = getPages $= EL.filter importantDocument $= EL.map extractPlainText $= EL.map tokenize $= EL.map toTokenSet $$ EL.fold Set.union Set.empty Saturday, May 11, 13

Slide 37

Slide 37 text

Enumerator obtainImportantKeywords :: Iteratee KeywordSet m KeywordSet obtainImportantKeywords = getPages $= EL.filter importantDocument $= EL.map extractPlainText $= EL.map tokenize $= EL.map toTokenSet $$ EL.fold Set.union Set.empty getPages Saturday, May 11, 13

Slide 38

Slide 38 text

Iteratee obtainImportantKeywords :: Iteratee KeywordSet m KeywordSet obtainImportantKeywords = getPages $= EL.filter importantDocument $= EL.map extractPlainText $= EL.map tokenize $= EL.map toTokenSet $$ EL.fold Set.union Set.empty EL.fold Set.union Set.empty Saturday, May 11, 13

Slide 39

Slide 39 text

Enumeratees obtainImportantKeywords :: Iteratee KeywordSet m KeywordSet obtainImportantKeywords = getPages $= EL.filter importantDocument $= EL.map extractPlainText $= EL.map tokenize $= EL.map toTokenSet $$ EL.fold Set.union Set.empty $= EL.filter importantDocument $= EL.map extractPlainText $= EL.map tokenize $= EL.map toTokenSet Saturday, May 11, 13

Slide 40

Slide 40 text

Operators obtainImportantKeywords :: Iteratee KeywordSet m KeywordSet obtainImportantKeywords = getPages $= EL.filter importantDocument $= EL.map extractPlainText $= EL.map tokenize $= EL.map toTokenSet $$ EL.fold Set.union Set.empty $= $= $= $= $$ Saturday, May 11, 13

Slide 41

Slide 41 text

Lettin the stream flow run obtainImportantKeywords Saturday, May 11, 13

Slide 42

Slide 42 text

Concatenatin concatEnums [enum1, enum2, enum3] Saturday, May 11, 13

Slide 43

Slide 43 text

Zippin Iteratees EL.zip iteratee1 iteratee2 Saturday, May 11, 13

Slide 44

Slide 44 text

Combinin Iteratees combinedIteratee = do a <- EL.take 3 b <- EL.take 3 return (a,b) Saturday, May 11, 13

Slide 45

Slide 45 text

Iteratee internals Saturday, May 11, 13

Slide 46

Slide 46 text

Chunks Yield Left overs Saturday, May 11, 13

Slide 47

Slide 47 text

Continue Yield Error Saturday, May 11, 13

Slide 48

Slide 48 text

Internals in code data Stream i = Chunks [i] | EOF deriving (Show, Eq) data Step i m o = Continue (Stream i -> Iteratee i m o) | Yield b (Stream o) | Error Exc.SomeException Saturday, May 11, 13

Slide 49

Slide 49 text

Constructin an Iteratee consume :: Monad m => Iteratee a m [a] consume = liftI $ step id where step acc chunk = case chunk of Chunks [] -> Continue $ returnI . step acc Chunks xs -> Continue $ returnI . (step $ acc . (xs ++)) EOF -> Yield (acc []) EOF Saturday, May 11, 13

Slide 50

Slide 50 text

Stream processin is ettin hip in other lan ua es as well! Saturday, May 11, 13

Slide 51

Slide 51 text

Iteratees are available in Scala (more on that later) Saturday, May 11, 13

Slide 52

Slide 52 text

Java 8 introduces a new stream API (althou h limited in comparison) Saturday, May 11, 13

Slide 53

Slide 53 text

Clojure has reducers Saturday, May 11, 13

Slide 54

Slide 54 text

case class LogMessage(ip: String, code: Int, message: String) val (enum, channels) = Concurrent.Broadcast[Array[Byte]] val converter = Enumeratee.map[Array[Byte]](ba => JsValue(ba)) val reader = Enumeratee.map[JsValue](evt => LogMessage(evt("id"), evt("code"), evt("message"))) val filterErrors = Enumeratee.filter(e => e.code == 500) def streamErrors = Action { request => enum &> converter &> reader &> filterErrors &> Comet(callback = "parent.message")) } Showin off Iteratees in Scala Saturday, May 11, 13

Slide 55

Slide 55 text

The state of the art in Python Saturday, May 11, 13

Slide 56

Slide 56 text

Generators available throu h yield Saturday, May 11, 13

Slide 57

Slide 57 text

Built-in map/filter in Python 3 return an iterable (this is ood!) Saturday, May 11, 13

Slide 58

Slide 58 text

streams.py import re from stream import filter, map, cut result = open('file') \ >> filter(re.compile(regex).search) \ >> map(re.compile(' |:|\.').split) \ >> cut[3] \ >> list Saturday, May 11, 13

Slide 59

Slide 59 text

If you are lookin to continue a interestin project, this may be it! Saturday, May 11, 13

Slide 60

Slide 60 text

Mutablility leads to non-determinism root of all evil Saturday, May 11, 13

Slide 61

Slide 61 text

Saturday, May 11, 13

Slide 62

Slide 62 text

First meetup in two weeks contact either of us for more info! http://www.meetup.com/Functional-Gronin en/ Saturday, May 11, 13