Vaclav Pech on GPars

Transcript

1. GPars Groovy Parallel Systems Václav Pech
2. About me  Passionate programmer  Concurrency enthusiast  GPars @ Codehaus lead  Technology evangelist @ JetBrains  JetBrains Academy member http://www.jroller.com/vaclav http://twitter.com/vaclav_pech
3. We’re quad core already Beware: More cores to come shortly!
4. Stone age of parallel SW  Dead-locks  Live-locks  Race conditions  Starvation  Shared Mutable State
5. Locks and threads Multithreaded programs today work mostly by accident!
6. Fork/Join Actors Collections Dataflow Safe Toolset
7. Call closures asynchronously def isSelfPortrait = {image -> image.contains me} SYNC: def flag = isSelfPortrait.call(img1) ASYNC: def future = isSelfPortrait.callAsync(img1) … def flag = future.get()
8. Asynchronous closures def resize = {img -> img.resize(64, 64)} def fastResize = resize.async() def resized = images.collect fastResize … createAlbum resized*.get()
9. Thread Pool Tasks Worker threads Queue
10. Thread Pool Contention!
11. Fork/Join Thread Pool
12. Fork/Join Thread Pool Work stealing
13. Fork/Join  Solve hierarchical problems  Divide and conquer  Merge sort, Quick sort  Tree traversal  File scan / search  … [a, b, c, d, e, f, g, h] [a, b, c, d] [e, f, g, h] [a, b] [c, d] [e, f] [g, h]
14. Fork/Join (GPars) runForkJoin(new File(“./src”)) {currentDir -> long count = 0; currentDir.eachFile { if (it.isDirectory()) { println "Forking a thread for $it" forkOffChild it } else { count++ } } return count + childrenResults.sum(0) } Waits for children without blocking the thread!
15. Collections (GPars) images.eachParallel {it.process()} documents.sumParallel() candidates.maxParallel {it.salary}.marry()
16. Transparently parallel withPool(4) { images.makeTransparent() .findAll {it.contains me} .collect {convert it} .groupBy {it.size()} }
17. Parallelize existing code buyBargain stocks.makeTransparent() def buyBargain(stocks) { buy stocks.findAll { it.download().price < 50 } } Use with CAUTION!
18. Functional flavor Map / Reduce map, reduce, filter, min, max, sum, … (1..n).parallel.filter {it%2==0} .map {it ** 2} .reduce {a, b -> a + b}
19. GroovyDSL – IntelliJ IDEA CE
20. Dataflow Concurrency  No race-conditions  No live-locks  Deterministic deadlocks Completely deterministic programs BEAUTIFUL code (Jonas Bonér)
21. Dataflow Variables / Promises main task2 task3 x y z task1
22. Dataflow Variables task { z << x.val + y.val } task { x << 40 } task { y << 2 } assert 42 == z.val  Single-assignment variables with blocking read
23. DataFlows def df = new DataFlows() task { df.z = df.x + df.y } task { df.x = 10 } task { df.y = 5 } assert 15 == df.z
24. Milestone  Asynchronous background tasks  Fork/Join  Parallel collection processing  Dataflow variables/streams
25. Message Passing Communicating Sequential Processes (CSP) Dataflow Operators Actors
26. Actors  Isolated  Communicating Immutable messages  Active Pooled shared threads  Activities Create a new actor Send a message Receive a message Actor Actor Actor Actor Actor Actor Actor TTT Thread pool
27. Actors use Gate Keeper Form HTTP Finger Prints Address Check Email Check Process Fraud Detect Response SOAP SMTP
28. Actors patterns Enricher Router Translator Endpoint Splitter Agregator Filter Resequencer Checker
29. Sending messages buddy.send 10.eur buddy << new Book(title:’Groovy Recipes’, author:’Scott Davis’) def canChat = buddy.sendAndWait ‘Got time?’ buddy.sendAndContinue ‘Need money!’, {cash-> pocket.add cash }
30. Stateless Actors (pure Java) class MyActor extends DynamicDispatchActor { Account account = ... public void onMessage(String msg) { String encripted = encrypt(msg); reply(encripted); } public void onMessage(Integer number) { reply(2 * number); } public void onMessage(Money cash) { System.out.println("Received a donation " + cash); account.deposit(cash); } }
31. Dynamic Dispatch Actor def actor = messageHandler { when {BigDecimal num -> println 'Received BigDecimal'} when {String code -> compileAndRun code } when {Book book -> read book } }
32. Creating Actors class MyActor extends AbstractPooledActor { void act() { def buddy = new YourActor() buddy << ‘Hi man, how\’re things?’ def response = receive() } }
33. Creating Actors def decryptor = actor { loop { react {msg -> reply msg.reverse() } } } decryptor << ‘noitcA nI yvoorG’
34. Implicit State in Actors val me = actor { loop { react {msg1 -> switch (msg1) { case Work: reply “I don't work so early” ; break; case Breakfast: b.eat react {msg2 → switch (msg2) { case Work: reply “OK, time to work”; w.do case Lunch: ... } } } }
35. Choosing the Reaction react / receive {gift -> switch (gift) { case Money:reply ‘Thanks’;pocket gift; break case [iPhone, iPod]:child << gift; break case (BigFlat..SmallHouse):moveIn(gift); break case Clothes: putOn(gift) fits(gift)?reply ‘Thanks’:reply gift break case EXIT:stop() } }
36. Continuation Style loop { … react { … react {/*schedule the block; throw CONTINUE*/ … } //Never reached } //Never reached } //Never reached
37. Dataflow Operators operator(inputs: [stocksStream], outputs: [pricedStocks]) {stock -> def price = getClosing(stock, 2008) bindOutput(0, [stock: stock, price: price]) } * + <>
38. Url resolver Url resolver Downloader Groovy scanner Scala scanner Url resolver Url resolver Reporter
39. Agent  Lock Shared Mutable State in a Safe Alter the State Alter the State
40. Agent inside Double Increment Add 25 Message Queue 36 thread
41. Agent List def jugMembers = new Agent(['Me']) //add Me task { jugMembers.send {it.add 'Joe'} //add Joe } task { jugMembers << {it.add 'Dave'} //add Dave jugMembers << {it.add 'Alice'} //add Alice } ... println jugMembers.val
42. Integration  Bundled with Groovy dists  Maven  Gradle  Grape (@Grab)  Griffon plugin  Grails plugin
43. Roadmap STM Actor and DataFlow remoting CSP improvements
44. No more threads and locks images.parallel.map { //concurrency agnostic code here } def myActor = actor { //concurrency agnostic code here } Identically with agents, fork/join, dataflow, …
45. Summary Parallelism is not hard, multi-threading is Jon Kerridge, Napier University
46. Questions? Find more at: http://gpars.codehaus.org http://www.jroller.com/vaclav http://twitter.com/vaclav_pech