Concurrency and Distributed Systems in JRuby

Transcript

1. CONCURRENCY AND DISTRIBUTED SYSTEMS IN JRUBY @iconara

2. speakerdeck.com/u/iconara (real time!)

3. Theo / @iconara

4. Chief Architect at

5. big data with Ruby since 2009

6. MAKING THIS

7. INTO THIS

8. big data with Ruby since 2009

9. APRIL 2011 stuck

10. MAY 2011 let’s try JRuby

11. JUNE 2011 wrapping Java libraries for fun and for profit

12. HOTBUNNIES natural Ruby API for RabbitMQ

13. JULY 2011 threads are better than processes

14. JRUBY

15. JRUBY

16. J.U.C (java.util.concurrent)

17. SHORTCUT Java Concurrency in Practice

18. EXECUTORS configurable thread pools, a better abstraction than using Thread

    directly

19. thread_pool = Executors.new_fixed_thread_pool(16) all_the_things.each do |thing| thread_pool.submit do crunch_crunch_crunch(thing) end

    end

20. BLOCKING QUEUES easy producer/consumer patterns

21. J.U.C.ATOMIC optimistic locking primitives, compare-and-swap, thread safe non-blocking counters

22. MORE AWESOMES ConcurrentHashMap, CountDownLatch, ForkJoinPool, Google Guava, LMAX distruptor

23. TL; DR use j.u.c and avoid mutable state and you’re

    safe, mostly

24. enough rope THREADS

25. AUGUST 2011 threads are a mess, let’s try actors

26. ACTORS a higher level abstraction of concurrency

27. AKKA concurrency library for Scala, famous actor implementation

28. MIKKA Akka wrapper for JRuby

29. DO ALL THE THINGS Programming Concurrency on the JVM

30. AUGUST 2011 async systems with no back pressure mechanism will

    die of memory starvation

31. STATE AGGREGATION INPUT PERSISTENCE OUTPUT TIMER

32. STATE AGGREGATION INPUT PERSISTENCE OUTPUT TIMER !

33. AUGUST 2011 actors are cool, but not really worth it

    when all the arrows in your flow chart point in the same direction

34. DIVIDE DIVIDE DIVIDE run multiple independent, synchronous workers, tune to

    the number of CPUs & IO wait

35. SEPTEMBER 2011 blocking all the way down, back pressure all

    the way up

36. AUGUST 2011 (short rant about MongoDB’s global write lock)

37. NEW SHINY TOYS for building new shiny products

38. CASSANDRA distributed database, mostly magic

39. CASSANDRA very verbose drivers ColumnFamilyUpdater<String, String> updater = template.createUpdater("a key");

    updater.setString("domain", "www.datastax.com"); updater.setLong("time", System.currentTimeMillis()); try { template.update(updater); } catch (HectorException e) { // hurgh }

40. EURYDICE JRuby nails it new_data = {:domain => 'www.datastax.com', :time

    => Time.now.to_i} column_family.update('a key', new_data)

41. STORM stream processing framework written in Clojure

42. STORM (it does the arrows, you do the blobs)

43. REDSTORM Storm for JRuby

44. DON’T DO THIS AT HOME KIDS! the key to building

    concurrent and distributed systems is to separate what you want to do from the concurrent- and distributed-ness, and let someone else take care of that YOUR CODE

45. JRUBY IS A SLIPPERY SLOPE TO JBOSS let there be

    no doubt about it

46. SCALING IS HARD go to Srdan’s talk tomorrow

47. KTHXBAI @iconara github.com/iconara architecturalatrocities.com burtcorp.com