Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Concurrency and Distributed Systems in JRuby

Concurrency and Distributed Systems in JRuby

Presentation given av JRubyConf EU/Eurucamp 2012

8c21306523b16ba5dd35c3549bf90994?s=128

Theo Hultberg

August 17, 2012
Tweet

Transcript

  1. CONCURRENCY AND DISTRIBUTED SYSTEMS IN JRUBY @iconara

  2. speakerdeck.com/u/iconara (real time!)

  3. Theo / @iconara

  4. Chief Architect at

  5. big data with Ruby since 2009

  6. MAKING THIS

  7. INTO THIS

  8. big data with Ruby since 2009

  9. APRIL 2011 stuck

  10. MAY 2011 let’s try JRuby

  11. JUNE 2011 wrapping Java libraries for fun and for profit

  12. HOTBUNNIES natural Ruby API for RabbitMQ

  13. JULY 2011 threads are better than processes

  14. JRUBY

  15. JRUBY

  16. J.U.C (java.util.concurrent)

  17. SHORTCUT Java Concurrency in Practice

  18. EXECUTORS configurable thread pools, a better abstraction than using Thread

    directly
  19. thread_pool = Executors.new_fixed_thread_pool(16) all_the_things.each do |thing| thread_pool.submit do crunch_crunch_crunch(thing) end

    end
  20. BLOCKING QUEUES easy producer/consumer patterns

  21. J.U.C.ATOMIC optimistic locking primitives, compare-and-swap, thread safe non-blocking counters

  22. MORE AWESOMES ConcurrentHashMap, CountDownLatch, ForkJoinPool, Google Guava, LMAX distruptor

  23. TL; DR use j.u.c and avoid mutable state and you’re

    safe, mostly
  24. enough rope THREADS

  25. AUGUST 2011 threads are a mess, let’s try actors

  26. ACTORS a higher level abstraction of concurrency

  27. AKKA concurrency library for Scala, famous actor implementation

  28. MIKKA Akka wrapper for JRuby

  29. DO ALL THE THINGS Programming Concurrency on the JVM

  30. AUGUST 2011 async systems with no back pressure mechanism will

    die of memory starvation
  31. STATE AGGREGATION INPUT PERSISTENCE OUTPUT TIMER

  32. STATE AGGREGATION INPUT PERSISTENCE OUTPUT TIMER !

  33. AUGUST 2011 actors are cool, but not really worth it

    when all the arrows in your flow chart point in the same direction
  34. DIVIDE DIVIDE DIVIDE run multiple independent, synchronous workers, tune to

    the number of CPUs & IO wait
  35. SEPTEMBER 2011 blocking all the way down, back pressure all

    the way up
  36. AUGUST 2011 (short rant about MongoDB’s global write lock)

  37. NEW SHINY TOYS for building new shiny products

  38. CASSANDRA distributed database, mostly magic

  39. CASSANDRA very verbose drivers ColumnFamilyUpdater<String, String> updater = template.createUpdater("a key");

    updater.setString("domain", "www.datastax.com"); updater.setLong("time", System.currentTimeMillis()); try { template.update(updater); } catch (HectorException e) { // hurgh }
  40. EURYDICE JRuby nails it new_data = {:domain => 'www.datastax.com', :time

    => Time.now.to_i} column_family.update('a key', new_data)
  41. STORM stream processing framework written in Clojure

  42. STORM (it does the arrows, you do the blobs)

  43. REDSTORM Storm for JRuby

  44. DON’T DO THIS AT HOME KIDS! the key to building

    concurrent and distributed systems is to separate what you want to do from the concurrent- and distributed-ness, and let someone else take care of that YOUR CODE
  45. JRUBY IS A SLIPPERY SLOPE TO JBOSS let there be

    no doubt about it
  46. SCALING IS HARD go to Srdan’s talk tomorrow

  47. KTHXBAI @iconara github.com/iconara architecturalatrocities.com burtcorp.com