Concurrency and Distributed Systems in JRuby

Slide 1

Slide 1 text

CONCURRENCY AND DISTRIBUTED SYSTEMS IN JRUBY @iconara

Slide 2

Slide 2 text

speakerdeck.com/u/iconara (real time!)

Slide 3

Slide 3 text

Theo / @iconara

Slide 4

Slide 4 text

Chief Architect at

Slide 5

Slide 5 text

big data with Ruby since 2009

Slide 6

Slide 6 text

MAKING THIS

Slide 7

Slide 7 text

INTO THIS

Slide 8

Slide 8 text

big data with Ruby since 2009

Slide 9

Slide 9 text

APRIL 2011 stuck

Slide 10

Slide 10 text

MAY 2011 let’s try JRuby

Slide 11

Slide 11 text

JUNE 2011 wrapping Java libraries for fun and for proﬁt

Slide 12

Slide 12 text

HOTBUNNIES natural Ruby API for RabbitMQ

Slide 13

Slide 13 text

JULY 2011 threads are better than processes

Slide 14

Slide 14 text

JRUBY

Slide 15

Slide 15 text

JRUBY

Slide 16

Slide 16 text

J.U.C (java.util.concurrent)

Slide 17

Slide 17 text

SHORTCUT Java Concurrency in Practice

Slide 18

Slide 18 text

EXECUTORS conﬁgurable thread pools, a better abstraction than using Thread directly

Slide 19

Slide 19 text

thread_pool = Executors.new_fixed_thread_pool(16) all_the_things.each do |thing| thread_pool.submit do crunch_crunch_crunch(thing) end end

Slide 20

Slide 20 text

BLOCKING QUEUES easy producer/consumer patterns

Slide 21

Slide 21 text

J.U.C.ATOMIC optimistic locking primitives, compare-and-swap, thread safe non-blocking counters

Slide 22

Slide 22 text

MORE AWESOMES ConcurrentHashMap, CountDownLatch, ForkJoinPool, Google Guava, LMAX distruptor

Slide 23

Slide 23 text

TL; DR use j.u.c and avoid mutable state and you’re safe, mostly

Slide 24

Slide 24 text

enough rope THREADS

Slide 25

Slide 25 text

AUGUST 2011 threads are a mess, let’s try actors

Slide 26

Slide 26 text

ACTORS a higher level abstraction of concurrency

Slide 27

Slide 27 text

AKKA concurrency library for Scala, famous actor implementation

Slide 28

Slide 28 text

MIKKA Akka wrapper for JRuby

Slide 29

Slide 29 text

DO ALL THE THINGS Programming Concurrency on the JVM

Slide 30

Slide 30 text

AUGUST 2011 async systems with no back pressure mechanism will die of memory starvation

Slide 31

Slide 31 text

STATE AGGREGATION INPUT PERSISTENCE OUTPUT TIMER

Slide 32

Slide 32 text

STATE AGGREGATION INPUT PERSISTENCE OUTPUT TIMER !

Slide 33

Slide 33 text

AUGUST 2011 actors are cool, but not really worth it when all the arrows in your ﬂow chart point in the same direction

Slide 34

Slide 34 text

DIVIDE DIVIDE DIVIDE run multiple independent, synchronous workers, tune to the number of CPUs & IO wait

Slide 35

Slide 35 text

SEPTEMBER 2011 blocking all the way down, back pressure all the way up

Slide 36

Slide 36 text

AUGUST 2011 (short rant about MongoDB’s global write lock)

Slide 37

Slide 37 text

NEW SHINY TOYS for building new shiny products

Slide 38

Slide 38 text

CASSANDRA distributed database, mostly magic

Slide 39

Slide 39 text

CASSANDRA very verbose drivers ColumnFamilyUpdater updater = template.createUpdater("a key"); updater.setString("domain", "www.datastax.com"); updater.setLong("time", System.currentTimeMillis()); try { template.update(updater); } catch (HectorException e) { // hurgh }

Slide 40

Slide 40 text

EURYDICE JRuby nails it new_data = {:domain => 'www.datastax.com', :time => Time.now.to_i} column_family.update('a key', new_data)

Slide 41

Slide 41 text

STORM stream processing framework written in Clojure

Slide 42

Slide 42 text

STORM (it does the arrows, you do the blobs)

Slide 43

Slide 43 text

REDSTORM Storm for JRuby

Slide 44

Slide 44 text

DON’T DO THIS AT HOME KIDS! the key to building concurrent and distributed systems is to separate what you want to do from the concurrent- and distributed-ness, and let someone else take care of that YOUR CODE