Upgrade to Pro — share decks privately, control downloads, hide ads and more …

ZOMGscale! with Celluloid & JRuby - RubyShift 2013 Kiev

146e52d49d361f85c0945487452fc6a0?s=47 Ben Lovell
September 27, 2013

ZOMGscale! with Celluloid & JRuby - RubyShift 2013 Kiev

ZOMG - your CPUs have chilly chills! Light a fire under your cores and keep warm at night by unleashing the awesomeness contained within Celluloid & JRuby!

We'll talk about the state of concurrency and parallelism in ruby (no GIL whining, I promise) and how the landscape is changing thanks to Moore's Law no longer standing stead. I'll muse upon the problems with processes and threads for concurrency and parallelism and how the actor model largely solves those issues. You won't even have to sport a huge unix beard to play along. Unless you want to. In which case - BEARD ON! Finally I will show via the medium of code how Celluloid & JRuby are a mature and simple to reason about combination of sheer awesomeness!

I'll give a brief overview of Celluloid and some pointers on how best to apply it. This stuff isn't witchcraft, do not be afraid!

146e52d49d361f85c0945487452fc6a0?s=128

Ben Lovell

September 27, 2013
Tweet

Transcript

  1. zomgscale! with Celluloid and JRuby Ben Lovell

  2. benlovell _j

  3. 113581334398839860922

  4. None
  5. None
  6. digital agency primarily rubyists! iOS and Android

  7. None
  8. we don’t always fight ...we do it dressed as sumo

    on a trampoline but when we do...
  9. Moore’s Law

  10. None
  11. I don’t like bungee jumping... ...but I do like skiing

    Roger Moore
  12. None
  13. every few years CPU clocks have doubled

  14. None
  15. but recently this growth has stalled

  16. cores ++++++++

  17. the free lunch is over Herb Sutter

  18. harness the POWER in those cores

  19. ?

  20. concurrency parallelism

  21. so there’s a difference?

  22. concurrent!

  23. parallel?

  24. None
  25. processes or threads

  26. processes memory constraints communication x cores == x processes?

  27. processes what about fork(2) and CoW friendly GC ?

  28. not CoW friendly

  29. threads shared state locks and lock granularity race conditions can

    be hard to reason about
  30. zomg! I love multithreaded code - NOBODY EVER

  31. None
  32. so what’s up with MRI?

  33. well, nothing but...

  34. GIL

  35. some things the GIL is responsible for...

  36. San Francisco Bay oil spillage

  37. ...maritime disasters

  38. THANKS, GIL

  39. GIL

  40. MRI not so bad if you’re I/O bound

  41. MRI but what about computation?

  42. meh. thread-level parallelism is available right now! ...just not with

    MRI
  43. None
  44. rubinius 2.0.0 due for release 2042

  45. so now that we have truly parallel threads is the

    problem solved?
  46. rules of threading

  47. Don’t do it!

  48. If you must do it don’t share data across threads

  49. If you must share data across threads don’t share mutable

    data
  50. If you must share mutable data across threads synchronise access

    to this data
  51. don’t communicate by sharing memory... ...share memory by communicating go

  52. None
  53. painless multithreaded programming for ruby

  54. Tony Arcieri Tim Carey-Smith Ben Langfeld @bascule @halorgium @benlangfeld The

    Maintainers
  55. a concurrent object oriented programming framework which lets you build

    multithreaded programs out of concurrent objects just as easily as you build sequential programs out of regular objects
  56. None
  57. based upon the actor model

  58. actor model first proposed way back in 1970

  59. actor model actors are isolated within lightweight processes

  60. actor model actors possess identity

  61. actor model absolutely no shared state

  62. actor model actors don’t need to compete for locks

  63. actor model are sent messages asynchronously

  64. actor model messages are buffered by a mailbox

  65. actor model the actor works off each message sequentially

  66. actor model has implementations in many languages

  67. None
  68. None
  69. None
  70. celluloid actors automatically synchronize state

  71. 1 class Actor 2 attr_reader :counter 3 4 def initialize

    5 @counter = 0 6 @mutex = Mutex.new 7 end 8 9 def increment 10 @mutex.synchronize do 11 @counter += 1 12 end 13 end 14 end
  72. with celluloid the same example...

  73. 1 require "celluloid" 2 3 class Actor 4 include Celluloid

    5 attr_reader :counter 6 7 def initialize 8 @counter = 0 9 end 10 11 def increment 12 @counter += 1 13 end 14 end
  74. None
  75. celluloid actors are active objects living within threads

  76. 1 require "celluloid" 2 3 class Actor 4 include Celluloid

    5 end 6 7 actor = Actor.new 8 actor.inspect 9 #=> <Celluloid::ActorProxy(Actor:0x3feaecbb38e0)> 10 11 Thread.main 12 #=> <Thread:0x007f86290b8ce8 run> 13 14 actor.thread 15 #=> <Thread:0x007f862ad27a78 sleep>
  77. 1 module Celluloid 2 module ClassMethods 3 # Create a

    new actor 4 def new(*args, &block) 5 proxy = Actor.new(allocate, actor_options).proxy 6 proxy._send_(:initialize, *args, &block) 7 proxy 8 end 9 #... 10 end 11 #... 12 end
  78. celluloid actors messages you send are buffered via the actor’s

    mailbox...
  79. celluloid actors ... until the actor is ready to act

    upon them
  80. ______________ < ETOOMANYACTS > -------------- \ ^__^ \ (oo)\_______ (__)\

    )\/\ ||----w | || ||
  81. celluloid actors no pattern matching just regular messages

  82. celluloid actors poll their mailbox in a message loop

  83. 1 class Actor 2 # Wrap the given subject with

    an Actor 3 def initialize(subject, options = {}) 4 @subject = subject 5 @mailbox = options[:mailbox] || Mailbox.new 6 @running = true 7 8 @thread = ThreadHandle.new(:actor) do 9 setup_thread 10 run 11 end 12 #... 13 end 14 #... 15 end
  84. 1 class Actor 2 def run 3 #... 4 while

    @running 5 if message = @mailbox.receive(timeout_interval) 6 handle_message message 7 else 8 # No message indicates a timeout 9 @timers.fire 10 @receivers.fire_timers 11 end 12 end 13 #... 14 shutdown 15 end 16 end
  85. celluloid actors act upon messages sequentially

  86. what about ordering? no guarantees

  87. celluloid actors dispatch calls within fibers

  88. fibers? cooperative lightweight user space some gotchas...

  89. celluloid actors can dispatch synchronously

  90. 1 require "celluloid" 2 3 class Actor 4 include Celluloid

    5 6 def compute_all_the_things 7 sleep 2 8 puts "42" 9 end 10 end 11 12 actor = Actor.new 13 actor.compute_all_the_things 14 puts "done!" #=> 42 #=> done! blocking
  91. celluloid actors can dispatch asynchronously

  92. 1 require "celluloid" 2 3 class Actor 4 include Celluloid

    5 6 def compute_all_the_things 7 sleep 2 8 puts "42" 9 end 10 end 11 12 actor = Actor.new 13 actor.async.compute_all_the_things 14 puts "done!" 15 16 #=> done! 17 #=> 42 returns immediately
  93. celluloid actors can perform tasks in futures

  94. 1 require "celluloid" 2 3 class Actor 4 include Celluloid

    5 6 def compute_all_the_things 7 sleep 2 8 "42" 9 end 10 end 11 12 actor = Actor.new 13 future = actor.future.compute_all_the_things 14 puts "done!" 15 puts future.value 16 17 #=> done! 18 #=> 42 returns immediately blocks until a value is yielded
  95. celluloid actors are accessible by reference or name

  96. 1 require "celluloid" 2 3 class Actor 4 include Celluloid

    5 6 def compute_all_the_things 7 sleep 2 8 puts "42" 9 end 10 end 11 12 actor = Actor.new 13 Celluloid::Actor[:foo] = actor 14 15 actor.inspect 16 #=> <Celluloid::ActorProxy(Actor:0x3feb3ec11308)> 17 Celluloid::Actor[:foo].inspect 18 #=> <Celluloid::ActorProxy(Actor:0x3feb3ec11308)>
  97. celluloid actors are fault tolerant ... let it crash!

  98. 1 require "celluloid/autostart" 2 3 class Actor 4 include Celluloid

    5 6 def compute_all_the_things 7 puts "42" 8 end 9 10 def zomg_crash 11 raise "derp!" 12 end 13 end 14 15 supervisor = Actor.supervise_as :foo 16 17 begin 18 Celluloid::Actor[:foo].zomg_crash 19 rescue 20 puts "whoops" 21 end 22 23 Celluloid::Actor[:foo].compute_all_the_things 24 25 #=> whoops 26 #=> 42 crash the actor fresh actor take care of me!
  99. celluloid actors can be arranged as pooled workers

  100. 1 require "celluloid" 2 3 class Actor 4 include Celluloid

    5 6 def compute_all_the_things 7 sleep 1 8 puts "42" 9 end 10 end 11 12 pool = Actor.pool 13 14 4.times { pool.compute_all_the_things } 15 16 #=> 42 17 #=> 42 and so on... size*cores load up the workers
  101. there’s more timers links supervision groups pub/sub conditions

  102. that low hanging fruit? yeah, about that...

  103. but there is one tip! blocking I/O... don’t

  104. None
  105. an event-driven IO system for building fast, scalable network applications

    that integrate directly with celluloid actors
  106. unlike certain other evented I/O systems which limit you to

    a single event loop per process Celluloid::IO lets you make as many actors as you want system resources permitting
  107. None
  108. a distributed extension to celluloid which provides distributed and concurrent

    objects for ruby that are both robust and fault-tolerant
  109. None
  110. a fast non-blocking and evented web server. Thanks to celluloid,

    Reel works great for multithreaded applications and provides traditional multithreaded blocking IO support too.
  111. EXPERIMENTAL or broken as it is known outside of the

    OSS world
  112. to summarise...

  113. the future of ruby concurrency and parallelism?

  114. thanks! @benlovell ?