Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Legion intro
Search
Nathan Hopkins
August 25, 2013
0
44
Legion intro
Ruby concurrency made easy
Nathan Hopkins
August 25, 2013
Tweet
Share
More Decks by Nathan Hopkins
See All by Nathan Hopkins
Ellington Intro
hopsoft
1
170
Featured
See All Featured
Designing Experiences People Love
moore
142
24k
Raft: Consensus for Rubyists
vanstee
140
7.1k
Git: the NoSQL Database
bkeepers
PRO
431
66k
Fantastic passwords and where to find them - at NoRuKo
philnash
52
3.4k
Context Engineering - Making Every Token Count
addyosmani
7
260
A Tale of Four Properties
chriscoyier
161
23k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
359
30k
No one is an island. Learnings from fostering a developers community.
thoeni
21
3.5k
The Invisible Side of Design
smashingmag
302
51k
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
508
140k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
194
16k
Why You Should Never Use an ORM
jnunemaker
PRO
59
9.6k
Transcript
My name is Legion for we are many Concurrency Made
Easy
HOW IT STARTED
• Rails project • Data import • Lots of data
• Data manipulation THE PROJECT
• Chicago crime data - 1 GB CSV • Save
CSV records to database • Convert longitude/latitude to static image of map • Various sizes thumbnail, medium, large • Rails + Carrierwave REQUIREMENTS
RAKE TASK?
class Importer def import_csv_row(row) record = ChicagoCrime.new( row.headers.reduce({}) do |memo,
name| column = name.downcase.gsub(/\s/, "_").to_s column = :external_id if column == :id memo[column] = row[name] memo end ) #sleep 1 record.static_map = File.open("/Users/nathan/wo record.save! end end
desc "Import data" task :import => :environment do require "csv"
require_relative "../importer" importer = Importer.new count = 0 path = File.expand_path("../../../data/Crimes_-_2 CSV.foreach(path, :headers => true) do |row| importer.import_csv_row row print "#{count += 1}," end end
SINGLE PROCESS
SLOW What to do?
CONCURRENCY • Threads with JRuby or Rubinius? • Celluloid with
JRuby or Rubinius? • with MRI? • Background Jobs? • fork?
MRI has a concurrency Story It’t called DRb - Distributed
Ruby
LEGION Makes it easy
class LegionImporter < Legion::Object before :import_csv_row do ActiveRecord::Base.establish_connection ActiveRe end
def import_csv_row(row) record = ChicagoCrime.new( row.headers.reduce({}) do |memo, name| column = name.downcase.gsub(/\s/, "_").to_sy column = :external_id if column == :id memo[column] = row[name] memo end ) # sleep 1 record.static_map = File.open("/Users/nathan/wor record.save! end end
class LegionImporter < Legion::Object before :import_csv_row do ActiveRecord::Base.establish_connection ActiveRe end
def import_csv_row(row) record = ChicagoCrime.new( row.headers.reduce({}) do |memo, name| column = name.downcase.gsub(/\s/, "_").to_sy column = :external_id if column == :id memo[column] = row[name] memo end ) # sleep 1 record.static_map = File.open("/Users/nathan/wor record.save! end end
desc "Import data with legion" task :import_with_legion => :environment do
require "csv" require_relative "../legion_importer" supervisor = Legion::Supervisor.new( LegionImporter, :processes => 8, :port => 42042 ) supervisor.start count = 0 path = File.expand_path("../../../data/Crimes_-_20 CSV.foreach(path, :headers => true) do |row| supervisor.import_csv_row row print "#{count += 1}," end supervisor.stop end
desc "Import data with legion" task :import_with_legion => :environment do
require "csv" require_relative "../legion_importer" supervisor = Legion::Supervisor.new( LegionImporter, :processes => 8, :port => 42042 ) supervisor.start count = 0 path = File.expand_path("../../../data/Crimes_-_20 CSV.foreach(path, :headers => true) do |row| supervisor.import_csv_row row print "#{count += 1}," end supervisor.stop end
MULTIPLE PROCESSES
ACTUALLY USE YOUR RESOURCES • More efficient • Faster •
Smarter • Better
HOW IT WORKS fork + DRb + background thread
Legion Objects can create remote instances of themselves (fork +
DRb)
Supervisors manage groups of remote objects
Remote objects provide async methods via a background thread
Supervisor round-robins work to remote objects in sequence
SMALL 120 lines of code
GOTCHAS • Legion::Object subclasses must define async methods directly •
Remember you’re dealing with a forked process • Rails must reconnect to the database with a :before callback • Relative file paths don’t work
@hopsoft Questions?