Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Legion intro
Search
Nathan Hopkins
August 25, 2013
0
43
Legion intro
Ruby concurrency made easy
Nathan Hopkins
August 25, 2013
Tweet
Share
More Decks by Nathan Hopkins
See All by Nathan Hopkins
Ellington Intro
hopsoft
1
170
Featured
See All Featured
KATA
mclloyd
30
14k
Six Lessons from altMBA
skipperchong
28
3.9k
Fantastic passwords and where to find them - at NoRuKo
philnash
51
3.3k
The Pragmatic Product Professional
lauravandoore
35
6.7k
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
29
2.7k
BBQ
matthewcrist
89
9.7k
Art, The Web, and Tiny UX
lynnandtonic
299
21k
Unsuck your backbone
ammeep
671
58k
Into the Great Unknown - MozCon
thekraken
39
1.9k
Large-scale JavaScript Application Architecture
addyosmani
512
110k
GraphQLとの向き合い方2022年版
quramy
49
14k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
107
19k
Transcript
My name is Legion for we are many Concurrency Made
Easy
HOW IT STARTED
• Rails project • Data import • Lots of data
• Data manipulation THE PROJECT
• Chicago crime data - 1 GB CSV • Save
CSV records to database • Convert longitude/latitude to static image of map • Various sizes thumbnail, medium, large • Rails + Carrierwave REQUIREMENTS
RAKE TASK?
class Importer def import_csv_row(row) record = ChicagoCrime.new( row.headers.reduce({}) do |memo,
name| column = name.downcase.gsub(/\s/, "_").to_s column = :external_id if column == :id memo[column] = row[name] memo end ) #sleep 1 record.static_map = File.open("/Users/nathan/wo record.save! end end
desc "Import data" task :import => :environment do require "csv"
require_relative "../importer" importer = Importer.new count = 0 path = File.expand_path("../../../data/Crimes_-_2 CSV.foreach(path, :headers => true) do |row| importer.import_csv_row row print "#{count += 1}," end end
SINGLE PROCESS
SLOW What to do?
CONCURRENCY • Threads with JRuby or Rubinius? • Celluloid with
JRuby or Rubinius? • with MRI? • Background Jobs? • fork?
MRI has a concurrency Story It’t called DRb - Distributed
Ruby
LEGION Makes it easy
class LegionImporter < Legion::Object before :import_csv_row do ActiveRecord::Base.establish_connection ActiveRe end
def import_csv_row(row) record = ChicagoCrime.new( row.headers.reduce({}) do |memo, name| column = name.downcase.gsub(/\s/, "_").to_sy column = :external_id if column == :id memo[column] = row[name] memo end ) # sleep 1 record.static_map = File.open("/Users/nathan/wor record.save! end end
class LegionImporter < Legion::Object before :import_csv_row do ActiveRecord::Base.establish_connection ActiveRe end
def import_csv_row(row) record = ChicagoCrime.new( row.headers.reduce({}) do |memo, name| column = name.downcase.gsub(/\s/, "_").to_sy column = :external_id if column == :id memo[column] = row[name] memo end ) # sleep 1 record.static_map = File.open("/Users/nathan/wor record.save! end end
desc "Import data with legion" task :import_with_legion => :environment do
require "csv" require_relative "../legion_importer" supervisor = Legion::Supervisor.new( LegionImporter, :processes => 8, :port => 42042 ) supervisor.start count = 0 path = File.expand_path("../../../data/Crimes_-_20 CSV.foreach(path, :headers => true) do |row| supervisor.import_csv_row row print "#{count += 1}," end supervisor.stop end
desc "Import data with legion" task :import_with_legion => :environment do
require "csv" require_relative "../legion_importer" supervisor = Legion::Supervisor.new( LegionImporter, :processes => 8, :port => 42042 ) supervisor.start count = 0 path = File.expand_path("../../../data/Crimes_-_20 CSV.foreach(path, :headers => true) do |row| supervisor.import_csv_row row print "#{count += 1}," end supervisor.stop end
MULTIPLE PROCESSES
ACTUALLY USE YOUR RESOURCES • More efficient • Faster •
Smarter • Better
HOW IT WORKS fork + DRb + background thread
Legion Objects can create remote instances of themselves (fork +
DRb)
Supervisors manage groups of remote objects
Remote objects provide async methods via a background thread
Supervisor round-robins work to remote objects in sequence
SMALL 120 lines of code
GOTCHAS • Legion::Object subclasses must define async methods directly •
Remember you’re dealing with a forked process • Rails must reconnect to the database with a :before callback • Relative file paths don’t work
@hopsoft Questions?