Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Legion intro
Search
Nathan Hopkins
August 25, 2013
0
38
Legion intro
Ruby concurrency made easy
Nathan Hopkins
August 25, 2013
Tweet
Share
More Decks by Nathan Hopkins
See All by Nathan Hopkins
Ellington Intro
hopsoft
1
140
Featured
See All Featured
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
79
43k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
275
13k
Reflections from 52 weeks, 52 projects
jeffersonlam
345
19k
Stop Working from a Prison Cell
hatefulcrawdad
267
19k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
323
20k
A Philosophy of Restraint
colly
197
16k
BBQ
matthewcrist
80
8.8k
Keith and Marios Guide to Fast Websites
keithpitt
408
22k
Optimising Largest Contentful Paint
csswizardry
12
2.4k
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
mongodb
34
8.9k
KATA
mclloyd
16
12k
The Language of Interfaces
destraynor
151
23k
Transcript
My name is Legion for we are many Concurrency Made
Easy
HOW IT STARTED
• Rails project • Data import • Lots of data
• Data manipulation THE PROJECT
• Chicago crime data - 1 GB CSV • Save
CSV records to database • Convert longitude/latitude to static image of map • Various sizes thumbnail, medium, large • Rails + Carrierwave REQUIREMENTS
RAKE TASK?
class Importer def import_csv_row(row) record = ChicagoCrime.new( row.headers.reduce({}) do |memo,
name| column = name.downcase.gsub(/\s/, "_").to_s column = :external_id if column == :id memo[column] = row[name] memo end ) #sleep 1 record.static_map = File.open("/Users/nathan/wo record.save! end end
desc "Import data" task :import => :environment do require "csv"
require_relative "../importer" importer = Importer.new count = 0 path = File.expand_path("../../../data/Crimes_-_2 CSV.foreach(path, :headers => true) do |row| importer.import_csv_row row print "#{count += 1}," end end
SINGLE PROCESS
SLOW What to do?
CONCURRENCY • Threads with JRuby or Rubinius? • Celluloid with
JRuby or Rubinius? • with MRI? • Background Jobs? • fork?
MRI has a concurrency Story It’t called DRb - Distributed
Ruby
LEGION Makes it easy
class LegionImporter < Legion::Object before :import_csv_row do ActiveRecord::Base.establish_connection ActiveRe end
def import_csv_row(row) record = ChicagoCrime.new( row.headers.reduce({}) do |memo, name| column = name.downcase.gsub(/\s/, "_").to_sy column = :external_id if column == :id memo[column] = row[name] memo end ) # sleep 1 record.static_map = File.open("/Users/nathan/wor record.save! end end
class LegionImporter < Legion::Object before :import_csv_row do ActiveRecord::Base.establish_connection ActiveRe end
def import_csv_row(row) record = ChicagoCrime.new( row.headers.reduce({}) do |memo, name| column = name.downcase.gsub(/\s/, "_").to_sy column = :external_id if column == :id memo[column] = row[name] memo end ) # sleep 1 record.static_map = File.open("/Users/nathan/wor record.save! end end
desc "Import data with legion" task :import_with_legion => :environment do
require "csv" require_relative "../legion_importer" supervisor = Legion::Supervisor.new( LegionImporter, :processes => 8, :port => 42042 ) supervisor.start count = 0 path = File.expand_path("../../../data/Crimes_-_20 CSV.foreach(path, :headers => true) do |row| supervisor.import_csv_row row print "#{count += 1}," end supervisor.stop end
desc "Import data with legion" task :import_with_legion => :environment do
require "csv" require_relative "../legion_importer" supervisor = Legion::Supervisor.new( LegionImporter, :processes => 8, :port => 42042 ) supervisor.start count = 0 path = File.expand_path("../../../data/Crimes_-_20 CSV.foreach(path, :headers => true) do |row| supervisor.import_csv_row row print "#{count += 1}," end supervisor.stop end
MULTIPLE PROCESSES
ACTUALLY USE YOUR RESOURCES • More efficient • Faster •
Smarter • Better
HOW IT WORKS fork + DRb + background thread
Legion Objects can create remote instances of themselves (fork +
DRb)
Supervisors manage groups of remote objects
Remote objects provide async methods via a background thread
Supervisor round-robins work to remote objects in sequence
SMALL 120 lines of code
GOTCHAS • Legion::Object subclasses must define async methods directly •
Remember you’re dealing with a forked process • Rails must reconnect to the database with a :before callback • Relative file paths don’t work
@hopsoft Questions?