Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Legion intro
Search
Nathan Hopkins
August 25, 2013
0
43
Legion intro
Ruby concurrency made easy
Nathan Hopkins
August 25, 2013
Tweet
Share
More Decks by Nathan Hopkins
See All by Nathan Hopkins
Ellington Intro
hopsoft
1
170
Featured
See All Featured
Rebuilding a faster, lazier Slack
samanthasiow
82
9.1k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
229
22k
The Power of CSS Pseudo Elements
geoffreycrofte
77
5.8k
[RailsConf 2023] Rails as a piece of cake
palkan
55
5.6k
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
irinanazarova
8
790
Mobile First: as difficult as doing things right
swwweet
223
9.7k
Fantastic passwords and where to find them - at NoRuKo
philnash
51
3.3k
A better future with KSS
kneath
239
17k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
48
5.4k
Designing for Performance
lara
609
69k
Statistics for Hackers
jakevdp
799
220k
Keith and Marios Guide to Fast Websites
keithpitt
411
22k
Transcript
My name is Legion for we are many Concurrency Made
Easy
HOW IT STARTED
• Rails project • Data import • Lots of data
• Data manipulation THE PROJECT
• Chicago crime data - 1 GB CSV • Save
CSV records to database • Convert longitude/latitude to static image of map • Various sizes thumbnail, medium, large • Rails + Carrierwave REQUIREMENTS
RAKE TASK?
class Importer def import_csv_row(row) record = ChicagoCrime.new( row.headers.reduce({}) do |memo,
name| column = name.downcase.gsub(/\s/, "_").to_s column = :external_id if column == :id memo[column] = row[name] memo end ) #sleep 1 record.static_map = File.open("/Users/nathan/wo record.save! end end
desc "Import data" task :import => :environment do require "csv"
require_relative "../importer" importer = Importer.new count = 0 path = File.expand_path("../../../data/Crimes_-_2 CSV.foreach(path, :headers => true) do |row| importer.import_csv_row row print "#{count += 1}," end end
SINGLE PROCESS
SLOW What to do?
CONCURRENCY • Threads with JRuby or Rubinius? • Celluloid with
JRuby or Rubinius? • with MRI? • Background Jobs? • fork?
MRI has a concurrency Story It’t called DRb - Distributed
Ruby
LEGION Makes it easy
class LegionImporter < Legion::Object before :import_csv_row do ActiveRecord::Base.establish_connection ActiveRe end
def import_csv_row(row) record = ChicagoCrime.new( row.headers.reduce({}) do |memo, name| column = name.downcase.gsub(/\s/, "_").to_sy column = :external_id if column == :id memo[column] = row[name] memo end ) # sleep 1 record.static_map = File.open("/Users/nathan/wor record.save! end end
class LegionImporter < Legion::Object before :import_csv_row do ActiveRecord::Base.establish_connection ActiveRe end
def import_csv_row(row) record = ChicagoCrime.new( row.headers.reduce({}) do |memo, name| column = name.downcase.gsub(/\s/, "_").to_sy column = :external_id if column == :id memo[column] = row[name] memo end ) # sleep 1 record.static_map = File.open("/Users/nathan/wor record.save! end end
desc "Import data with legion" task :import_with_legion => :environment do
require "csv" require_relative "../legion_importer" supervisor = Legion::Supervisor.new( LegionImporter, :processes => 8, :port => 42042 ) supervisor.start count = 0 path = File.expand_path("../../../data/Crimes_-_20 CSV.foreach(path, :headers => true) do |row| supervisor.import_csv_row row print "#{count += 1}," end supervisor.stop end
desc "Import data with legion" task :import_with_legion => :environment do
require "csv" require_relative "../legion_importer" supervisor = Legion::Supervisor.new( LegionImporter, :processes => 8, :port => 42042 ) supervisor.start count = 0 path = File.expand_path("../../../data/Crimes_-_20 CSV.foreach(path, :headers => true) do |row| supervisor.import_csv_row row print "#{count += 1}," end supervisor.stop end
MULTIPLE PROCESSES
ACTUALLY USE YOUR RESOURCES • More efficient • Faster •
Smarter • Better
HOW IT WORKS fork + DRb + background thread
Legion Objects can create remote instances of themselves (fork +
DRb)
Supervisors manage groups of remote objects
Remote objects provide async methods via a background thread
Supervisor round-robins work to remote objects in sequence
SMALL 120 lines of code
GOTCHAS • Legion::Object subclasses must define async methods directly •
Remember you’re dealing with a forked process • Rails must reconnect to the database with a :before callback • Relative file paths don’t work
@hopsoft Questions?