Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Legion intro
Search
Nathan Hopkins
August 25, 2013
0
43
Legion intro
Ruby concurrency made easy
Nathan Hopkins
August 25, 2013
Tweet
Share
More Decks by Nathan Hopkins
See All by Nathan Hopkins
Ellington Intro
hopsoft
1
170
Featured
See All Featured
Documentation Writing (for coders)
carmenintech
74
5k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
131
19k
We Have a Design System, Now What?
morganepeng
53
7.8k
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
eileencodes
26
3k
What's in a price? How to price your products and services
michaelherold
246
12k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
252
21k
Building Applications with DynamoDB
mza
96
6.6k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
229
22k
[RailsConf 2023] Rails as a piece of cake
palkan
57
5.8k
[RailsConf 2023 Opening Keynote] The Magic of Rails
eileencodes
30
9.6k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
126
53k
Understanding Cognitive Biases in Performance Measurement
bluesmoon
29
1.9k
Transcript
My name is Legion for we are many Concurrency Made
Easy
HOW IT STARTED
• Rails project • Data import • Lots of data
• Data manipulation THE PROJECT
• Chicago crime data - 1 GB CSV • Save
CSV records to database • Convert longitude/latitude to static image of map • Various sizes thumbnail, medium, large • Rails + Carrierwave REQUIREMENTS
RAKE TASK?
class Importer def import_csv_row(row) record = ChicagoCrime.new( row.headers.reduce({}) do |memo,
name| column = name.downcase.gsub(/\s/, "_").to_s column = :external_id if column == :id memo[column] = row[name] memo end ) #sleep 1 record.static_map = File.open("/Users/nathan/wo record.save! end end
desc "Import data" task :import => :environment do require "csv"
require_relative "../importer" importer = Importer.new count = 0 path = File.expand_path("../../../data/Crimes_-_2 CSV.foreach(path, :headers => true) do |row| importer.import_csv_row row print "#{count += 1}," end end
SINGLE PROCESS
SLOW What to do?
CONCURRENCY • Threads with JRuby or Rubinius? • Celluloid with
JRuby or Rubinius? • with MRI? • Background Jobs? • fork?
MRI has a concurrency Story It’t called DRb - Distributed
Ruby
LEGION Makes it easy
class LegionImporter < Legion::Object before :import_csv_row do ActiveRecord::Base.establish_connection ActiveRe end
def import_csv_row(row) record = ChicagoCrime.new( row.headers.reduce({}) do |memo, name| column = name.downcase.gsub(/\s/, "_").to_sy column = :external_id if column == :id memo[column] = row[name] memo end ) # sleep 1 record.static_map = File.open("/Users/nathan/wor record.save! end end
class LegionImporter < Legion::Object before :import_csv_row do ActiveRecord::Base.establish_connection ActiveRe end
def import_csv_row(row) record = ChicagoCrime.new( row.headers.reduce({}) do |memo, name| column = name.downcase.gsub(/\s/, "_").to_sy column = :external_id if column == :id memo[column] = row[name] memo end ) # sleep 1 record.static_map = File.open("/Users/nathan/wor record.save! end end
desc "Import data with legion" task :import_with_legion => :environment do
require "csv" require_relative "../legion_importer" supervisor = Legion::Supervisor.new( LegionImporter, :processes => 8, :port => 42042 ) supervisor.start count = 0 path = File.expand_path("../../../data/Crimes_-_20 CSV.foreach(path, :headers => true) do |row| supervisor.import_csv_row row print "#{count += 1}," end supervisor.stop end
desc "Import data with legion" task :import_with_legion => :environment do
require "csv" require_relative "../legion_importer" supervisor = Legion::Supervisor.new( LegionImporter, :processes => 8, :port => 42042 ) supervisor.start count = 0 path = File.expand_path("../../../data/Crimes_-_20 CSV.foreach(path, :headers => true) do |row| supervisor.import_csv_row row print "#{count += 1}," end supervisor.stop end
MULTIPLE PROCESSES
ACTUALLY USE YOUR RESOURCES • More efficient • Faster •
Smarter • Better
HOW IT WORKS fork + DRb + background thread
Legion Objects can create remote instances of themselves (fork +
DRb)
Supervisors manage groups of remote objects
Remote objects provide async methods via a background thread
Supervisor round-robins work to remote objects in sequence
SMALL 120 lines of code
GOTCHAS • Legion::Object subclasses must define async methods directly •
Remember you’re dealing with a forked process • Rails must reconnect to the database with a :before callback • Relative file paths don’t work
@hopsoft Questions?