Legion intro

Ruby concurrency made easy

Transcript

1. My name is Legion for we are many Concurrency Made Easy
2. HOW IT STARTED
3. • Rails project • Data import • Lots of data • Data manipulation THE PROJECT
4. • Chicago crime data - 1 GB CSV • Save CSV records to database • Convert longitude/latitude to static image of map • Various sizes thumbnail, medium, large • Rails + Carrierwave REQUIREMENTS
5. RAKE TASK?
6. class Importer def import_csv_row(row) record = ChicagoCrime.new( row.headers.reduce({}) do |memo, name| column = name.downcase.gsub(/\s/, "_").to_s column = :external_id if column == :id memo[column] = row[name] memo end ) #sleep 1 record.static_map = File.open("/Users/nathan/wo record.save! end end
7. desc "Import data" task :import => :environment do require "csv" require_relative "../importer" importer = Importer.new count = 0 path = File.expand_path("../../../data/Crimes_-_2 CSV.foreach(path, :headers => true) do |row| importer.import_csv_row row print "#{count += 1}," end end
8. SINGLE PROCESS
9. SLOW What to do?
10. CONCURRENCY • Threads with JRuby or Rubinius? • Celluloid with JRuby or Rubinius? • with MRI? • Background Jobs? • fork?
11. MRI has a concurrency Story It’t called DRb - Distributed Ruby
12. LEGION Makes it easy
13. class LegionImporter < Legion::Object before :import_csv_row do ActiveRecord::Base.establish_connection ActiveRe end def import_csv_row(row) record = ChicagoCrime.new( row.headers.reduce({}) do |memo, name| column = name.downcase.gsub(/\s/, "_").to_sy column = :external_id if column == :id memo[column] = row[name] memo end ) # sleep 1 record.static_map = File.open("/Users/nathan/wor record.save! end end
14. class LegionImporter < Legion::Object before :import_csv_row do ActiveRecord::Base.establish_connection ActiveRe end def import_csv_row(row) record = ChicagoCrime.new( row.headers.reduce({}) do |memo, name| column = name.downcase.gsub(/\s/, "_").to_sy column = :external_id if column == :id memo[column] = row[name] memo end ) # sleep 1 record.static_map = File.open("/Users/nathan/wor record.save! end end
15. desc "Import data with legion" task :import_with_legion => :environment do require "csv" require_relative "../legion_importer" supervisor = Legion::Supervisor.new( LegionImporter, :processes => 8, :port => 42042 ) supervisor.start count = 0 path = File.expand_path("../../../data/Crimes_-_20 CSV.foreach(path, :headers => true) do |row| supervisor.import_csv_row row print "#{count += 1}," end supervisor.stop end
16. desc "Import data with legion" task :import_with_legion => :environment do require "csv" require_relative "../legion_importer" supervisor = Legion::Supervisor.new( LegionImporter, :processes => 8, :port => 42042 ) supervisor.start count = 0 path = File.expand_path("../../../data/Crimes_-_20 CSV.foreach(path, :headers => true) do |row| supervisor.import_csv_row row print "#{count += 1}," end supervisor.stop end
17. MULTIPLE PROCESSES
18. ACTUALLY USE YOUR RESOURCES • More efficient • Faster • Smarter • Better
19. HOW IT WORKS fork + DRb + background thread
20. Legion Objects can create remote instances of themselves (fork + DRb)
21. Supervisors manage groups of remote objects
22. Remote objects provide async methods via a background thread
23. Supervisor round-robins work to remote objects in sequence
24. SMALL 120 lines of code
25. GOTCHAS • Legion::Object subclasses must define async methods directly • Remember you’re dealing with a forked process • Rails must reconnect to the database with a :before callback • Relative file paths don’t work
26. @hopsoft Questions?