Pro Yearly is on sale from $80 to $50! »

Hidden Gems

259f23c3b129f07b0c496b9f0495f07e?s=47 jeg2
September 05, 2008

Hidden Gems

Some ways to coax some extra speed out of Ruby processes.

259f23c3b129f07b0c496b9f0495f07e?s=128

jeg2

September 05, 2008
Tweet

Transcript

  1. Ruby in the Fast Lane Hidden Gems

  2. Back at Lone Star

  3. Back at Lone Star ‣I am James Edward Gray II

  4. Back at Lone Star ‣I am James Edward Gray II

    ‣Created the Ruby Quiz
  5. Back at Lone Star ‣I am James Edward Gray II

    ‣Created the Ruby Quiz ‣released FasterCSV, HighLine, and Elif
  6. Back at Lone Star ‣I am James Edward Gray II

    ‣Created the Ruby Quiz ‣released FasterCSV, HighLine, and Elif ‣Wrote a couple of Pragmatic Programmer books with a lot of Ruby in them
  7. Back at Lone Star ‣I am James Edward Gray II

    ‣Created the Ruby Quiz ‣released FasterCSV, HighLine, and Elif ‣Wrote a couple of Pragmatic Programmer books with a lot of Ruby in them ‣I’ve given a talk at every LSRC so far
  8. None
  9. None
  10. Why BSG?

  11. Why BSG? ‣The old ship that was to be decommissioned

    becomes the only thing keeping the human race alive
  12. Why BSG? ‣The old ship that was to be decommissioned

    becomes the only thing keeping the human race alive ‣Those are my kind of odds
  13. Ugly Rumor

  14. “Ruby is slow.” Ugly Rumor

  15. My Opinion of the Speed Rumor

  16. BS My Opinion of the Speed Rumor

  17. Ruby is as Fast as we Want her to Be!

  18. Ruby is as Fast as we Want her to Be!

    ‣We can always write C extensions for speed-critical portions
  19. Ruby is as Fast as we Want her to Be!

    ‣We can always write C extensions for speed-critical portions ‣This is rarely actually needed though
  20. Ruby is as Fast as we Want her to Be!

    ‣We can always write C extensions for speed-critical portions ‣This is rarely actually needed though ‣We can make use of libraries, some written in C, that help with our problem
  21. Ruby is as Fast as we Want her to Be!

    ‣We can always write C extensions for speed-critical portions ‣This is rarely actually needed though ‣We can make use of libraries, some written in C, that help with our problem ‣We can add more processing power
  22. Ruby is as Fast as we Want her to Be!

    ‣We can always write C extensions for speed-critical portions ‣This is rarely actually needed though ‣We can make use of libraries, some written in C, that help with our problem ‣We can add more processing power ‣We can rework our data structures to better support the task at hand
  23. Ruby is as Fast as we Want her to Be!

    ‣We can always write C extensions for speed-critical portions ‣This is rarely actually needed though ‣We can make use of libraries, some written in C, that help with our problem ‣We can add more processing power ‣We can rework our data structures to better support the task at hand ‣Always the big win, in my opinion
  24. Ruby in the Fast Lane

  25. Ruby in the Fast Lane ‣Let’s see how fast Ruby

    can run using:
  26. Ruby in the Fast Lane ‣Let’s see how fast Ruby

    can run using: ‣NArray
  27. Ruby in the Fast Lane ‣Let’s see how fast Ruby

    can run using: ‣NArray ‣SQLite
  28. Ruby in the Fast Lane ‣Let’s see how fast Ruby

    can run using: ‣NArray ‣SQLite ‣RBTree
  29. Ruby in the Fast Lane ‣Let’s see how fast Ruby

    can run using: ‣NArray ‣SQLite ‣RBTree ‣FSDB
  30. Ruby in the Fast Lane ‣Let’s see how fast Ruby

    can run using: ‣NArray ‣SQLite ‣RBTree ‣FSDB ‣Rinda
  31. Ruby in the Fast Lane ‣Let’s see how fast Ruby

    can run using: ‣NArray ‣SQLite ‣RBTree ‣FSDB ‣Rinda ‣Thinking outside the box
  32. Super Fast Number crunching

  33. Super Fast Number crunching NArray

  34. When the Numbers Count

  35. When the Numbers Count ‣Ruby’s Numeric family of objects were

    built for ease of use
  36. When the Numbers Count ‣Ruby’s Numeric family of objects were

    built for ease of use ‣This makes them a bit slower
  37. When the Numbers Count ‣Ruby’s Numeric family of objects were

    built for ease of use ‣This makes them a bit slower ‣C’s numbers were built for speed
  38. When the Numbers Count ‣Ruby’s Numeric family of objects were

    built for ease of use ‣This makes them a bit slower ‣C’s numbers were built for speed ‣Ruby can borrow them with NArray
  39. Problem: Faster Imaging

  40. Problem: Faster Imaging ‣I use a trivial PPM library to

    generate images for one project
  41. Problem: Faster Imaging ‣I use a trivial PPM library to

    generate images for one project ‣The PPM code was taking about 1.3 seconds for a 400 by 200 pixel image
  42. Problem: Faster Imaging ‣I use a trivial PPM library to

    generate images for one project ‣The PPM code was taking about 1.3 seconds for a 400 by 200 pixel image ‣I replaced a two dimensional Array of Color objects with a 3D NArray
  43. Problem: Faster Imaging ‣I use a trivial PPM library to

    generate images for one project ‣The PPM code was taking about 1.3 seconds for a 400 by 200 pixel image ‣I replaced a two dimensional Array of Color objects with a 3D NArray ‣I changed less than ten lines of code
  44. Problem: Faster Imaging ‣I use a trivial PPM library to

    generate images for one project ‣The PPM code was taking about 1.3 seconds for a 400 by 200 pixel image ‣I replaced a two dimensional Array of Color objects with a 3D NArray ‣I changed less than ten lines of code ‣The speed on the same image dropped to about 1/100th of a second
  45. Creating The Canvas

  46. Creating The Canvas def initialize(options = Hash.new) options = DEFAULT_OPTIONS.merge(options)

    @width = options[:width] @height = options[:height] @background = options[:background] @foreground = options[:foreground] @mode = options[:mode] @canvas = Array.new(@height) { Array.new(@width) { @background } } end
  47. Creating The Canvas def initialize(options = Hash.new) options = DEFAULT_OPTIONS.merge(options)

    @width = options[:width] @height = options[:height] @background = options[:background] @foreground = options[:foreground] @mode = options[:mode] @canvas = Array.new(@height) { Array.new(@width) { @background } } end
  48. Creating The Canvas def initialize(options = Hash.new) require "rubygems" require

    "narray" ! options = DEFAULT_OPTIONS.merge(options) @width = options[:width] @height = options[:height] @background = options[:background] @foreground = options[:foreground] @mode = options[:mode] @canvas = NArray.byte(@width, @height, 3) end
  49. Creating The Canvas def initialize(options = Hash.new) require "rubygems" require

    "narray" ! options = DEFAULT_OPTIONS.merge(options) @width = options[:width] @height = options[:height] @background = options[:background] @foreground = options[:foreground] @mode = options[:mode] @canvas = NArray.byte(@width, @height, 3) end
  50. Marking Pixels

  51. Marking Pixels def draw_point(x, y, color = @foreground) return unless

    x.between? 0, @width - 1 return unless y.between? 0, @height - 1 ! @canvas[y][x] = color end
  52. Marking Pixels def draw_point(x, y, color = @foreground) return unless

    x.between? 0, @width - 1 return unless y.between? 0, @height - 1 ! @canvas[y][x] = color end
  53. Marking Pixels def draw_point(x, y, color = @foreground) return unless

    x.between? 0, @width - 1 return unless y.between? 0, @height - 1 ! @canvas[x, y, 0..2] = color.to_a end
  54. Marking Pixels def draw_point(x, y, color = @foreground) return unless

    x.between? 0, @width - 1 return unless y.between? 0, @height - 1 ! @canvas[x, y, 0..2] = color.to_a end
  55. Drawing an Image

  56. Drawing an Image def save(file) File.open(file.sub(/\.ppm$/i, "") + ".ppm", "w")

    do |image| image.puts @mode image.puts "#{@width} #{@height} 255" @canvas.each do |row| pixels = row.map { |pixel| pixel.to_s(@mode) } image.send( @mode == "P6" ? :print : :puts, pixels.join(@mode == "P6" ? "" : " ") ) end end end
  57. Drawing an Image def save(file) File.open(file.sub(/\.ppm$/i, "") + ".ppm", "w")

    do |image| image.puts @mode image.puts "#{@width} #{@height} 255" @canvas.each do |row| pixels = row.map { |pixel| pixel.to_s(@mode) } image.send( @mode == "P6" ? :print : :puts, pixels.join(@mode == "P6" ? "" : " ") ) end end end
  58. Drawing an Image def save(file) File.open(file.sub(/\.ppm$/i, "") + ".ppm", "w")

    do |image| image.puts @mode image.puts "#{@width} #{@height} 255" 0.upto(@height - 1) do |y| row = @canvas[0...@width, y, 0..2].transpose(-1, 0) image.send( @mode == "P6" ? :print : :puts, @mode == "P6" ? row.to_s : row.to_a.join(" ") ) end end end
  59. Drawing an Image def save(file) File.open(file.sub(/\.ppm$/i, "") + ".ppm", "w")

    do |image| image.puts @mode image.puts "#{@width} #{@height} 255" 0.upto(@height - 1) do |y| row = @canvas[0...@width, y, 0..2].transpose(-1, 0) image.send( @mode == "P6" ? :print : :puts, @mode == "P6" ? row.to_s : row.to_a.join(" ") ) end end end
  60. Other Nice Features

  61. Other Nice Features ‣NArray supports integers, floats, and even complex

    numbers in various sizes
  62. Other Nice Features ‣NArray supports integers, floats, and even complex

    numbers in various sizes ‣Aside from indexing and iteration, NArray supports data generation, arithmetic operations, comparisons, bitwise manipulations, statistic calculations, and more
  63. Other Nice Features ‣NArray supports integers, floats, and even complex

    numbers in various sizes ‣Aside from indexing and iteration, NArray supports data generation, arithmetic operations, comparisons, bitwise manipulations, statistic calculations, and more ‣View the large API with examples At: http://narray.rubyforge.org/
  64. Conway’s Game of Life

  65. Conway’s Game of Life #!/usr/bin/env ruby -wKU ! require "rubygems"

    require "narray" ! # build cells life = NArray.byte(5, 5) life[1, 1] = NArray.byte(3, 3).random!(2) p life ! # count neighbors counts = NArray.byte(*life.shape) counts[1..-2, 1..-2] = life[0..-3, 0..-3] + life[0..-3, 1..-2] + life[0..-3, 2..-1] + life[1..-2, 0..-3] + life[1..-2, 2..-1] + life[2..-1, 0..-3] + life[2..-1, 1..-2] + life[2..-1, 2..-1] p counts ! # one step of the game life[] = counts.eq(3) | (counts.eq(2) & life) p life
  66. Conway’s Game of Life #!/usr/bin/env ruby -wKU ! require "rubygems"

    require "narray" ! # build cells life = NArray.byte(5, 5) life[1, 1] = NArray.byte(3, 3).random!(2) p life ! # count neighbors counts = NArray.byte(*life.shape) counts[1..-2, 1..-2] = life[0..-3, 0..-3] + life[0..-3, 1..-2] + life[0..-3, 2..-1] + life[1..-2, 0..-3] + life[1..-2, 2..-1] + life[2..-1, 0..-3] + life[2..-1, 1..-2] + life[2..-1, 2..-1] p counts ! # one step of the game life[] = counts.eq(3) | (counts.eq(2) & life) p life
  67. Conway’s Game of Life NArray.byte(5,5): [ [ 0, 0, 0,

    0, 0 ], [ 0, 1, 0, 1, 0 ], [ 0, 0, 0, 0, 0 ], [ 0, 1, 1, 0, 0 ], [ 0, 0, 0, 0, 0 ] ] #!/usr/bin/env ruby -wKU ! require "rubygems" require "narray" ! # build cells life = NArray.byte(5, 5) life[1, 1] = NArray.byte(3, 3).random!(2) p life ! # count neighbors counts = NArray.byte(*life.shape) counts[1..-2, 1..-2] = life[0..-3, 0..-3] + life[0..-3, 1..-2] + life[0..-3, 2..-1] + life[1..-2, 0..-3] + life[1..-2, 2..-1] + life[2..-1, 0..-3] + life[2..-1, 1..-2] + life[2..-1, 2..-1] p counts ! # one step of the game life[] = counts.eq(3) | (counts.eq(2) & life) p life
  68. Conway’s Game of Life NArray.byte(5,5): [ [ 0, 0, 0,

    0, 0 ], [ 0, 1, 0, 1, 0 ], [ 0, 0, 0, 0, 0 ], [ 0, 1, 1, 0, 0 ], [ 0, 0, 0, 0, 0 ] ] #!/usr/bin/env ruby -wKU ! require "rubygems" require "narray" ! # build cells life = NArray.byte(5, 5) life[1, 1] = NArray.byte(3, 3).random!(2) p life ! # count neighbors counts = NArray.byte(*life.shape) counts[1..-2, 1..-2] = life[0..-3, 0..-3] + life[0..-3, 1..-2] + life[0..-3, 2..-1] + life[1..-2, 0..-3] + life[1..-2, 2..-1] + life[2..-1, 0..-3] + life[2..-1, 1..-2] + life[2..-1, 2..-1] p counts ! # one step of the game life[] = counts.eq(3) | (counts.eq(2) & life) p life
  69. Conway’s Game of Life NArray.byte(5,5): [ [ 0, 0, 0,

    0, 0 ], [ 0, 1, 0, 1, 0 ], [ 0, 0, 0, 0, 0 ], [ 0, 1, 1, 0, 0 ], [ 0, 0, 0, 0, 0 ] ] NArray.byte(5,5): [ [ 0, 0, 0, 0, 0 ], [ 0, 0, 2, 0, 0 ], [ 0, 3, 4, 2, 0 ], [ 0, 1, 1, 1, 0 ], [ 0, 0, 0, 0, 0 ] ] #!/usr/bin/env ruby -wKU ! require "rubygems" require "narray" ! # build cells life = NArray.byte(5, 5) life[1, 1] = NArray.byte(3, 3).random!(2) p life ! # count neighbors counts = NArray.byte(*life.shape) counts[1..-2, 1..-2] = life[0..-3, 0..-3] + life[0..-3, 1..-2] + life[0..-3, 2..-1] + life[1..-2, 0..-3] + life[1..-2, 2..-1] + life[2..-1, 0..-3] + life[2..-1, 1..-2] + life[2..-1, 2..-1] p counts ! # one step of the game life[] = counts.eq(3) | (counts.eq(2) & life) p life
  70. Conway’s Game of Life NArray.byte(5,5): [ [ 0, 0, 0,

    0, 0 ], [ 0, 1, 0, 1, 0 ], [ 0, 0, 0, 0, 0 ], [ 0, 1, 1, 0, 0 ], [ 0, 0, 0, 0, 0 ] ] NArray.byte(5,5): [ [ 0, 0, 0, 0, 0 ], [ 0, 0, 2, 0, 0 ], [ 0, 3, 4, 2, 0 ], [ 0, 1, 1, 1, 0 ], [ 0, 0, 0, 0, 0 ] ] #!/usr/bin/env ruby -wKU ! require "rubygems" require "narray" ! # build cells life = NArray.byte(5, 5) life[1, 1] = NArray.byte(3, 3).random!(2) p life ! # count neighbors counts = NArray.byte(*life.shape) counts[1..-2, 1..-2] = life[0..-3, 0..-3] + life[0..-3, 1..-2] + life[0..-3, 2..-1] + life[1..-2, 0..-3] + life[1..-2, 2..-1] + life[2..-1, 0..-3] + life[2..-1, 1..-2] + life[2..-1, 2..-1] p counts ! # one step of the game life[] = counts.eq(3) | (counts.eq(2) & life) p life
  71. Conway’s Game of Life NArray.byte(5,5): [ [ 0, 0, 0,

    0, 0 ], [ 0, 1, 0, 1, 0 ], [ 0, 0, 0, 0, 0 ], [ 0, 1, 1, 0, 0 ], [ 0, 0, 0, 0, 0 ] ] NArray.byte(5,5): [ [ 0, 0, 0, 0, 0 ], [ 0, 0, 2, 0, 0 ], [ 0, 3, 4, 2, 0 ], [ 0, 1, 1, 1, 0 ], [ 0, 0, 0, 0, 0 ] ] NArray.byte(5,5): [ [ 0, 0, 0, 0, 0 ], [ 0, 0, 0, 0, 0 ], [ 0, 1, 0, 0, 0 ], [ 0, 0, 0, 0, 0 ], [ 0, 0, 0, 0, 0 ] ] #!/usr/bin/env ruby -wKU ! require "rubygems" require "narray" ! # build cells life = NArray.byte(5, 5) life[1, 1] = NArray.byte(3, 3).random!(2) p life ! # count neighbors counts = NArray.byte(*life.shape) counts[1..-2, 1..-2] = life[0..-3, 0..-3] + life[0..-3, 1..-2] + life[0..-3, 2..-1] + life[1..-2, 0..-3] + life[1..-2, 2..-1] + life[2..-1, 0..-3] + life[2..-1, 1..-2] + life[2..-1, 2..-1] p counts ! # one step of the game life[] = counts.eq(3) | (counts.eq(2) & life) p life
  72. A Data DSL

  73. A Data DSL SQLite

  74. Thinking About Data can be Hard

  75. Thinking About Data can be Hard ‣SQLite has already solved

    many hard problems for data storage and retrieval
  76. Thinking About Data can be Hard ‣SQLite has already solved

    many hard problems for data storage and retrieval ‣It gives you an entire language to express your data needs
  77. Problem: IP to Country

  78. Problem: IP to Country ‣Given an IP address, return the

    country for that IP
  79. Problem: IP to Country ‣Given an IP address, return the

    country for that IP ‣this was a Ruby Quiz
  80. Problem: IP to Country ‣Given an IP address, return the

    country for that IP ‣this was a Ruby Quiz ‣Solutions were also expected to be efficient in memory and speed
  81. Problem: IP to Country ‣Given an IP address, return the

    country for that IP ‣this was a Ruby Quiz ‣Solutions were also expected to be efficient in memory and speed ‣This is a real world task I’ve had to do for my job
  82. The Data

  83. The Data # © 2002-2008 Webnet77.com # # # #

    "0","16777215","IANA","410227200","ZZ","ZZZ","RESERVED" "50331648","67108863","ARIN","572572800","US","USA","UNITED STATES" "67108864","83886079","ARIN","0","US","USA","UNITED STATES" "100663296","117440511","ARIN","0","US","USA","UNITED STATES" "117440512","134217727","ARIN","880329600","US","USA","UNITED STATES"
  84. The Data # © 2002-2008 Webnet77.com # # # #

    "0","16777215","IANA","410227200","ZZ","ZZZ","RESERVED" "50331648","67108863","ARIN","572572800","US","USA","UNITED STATES" "67108864","83886079","ARIN","0","US","USA","UNITED STATES" "100663296","117440511","ARIN","0","US","USA","UNITED STATES" "117440512","134217727","ARIN","880329600","US","USA","UNITED STATES"
  85. Solutions

  86. Solutions ‣Many solved the problem with a binary search on

    the file
  87. Solutions ‣Many solved the problem with a binary search on

    the file ‣Most of those prepossessed the file to make that search easier
  88. Solutions ‣Many solved the problem with a binary search on

    the file ‣Most of those prepossessed the file to make that search easier ‣I’m going to show a SQLite solution
  89. Solutions ‣Many solved the problem with a binary search on

    the file ‣Most of those prepossessed the file to make that search easier ‣I’m going to show a SQLite solution ‣It’s very close to the same speed (about 1/3rd of a second to lookup an IP)
  90. Solutions ‣Many solved the problem with a binary search on

    the file ‣Most of those prepossessed the file to make that search easier ‣I’m going to show a SQLite solution ‣It’s very close to the same speed (about 1/3rd of a second to lookup an IP) ‣I didn’t have to be clever or even add an index
  91. Solutions ‣Many solved the problem with a binary search on

    the file ‣Most of those prepossessed the file to make that search easier ‣I’m going to show a SQLite solution ‣It’s very close to the same speed (about 1/3rd of a second to lookup an IP) ‣I didn’t have to be clever or even add an index ‣It was easier for me to use full country names
  92. Setup

  93. Setup #!/usr/bin/env ruby -KU ! require "open-uri" require "zlib" !

    require "rubygems" require "faster_csv" require "sqlite3" ! REMOTE_DB = "http://software77.net/cgi-bin/" + "ip-country/geo-ip.pl?action=download" LOCAL_DB = "country_ips.sqlite" ! File.unlink(LOCAL_DB) if ARGV.delete("-r") and File.exist? LOCAL_DB ! # ...
  94. Build The Database

  95. Build The Database # ... ! unless File.exist? LOCAL_DB db

    = SQLite3::Database.new(LOCAL_DB) db.execute(<<-END_TABLE.strip) CREATE TABLE ips ( low_ip INTEGER, high_ip INTEGER, country TEXT ) END_TABLE open(REMOTE_DB) do |url| Zlib::GzipReader.new(url).each do |line| next if line =~ /\A\s*(?:#|\z)/ args = FCSV.parse_line(line).values_at(0..1, 6) db.execute(<<-END_INSERT.strip, *args) INSERT INTO ips( low_ip, high_ip, country) VALUES( ?, ?, ?) END_INSERT end end db.close end ! # ...
  96. Build The Database # ... ! unless File.exist? LOCAL_DB db

    = SQLite3::Database.new(LOCAL_DB) db.execute(<<-END_TABLE.strip) CREATE TABLE ips ( low_ip INTEGER, high_ip INTEGER, country TEXT ) END_TABLE open(REMOTE_DB) do |url| Zlib::GzipReader.new(url).each do |line| next if line =~ /\A\s*(?:#|\z)/ args = FCSV.parse_line(line).values_at(0..1, 6) db.execute(<<-END_INSERT.strip, *args) INSERT INTO ips( low_ip, high_ip, country) VALUES( ?, ?, ?) END_INSERT end end db.close end ! # ...
  97. Build The Database # ... ! unless File.exist? LOCAL_DB db

    = SQLite3::Database.new(LOCAL_DB) db.execute(<<-END_TABLE.strip) CREATE TABLE ips ( low_ip INTEGER, high_ip INTEGER, country TEXT ) END_TABLE open(REMOTE_DB) do |url| Zlib::GzipReader.new(url).each do |line| next if line =~ /\A\s*(?:#|\z)/ args = FCSV.parse_line(line).values_at(0..1, 6) db.execute(<<-END_INSERT.strip, *args) INSERT INTO ips( low_ip, high_ip, country) VALUES( ?, ?, ?) END_INSERT end end db.close end ! # ...
  98. Query

  99. Query # ... ! ip = ARGV.shift or abort "Usage:

    #{File.basename($PROGRAM_NAME)} IP" ip_int = ip.split(".").map { |n| Integer(n) }. pack("C*").unpack("N").first db = SQLite3::Database.new(LOCAL_DB) ! puts db.get_first_value(<<-END_SELECT.strip, :ip => ip_int) || "Unknown" SELECT country FROM ips WHERE low_ip <= :ip AND :ip <= high_ip END_SELECT
  100. Query # ... ! ip = ARGV.shift or abort "Usage:

    #{File.basename($PROGRAM_NAME)} IP" ip_int = ip.split(".").map { |n| Integer(n) }. pack("C*").unpack("N").first db = SQLite3::Database.new(LOCAL_DB) ! puts db.get_first_value(<<-END_SELECT.strip, :ip => ip_int) || "Unknown" SELECT country FROM ips WHERE low_ip <= :ip AND :ip <= high_ip END_SELECT
  101. Did You Know?

  102. Did You Know? ‣SQLite is totally free

  103. Did You Know? ‣SQLite is totally free ‣You can receive

    query results in a Hash to index by column name and/or convert column values to Ruby objects based on type
  104. Did You Know? ‣SQLite is totally free ‣You can receive

    query results in a Hash to index by column name and/or convert column values to Ruby objects based on type ‣It can work with in-memory databases
  105. Did You Know? ‣SQLite is totally free ‣You can receive

    query results in a Hash to index by column name and/or convert column values to Ruby objects based on type ‣It can work with in-memory databases ‣It can run queries across tables in multiple database files
  106. Did You Know? ‣SQLite is totally free ‣You can receive

    query results in a Hash to index by column name and/or convert column values to Ruby objects based on type ‣It can work with in-memory databases ‣It can run queries across tables in multiple database files ‣You can define SQL functions and aggregates for it in Ruby code
  107. Ruby Friendly Data

  108. Ruby Friendly Data #!/usr/bin/env ruby -KU ! require "rubygems" require

    "sqlite3" ! country = ARGV.shift or abort "Usage: #{File.basename($PROGRAM_NAME)} COUNTRY" db = SQLite3::Database.new("country_ips.sqlite") db.results_as_hash = true db.type_translation = true ! db.execute( "SELECT * FROM ips WHERE country LIKE ?", "%#{country}%" ) do |match| low, high = match.values_at("low_ip", "high_ip"). map { |i| [i].pack("N").unpack("C*").join(".") } puts "%s: %15s - %15s" % [match["country"], low, high] end
  109. Ruby Friendly Data #!/usr/bin/env ruby -KU ! require "rubygems" require

    "sqlite3" ! country = ARGV.shift or abort "Usage: #{File.basename($PROGRAM_NAME)} COUNTRY" db = SQLite3::Database.new("country_ips.sqlite") db.results_as_hash = true db.type_translation = true ! db.execute( "SELECT * FROM ips WHERE country LIKE ?", "%#{country}%" ) do |match| low, high = match.values_at("low_ip", "high_ip"). map { |i| [i].pack("N").unpack("C*").join(".") } puts "%s: %15s - %15s" % [match["country"], low, high] end
  110. Ruby Friendly Data #!/usr/bin/env ruby -KU ! require "rubygems" require

    "sqlite3" ! country = ARGV.shift or abort "Usage: #{File.basename($PROGRAM_NAME)} COUNTRY" db = SQLite3::Database.new("country_ips.sqlite") db.results_as_hash = true db.type_translation = true ! db.execute( "SELECT * FROM ips WHERE country LIKE ?", "%#{country}%" ) do |match| low, high = match.values_at("low_ip", "high_ip"). map { |i| [i].pack("N").unpack("C*").join(".") } puts "%s: %15s - %15s" % [match["country"], low, high] end
  111. In-Memory

  112. In-Memory #!/usr/bin/env ruby -KU ! # ... requires unchanged ...

    ! REMOTE_DB = "http://software77.net/cgi-bin/" + "ip-country/geo-ip.pl?action=download" ! db = SQLite3::Database.new(":memory:") ! # ... database loading unchanged ... ! stmt = db.prepare(<<-END_SELECT.strip) SELECT country FROM ips WHERE low_ip <= :ip AND :ip <= high_ip END_SELECT loop do print "IP address? " ip = gets.to_s.strip if ip =~ /\S/ ip_int = ip.split(".").map { |n| Integer(n) }. pack("C*").unpack("N").first puts stmt.execute(:ip => ip_int).each { |c| break c[0] } || "Unknown" else break end end
  113. In-Memory #!/usr/bin/env ruby -KU ! # ... requires unchanged ...

    ! REMOTE_DB = "http://software77.net/cgi-bin/" + "ip-country/geo-ip.pl?action=download" ! db = SQLite3::Database.new(":memory:") ! # ... database loading unchanged ... ! stmt = db.prepare(<<-END_SELECT.strip) SELECT country FROM ips WHERE low_ip <= :ip AND :ip <= high_ip END_SELECT loop do print "IP address? " ip = gets.to_s.strip if ip =~ /\S/ ip_int = ip.split(".").map { |n| Integer(n) }. pack("C*").unpack("N").first puts stmt.execute(:ip => ip_int).each { |c| break c[0] } || "Unknown" else break end end
  114. In-Memory #!/usr/bin/env ruby -KU ! # ... requires unchanged ...

    ! REMOTE_DB = "http://software77.net/cgi-bin/" + "ip-country/geo-ip.pl?action=download" ! db = SQLite3::Database.new(":memory:") ! # ... database loading unchanged ... ! stmt = db.prepare(<<-END_SELECT.strip) SELECT country FROM ips WHERE low_ip <= :ip AND :ip <= high_ip END_SELECT loop do print "IP address? " ip = gets.to_s.strip if ip =~ /\S/ ip_int = ip.split(".").map { |n| Integer(n) }. pack("C*").unpack("N").first puts stmt.execute(:ip => ip_int).each { |c| break c[0] } || "Unknown" else break end end
  115. In-Memory #!/usr/bin/env ruby -KU ! # ... requires unchanged ...

    ! REMOTE_DB = "http://software77.net/cgi-bin/" + "ip-country/geo-ip.pl?action=download" ! db = SQLite3::Database.new(":memory:") ! # ... database loading unchanged ... ! stmt = db.prepare(<<-END_SELECT.strip) SELECT country FROM ips WHERE low_ip <= :ip AND :ip <= high_ip END_SELECT loop do print "IP address? " ip = gets.to_s.strip if ip =~ /\S/ ip_int = ip.split(".").map { |n| Integer(n) }. pack("C*").unpack("N").first puts stmt.execute(:ip => ip_int).each { |c| break c[0] } || "Unknown" else break end end
  116. Attach and Functions

  117. Attach and Functions #!/usr/bin/env ruby -KU ! require "rubygems" require

    "sqlite3" ! user = ARGV.shift or abort "Usage: #{File.basename($PROGRAM_NAME)} USER" db = SQLite3::Database.new("users.sqlite") db.execute("ATTACH DATABASE 'country_ips.sqlite' AS country_ips") ! db.create_function("IP2INT", 1) do |func, ip| func.result = ip.to_s.split(".").map { |n| Integer(n) }. pack("C*").unpack("N").first end ! sql = <<END_SQL SELECT users.name, users.ip, ips.country FROM users INNER JOIN ips ON ips.low_ip <= IP2INT(users.ip) AND IP2INT(users.ip) <= ips.high_ip WHERE users.name LIKE ? LIMIT 1 END_SQL puts db.get_first_row(sql, "%#{user}%").join(", ")
  118. Attach and Functions #!/usr/bin/env ruby -KU ! require "rubygems" require

    "sqlite3" ! user = ARGV.shift or abort "Usage: #{File.basename($PROGRAM_NAME)} USER" db = SQLite3::Database.new("users.sqlite") db.execute("ATTACH DATABASE 'country_ips.sqlite' AS country_ips") ! db.create_function("IP2INT", 1) do |func, ip| func.result = ip.to_s.split(".").map { |n| Integer(n) }. pack("C*").unpack("N").first end ! sql = <<END_SQL SELECT users.name, users.ip, ips.country FROM users INNER JOIN ips ON ips.low_ip <= IP2INT(users.ip) AND IP2INT(users.ip) <= ips.high_ip WHERE users.name LIKE ? LIMIT 1 END_SQL puts db.get_first_row(sql, "%#{user}%").join(", ")
  119. Attach and Functions #!/usr/bin/env ruby -KU ! require "rubygems" require

    "sqlite3" ! user = ARGV.shift or abort "Usage: #{File.basename($PROGRAM_NAME)} USER" db = SQLite3::Database.new("users.sqlite") db.execute("ATTACH DATABASE 'country_ips.sqlite' AS country_ips") ! db.create_function("IP2INT", 1) do |func, ip| func.result = ip.to_s.split(".").map { |n| Integer(n) }. pack("C*").unpack("N").first end ! sql = <<END_SQL SELECT users.name, users.ip, ips.country FROM users INNER JOIN ips ON ips.low_ip <= IP2INT(users.ip) AND IP2INT(users.ip) <= ips.high_ip WHERE users.name LIKE ? LIMIT 1 END_SQL puts db.get_first_row(sql, "%#{user}%").join(", ")
  120. Attach and Functions #!/usr/bin/env ruby -KU ! require "rubygems" require

    "sqlite3" ! user = ARGV.shift or abort "Usage: #{File.basename($PROGRAM_NAME)} USER" db = SQLite3::Database.new("users.sqlite") db.execute("ATTACH DATABASE 'country_ips.sqlite' AS country_ips") ! db.create_function("IP2INT", 1) do |func, ip| func.result = ip.to_s.split(".").map { |n| Integer(n) }. pack("C*").unpack("N").first end ! sql = <<END_SQL SELECT users.name, users.ip, ips.country FROM users INNER JOIN ips ON ips.low_ip <= IP2INT(users.ip) AND IP2INT(users.ip) <= ips.high_ip WHERE users.name LIKE ? LIMIT 1 END_SQL puts db.get_first_row(sql, "%#{user}%").join(", ")
  121. A Binary Tree

  122. A Binary Tree RBTree

  123. Sometimes you Just Need the big Guns

  124. Sometimes you Just Need the big Guns ‣Binary search and

    binary trees are pretty big guns in computing
  125. Sometimes you Just Need the big Guns ‣Binary search and

    binary trees are pretty big guns in computing ‣RBTree provides a super efficient binary tree implementation
  126. Sometimes you Just Need the big Guns ‣Binary search and

    binary trees are pretty big guns in computing ‣RBTree provides a super efficient binary tree implementation ‣It’s written in C
  127. Another Solution: IP to Country

  128. Another Solution: IP to Country ‣This time we will use

    a real binary search
  129. Another Solution: IP to Country ‣This time we will use

    a real binary search ‣But we don’t have to write it
  130. Another Solution: IP to Country ‣This time we will use

    a real binary search ‣But we don’t have to write it ‣This drops the search time below 1/1,000th of a second
  131. Another Solution: IP to Country ‣This time we will use

    a real binary search ‣But we don’t have to write it ‣This drops the search time below 1/1,000th of a second ‣We will Marshal RBTree to build our persistent database
  132. Another Solution: IP to Country ‣This time we will use

    a real binary search ‣But we don’t have to write it ‣This drops the search time below 1/1,000th of a second ‣We will Marshal RBTree to build our persistent database ‣We will use RBTree’s bounds search methods to perform the search
  133. The Same Setup

  134. The Same Setup #!/usr/bin/env ruby -wKU ! require "open-uri" require

    "zlib" ! require "rubygems" require "faster_csv" require "rbtree" ! REMOTE_DB = "http://software77.net/cgi-bin/" + "ip-country/geo-ip.pl?action=download" LOCAL_DB = "country_ips.marshal" ! File.unlink(LOCAL_DB) if ARGV.delete("-r") and File.exist? LOCAL_DB ! # ...
  135. An Easier Load

  136. An Easier Load # ... ! unless File.exist? LOCAL_DB ips

    = RBTree.new open(REMOTE_DB) do |url| Zlib::GzipReader.new(url).each do |line| next if line =~ /\A\s*(?:#|\z)/ low, high, country = FCSV.parse_line(line).values_at(0..1, 6) ips[Integer(low)] = [Integer(high), country] end end File.open(LOCAL_DB, "wb") { |file| Marshal.dump(ips, file) } end ! # ...
  137. An Easier Load # ... ! unless File.exist? LOCAL_DB ips

    = RBTree.new open(REMOTE_DB) do |url| Zlib::GzipReader.new(url).each do |line| next if line =~ /\A\s*(?:#|\z)/ low, high, country = FCSV.parse_line(line).values_at(0..1, 6) ips[Integer(low)] = [Integer(high), country] end end File.open(LOCAL_DB, "wb") { |file| Marshal.dump(ips, file) } end ! # ...
  138. An Easier Load # ... ! unless File.exist? LOCAL_DB ips

    = RBTree.new open(REMOTE_DB) do |url| Zlib::GzipReader.new(url).each do |line| next if line =~ /\A\s*(?:#|\z)/ low, high, country = FCSV.parse_line(line).values_at(0..1, 6) ips[Integer(low)] = [Integer(high), country] end end File.open(LOCAL_DB, "wb") { |file| Marshal.dump(ips, file) } end ! # ...
  139. A Much Faster Search

  140. A Much Faster Search # ... ! ips = File.open(LOCAL_DB,

    "rb") { |file| Marshal.load(file) } loop do print "IP address? " ip = gets.to_s.strip if ip =~ /\S/ ip_int = ip.split(".").map { |n| Integer(n) }. pack("C*").unpack("N").first match = ips.upper_bound(ip_int) puts match && ip_int <= match.last.first ? match.last.last : "Unknown" else break end end
  141. A Much Faster Search # ... ! ips = File.open(LOCAL_DB,

    "rb") { |file| Marshal.load(file) } loop do print "IP address? " ip = gets.to_s.strip if ip =~ /\S/ ip_int = ip.split(".").map { |n| Integer(n) }. pack("C*").unpack("N").first match = ips.upper_bound(ip_int) puts match && ip_int <= match.last.first ? match.last.last : "Unknown" else break end end
  142. A Much Faster Search # ... ! ips = File.open(LOCAL_DB,

    "rb") { |file| Marshal.load(file) } loop do print "IP address? " ip = gets.to_s.strip if ip =~ /\S/ ip_int = ip.split(".").map { |n| Integer(n) }. pack("C*").unpack("N").first match = ips.upper_bound(ip_int) puts match && ip_int <= match.last.first ? match.last.last : "Unknown" else break end end
  143. Other Nice Features

  144. Other Nice Features ‣RBTree is pretty much a drop in

    replacement for a Hash you want to keep ordered by keys
  145. Other Nice Features ‣RBTree is pretty much a drop in

    replacement for a Hash you want to keep ordered by keys ‣Just having RBTree available magically speeds up Ruby SortedSet (over 15 times faster for simple iteration) in the standard library
  146. An Ordered Hash

  147. An Ordered Hash #!/usr/bin/env ruby -wKU ! require "rubygems" require

    "rbtree" ! ordered_hash = RBTree.new ! ordered_hash[2] = "two" ordered_hash[1] = "one" ordered_hash[3] = "three" ! ordered_hash.each do |key, value| puts "#{key}: #{value}" end # >> 1: one # >> 2: two # >> 3: three
  148. Magically Improving SortedSet

  149. Magically Improving SortedSet #!/usr/bin/env ruby -wKU ! # pass an

    argument, like --rbtree, and this sets up the load require "rubygems" unless ARGV.empty? ! require "set" dictionary = SortedSet.new File.foreach("/usr/share/dict/words") do |word| dictionary << word.strip if word =~ /\S/ end ! start = Time.now dictionary.to_a # force the set into order puts "Time to order: #{Time.now - start}"
  150. The Filesystem as a Hash

  151. The Filesystem as a Hash FSDB

  152. Stay Flexible

  153. Stay Flexible ‣Data can be in many different formats and

    related in many different ways
  154. Stay Flexible ‣Data can be in many different formats and

    related in many different ways ‣FSDB gives you a lot of flexibility in these areas
  155. Stay Flexible ‣Data can be in many different formats and

    related in many different ways ‣FSDB gives you a lot of flexibility in these areas ‣Get from: http:// redshift.sourcefor ge.net/fsdb/
  156. Problem: Server Monitoring

  157. Problem: Server Monitoring ‣At my job we do a lot

    of server monitoring
  158. Problem: Server Monitoring ‣At my job we do a lot

    of server monitoring ‣We collect various statistics from servers at regular intervals
  159. Problem: Server Monitoring ‣At my job we do a lot

    of server monitoring ‣We collect various statistics from servers at regular intervals ‣We later analyze this data for spikes and trends
  160. Problem: Server Monitoring ‣At my job we do a lot

    of server monitoring ‣We collect various statistics from servers at regular intervals ‣We later analyze this data for spikes and trends ‣Time Series data is one thing RDBMS don’t do well
  161. A Solution

  162. A Solution ‣Store the data so it is easy to

    focus in on the parts that matter to you now
  163. A Solution ‣Store the data so it is easy to

    focus in on the parts that matter to you now ‣FSDB is essentially a Hash backed by the filesystem
  164. A Solution ‣Store the data so it is easy to

    focus in on the parts that matter to you now ‣FSDB is essentially a Hash backed by the filesystem ‣This allows you to use paths to drill down to subsets of the data
  165. A Solution ‣Store the data so it is easy to

    focus in on the parts that matter to you now ‣FSDB is essentially a Hash backed by the filesystem ‣This allows you to use paths to drill down to subsets of the data ‣It avoids irrelevant data and even the need for an index in some cases
  166. A Solution ‣Store the data so it is easy to

    focus in on the parts that matter to you now ‣FSDB is essentially a Hash backed by the filesystem ‣This allows you to use paths to drill down to subsets of the data ‣It avoids irrelevant data and even the need for an index in some cases ‣Techniques like this improved our graphing speed from almost four seconds to well under one
  167. FSDB Structure

  168. FSDB Structure

  169. Creating a Database

  170. Creating a Database #!/usr/bin/env ruby -wKU ! require "fsdb" !

    module TimeSeries DB = FSDB::Database.new("server_stats/") ! module_function def record(data, time = Time.now) DB[time.strftime("%Y/%m/%d/%H/%M.obj")] = data end # ...
  171. Creating a Database #!/usr/bin/env ruby -wKU ! require "fsdb" !

    module TimeSeries DB = FSDB::Database.new("server_stats/") ! module_function def record(data, time = Time.now) DB[time.strftime("%Y/%m/%d/%H/%M.obj")] = data end # ...
  172. Creating a Database #!/usr/bin/env ruby -wKU ! require "fsdb" !

    module TimeSeries DB = FSDB::Database.new("server_stats/") ! module_function def record(data, time = Time.now) DB[time.strftime("%Y/%m/%d/%H/%M.obj")] = data end # ...
  173. Query

  174. Query # ... def sum(year, *args, &block) total = 0

    path = [year, *args][0..5].join("/").gsub(/\b\d\b/, '0\0') if File.extname(path) == ".obj" total += block[DB[path]] else (DB[path] || []).each do |new_path| total += sum(File.join(path, new_path), &block) end end total end def average(*args) count = 0 sum(*args) { |data| count += 1; yield data } / count.to_f end end ! # ...
  175. Query # ... def sum(year, *args, &block) total = 0

    path = [year, *args][0..5].join("/").gsub(/\b\d\b/, '0\0') if File.extname(path) == ".obj" total += block[DB[path]] else (DB[path] || []).each do |new_path| total += sum(File.join(path, new_path), &block) end end total end def average(*args) count = 0 sum(*args) { |data| count += 1; yield data } / count.to_f end end ! # ...
  176. Query # ... def sum(year, *args, &block) total = 0

    path = [year, *args][0..5].join("/").gsub(/\b\d\b/, '0\0') if File.extname(path) == ".obj" total += block[DB[path]] else (DB[path] || []).each do |new_path| total += sum(File.join(path, new_path), &block) end end total end def average(*args) count = 0 sum(*args) { |data| count += 1; yield data } / count.to_f end end ! # ...
  177. Query # ... def sum(year, *args, &block) total = 0

    path = [year, *args][0..5].join("/").gsub(/\b\d\b/, '0\0') if File.extname(path) == ".obj" total += block[DB[path]] else (DB[path] || []).each do |new_path| total += sum(File.join(path, new_path), &block) end end total end def average(*args) count = 0 sum(*args) { |data| count += 1; yield data } / count.to_f end end ! # ...
  178. sample Usage

  179. sample Usage # ... ! if __FILE__ == $PROGRAM_NAME include

    TimeSeries record( { :load_average => 70, :disk_free => 50 }, Time.local(2008, 9, 4, 12, 0) ) record( { :load_average => 10, :disk_free => 50 }, Time.local(2008, 9, 4, 12, 30) ) record( { :load_average => 20, :disk_free => 51 }, Time.local(2008, 9, 4, 13, 0) ) p average(2008, 9, 4) { |data| data[:load_average] } # >> 33.3333333333333 p average(2008, 9, 4, 12) { |data| data[:load_average] } # >> 40.0 end
  180. Other Nice Features

  181. Other Nice Features ‣FSDB is multi-thread and multi-process safe on

    most platforms
  182. Other Nice Features ‣FSDB is multi-thread and multi-process safe on

    most platforms ‣It supports read only and read/write transactions and they can even be nested
  183. Other Nice Features ‣FSDB is multi-thread and multi-process safe on

    most platforms ‣It supports read only and read/write transactions and they can even be nested ‣You can define your own formats for files
  184. Transactions

  185. Transactions #!/usr/bin/env ruby -wKU ! require "fsdb" db = FSDB::Database.new("server_stats/")

    ! # a read only transaction (shared lock) db.browse "2008/09/04/12/00.obj" do |data| p data[:load_average] # >> 70 p data[:disk_free] # >> 50 end ! # a read/write transaction (exclusive lock) db.replace "2008/09/04/12/00.obj" do |data| data.merge(:uptime => 21 * 60) end ! p db["2008/09/04/12/00.obj"][:uptime] # >> 1260
  186. Transactions #!/usr/bin/env ruby -wKU ! require "fsdb" db = FSDB::Database.new("server_stats/")

    ! # a read only transaction (shared lock) db.browse "2008/09/04/12/00.obj" do |data| p data[:load_average] # >> 70 p data[:disk_free] # >> 50 end ! # a read/write transaction (exclusive lock) db.replace "2008/09/04/12/00.obj" do |data| data.merge(:uptime => 21 * 60) end ! p db["2008/09/04/12/00.obj"][:uptime] # >> 1260
  187. Transactions #!/usr/bin/env ruby -wKU ! require "fsdb" db = FSDB::Database.new("server_stats/")

    ! # a read only transaction (shared lock) db.browse "2008/09/04/12/00.obj" do |data| p data[:load_average] # >> 70 p data[:disk_free] # >> 50 end ! # a read/write transaction (exclusive lock) db.replace "2008/09/04/12/00.obj" do |data| data.merge(:uptime => 21 * 60) end ! p db["2008/09/04/12/00.obj"][:uptime] # >> 1260
  188. A Custom Format

  189. A Custom Format #!/usr/bin/env ruby -wKU ! require "fsdb" db

    = FSDB::Database.new("images/") ! PNG_FORMAT = FSDB::Format.new( /\.png\z/i, :binary, :name => "PNG_FORMAT", :load => lambda { |f| f.seek(16) w, h = f.read(8).unpack("N2") {:width => w, :height => h} }, :dump => lambda { raise "Read only format." } ) db.formats = [PNG_FORMAT] ! db.browse_each_child "/" do |image, details| puts "%s: %p" % [image, details] end # >> /fsdb_tree.png: {:width=>727, :height=>249} # >> /ip_to_country_quiz.png: {:width=>798, :height=>732} # >> /scout.png: {:width=>865, :height=>713}
  190. A Custom Format #!/usr/bin/env ruby -wKU ! require "fsdb" db

    = FSDB::Database.new("images/") ! PNG_FORMAT = FSDB::Format.new( /\.png\z/i, :binary, :name => "PNG_FORMAT", :load => lambda { |f| f.seek(16) w, h = f.read(8).unpack("N2") {:width => w, :height => h} }, :dump => lambda { raise "Read only format." } ) db.formats = [PNG_FORMAT] ! db.browse_each_child "/" do |image, details| puts "%s: %p" % [image, details] end # >> /fsdb_tree.png: {:width=>727, :height=>249} # >> /ip_to_country_quiz.png: {:width=>798, :height=>732} # >> /scout.png: {:width=>865, :height=>713}
  191. A Custom Format #!/usr/bin/env ruby -wKU ! require "fsdb" db

    = FSDB::Database.new("images/") ! PNG_FORMAT = FSDB::Format.new( /\.png\z/i, :binary, :name => "PNG_FORMAT", :load => lambda { |f| f.seek(16) w, h = f.read(8).unpack("N2") {:width => w, :height => h} }, :dump => lambda { raise "Read only format." } ) db.formats = [PNG_FORMAT] ! db.browse_each_child "/" do |image, details| puts "%s: %p" % [image, details] end # >> /fsdb_tree.png: {:width=>727, :height=>249} # >> /ip_to_country_quiz.png: {:width=>798, :height=>732} # >> /scout.png: {:width=>865, :height=>713}
  192. Dirt Simple IPC

  193. Dirt Simple IPC Rinda

  194. The Importance of Networking

  195. The Importance of Networking ‣When you need more processing power,

    you have to start hooking CPU’s together
  196. The Importance of Networking ‣When you need more processing power,

    you have to start hooking CPU’s together ‣Rinda can make the communication between processes a snap
  197. Problem: Descrambling

  198. Problem: Descrambling ‣Given some scrambled letters, find all dictionary words

    that can be formed by rearranging those letters
  199. Problem: Descrambling ‣Given some scrambled letters, find all dictionary words

    that can be formed by rearranging those letters ‣This task represents any task that just needs some processing time to sort out
  200. Problem: Descrambling ‣Given some scrambled letters, find all dictionary words

    that can be formed by rearranging those letters ‣This task represents any task that just needs some processing time to sort out ‣All this requires is some I/O and simple comparisons
  201. The Trivial Solution

  202. The Trivial Solution #!/usr/bin/env ruby -wKU ! class String def

    signature strip.downcase.delete("^a-z").split("").sort.join end end ! pattern = ARGV.shift.signature descrambled = [ ] ! File.foreach("/usr/share/dict/words") do |word| descrambled << word if word.signature == pattern end ! puts descrambled
  203. The Trivial Solution #!/usr/bin/env ruby -wKU ! class String def

    signature strip.downcase.delete("^a-z").split("").sort.join end end ! pattern = ARGV.shift.signature descrambled = [ ] ! File.foreach("/usr/share/dict/words") do |word| descrambled << word if word.signature == pattern end ! puts descrambled
  204. Dividing the Work

  205. Dividing the Work ‣Heavy processing almost always benefits from more

    processes doing the work
  206. Dividing the Work ‣Heavy processing almost always benefits from more

    processes doing the work ‣This is true multiprocessing, unlike Ruby’s thread model
  207. Dividing the Work ‣Heavy processing almost always benefits from more

    processes doing the work ‣This is true multiprocessing, unlike Ruby’s thread model ‣Rinda’s TupleSpace makes the Inter- Process Communication super easy
  208. Dividing the Work ‣Heavy processing almost always benefits from more

    processes doing the work ‣This is true multiprocessing, unlike Ruby’s thread model ‣Rinda’s TupleSpace makes the Inter- Process Communication super easy ‣This task is I/O bound, but it still halved the time to put four processes on it
  209. Setup

  210. Setup #!/usr/bin/env ruby -wKU ! require "rinda/tuplespace" ! class String

    def signature strip.downcase.delete("^a-z").split("").sort.join end end ! DICT = "/usr/share/dict/words" workers = ARGV.first =~ /\A\d+\z/ ? ARGV.shift.to_i : 4 pattern = ARGV.shift.signature chunk_size = File.stat(DICT).size / workers ! # ...
  211. Spawn Workers

  212. Spawn Workers # ... ! workers.times do |n| fork do

    descrambled = [ ] File.open(DICT) do |words| my_start = chunk_size * n my_end = my_start + chunk_size words.seek(my_start) words.gets unless my_start.zero? words.each do |word| descrambled << word if word.signature == pattern break if words.pos > my_end end end ! results = Rinda::TupleSpaceProxy.new( DRbObject.new_with_uri("druby://localhost:61676") ) results.write([pattern, descrambled]) end end ! # ...
  213. Spawn Workers # ... ! workers.times do |n| fork do

    descrambled = [ ] File.open(DICT) do |words| my_start = chunk_size * n my_end = my_start + chunk_size words.seek(my_start) words.gets unless my_start.zero? words.each do |word| descrambled << word if word.signature == pattern break if words.pos > my_end end end ! results = Rinda::TupleSpaceProxy.new( DRbObject.new_with_uri("druby://localhost:61676") ) results.write([pattern, descrambled]) end end ! # ...
  214. Collect Results

  215. Collect Results # ... ! results = Rinda::TupleSpace.new DRb.start_service("druby://localhost:61676", results)

    workers.times do descrambled = results.take([/\b#{Regexp.escape(pattern)}\b/, Array]) puts descrambled.last end Process.waitall
  216. Collect Results # ... ! results = Rinda::TupleSpace.new DRb.start_service("druby://localhost:61676", results)

    workers.times do descrambled = results.take([/\b#{Regexp.escape(pattern)}\b/, Array]) puts descrambled.last end Process.waitall
  217. Collect Results # ... ! results = Rinda::TupleSpace.new DRb.start_service("druby://localhost:61676", results)

    workers.times do descrambled = results.take([/\b#{Regexp.escape(pattern)}\b/, Array]) puts descrambled.last end Process.waitall
  218. Other Nice Features

  219. Other Nice Features ‣You can set expiration times for tuples

    added to a TupleSpace
  220. Other Nice Features ‣You can set expiration times for tuples

    added to a TupleSpace ‣Rinda also comes with a RingServer for zero configuration networking
  221. Using RingServer

  222. Using RingServer #!/usr/bin/env ruby -wKU ! require "rinda/ring" # for

    RingServer require "rinda/tuplespace" # for TupleSpace ! # start a RingServer DRb.start_service Rinda::RingServer.new(Rinda::TupleSpace.new) ! # ...
  223. Using RingServer #!/usr/bin/env ruby -wKU ! require "rinda/ring" # for

    RingServer require "rinda/tuplespace" # for TupleSpace ! # start a RingServer DRb.start_service Rinda::RingServer.new(Rinda::TupleSpace.new) ! # ...
  224. Using RingServer #!/usr/bin/env ruby -wKU ! require "rinda/ring" # for

    RingServer require "rinda/tuplespace" # for TupleSpace ! # start a RingServer DRb.start_service Rinda::RingServer.new(Rinda::TupleSpace.new) ! # ... #!/usr/bin/env ruby -wKU ! require "rinda/ring" # for RingFinger require "rinda/tuplespace" # for TupleSpace ! # find a RingServer DRb.start_service ring_server = Rinda::RingFinger.primary ! # ...
  225. Questions?