Tweet
Tweet

Slide 1

Slide 1 text

Ruby in the Fast Lane Hidden Gems

Slide 2

Slide 2 text

Back at Lone Star

Slide 3

Slide 3 text

Back at Lone Star ‣I am James Edward Gray II

Slide 4

Slide 4 text

Back at Lone Star ‣I am James Edward Gray II ‣Created the Ruby Quiz

Slide 5

Slide 5 text

Back at Lone Star ‣I am James Edward Gray II ‣Created the Ruby Quiz ‣released FasterCSV, HighLine, and Elif

Slide 6

Slide 6 text

Back at Lone Star ‣I am James Edward Gray II ‣Created the Ruby Quiz ‣released FasterCSV, HighLine, and Elif ‣Wrote a couple of Pragmatic Programmer books with a lot of Ruby in them

Slide 7

Slide 7 text

Back at Lone Star ‣I am James Edward Gray II ‣Created the Ruby Quiz ‣released FasterCSV, HighLine, and Elif ‣Wrote a couple of Pragmatic Programmer books with a lot of Ruby in them ‣I’ve given a talk at every LSRC so far

Slide 8

Slide 8 text

No content

Slide 9

Slide 9 text

No content

Slide 10

Slide 10 text

Why BSG?

Slide 11

Slide 11 text

Why BSG? ‣The old ship that was to be decommissioned becomes the only thing keeping the human race alive

Slide 12

Slide 12 text

Why BSG? ‣The old ship that was to be decommissioned becomes the only thing keeping the human race alive ‣Those are my kind of odds

Slide 13

Slide 13 text

Ugly Rumor

Slide 14

Slide 14 text

“Ruby is slow.” Ugly Rumor

Slide 15

Slide 15 text

My Opinion of the Speed Rumor

Slide 16

Slide 16 text

BS My Opinion of the Speed Rumor

Slide 17

Slide 17 text

Ruby is as Fast as we Want her to Be!

Slide 18

Slide 18 text

Ruby is as Fast as we Want her to Be! ‣We can always write C extensions for speed-critical portions

Slide 19

Slide 19 text

Ruby is as Fast as we Want her to Be! ‣We can always write C extensions for speed-critical portions ‣This is rarely actually needed though

Slide 20

Slide 20 text

Ruby is as Fast as we Want her to Be! ‣We can always write C extensions for speed-critical portions ‣This is rarely actually needed though ‣We can make use of libraries, some written in C, that help with our problem

Slide 21

Slide 21 text

Ruby is as Fast as we Want her to Be! ‣We can always write C extensions for speed-critical portions ‣This is rarely actually needed though ‣We can make use of libraries, some written in C, that help with our problem ‣We can add more processing power

Slide 22

Slide 22 text

Ruby is as Fast as we Want her to Be! ‣We can always write C extensions for speed-critical portions ‣This is rarely actually needed though ‣We can make use of libraries, some written in C, that help with our problem ‣We can add more processing power ‣We can rework our data structures to better support the task at hand

Slide 23

Slide 23 text

Ruby is as Fast as we Want her to Be! ‣We can always write C extensions for speed-critical portions ‣This is rarely actually needed though ‣We can make use of libraries, some written in C, that help with our problem ‣We can add more processing power ‣We can rework our data structures to better support the task at hand ‣Always the big win, in my opinion

Slide 24

Slide 24 text

Ruby in the Fast Lane

Slide 25

Slide 25 text

Ruby in the Fast Lane ‣Let’s see how fast Ruby can run using:

Slide 26

Slide 26 text

Ruby in the Fast Lane ‣Let’s see how fast Ruby can run using: ‣NArray

Slide 27

Slide 27 text

Ruby in the Fast Lane ‣Let’s see how fast Ruby can run using: ‣NArray ‣SQLite

Slide 28

Slide 28 text

Ruby in the Fast Lane ‣Let’s see how fast Ruby can run using: ‣NArray ‣SQLite ‣RBTree

Slide 29

Slide 29 text

Ruby in the Fast Lane ‣Let’s see how fast Ruby can run using: ‣NArray ‣SQLite ‣RBTree ‣FSDB

Slide 30

Slide 30 text

Ruby in the Fast Lane ‣Let’s see how fast Ruby can run using: ‣NArray ‣SQLite ‣RBTree ‣FSDB ‣Rinda

Slide 31

Slide 31 text

Ruby in the Fast Lane ‣Let’s see how fast Ruby can run using: ‣NArray ‣SQLite ‣RBTree ‣FSDB ‣Rinda ‣Thinking outside the box

Slide 32

Slide 32 text

Super Fast Number crunching

Slide 33

Slide 33 text

Super Fast Number crunching NArray

Slide 34

Slide 34 text

When the Numbers Count

Slide 35

Slide 35 text

When the Numbers Count ‣Ruby’s Numeric family of objects were built for ease of use

Slide 36

Slide 36 text

When the Numbers Count ‣Ruby’s Numeric family of objects were built for ease of use ‣This makes them a bit slower

Slide 37

Slide 37 text

When the Numbers Count ‣Ruby’s Numeric family of objects were built for ease of use ‣This makes them a bit slower ‣C’s numbers were built for speed

Slide 38

Slide 38 text

When the Numbers Count ‣Ruby’s Numeric family of objects were built for ease of use ‣This makes them a bit slower ‣C’s numbers were built for speed ‣Ruby can borrow them with NArray

Slide 39

Slide 39 text

Problem: Faster Imaging

Slide 40

Slide 40 text

Problem: Faster Imaging ‣I use a trivial PPM library to generate images for one project

Slide 41

Slide 41 text

Problem: Faster Imaging ‣I use a trivial PPM library to generate images for one project ‣The PPM code was taking about 1.3 seconds for a 400 by 200 pixel image

Slide 42

Slide 42 text

Problem: Faster Imaging ‣I use a trivial PPM library to generate images for one project ‣The PPM code was taking about 1.3 seconds for a 400 by 200 pixel image ‣I replaced a two dimensional Array of Color objects with a 3D NArray

Slide 43

Slide 43 text

Problem: Faster Imaging ‣I use a trivial PPM library to generate images for one project ‣The PPM code was taking about 1.3 seconds for a 400 by 200 pixel image ‣I replaced a two dimensional Array of Color objects with a 3D NArray ‣I changed less than ten lines of code

Slide 44

Slide 44 text

Problem: Faster Imaging ‣I use a trivial PPM library to generate images for one project ‣The PPM code was taking about 1.3 seconds for a 400 by 200 pixel image ‣I replaced a two dimensional Array of Color objects with a 3D NArray ‣I changed less than ten lines of code ‣The speed on the same image dropped to about 1/100th of a second

Slide 45

Slide 45 text

Creating The Canvas

Slide 46

Slide 46 text

Creating The Canvas def initialize(options = Hash.new) options = DEFAULT_OPTIONS.merge(options) @width = options[:width] @height = options[:height] @background = options[:background] @foreground = options[:foreground] @mode = options[:mode] @canvas = Array.new(@height) { Array.new(@width) { @background } } end

Slide 47

Slide 47 text

Creating The Canvas def initialize(options = Hash.new) options = DEFAULT_OPTIONS.merge(options) @width = options[:width] @height = options[:height] @background = options[:background] @foreground = options[:foreground] @mode = options[:mode] @canvas = Array.new(@height) { Array.new(@width) { @background } } end

Slide 48

Slide 48 text

Creating The Canvas def initialize(options = Hash.new) require "rubygems" require "narray" ! options = DEFAULT_OPTIONS.merge(options) @width = options[:width] @height = options[:height] @background = options[:background] @foreground = options[:foreground] @mode = options[:mode] @canvas = NArray.byte(@width, @height, 3) end

Slide 49

Slide 49 text

Creating The Canvas def initialize(options = Hash.new) require "rubygems" require "narray" ! options = DEFAULT_OPTIONS.merge(options) @width = options[:width] @height = options[:height] @background = options[:background] @foreground = options[:foreground] @mode = options[:mode] @canvas = NArray.byte(@width, @height, 3) end

Slide 50

Slide 50 text

Marking Pixels

Slide 51

Slide 51 text

Marking Pixels def draw_point(x, y, color = @foreground) return unless x.between? 0, @width - 1 return unless y.between? 0, @height - 1 ! @canvas[y][x] = color end

Slide 52

Slide 52 text

Marking Pixels def draw_point(x, y, color = @foreground) return unless x.between? 0, @width - 1 return unless y.between? 0, @height - 1 ! @canvas[y][x] = color end

Slide 53

Slide 53 text

Marking Pixels def draw_point(x, y, color = @foreground) return unless x.between? 0, @width - 1 return unless y.between? 0, @height - 1 ! @canvas[x, y, 0..2] = color.to_a end

Slide 54

Slide 54 text

Marking Pixels def draw_point(x, y, color = @foreground) return unless x.between? 0, @width - 1 return unless y.between? 0, @height - 1 ! @canvas[x, y, 0..2] = color.to_a end

Slide 55

Slide 55 text

Drawing an Image

Slide 56

Slide 56 text

Drawing an Image def save(file) File.open(file.sub(/\.ppm$/i, "") + ".ppm", "w") do |image| image.puts @mode image.puts "#{@width} #{@height} 255" @canvas.each do |row| pixels = row.map { |pixel| pixel.to_s(@mode) } image.send( @mode == "P6" ? :print : :puts, pixels.join(@mode == "P6" ? "" : " ") ) end end end

Slide 57

Slide 57 text

Drawing an Image def save(file) File.open(file.sub(/\.ppm$/i, "") + ".ppm", "w") do |image| image.puts @mode image.puts "#{@width} #{@height} 255" @canvas.each do |row| pixels = row.map { |pixel| pixel.to_s(@mode) } image.send( @mode == "P6" ? :print : :puts, pixels.join(@mode == "P6" ? "" : " ") ) end end end

Slide 58

Slide 58 text

Drawing an Image def save(file) File.open(file.sub(/\.ppm$/i, "") + ".ppm", "w") do |image| image.puts @mode image.puts "#{@width} #{@height} 255" 0.upto(@height - 1) do |y| row = @canvas[0...@width, y, 0..2].transpose(-1, 0) image.send( @mode == "P6" ? :print : :puts, @mode == "P6" ? row.to_s : row.to_a.join(" ") ) end end end

Slide 59

Slide 59 text

Drawing an Image def save(file) File.open(file.sub(/\.ppm$/i, "") + ".ppm", "w") do |image| image.puts @mode image.puts "#{@width} #{@height} 255" 0.upto(@height - 1) do |y| row = @canvas[0...@width, y, 0..2].transpose(-1, 0) image.send( @mode == "P6" ? :print : :puts, @mode == "P6" ? row.to_s : row.to_a.join(" ") ) end end end

Slide 60

Slide 60 text

Other Nice Features

Slide 61

Slide 61 text

Other Nice Features ‣NArray supports integers, floats, and even complex numbers in various sizes

Slide 62

Slide 62 text

Other Nice Features ‣NArray supports integers, floats, and even complex numbers in various sizes ‣Aside from indexing and iteration, NArray supports data generation, arithmetic operations, comparisons, bitwise manipulations, statistic calculations, and more

Slide 63

Slide 63 text

Other Nice Features ‣NArray supports integers, floats, and even complex numbers in various sizes ‣Aside from indexing and iteration, NArray supports data generation, arithmetic operations, comparisons, bitwise manipulations, statistic calculations, and more ‣View the large API with examples At: http://narray.rubyforge.org/

Slide 64

Slide 64 text

Conway’s Game of Life

Slide 65

Slide 65 text

Conway’s Game of Life #!/usr/bin/env ruby -wKU ! require "rubygems" require "narray" ! # build cells life = NArray.byte(5, 5) life[1, 1] = NArray.byte(3, 3).random!(2) p life ! # count neighbors counts = NArray.byte(*life.shape) counts[1..-2, 1..-2] = life[0..-3, 0..-3] + life[0..-3, 1..-2] + life[0..-3, 2..-1] + life[1..-2, 0..-3] + life[1..-2, 2..-1] + life[2..-1, 0..-3] + life[2..-1, 1..-2] + life[2..-1, 2..-1] p counts ! # one step of the game life[] = counts.eq(3) | (counts.eq(2) & life) p life

Slide 66

Slide 66 text

Conway’s Game of Life #!/usr/bin/env ruby -wKU ! require "rubygems" require "narray" ! # build cells life = NArray.byte(5, 5) life[1, 1] = NArray.byte(3, 3).random!(2) p life ! # count neighbors counts = NArray.byte(*life.shape) counts[1..-2, 1..-2] = life[0..-3, 0..-3] + life[0..-3, 1..-2] + life[0..-3, 2..-1] + life[1..-2, 0..-3] + life[1..-2, 2..-1] + life[2..-1, 0..-3] + life[2..-1, 1..-2] + life[2..-1, 2..-1] p counts ! # one step of the game life[] = counts.eq(3) | (counts.eq(2) & life) p life

Slide 67

Slide 67 text

Conway’s Game of Life NArray.byte(5,5): [ [ 0, 0, 0, 0, 0 ], [ 0, 1, 0, 1, 0 ], [ 0, 0, 0, 0, 0 ], [ 0, 1, 1, 0, 0 ], [ 0, 0, 0, 0, 0 ] ] #!/usr/bin/env ruby -wKU ! require "rubygems" require "narray" ! # build cells life = NArray.byte(5, 5) life[1, 1] = NArray.byte(3, 3).random!(2) p life ! # count neighbors counts = NArray.byte(*life.shape) counts[1..-2, 1..-2] = life[0..-3, 0..-3] + life[0..-3, 1..-2] + life[0..-3, 2..-1] + life[1..-2, 0..-3] + life[1..-2, 2..-1] + life[2..-1, 0..-3] + life[2..-1, 1..-2] + life[2..-1, 2..-1] p counts ! # one step of the game life[] = counts.eq(3) | (counts.eq(2) & life) p life

Slide 68

Slide 68 text

Conway’s Game of Life NArray.byte(5,5): [ [ 0, 0, 0, 0, 0 ], [ 0, 1, 0, 1, 0 ], [ 0, 0, 0, 0, 0 ], [ 0, 1, 1, 0, 0 ], [ 0, 0, 0, 0, 0 ] ] #!/usr/bin/env ruby -wKU ! require "rubygems" require "narray" ! # build cells life = NArray.byte(5, 5) life[1, 1] = NArray.byte(3, 3).random!(2) p life ! # count neighbors counts = NArray.byte(*life.shape) counts[1..-2, 1..-2] = life[0..-3, 0..-3] + life[0..-3, 1..-2] + life[0..-3, 2..-1] + life[1..-2, 0..-3] + life[1..-2, 2..-1] + life[2..-1, 0..-3] + life[2..-1, 1..-2] + life[2..-1, 2..-1] p counts ! # one step of the game life[] = counts.eq(3) | (counts.eq(2) & life) p life

Slide 69

Slide 69 text

Conway’s Game of Life NArray.byte(5,5): [ [ 0, 0, 0, 0, 0 ], [ 0, 1, 0, 1, 0 ], [ 0, 0, 0, 0, 0 ], [ 0, 1, 1, 0, 0 ], [ 0, 0, 0, 0, 0 ] ] NArray.byte(5,5): [ [ 0, 0, 0, 0, 0 ], [ 0, 0, 2, 0, 0 ], [ 0, 3, 4, 2, 0 ], [ 0, 1, 1, 1, 0 ], [ 0, 0, 0, 0, 0 ] ] #!/usr/bin/env ruby -wKU ! require "rubygems" require "narray" ! # build cells life = NArray.byte(5, 5) life[1, 1] = NArray.byte(3, 3).random!(2) p life ! # count neighbors counts = NArray.byte(*life.shape) counts[1..-2, 1..-2] = life[0..-3, 0..-3] + life[0..-3, 1..-2] + life[0..-3, 2..-1] + life[1..-2, 0..-3] + life[1..-2, 2..-1] + life[2..-1, 0..-3] + life[2..-1, 1..-2] + life[2..-1, 2..-1] p counts ! # one step of the game life[] = counts.eq(3) | (counts.eq(2) & life) p life

Slide 70

Slide 70 text

Conway’s Game of Life NArray.byte(5,5): [ [ 0, 0, 0, 0, 0 ], [ 0, 1, 0, 1, 0 ], [ 0, 0, 0, 0, 0 ], [ 0, 1, 1, 0, 0 ], [ 0, 0, 0, 0, 0 ] ] NArray.byte(5,5): [ [ 0, 0, 0, 0, 0 ], [ 0, 0, 2, 0, 0 ], [ 0, 3, 4, 2, 0 ], [ 0, 1, 1, 1, 0 ], [ 0, 0, 0, 0, 0 ] ] #!/usr/bin/env ruby -wKU ! require "rubygems" require "narray" ! # build cells life = NArray.byte(5, 5) life[1, 1] = NArray.byte(3, 3).random!(2) p life ! # count neighbors counts = NArray.byte(*life.shape) counts[1..-2, 1..-2] = life[0..-3, 0..-3] + life[0..-3, 1..-2] + life[0..-3, 2..-1] + life[1..-2, 0..-3] + life[1..-2, 2..-1] + life[2..-1, 0..-3] + life[2..-1, 1..-2] + life[2..-1, 2..-1] p counts ! # one step of the game life[] = counts.eq(3) | (counts.eq(2) & life) p life

Slide 71

Slide 71 text

Conway’s Game of Life NArray.byte(5,5): [ [ 0, 0, 0, 0, 0 ], [ 0, 1, 0, 1, 0 ], [ 0, 0, 0, 0, 0 ], [ 0, 1, 1, 0, 0 ], [ 0, 0, 0, 0, 0 ] ] NArray.byte(5,5): [ [ 0, 0, 0, 0, 0 ], [ 0, 0, 2, 0, 0 ], [ 0, 3, 4, 2, 0 ], [ 0, 1, 1, 1, 0 ], [ 0, 0, 0, 0, 0 ] ] NArray.byte(5,5): [ [ 0, 0, 0, 0, 0 ], [ 0, 0, 0, 0, 0 ], [ 0, 1, 0, 0, 0 ], [ 0, 0, 0, 0, 0 ], [ 0, 0, 0, 0, 0 ] ] #!/usr/bin/env ruby -wKU ! require "rubygems" require "narray" ! # build cells life = NArray.byte(5, 5) life[1, 1] = NArray.byte(3, 3).random!(2) p life ! # count neighbors counts = NArray.byte(*life.shape) counts[1..-2, 1..-2] = life[0..-3, 0..-3] + life[0..-3, 1..-2] + life[0..-3, 2..-1] + life[1..-2, 0..-3] + life[1..-2, 2..-1] + life[2..-1, 0..-3] + life[2..-1, 1..-2] + life[2..-1, 2..-1] p counts ! # one step of the game life[] = counts.eq(3) | (counts.eq(2) & life) p life

Slide 72

Slide 72 text

A Data DSL

Slide 73

Slide 73 text

A Data DSL SQLite

Slide 74

Slide 74 text

Thinking About Data can be Hard

Slide 75

Slide 75 text

Thinking About Data can be Hard ‣SQLite has already solved many hard problems for data storage and retrieval

Slide 76

Slide 76 text

Thinking About Data can be Hard ‣SQLite has already solved many hard problems for data storage and retrieval ‣It gives you an entire language to express your data needs

Slide 77

Slide 77 text

Problem: IP to Country

Slide 78

Slide 78 text

Problem: IP to Country ‣Given an IP address, return the country for that IP

Slide 79

Slide 79 text

Problem: IP to Country ‣Given an IP address, return the country for that IP ‣this was a Ruby Quiz

Slide 80

Slide 80 text

Problem: IP to Country ‣Given an IP address, return the country for that IP ‣this was a Ruby Quiz ‣Solutions were also expected to be efficient in memory and speed

Slide 81

Slide 81 text

Problem: IP to Country ‣Given an IP address, return the country for that IP ‣this was a Ruby Quiz ‣Solutions were also expected to be efficient in memory and speed ‣This is a real world task I’ve had to do for my job

Slide 82

Slide 82 text

The Data

Slide 83

Slide 83 text

The Data # © 2002-2008 Webnet77.com # # # # "0","16777215","IANA","410227200","ZZ","ZZZ","RESERVED" "50331648","67108863","ARIN","572572800","US","USA","UNITED STATES" "67108864","83886079","ARIN","0","US","USA","UNITED STATES" "100663296","117440511","ARIN","0","US","USA","UNITED STATES" "117440512","134217727","ARIN","880329600","US","USA","UNITED STATES"

Slide 84

Slide 84 text

The Data # © 2002-2008 Webnet77.com # # # # "0","16777215","IANA","410227200","ZZ","ZZZ","RESERVED" "50331648","67108863","ARIN","572572800","US","USA","UNITED STATES" "67108864","83886079","ARIN","0","US","USA","UNITED STATES" "100663296","117440511","ARIN","0","US","USA","UNITED STATES" "117440512","134217727","ARIN","880329600","US","USA","UNITED STATES"

Slide 85

Slide 85 text

Solutions

Slide 86

Slide 86 text

Solutions ‣Many solved the problem with a binary search on the file

Slide 87

Slide 87 text

Solutions ‣Many solved the problem with a binary search on the file ‣Most of those prepossessed the file to make that search easier

Slide 88

Slide 88 text

Solutions ‣Many solved the problem with a binary search on the file ‣Most of those prepossessed the file to make that search easier ‣I’m going to show a SQLite solution

Slide 89

Slide 89 text

Solutions ‣Many solved the problem with a binary search on the file ‣Most of those prepossessed the file to make that search easier ‣I’m going to show a SQLite solution ‣It’s very close to the same speed (about 1/3rd of a second to lookup an IP)

Slide 90

Slide 90 text

Solutions ‣Many solved the problem with a binary search on the file ‣Most of those prepossessed the file to make that search easier ‣I’m going to show a SQLite solution ‣It’s very close to the same speed (about 1/3rd of a second to lookup an IP) ‣I didn’t have to be clever or even add an index

Slide 91

Slide 91 text

Solutions ‣Many solved the problem with a binary search on the file ‣Most of those prepossessed the file to make that search easier ‣I’m going to show a SQLite solution ‣It’s very close to the same speed (about 1/3rd of a second to lookup an IP) ‣I didn’t have to be clever or even add an index ‣It was easier for me to use full country names

Slide 92

Slide 92 text

Setup

Slide 93

Slide 93 text

Setup #!/usr/bin/env ruby -KU ! require "open-uri" require "zlib" ! require "rubygems" require "faster_csv" require "sqlite3" ! REMOTE_DB = "http://software77.net/cgi-bin/" + "ip-country/geo-ip.pl?action=download" LOCAL_DB = "country_ips.sqlite" ! File.unlink(LOCAL_DB) if ARGV.delete("-r") and File.exist? LOCAL_DB ! # ...

Slide 94

Slide 94 text

Build The Database

Slide 95

Slide 95 text

Build The Database # ... ! unless File.exist? LOCAL_DB db = SQLite3::Database.new(LOCAL_DB) db.execute(<<-END_TABLE.strip) CREATE TABLE ips ( low_ip INTEGER, high_ip INTEGER, country TEXT ) END_TABLE open(REMOTE_DB) do |url| Zlib::GzipReader.new(url).each do |line| next if line =~ /\A\s*(?:#|\z)/ args = FCSV.parse_line(line).values_at(0..1, 6) db.execute(<<-END_INSERT.strip, *args) INSERT INTO ips( low_ip, high_ip, country) VALUES( ?, ?, ?) END_INSERT end end db.close end ! # ...

Slide 96

Slide 96 text

Build The Database # ... ! unless File.exist? LOCAL_DB db = SQLite3::Database.new(LOCAL_DB) db.execute(<<-END_TABLE.strip) CREATE TABLE ips ( low_ip INTEGER, high_ip INTEGER, country TEXT ) END_TABLE open(REMOTE_DB) do |url| Zlib::GzipReader.new(url).each do |line| next if line =~ /\A\s*(?:#|\z)/ args = FCSV.parse_line(line).values_at(0..1, 6) db.execute(<<-END_INSERT.strip, *args) INSERT INTO ips( low_ip, high_ip, country) VALUES( ?, ?, ?) END_INSERT end end db.close end ! # ...

Slide 97

Slide 97 text

Build The Database # ... ! unless File.exist? LOCAL_DB db = SQLite3::Database.new(LOCAL_DB) db.execute(<<-END_TABLE.strip) CREATE TABLE ips ( low_ip INTEGER, high_ip INTEGER, country TEXT ) END_TABLE open(REMOTE_DB) do |url| Zlib::GzipReader.new(url).each do |line| next if line =~ /\A\s*(?:#|\z)/ args = FCSV.parse_line(line).values_at(0..1, 6) db.execute(<<-END_INSERT.strip, *args) INSERT INTO ips( low_ip, high_ip, country) VALUES( ?, ?, ?) END_INSERT end end db.close end ! # ...

Slide 98

Slide 98 text

Query

Slide 99

Slide 99 text

Query # ... ! ip = ARGV.shift or abort "Usage: #{File.basename($PROGRAM_NAME)} IP" ip_int = ip.split(".").map { |n| Integer(n) }. pack("C*").unpack("N").first db = SQLite3::Database.new(LOCAL_DB) ! puts db.get_first_value(<<-END_SELECT.strip, :ip => ip_int) || "Unknown" SELECT country FROM ips WHERE low_ip <= :ip AND :ip <= high_ip END_SELECT

Slide 100

Slide 100 text

Query # ... ! ip = ARGV.shift or abort "Usage: #{File.basename($PROGRAM_NAME)} IP" ip_int = ip.split(".").map { |n| Integer(n) }. pack("C*").unpack("N").first db = SQLite3::Database.new(LOCAL_DB) ! puts db.get_first_value(<<-END_SELECT.strip, :ip => ip_int) || "Unknown" SELECT country FROM ips WHERE low_ip <= :ip AND :ip <= high_ip END_SELECT

Slide 101

Slide 101 text

Did You Know?

Slide 102

Slide 102 text

Did You Know? ‣SQLite is totally free

Slide 103

Slide 103 text

Did You Know? ‣SQLite is totally free ‣You can receive query results in a Hash to index by column name and/or convert column values to Ruby objects based on type

Slide 104

Slide 104 text

Did You Know? ‣SQLite is totally free ‣You can receive query results in a Hash to index by column name and/or convert column values to Ruby objects based on type ‣It can work with in-memory databases

Slide 105

Slide 105 text

Did You Know? ‣SQLite is totally free ‣You can receive query results in a Hash to index by column name and/or convert column values to Ruby objects based on type ‣It can work with in-memory databases ‣It can run queries across tables in multiple database files

Slide 106

Slide 106 text

Did You Know? ‣SQLite is totally free ‣You can receive query results in a Hash to index by column name and/or convert column values to Ruby objects based on type ‣It can work with in-memory databases ‣It can run queries across tables in multiple database files ‣You can define SQL functions and aggregates for it in Ruby code

Slide 107

Slide 107 text

Ruby Friendly Data

Slide 108

Slide 108 text

Ruby Friendly Data #!/usr/bin/env ruby -KU ! require "rubygems" require "sqlite3" ! country = ARGV.shift or abort "Usage: #{File.basename($PROGRAM_NAME)} COUNTRY" db = SQLite3::Database.new("country_ips.sqlite") db.results_as_hash = true db.type_translation = true ! db.execute( "SELECT * FROM ips WHERE country LIKE ?", "%#{country}%" ) do |match| low, high = match.values_at("low_ip", "high_ip"). map { |i| [i].pack("N").unpack("C*").join(".") } puts "%s: %15s - %15s" % [match["country"], low, high] end

Slide 109

Slide 109 text

Ruby Friendly Data #!/usr/bin/env ruby -KU ! require "rubygems" require "sqlite3" ! country = ARGV.shift or abort "Usage: #{File.basename($PROGRAM_NAME)} COUNTRY" db = SQLite3::Database.new("country_ips.sqlite") db.results_as_hash = true db.type_translation = true ! db.execute( "SELECT * FROM ips WHERE country LIKE ?", "%#{country}%" ) do |match| low, high = match.values_at("low_ip", "high_ip"). map { |i| [i].pack("N").unpack("C*").join(".") } puts "%s: %15s - %15s" % [match["country"], low, high] end

Slide 110

Slide 110 text

Ruby Friendly Data #!/usr/bin/env ruby -KU ! require "rubygems" require "sqlite3" ! country = ARGV.shift or abort "Usage: #{File.basename($PROGRAM_NAME)} COUNTRY" db = SQLite3::Database.new("country_ips.sqlite") db.results_as_hash = true db.type_translation = true ! db.execute( "SELECT * FROM ips WHERE country LIKE ?", "%#{country}%" ) do |match| low, high = match.values_at("low_ip", "high_ip"). map { |i| [i].pack("N").unpack("C*").join(".") } puts "%s: %15s - %15s" % [match["country"], low, high] end

Slide 111

Slide 111 text

In-Memory

Slide 112

Slide 112 text

In-Memory #!/usr/bin/env ruby -KU ! # ... requires unchanged ... ! REMOTE_DB = "http://software77.net/cgi-bin/" + "ip-country/geo-ip.pl?action=download" ! db = SQLite3::Database.new(":memory:") ! # ... database loading unchanged ... ! stmt = db.prepare(<<-END_SELECT.strip) SELECT country FROM ips WHERE low_ip <= :ip AND :ip <= high_ip END_SELECT loop do print "IP address? " ip = gets.to_s.strip if ip =~ /\S/ ip_int = ip.split(".").map { |n| Integer(n) }. pack("C*").unpack("N").first puts stmt.execute(:ip => ip_int).each { |c| break c[0] } || "Unknown" else break end end

Slide 113

Slide 113 text

In-Memory #!/usr/bin/env ruby -KU ! # ... requires unchanged ... ! REMOTE_DB = "http://software77.net/cgi-bin/" + "ip-country/geo-ip.pl?action=download" ! db = SQLite3::Database.new(":memory:") ! # ... database loading unchanged ... ! stmt = db.prepare(<<-END_SELECT.strip) SELECT country FROM ips WHERE low_ip <= :ip AND :ip <= high_ip END_SELECT loop do print "IP address? " ip = gets.to_s.strip if ip =~ /\S/ ip_int = ip.split(".").map { |n| Integer(n) }. pack("C*").unpack("N").first puts stmt.execute(:ip => ip_int).each { |c| break c[0] } || "Unknown" else break end end

Slide 114

Slide 114 text

In-Memory #!/usr/bin/env ruby -KU ! # ... requires unchanged ... ! REMOTE_DB = "http://software77.net/cgi-bin/" + "ip-country/geo-ip.pl?action=download" ! db = SQLite3::Database.new(":memory:") ! # ... database loading unchanged ... ! stmt = db.prepare(<<-END_SELECT.strip) SELECT country FROM ips WHERE low_ip <= :ip AND :ip <= high_ip END_SELECT loop do print "IP address? " ip = gets.to_s.strip if ip =~ /\S/ ip_int = ip.split(".").map { |n| Integer(n) }. pack("C*").unpack("N").first puts stmt.execute(:ip => ip_int).each { |c| break c[0] } || "Unknown" else break end end

Slide 115

Slide 115 text

In-Memory #!/usr/bin/env ruby -KU ! # ... requires unchanged ... ! REMOTE_DB = "http://software77.net/cgi-bin/" + "ip-country/geo-ip.pl?action=download" ! db = SQLite3::Database.new(":memory:") ! # ... database loading unchanged ... ! stmt = db.prepare(<<-END_SELECT.strip) SELECT country FROM ips WHERE low_ip <= :ip AND :ip <= high_ip END_SELECT loop do print "IP address? " ip = gets.to_s.strip if ip =~ /\S/ ip_int = ip.split(".").map { |n| Integer(n) }. pack("C*").unpack("N").first puts stmt.execute(:ip => ip_int).each { |c| break c[0] } || "Unknown" else break end end

Slide 116

Slide 116 text

Attach and Functions

Slide 117

Slide 117 text

Attach and Functions #!/usr/bin/env ruby -KU ! require "rubygems" require "sqlite3" ! user = ARGV.shift or abort "Usage: #{File.basename($PROGRAM_NAME)} USER" db = SQLite3::Database.new("users.sqlite") db.execute("ATTACH DATABASE 'country_ips.sqlite' AS country_ips") ! db.create_function("IP2INT", 1) do |func, ip| func.result = ip.to_s.split(".").map { |n| Integer(n) }. pack("C*").unpack("N").first end ! sql = <

Slide 118

Slide 118 text

Attach and Functions #!/usr/bin/env ruby -KU ! require "rubygems" require "sqlite3" ! user = ARGV.shift or abort "Usage: #{File.basename($PROGRAM_NAME)} USER" db = SQLite3::Database.new("users.sqlite") db.execute("ATTACH DATABASE 'country_ips.sqlite' AS country_ips") ! db.create_function("IP2INT", 1) do |func, ip| func.result = ip.to_s.split(".").map { |n| Integer(n) }. pack("C*").unpack("N").first end ! sql = <

Slide 119

Slide 119 text

Attach and Functions #!/usr/bin/env ruby -KU ! require "rubygems" require "sqlite3" ! user = ARGV.shift or abort "Usage: #{File.basename($PROGRAM_NAME)} USER" db = SQLite3::Database.new("users.sqlite") db.execute("ATTACH DATABASE 'country_ips.sqlite' AS country_ips") ! db.create_function("IP2INT", 1) do |func, ip| func.result = ip.to_s.split(".").map { |n| Integer(n) }. pack("C*").unpack("N").first end ! sql = <

Slide 120

Slide 120 text

Attach and Functions #!/usr/bin/env ruby -KU ! require "rubygems" require "sqlite3" ! user = ARGV.shift or abort "Usage: #{File.basename($PROGRAM_NAME)} USER" db = SQLite3::Database.new("users.sqlite") db.execute("ATTACH DATABASE 'country_ips.sqlite' AS country_ips") ! db.create_function("IP2INT", 1) do |func, ip| func.result = ip.to_s.split(".").map { |n| Integer(n) }. pack("C*").unpack("N").first end ! sql = <

Slide 121

Slide 121 text

A Binary Tree

Slide 122

Slide 122 text

A Binary Tree RBTree

Slide 123

Slide 123 text

Sometimes you Just Need the big Guns

Slide 124

Slide 124 text

Sometimes you Just Need the big Guns ‣Binary search and binary trees are pretty big guns in computing

Slide 125

Slide 125 text

Sometimes you Just Need the big Guns ‣Binary search and binary trees are pretty big guns in computing ‣RBTree provides a super efficient binary tree implementation

Slide 126

Slide 126 text

Sometimes you Just Need the big Guns ‣Binary search and binary trees are pretty big guns in computing ‣RBTree provides a super efficient binary tree implementation ‣It’s written in C

Slide 127

Slide 127 text

Another Solution: IP to Country

Slide 128

Slide 128 text

Another Solution: IP to Country ‣This time we will use a real binary search

Slide 129

Slide 129 text

Another Solution: IP to Country ‣This time we will use a real binary search ‣But we don’t have to write it

Slide 130

Slide 130 text

Another Solution: IP to Country ‣This time we will use a real binary search ‣But we don’t have to write it ‣This drops the search time below 1/1,000th of a second

Slide 131

Slide 131 text

Another Solution: IP to Country ‣This time we will use a real binary search ‣But we don’t have to write it ‣This drops the search time below 1/1,000th of a second ‣We will Marshal RBTree to build our persistent database

Slide 132

Slide 132 text

Another Solution: IP to Country ‣This time we will use a real binary search ‣But we don’t have to write it ‣This drops the search time below 1/1,000th of a second ‣We will Marshal RBTree to build our persistent database ‣We will use RBTree’s bounds search methods to perform the search

Slide 133

Slide 133 text

The Same Setup

Slide 134

Slide 134 text

The Same Setup #!/usr/bin/env ruby -wKU ! require "open-uri" require "zlib" ! require "rubygems" require "faster_csv" require "rbtree" ! REMOTE_DB = "http://software77.net/cgi-bin/" + "ip-country/geo-ip.pl?action=download" LOCAL_DB = "country_ips.marshal" ! File.unlink(LOCAL_DB) if ARGV.delete("-r") and File.exist? LOCAL_DB ! # ...

Slide 135

Slide 135 text

An Easier Load

Slide 136

Slide 136 text

An Easier Load # ... ! unless File.exist? LOCAL_DB ips = RBTree.new open(REMOTE_DB) do |url| Zlib::GzipReader.new(url).each do |line| next if line =~ /\A\s*(?:#|\z)/ low, high, country = FCSV.parse_line(line).values_at(0..1, 6) ips[Integer(low)] = [Integer(high), country] end end File.open(LOCAL_DB, "wb") { |file| Marshal.dump(ips, file) } end ! # ...

Slide 137

Slide 137 text

An Easier Load # ... ! unless File.exist? LOCAL_DB ips = RBTree.new open(REMOTE_DB) do |url| Zlib::GzipReader.new(url).each do |line| next if line =~ /\A\s*(?:#|\z)/ low, high, country = FCSV.parse_line(line).values_at(0..1, 6) ips[Integer(low)] = [Integer(high), country] end end File.open(LOCAL_DB, "wb") { |file| Marshal.dump(ips, file) } end ! # ...

Slide 138

Slide 138 text

An Easier Load # ... ! unless File.exist? LOCAL_DB ips = RBTree.new open(REMOTE_DB) do |url| Zlib::GzipReader.new(url).each do |line| next if line =~ /\A\s*(?:#|\z)/ low, high, country = FCSV.parse_line(line).values_at(0..1, 6) ips[Integer(low)] = [Integer(high), country] end end File.open(LOCAL_DB, "wb") { |file| Marshal.dump(ips, file) } end ! # ...

Slide 139

Slide 139 text

A Much Faster Search

Slide 140

Slide 140 text

A Much Faster Search # ... ! ips = File.open(LOCAL_DB, "rb") { |file| Marshal.load(file) } loop do print "IP address? " ip = gets.to_s.strip if ip =~ /\S/ ip_int = ip.split(".").map { |n| Integer(n) }. pack("C*").unpack("N").first match = ips.upper_bound(ip_int) puts match && ip_int <= match.last.first ? match.last.last : "Unknown" else break end end

Slide 141

Slide 141 text

A Much Faster Search # ... ! ips = File.open(LOCAL_DB, "rb") { |file| Marshal.load(file) } loop do print "IP address? " ip = gets.to_s.strip if ip =~ /\S/ ip_int = ip.split(".").map { |n| Integer(n) }. pack("C*").unpack("N").first match = ips.upper_bound(ip_int) puts match && ip_int <= match.last.first ? match.last.last : "Unknown" else break end end

Slide 142

Slide 142 text

A Much Faster Search # ... ! ips = File.open(LOCAL_DB, "rb") { |file| Marshal.load(file) } loop do print "IP address? " ip = gets.to_s.strip if ip =~ /\S/ ip_int = ip.split(".").map { |n| Integer(n) }. pack("C*").unpack("N").first match = ips.upper_bound(ip_int) puts match && ip_int <= match.last.first ? match.last.last : "Unknown" else break end end

Slide 143

Slide 143 text

Other Nice Features

Slide 144

Slide 144 text

Other Nice Features ‣RBTree is pretty much a drop in replacement for a Hash you want to keep ordered by keys

Slide 145

Slide 145 text

Other Nice Features ‣RBTree is pretty much a drop in replacement for a Hash you want to keep ordered by keys ‣Just having RBTree available magically speeds up Ruby SortedSet (over 15 times faster for simple iteration) in the standard library

Slide 146

Slide 146 text

An Ordered Hash

Slide 147

Slide 147 text

An Ordered Hash #!/usr/bin/env ruby -wKU ! require "rubygems" require "rbtree" ! ordered_hash = RBTree.new ! ordered_hash[2] = "two" ordered_hash[1] = "one" ordered_hash[3] = "three" ! ordered_hash.each do |key, value| puts "#{key}: #{value}" end # >> 1: one # >> 2: two # >> 3: three

Slide 148

Slide 148 text

Magically Improving SortedSet

Slide 149

Slide 149 text

Magically Improving SortedSet #!/usr/bin/env ruby -wKU ! # pass an argument, like --rbtree, and this sets up the load require "rubygems" unless ARGV.empty? ! require "set" dictionary = SortedSet.new File.foreach("/usr/share/dict/words") do |word| dictionary << word.strip if word =~ /\S/ end ! start = Time.now dictionary.to_a # force the set into order puts "Time to order: #{Time.now - start}"

Slide 150

Slide 150 text

The Filesystem as a Hash

Slide 151

Slide 151 text

The Filesystem as a Hash FSDB

Slide 152

Slide 152 text

Stay Flexible

Slide 153

Slide 153 text

Stay Flexible ‣Data can be in many different formats and related in many different ways

Slide 154

Slide 154 text

Stay Flexible ‣Data can be in many different formats and related in many different ways ‣FSDB gives you a lot of flexibility in these areas

Slide 155

Slide 155 text

Stay Flexible ‣Data can be in many different formats and related in many different ways ‣FSDB gives you a lot of flexibility in these areas ‣Get from: http:// redshift.sourcefor ge.net/fsdb/

Slide 156

Slide 156 text

Problem: Server Monitoring

Slide 157

Slide 157 text

Problem: Server Monitoring ‣At my job we do a lot of server monitoring

Slide 158

Slide 158 text

Problem: Server Monitoring ‣At my job we do a lot of server monitoring ‣We collect various statistics from servers at regular intervals

Slide 159

Slide 159 text

Problem: Server Monitoring ‣At my job we do a lot of server monitoring ‣We collect various statistics from servers at regular intervals ‣We later analyze this data for spikes and trends

Slide 160

Slide 160 text

Problem: Server Monitoring ‣At my job we do a lot of server monitoring ‣We collect various statistics from servers at regular intervals ‣We later analyze this data for spikes and trends ‣Time Series data is one thing RDBMS don’t do well

Slide 161

Slide 161 text

A Solution

Slide 162

Slide 162 text

A Solution ‣Store the data so it is easy to focus in on the parts that matter to you now

Slide 163

Slide 163 text

A Solution ‣Store the data so it is easy to focus in on the parts that matter to you now ‣FSDB is essentially a Hash backed by the filesystem

Slide 164

Slide 164 text

A Solution ‣Store the data so it is easy to focus in on the parts that matter to you now ‣FSDB is essentially a Hash backed by the filesystem ‣This allows you to use paths to drill down to subsets of the data

Slide 165

Slide 165 text

A Solution ‣Store the data so it is easy to focus in on the parts that matter to you now ‣FSDB is essentially a Hash backed by the filesystem ‣This allows you to use paths to drill down to subsets of the data ‣It avoids irrelevant data and even the need for an index in some cases

Slide 166

Slide 166 text

A Solution ‣Store the data so it is easy to focus in on the parts that matter to you now ‣FSDB is essentially a Hash backed by the filesystem ‣This allows you to use paths to drill down to subsets of the data ‣It avoids irrelevant data and even the need for an index in some cases ‣Techniques like this improved our graphing speed from almost four seconds to well under one

Slide 167

Slide 167 text

FSDB Structure

Slide 168

Slide 168 text

FSDB Structure

Slide 169

Slide 169 text

Creating a Database

Slide 170

Slide 170 text

Creating a Database #!/usr/bin/env ruby -wKU ! require "fsdb" ! module TimeSeries DB = FSDB::Database.new("server_stats/") ! module_function def record(data, time = Time.now) DB[time.strftime("%Y/%m/%d/%H/%M.obj")] = data end # ...

Slide 171

Slide 171 text

Creating a Database #!/usr/bin/env ruby -wKU ! require "fsdb" ! module TimeSeries DB = FSDB::Database.new("server_stats/") ! module_function def record(data, time = Time.now) DB[time.strftime("%Y/%m/%d/%H/%M.obj")] = data end # ...

Slide 172

Slide 172 text

Creating a Database #!/usr/bin/env ruby -wKU ! require "fsdb" ! module TimeSeries DB = FSDB::Database.new("server_stats/") ! module_function def record(data, time = Time.now) DB[time.strftime("%Y/%m/%d/%H/%M.obj")] = data end # ...

Slide 173

Slide 173 text

Query

Slide 174

Slide 174 text

Query # ... def sum(year, *args, &block) total = 0 path = [year, *args][0..5].join("/").gsub(/\b\d\b/, '0\0') if File.extname(path) == ".obj" total += block[DB[path]] else (DB[path] || []).each do |new_path| total += sum(File.join(path, new_path), &block) end end total end def average(*args) count = 0 sum(*args) { |data| count += 1; yield data } / count.to_f end end ! # ...

Slide 175

Slide 175 text

Query # ... def sum(year, *args, &block) total = 0 path = [year, *args][0..5].join("/").gsub(/\b\d\b/, '0\0') if File.extname(path) == ".obj" total += block[DB[path]] else (DB[path] || []).each do |new_path| total += sum(File.join(path, new_path), &block) end end total end def average(*args) count = 0 sum(*args) { |data| count += 1; yield data } / count.to_f end end ! # ...

Slide 176

Slide 176 text

Query # ... def sum(year, *args, &block) total = 0 path = [year, *args][0..5].join("/").gsub(/\b\d\b/, '0\0') if File.extname(path) == ".obj" total += block[DB[path]] else (DB[path] || []).each do |new_path| total += sum(File.join(path, new_path), &block) end end total end def average(*args) count = 0 sum(*args) { |data| count += 1; yield data } / count.to_f end end ! # ...

Slide 177

Slide 177 text

Query # ... def sum(year, *args, &block) total = 0 path = [year, *args][0..5].join("/").gsub(/\b\d\b/, '0\0') if File.extname(path) == ".obj" total += block[DB[path]] else (DB[path] || []).each do |new_path| total += sum(File.join(path, new_path), &block) end end total end def average(*args) count = 0 sum(*args) { |data| count += 1; yield data } / count.to_f end end ! # ...

Slide 178

Slide 178 text

sample Usage

Slide 179

Slide 179 text

sample Usage # ... ! if __FILE__ == $PROGRAM_NAME include TimeSeries record( { :load_average => 70, :disk_free => 50 }, Time.local(2008, 9, 4, 12, 0) ) record( { :load_average => 10, :disk_free => 50 }, Time.local(2008, 9, 4, 12, 30) ) record( { :load_average => 20, :disk_free => 51 }, Time.local(2008, 9, 4, 13, 0) ) p average(2008, 9, 4) { |data| data[:load_average] } # >> 33.3333333333333 p average(2008, 9, 4, 12) { |data| data[:load_average] } # >> 40.0 end

Slide 180

Slide 180 text

Other Nice Features

Slide 181

Slide 181 text

Other Nice Features ‣FSDB is multi-thread and multi-process safe on most platforms

Slide 182

Slide 182 text

Other Nice Features ‣FSDB is multi-thread and multi-process safe on most platforms ‣It supports read only and read/write transactions and they can even be nested

Slide 183

Slide 183 text

Other Nice Features ‣FSDB is multi-thread and multi-process safe on most platforms ‣It supports read only and read/write transactions and they can even be nested ‣You can define your own formats for files

Slide 184

Slide 184 text

Transactions

Slide 185

Slide 185 text

Transactions #!/usr/bin/env ruby -wKU ! require "fsdb" db = FSDB::Database.new("server_stats/") ! # a read only transaction (shared lock) db.browse "2008/09/04/12/00.obj" do |data| p data[:load_average] # >> 70 p data[:disk_free] # >> 50 end ! # a read/write transaction (exclusive lock) db.replace "2008/09/04/12/00.obj" do |data| data.merge(:uptime => 21 * 60) end ! p db["2008/09/04/12/00.obj"][:uptime] # >> 1260

Slide 186

Slide 186 text

Transactions #!/usr/bin/env ruby -wKU ! require "fsdb" db = FSDB::Database.new("server_stats/") ! # a read only transaction (shared lock) db.browse "2008/09/04/12/00.obj" do |data| p data[:load_average] # >> 70 p data[:disk_free] # >> 50 end ! # a read/write transaction (exclusive lock) db.replace "2008/09/04/12/00.obj" do |data| data.merge(:uptime => 21 * 60) end ! p db["2008/09/04/12/00.obj"][:uptime] # >> 1260

Slide 187

Slide 187 text

Transactions #!/usr/bin/env ruby -wKU ! require "fsdb" db = FSDB::Database.new("server_stats/") ! # a read only transaction (shared lock) db.browse "2008/09/04/12/00.obj" do |data| p data[:load_average] # >> 70 p data[:disk_free] # >> 50 end ! # a read/write transaction (exclusive lock) db.replace "2008/09/04/12/00.obj" do |data| data.merge(:uptime => 21 * 60) end ! p db["2008/09/04/12/00.obj"][:uptime] # >> 1260

Slide 188

Slide 188 text

A Custom Format

Slide 189

Slide 189 text

A Custom Format #!/usr/bin/env ruby -wKU ! require "fsdb" db = FSDB::Database.new("images/") ! PNG_FORMAT = FSDB::Format.new( /\.png\z/i, :binary, :name => "PNG_FORMAT", :load => lambda { |f| f.seek(16) w, h = f.read(8).unpack("N2") {:width => w, :height => h} }, :dump => lambda { raise "Read only format." } ) db.formats = [PNG_FORMAT] ! db.browse_each_child "/" do |image, details| puts "%s: %p" % [image, details] end # >> /fsdb_tree.png: {:width=>727, :height=>249} # >> /ip_to_country_quiz.png: {:width=>798, :height=>732} # >> /scout.png: {:width=>865, :height=>713}

Slide 190

Slide 190 text

A Custom Format #!/usr/bin/env ruby -wKU ! require "fsdb" db = FSDB::Database.new("images/") ! PNG_FORMAT = FSDB::Format.new( /\.png\z/i, :binary, :name => "PNG_FORMAT", :load => lambda { |f| f.seek(16) w, h = f.read(8).unpack("N2") {:width => w, :height => h} }, :dump => lambda { raise "Read only format." } ) db.formats = [PNG_FORMAT] ! db.browse_each_child "/" do |image, details| puts "%s: %p" % [image, details] end # >> /fsdb_tree.png: {:width=>727, :height=>249} # >> /ip_to_country_quiz.png: {:width=>798, :height=>732} # >> /scout.png: {:width=>865, :height=>713}

Slide 191

Slide 191 text

A Custom Format #!/usr/bin/env ruby -wKU ! require "fsdb" db = FSDB::Database.new("images/") ! PNG_FORMAT = FSDB::Format.new( /\.png\z/i, :binary, :name => "PNG_FORMAT", :load => lambda { |f| f.seek(16) w, h = f.read(8).unpack("N2") {:width => w, :height => h} }, :dump => lambda { raise "Read only format." } ) db.formats = [PNG_FORMAT] ! db.browse_each_child "/" do |image, details| puts "%s: %p" % [image, details] end # >> /fsdb_tree.png: {:width=>727, :height=>249} # >> /ip_to_country_quiz.png: {:width=>798, :height=>732} # >> /scout.png: {:width=>865, :height=>713}

Slide 192

Slide 192 text

Dirt Simple IPC

Slide 193

Slide 193 text

Dirt Simple IPC Rinda

Slide 194

Slide 194 text

The Importance of Networking

Slide 195

Slide 195 text

The Importance of Networking ‣When you need more processing power, you have to start hooking CPU’s together

Slide 196

Slide 196 text

The Importance of Networking ‣When you need more processing power, you have to start hooking CPU’s together ‣Rinda can make the communication between processes a snap

Slide 197

Slide 197 text

Problem: Descrambling

Slide 198

Slide 198 text

Problem: Descrambling ‣Given some scrambled letters, find all dictionary words that can be formed by rearranging those letters

Slide 199

Slide 199 text

Problem: Descrambling ‣Given some scrambled letters, find all dictionary words that can be formed by rearranging those letters ‣This task represents any task that just needs some processing time to sort out

Slide 200

Slide 200 text

Problem: Descrambling ‣Given some scrambled letters, find all dictionary words that can be formed by rearranging those letters ‣This task represents any task that just needs some processing time to sort out ‣All this requires is some I/O and simple comparisons

Slide 201

Slide 201 text

The Trivial Solution

Slide 202

Slide 202 text

The Trivial Solution #!/usr/bin/env ruby -wKU ! class String def signature strip.downcase.delete("^a-z").split("").sort.join end end ! pattern = ARGV.shift.signature descrambled = [ ] ! File.foreach("/usr/share/dict/words") do |word| descrambled << word if word.signature == pattern end ! puts descrambled

Slide 203

Slide 203 text

The Trivial Solution #!/usr/bin/env ruby -wKU ! class String def signature strip.downcase.delete("^a-z").split("").sort.join end end ! pattern = ARGV.shift.signature descrambled = [ ] ! File.foreach("/usr/share/dict/words") do |word| descrambled << word if word.signature == pattern end ! puts descrambled

Slide 204

Slide 204 text

Dividing the Work

Slide 205

Slide 205 text

Dividing the Work ‣Heavy processing almost always benefits from more processes doing the work

Slide 206

Slide 206 text

Dividing the Work ‣Heavy processing almost always benefits from more processes doing the work ‣This is true multiprocessing, unlike Ruby’s thread model

Slide 207

Slide 207 text

Dividing the Work ‣Heavy processing almost always benefits from more processes doing the work ‣This is true multiprocessing, unlike Ruby’s thread model ‣Rinda’s TupleSpace makes the Inter- Process Communication super easy

Slide 208

Slide 208 text

Dividing the Work ‣Heavy processing almost always benefits from more processes doing the work ‣This is true multiprocessing, unlike Ruby’s thread model ‣Rinda’s TupleSpace makes the Inter- Process Communication super easy ‣This task is I/O bound, but it still halved the time to put four processes on it

Slide 209

Slide 209 text

Setup

Slide 210

Slide 210 text

Setup #!/usr/bin/env ruby -wKU ! require "rinda/tuplespace" ! class String def signature strip.downcase.delete("^a-z").split("").sort.join end end ! DICT = "/usr/share/dict/words" workers = ARGV.first =~ /\A\d+\z/ ? ARGV.shift.to_i : 4 pattern = ARGV.shift.signature chunk_size = File.stat(DICT).size / workers ! # ...

Slide 211

Slide 211 text

Spawn Workers

Slide 212

Slide 212 text

Spawn Workers # ... ! workers.times do |n| fork do descrambled = [ ] File.open(DICT) do |words| my_start = chunk_size * n my_end = my_start + chunk_size words.seek(my_start) words.gets unless my_start.zero? words.each do |word| descrambled << word if word.signature == pattern break if words.pos > my_end end end ! results = Rinda::TupleSpaceProxy.new( DRbObject.new_with_uri("druby://localhost:61676") ) results.write([pattern, descrambled]) end end ! # ...

Slide 213

Slide 213 text

Spawn Workers # ... ! workers.times do |n| fork do descrambled = [ ] File.open(DICT) do |words| my_start = chunk_size * n my_end = my_start + chunk_size words.seek(my_start) words.gets unless my_start.zero? words.each do |word| descrambled << word if word.signature == pattern break if words.pos > my_end end end ! results = Rinda::TupleSpaceProxy.new( DRbObject.new_with_uri("druby://localhost:61676") ) results.write([pattern, descrambled]) end end ! # ...

Slide 214

Slide 214 text

Collect Results

Slide 215

Slide 215 text

Collect Results # ... ! results = Rinda::TupleSpace.new DRb.start_service("druby://localhost:61676", results) workers.times do descrambled = results.take([/\b#{Regexp.escape(pattern)}\b/, Array]) puts descrambled.last end Process.waitall

Slide 216

Slide 216 text

Collect Results # ... ! results = Rinda::TupleSpace.new DRb.start_service("druby://localhost:61676", results) workers.times do descrambled = results.take([/\b#{Regexp.escape(pattern)}\b/, Array]) puts descrambled.last end Process.waitall

Slide 217

Slide 217 text

Collect Results # ... ! results = Rinda::TupleSpace.new DRb.start_service("druby://localhost:61676", results) workers.times do descrambled = results.take([/\b#{Regexp.escape(pattern)}\b/, Array]) puts descrambled.last end Process.waitall

Slide 218

Slide 218 text

Other Nice Features

Slide 219

Slide 219 text

Other Nice Features ‣You can set expiration times for tuples added to a TupleSpace

Slide 220

Slide 220 text

Other Nice Features ‣You can set expiration times for tuples added to a TupleSpace ‣Rinda also comes with a RingServer for zero configuration networking

Slide 221

Slide 221 text

Using RingServer

Slide 222

Slide 222 text

Using RingServer #!/usr/bin/env ruby -wKU ! require "rinda/ring" # for RingServer require "rinda/tuplespace" # for TupleSpace ! # start a RingServer DRb.start_service Rinda::RingServer.new(Rinda::TupleSpace.new) ! # ...

Slide 223

Slide 223 text

Using RingServer #!/usr/bin/env ruby -wKU ! require "rinda/ring" # for RingServer require "rinda/tuplespace" # for TupleSpace ! # start a RingServer DRb.start_service Rinda::RingServer.new(Rinda::TupleSpace.new) ! # ...

Slide 224

Slide 224 text

Using RingServer #!/usr/bin/env ruby -wKU ! require "rinda/ring" # for RingServer require "rinda/tuplespace" # for TupleSpace ! # start a RingServer DRb.start_service Rinda::RingServer.new(Rinda::TupleSpace.new) ! # ... #!/usr/bin/env ruby -wKU ! require "rinda/ring" # for RingFinger require "rinda/tuplespace" # for TupleSpace ! # find a RingServer DRb.start_service ring_server = Rinda::RingFinger.primary ! # ...

Slide 225

Slide 225 text

Questions?