Share
Embed
Ruby as a Glue Language
An attempt to defend the merits of "shelling out" in Ruby code.
jeg2
September 07, 2007
Transcript
148 slides
-
James Edward Gray II ‣I run the Ruby Quiz ‣I
wrote some open source libraries ‣FasterCSV ‣HighLine -
James Edward Gray II ‣I run the Ruby Quiz ‣I
wrote some open source libraries ‣FasterCSV ‣HighLine ‣I authored a couple of Pragmatic books with Ruby in them -
James Edward Gray II ‣I run the Ruby Quiz ‣I
wrote some open source libraries ‣FasterCSV ‣HighLine ‣I authored a couple of Pragmatic books with Ruby in them ‣I maintain the Ruby bundle for TextMate -
What is Heroes? ‣A weekly TV show on NBC ‣The
premise is that a few ordinary people realize they have super powers -
Good Programmers are Heroes ‣They are seemingly ordinary people ‣They
constantly do what seems impossible ‣They use their super powers -
Ruby Makes A Great Sidekick ‣Ruby has many powers of
her own ‣Including the much desired power to borrow the powers of others -
Glue Languages ‣A design goal of Perl was to make
it a good “glue language” ‣Glue languages are used to join a set of external tools together to get work done -
Glue Languages ‣A design goal of Perl was to make
it a good “glue language” ‣Glue languages are used to join a set of external tools together to get work done ‣Ruby copied this Perlism -
Evil Experts ‣Multiple books warn programmers away from glue features
‣Experts claim ‣Using these features hurts portability ‣Using these features adds failure points -
I Have a Super Power ‣I’m immune to the word
“can’t” ‣We, as an industry, sometimes struggle with that word -
I Have a Super Power ‣I’m immune to the word
“can’t” ‣We, as an industry, sometimes struggle with that word ‣MJD once said: Programming is a young field and when alchemy was as young as we are now, they were still trying to turn lead into gold -
We May not Need/ Want Portability ‣If we know where
the code will run, there’s no problem ‣TextMate uses Mac OS X glue code ‣Rails applications deployed to a company server have a known platform -
We May not Need/ Want Portability ‣If we know where
the code will run, there’s no problem ‣TextMate uses Mac OS X glue code ‣Rails applications deployed to a company server have a known platform ‣We may be accessing platform specific features like AppleScript, Spotlight, or Plist API’s -
Libraries Fail Too ‣C extensions can have non-trivial or non-portable
installs ‣Dependencies make this even worse -
Libraries Fail Too ‣C extensions can have non-trivial or non-portable
installs ‣Dependencies make this even worse ‣Libraries throw errors you must handle as well -
Counter Argument: It’s Fast! ‣At work, I investigated options for
an HTML to PDF conversion job ‣The Good Way: PDF::Writer ‣The Evil Way: wrap `html2ps | ps2pdf` -
Counter Argument: It’s Fast! ‣At work, I investigated options for
an HTML to PDF conversion job ‣The Good Way: PDF::Writer ‣The Evil Way: wrap `html2ps | ps2pdf` ‣I gave each approach three hours of my time ‣I estimated PDF::Writer would take weeks ‣I basically finished the job with glue code -
Example: A Unique ID ‣A common need ‣Asked a lot
on Ruby Talk ‣The last thread included ideas from a lot of smart people -
Example: A Unique ID ‣A common need ‣Asked a lot
on Ruby Talk ‣The last thread included ideas from a lot of smart people ‣There are multiple Libraries for this -
Alternate Syntax ‣Use this syntax when you need backticks in
your command ‣any symbol can be a delimiter id = %x{uuidgen} id = %x@uuidgen@ -
Alternate Syntax ‣Use this syntax when you need backticks in
your command ‣any symbol can be a delimiter ‣You can also use the matching pairs: (…), […], {…}, and <…> ‣These nest properly id = %x{uuidgen} id = %x@uuidgen@ -
Example: The Pasteboard ‣I want to put a search string
on OS X’s find “pasteboard” (clipboard) -
Example: The Pasteboard ‣I want to put a search string
on OS X’s find “pasteboard” (clipboard) ‣I don’t need any output for this operation -
Example: The Pasteboard ‣I want to put a search string
on OS X’s find “pasteboard” (clipboard) ‣I don’t need any output for this operation ‣I just need to know if the operation succeeded ‣A simple true or false will do -
Ran or Didn’t Run if system "pbcopy -pboard find <<<
'New Search String'" puts "Search string set." else puts "Could not search string." end -
Shell Expansion ENV["MY_VAR"] = "Set from Ruby" ! system "echo
$MY_VAR" # >> Set from Ruby ! system "echo", "$MY_VAR" # >> $MY_VAR -
Shell Expansion ENV["MY_VAR"] = "Set from Ruby" ! system "echo
$MY_VAR" # >> Set from Ruby ! system "echo", "$MY_VAR" # >> $MY_VAR ‣A single argument goes through shell expansion ‣File glob patterns ‣Environment variables -
Shell Expansion ENV["MY_VAR"] = "Set from Ruby" ! system "echo
$MY_VAR" # >> Set from Ruby ! system "echo", "$MY_VAR" # >> $MY_VAR ‣A single argument goes through shell expansion ‣File glob patterns ‣Environment variables ‣Multiple arguments are passed without going through expansion -
When Trouble strikes ‣Remember to handle STDERR ‣Check process exit
status ‣Use popen3() when things get complicated -
Example: Backups ‣I want to backup a directory as part
of a larger automation -
Example: Backups ‣I want to backup a directory as part
of a larger automation ‣The rsync program can do what I need -
Example: Backups ‣I want to backup a directory as part
of a larger automation ‣The rsync program can do what I need ‣I need to watch for problems and handle them gracefully ‣Possibly emailing a warning to the user -
Taming STDERR dir = ARGV.shift or abort "USAGE: #{File.basename($PROGRAM_NAME)} DIR"
results = `rsync -av --exclude '*.DS_Store' #{dir} #{dir}_backup 2>&1` if $?.success? # require "English"; $CHILD_STATUS.success? puts results.grep(/\A#{Regexp.escape(dir)}/) else puts "Error: Couldn't back up #{dir}" # … end -
Taming STDERR dir = ARGV.shift or abort "USAGE: #{File.basename($PROGRAM_NAME)} DIR"
results = `rsync -av --exclude '*.DS_Store' #{dir} #{dir}_backup 2>&1` if $?.success? # require "English"; $CHILD_STATUS.success? puts results.grep(/\A#{Regexp.escape(dir)}/) else puts "Error: Couldn't back up #{dir}" # … end -
Taming STDERR dir = ARGV.shift or abort "USAGE: #{File.basename($PROGRAM_NAME)} DIR"
results = `rsync -av --exclude '*.DS_Store' #{dir} #{dir}_backup 2>&1` if $?.success? # require "English"; $CHILD_STATUS.success? puts results.grep(/\A#{Regexp.escape(dir)}/) else puts "Error: Couldn't back up #{dir}" # … end -
Taming STDERR dir = ARGV.shift or abort "USAGE: #{File.basename($PROGRAM_NAME)} DIR"
results = `rsync -av --exclude '*.DS_Store' #{dir} #{dir}_backup 2>&1` if $?.success? # require "English"; $CHILD_STATUS.success? puts results.grep(/\A#{Regexp.escape(dir)}/) else puts "Error: Couldn't back up #{dir}" # … end -
Proper Shell Escaping # escape text to make it useable
in a shell script as # one “word” (string) def escape_for_shell(str) str.to_s.gsub( /(?=[^a-zA-Z0-9_.\/\-\x7F-\xFF\n])/, '\\' ). gsub( /\n/, "'\n'" ). sub( /^$/, "''" ) end -
Tips for Avoiding Errors ‣Use full paths to programs and
files whenever possible -
Tips for Avoiding Errors ‣Use full paths to programs and
files whenever possible ‣Send data to STDIN when you can -
Tips for Avoiding Errors ‣Use full paths to programs and
files whenever possible ‣Send data to STDIN when you can ‣If you can’t send it to STDIN, dump the data to a Tempfile and send that path -
Tips for Avoiding Errors ‣Use full paths to programs and
files whenever possible ‣Send data to STDIN when you can ‣If you can’t send it to STDIN, dump the data to a Tempfile and send that path ‣Remember to shell escape any command- line arguments that could contain dangerous characters (even spaces) -
Managing Streams ‣Use popen() to manage STDIN and STDOUT ‣Use
popen3() to manage STDIN, STDOUT, and STDERR ‣Use popen4() if you also need the PID -
Example: Formatting Prose ‣I want to rewrap some prose provided
by the user ‣Command-line arguments are not appropriate here ‣Complex shell Escaping ‣Size limit -
Example: Formatting Prose ‣I want to rewrap some prose provided
by the user ‣Command-line arguments are not appropriate here ‣Complex shell Escaping ‣Size limit ‣I need to send the prose to fmt via STDIN -
Reading and Writing prose = <<END_PROSE Lorem ipsum dolor sit
amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. END_PROSE ! formatted = IO.popen("fmt -w 30", "r+") do |pipe| # open("| fmt -w 30", "r+") do |pipe| pipe << prose pipe.close_write pipe.read end -
Reading and Writing prose = <<END_PROSE Lorem ipsum dolor sit
amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. END_PROSE ! formatted = IO.popen("fmt -w 30", "r+") do |pipe| # open("| fmt -w 30", "r+") do |pipe| pipe << prose pipe.close_write pipe.read end -
Reading and Writing prose = <<END_PROSE Lorem ipsum dolor sit
amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. END_PROSE ! formatted = IO.popen("fmt -w 30", "r+") do |pipe| # open("| fmt -w 30", "r+") do |pipe| pipe << prose pipe.close_write pipe.read end -
Reading and Writing prose = <<END_PROSE Lorem ipsum dolor sit
amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. END_PROSE ! formatted = IO.popen("fmt -w 30", "r+") do |pipe| # open("| fmt -w 30", "r+") do |pipe| pipe << prose pipe.close_write pipe.read end -
Reading and Writing prose = <<END_PROSE Lorem ipsum dolor sit
amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. END_PROSE ! formatted = IO.popen("fmt -w 30", "r+") do |pipe| # open("| fmt -w 30", "r+") do |pipe| pipe << prose pipe.close_write pipe.read end -
Example: A Ruby Session ‣I want to run some Ruby
code ‣I don’t want that code to affect my current Ruby process -
Example: A Ruby Session ‣I want to run some Ruby
code ‣I don’t want that code to affect my current Ruby process ‣I may also need to do some special setup, hacking Ruby’s core, before this code is run -
Example: A Ruby Session ‣I want to run some Ruby
code ‣I don’t want that code to affect my current Ruby process ‣I may also need to do some special setup, hacking Ruby’s core, before this code is run ‣I need to format STDOUT and STDERR differently -
With Error Handling require "open3" ! Open3.popen3("ruby") do |stdin, stdout,
stderr| stdin << %Q{puts "I am a puppet."; oops!()} stdin.close puts "Output:" puts stdout.read puts "Errors:" puts stderr.read end -
With Error Handling require "open3" ! Open3.popen3("ruby") do |stdin, stdout,
stderr| stdin << %Q{puts "I am a puppet."; oops!()} stdin.close puts "Output:" puts stdout.read puts "Errors:" puts stderr.read end -
With Error Handling require "open3" ! Open3.popen3("ruby") do |stdin, stdout,
stderr| stdin << %Q{puts "I am a puppet."; oops!()} stdin.close puts "Output:" puts stdout.read puts "Errors:" puts stderr.read end -
With Error Handling require "open3" ! Open3.popen3("ruby") do |stdin, stdout,
stderr| stdin << %Q{puts "I am a puppet."; oops!()} stdin.close puts "Output:" puts stdout.read puts "Errors:" puts stderr.read end -
When you Also Need a PID ‣Install the POpen4 gem
‣Unix version ‣Windows versions -
When you Also Need a PID ‣Install the POpen4 gem
‣Unix version ‣Windows versions ‣popen4() works like popen3() but it also passes you the PID for the child process ‣The PID is useful for sending the child process signals, possibly to kill the process -
“If it’s on the Web, it has an API.” !
James Britt Using Web Tools -
Don’t Forget the Web If you need to… Use the
tool… Read Content open-uri -
Don’t Forget the Web If you need to… Use the
tool… Read Content open-uri Write Form Data Net::HTTP -
Don’t Forget the Web If you need to… Use the
tool… Read Content open-uri Write Form Data Net::HTTP Emulate a Browser Mechanize -
Don’t Forget the Web If you need to… Use the
tool… Read Content open-uri Write Form Data Net::HTTP Emulate a Browser Mechanize Scrape HTML Hpricot -
Example: Tracking Ruby ‣I want to download the latest version
of Ruby as part of a larger automation -
Example: Tracking Ruby ‣I want to download the latest version
of Ruby as part of a larger automation ‣I want to verify the contents of the download -
Example: Tracking Ruby ‣I want to download the latest version
of Ruby as part of a larger automation ‣I want to verify the contents of the download ‣I want to expand the compressed archive -
Simple Scraping require "open-uri" require "digest/md5" ! require "rubygems" require
"hpricot" ! dl = Hpricot(open("http://www.ruby-lang.org/en/downloads/")) li = (dl / "div#content" / "ul" / "li").first url = (li / "a").first.attributes["href"] md5 = li.inner_html[/md5:.+?([A-Za-z0-9]{32})/, 1] ! rb = open(url) { |ftp| ftp.read } if Digest::MD5.hexdigest(rb) == md5 IO.popen("tar xvz", "wb") { |tar| tar << rb } else abort "Corrupt download" end -
Simple Scraping require "open-uri" require "digest/md5" ! require "rubygems" require
"hpricot" ! dl = Hpricot(open("http://www.ruby-lang.org/en/downloads/")) li = (dl / "div#content" / "ul" / "li").first url = (li / "a").first.attributes["href"] md5 = li.inner_html[/md5:.+?([A-Za-z0-9]{32})/, 1] ! rb = open(url) { |ftp| ftp.read } if Digest::MD5.hexdigest(rb) == md5 IO.popen("tar xvz", "wb") { |tar| tar << rb } else abort "Corrupt download" end -
Simple Scraping require "open-uri" require "digest/md5" ! require "rubygems" require
"hpricot" ! dl = Hpricot(open("http://www.ruby-lang.org/en/downloads/")) li = (dl / "div#content" / "ul" / "li").first url = (li / "a").first.attributes["href"] md5 = li.inner_html[/md5:.+?([A-Za-z0-9]{32})/, 1] ! rb = open(url) { |ftp| ftp.read } if Digest::MD5.hexdigest(rb) == md5 IO.popen("tar xvz", "wb") { |tar| tar << rb } else abort "Corrupt download" end -
Simple Scraping require "open-uri" require "digest/md5" ! require "rubygems" require
"hpricot" ! dl = Hpricot(open("http://www.ruby-lang.org/en/downloads/")) li = (dl / "div#content" / "ul" / "li").first url = (li / "a").first.attributes["href"] md5 = li.inner_html[/md5:.+?([A-Za-z0-9]{32})/, 1] ! rb = open(url) { |ftp| ftp.read } if Digest::MD5.hexdigest(rb) == md5 IO.popen("tar xvz", "wb") { |tar| tar << rb } else abort "Corrupt download" end -
Use Caution ‣These scraping techniques see wider use than talking
to external processes -
Use Caution ‣These scraping techniques see wider use than talking
to external processes ‣Ironically, they really do seem to be more fragile -
Use Caution ‣These scraping techniques see wider use than talking
to external processes ‣Ironically, they really do seem to be more fragile ‣tips for managing scraping code: ‣Abstract out the scraping code ‣Use more aggressive error handling ‣Make sure the maintenance is worth it -
Pop Quiz Out of the box, can Ruby… Apply a
difference algorithm to the contents of two Strings? -
Pop Quiz Out of the box, can Ruby… Apply a
difference algorithm to the contents of two Strings? Efficiently read a file line by line in reverse? -
YES! ‣Don’t be afraid to use your powers ‣You will
literally be able to accomplish anything -
String Diff require "tempfile" ! class String def diff(other) st
= Tempfile.new("diff_self") ot = Tempfile.new("diff_other") st << self ot << other [st, ot].each { |t| t.flush } `diff -u #{st.path} #{ot.path}`[/^@.+\z/m] end end ! puts "one\ntwo\n".diff("one\nthree\n") # >> @@ -1,2 +1,2 @@ # >> one # >> -two # >> +three -
Reading Backwards unless ARGV.size == 1 and File.exist? ARGV.first abort
"Usage: #{File.basename($PROGRAM_NAME)} FILE" end ! last_five_lines = Array.new ! IO.popen("tail -r #{ARGV.shift}") do |tail| tail.each do |line| last_five_lines << line break if last_five_lines.size == 5 end end last_five_lines.reverse! ! puts last_five_lines