Ruby as a Glue Language

259f23c3b129f07b0c496b9f0495f07e?s=47 jeg2
September 07, 2007

Ruby as a Glue Language

An attempt to defend the merits of "shelling out" in Ruby code.

259f23c3b129f07b0c496b9f0495f07e?s=128

jeg2

September 07, 2007
Tweet

Transcript

  1. Ruby as a Glue Language Claiming Your Super Powers

  2. James Edward Gray II

  3. James Edward Gray II ‣I run the Ruby Quiz

  4. James Edward Gray II ‣I run the Ruby Quiz ‣I

    wrote some open source libraries ‣FasterCSV ‣HighLine
  5. James Edward Gray II ‣I run the Ruby Quiz ‣I

    wrote some open source libraries ‣FasterCSV ‣HighLine ‣I authored a couple of Pragmatic books with Ruby in them
  6. James Edward Gray II ‣I run the Ruby Quiz ‣I

    wrote some open source libraries ‣FasterCSV ‣HighLine ‣I authored a couple of Pragmatic books with Ruby in them ‣I maintain the Ruby bundle for TextMate
  7. None
  8. None
  9. What is Heroes?

  10. What is Heroes? ‣A weekly TV show on NBC

  11. What is Heroes? ‣A weekly TV show on NBC ‣The

    premise is that a few ordinary people realize they have super powers
  12. Good Programmers are Heroes

  13. Good Programmers are Heroes ‣They are seemingly ordinary people

  14. Good Programmers are Heroes ‣They are seemingly ordinary people ‣They

    constantly do what seems impossible ‣They use their super powers
  15. Ruby Makes A Great Sidekick

  16. Ruby Makes A Great Sidekick ‣Ruby has many powers of

    her own
  17. Ruby Makes A Great Sidekick ‣Ruby has many powers of

    her own ‣Including the much desired power to borrow the powers of others
  18. Ruby Glue Good or bad?

  19. Glue Languages

  20. Glue Languages ‣A design goal of Perl was to make

    it a good “glue language” ‣Glue languages are used to join a set of external tools together to get work done
  21. Glue Languages ‣A design goal of Perl was to make

    it a good “glue language” ‣Glue languages are used to join a set of external tools together to get work done ‣Ruby copied this Perlism
  22. Evil Experts

  23. Evil Experts ‣Multiple books warn programmers away from glue features

  24. Evil Experts ‣Multiple books warn programmers away from glue features

    ‣Experts claim ‣Using these features hurts portability ‣Using these features adds failure points
  25. I Have a Super Power

  26. I Have a Super Power ‣I’m immune to the word

    “can’t”
  27. I Have a Super Power ‣I’m immune to the word

    “can’t” ‣We, as an industry, sometimes struggle with that word
  28. I Have a Super Power ‣I’m immune to the word

    “can’t” ‣We, as an industry, sometimes struggle with that word ‣MJD once said: Programming is a young field and when alchemy was as young as we are now, they were still trying to turn lead into gold
  29. My Opinion of the Expert Advice

  30. BS My Opinion of the Expert Advice

  31. We May not Need/ Want Portability

  32. We May not Need/ Want Portability ‣If we know where

    the code will run, there’s no problem ‣TextMate uses Mac OS X glue code ‣Rails applications deployed to a company server have a known platform
  33. We May not Need/ Want Portability ‣If we know where

    the code will run, there’s no problem ‣TextMate uses Mac OS X glue code ‣Rails applications deployed to a company server have a known platform ‣We may be accessing platform specific features like AppleScript, Spotlight, or Plist API’s
  34. Libraries Fail Too

  35. Libraries Fail Too ‣C extensions can have non-trivial or non-portable

    installs ‣Dependencies make this even worse
  36. Libraries Fail Too ‣C extensions can have non-trivial or non-portable

    installs ‣Dependencies make this even worse ‣Libraries throw errors you must handle as well
  37. Counter Argument: It’s Fast!

  38. Counter Argument: It’s Fast! ‣At work, I investigated options for

    an HTML to PDF conversion job ‣The Good Way: PDF::Writer ‣The Evil Way: wrap `html2ps | ps2pdf`
  39. Counter Argument: It’s Fast! ‣At work, I investigated options for

    an HTML to PDF conversion job ‣The Good Way: PDF::Writer ‣The Evil Way: wrap `html2ps | ps2pdf` ‣I gave each approach three hours of my time ‣I estimated PDF::Writer would take weeks ‣I basically finished the job with glue code
  40. Shelling Out Using Backticks

  41. Example: A Unique ID

  42. Example: A Unique ID ‣A common need

  43. Example: A Unique ID ‣A common need ‣Asked a lot

    on Ruby Talk ‣The last thread included ideas from a lot of smart people
  44. Example: A Unique ID ‣A common need ‣Asked a lot

    on Ruby Talk ‣The last thread included ideas from a lot of smart people ‣There are multiple Libraries for this
  45. A UUID from Glue Code

  46. A UUID from Glue Code id = `uuidgen`

  47. Alternate Syntax

  48. Alternate Syntax id = %x{uuidgen} id = %x@uuidgen@

  49. Alternate Syntax ‣Use this syntax when you need backticks in

    your command ‣any symbol can be a delimiter id = %x{uuidgen} id = %x@uuidgen@
  50. Alternate Syntax ‣Use this syntax when you need backticks in

    your command ‣any symbol can be a delimiter ‣You can also use the matching pairs: (…), […], {…}, and <…> ‣These nest properly id = %x{uuidgen} id = %x@uuidgen@
  51. No Output Needed Using system()

  52. Example: The Pasteboard

  53. Example: The Pasteboard ‣I want to put a search string

    on OS X’s find “pasteboard” (clipboard)
  54. Example: The Pasteboard ‣I want to put a search string

    on OS X’s find “pasteboard” (clipboard) ‣I don’t need any output for this operation
  55. Example: The Pasteboard ‣I want to put a search string

    on OS X’s find “pasteboard” (clipboard) ‣I don’t need any output for this operation ‣I just need to know if the operation succeeded ‣A simple true or false will do
  56. Ran or Didn’t Run

  57. Ran or Didn’t Run if system "pbcopy -pboard find <<<

    'New Search String'" puts "Search string set." else puts "Could not search string." end
  58. Shell Expansion

  59. Shell Expansion ENV["MY_VAR"] = "Set from Ruby" ! system "echo

    $MY_VAR" # >> Set from Ruby ! system "echo", "$MY_VAR" # >> $MY_VAR
  60. Shell Expansion ENV["MY_VAR"] = "Set from Ruby" ! system "echo

    $MY_VAR" # >> Set from Ruby ! system "echo", "$MY_VAR" # >> $MY_VAR ‣A single argument goes through shell expansion ‣File glob patterns ‣Environment variables
  61. Shell Expansion ENV["MY_VAR"] = "Set from Ruby" ! system "echo

    $MY_VAR" # >> Set from Ruby ! system "echo", "$MY_VAR" # >> $MY_VAR ‣A single argument goes through shell expansion ‣File glob patterns ‣Environment variables ‣Multiple arguments are passed without going through expansion
  62. Handling Errors Mind the Expert Warnings

  63. When Trouble strikes

  64. When Trouble strikes ‣Remember to handle STDERR

  65. When Trouble strikes ‣Remember to handle STDERR ‣Check process exit

    status
  66. When Trouble strikes ‣Remember to handle STDERR ‣Check process exit

    status ‣Use popen3() when things get complicated
  67. Example: Backups

  68. Example: Backups ‣I want to backup a directory as part

    of a larger automation
  69. Example: Backups ‣I want to backup a directory as part

    of a larger automation ‣The rsync program can do what I need
  70. Example: Backups ‣I want to backup a directory as part

    of a larger automation ‣The rsync program can do what I need ‣I need to watch for problems and handle them gracefully ‣Possibly emailing a warning to the user
  71. STDERR, The Problem Child

  72. STDERR, The Problem Child

  73. STDERR, The Problem Child

  74. STDERR, The Problem Child

  75. STDERR, The Problem Child

  76. STDERR, The Problem Child

  77. Taming STDERR

  78. Taming STDERR dir = ARGV.shift or abort "USAGE: #{File.basename($PROGRAM_NAME)} DIR"

    results = `rsync -av --exclude '*.DS_Store' #{dir} #{dir}_backup 2>&1` if $?.success? # require "English"; $CHILD_STATUS.success? puts results.grep(/\A#{Regexp.escape(dir)}/) else puts "Error: Couldn't back up #{dir}" # … end
  79. Taming STDERR dir = ARGV.shift or abort "USAGE: #{File.basename($PROGRAM_NAME)} DIR"

    results = `rsync -av --exclude '*.DS_Store' #{dir} #{dir}_backup 2>&1` if $?.success? # require "English"; $CHILD_STATUS.success? puts results.grep(/\A#{Regexp.escape(dir)}/) else puts "Error: Couldn't back up #{dir}" # … end
  80. Taming STDERR dir = ARGV.shift or abort "USAGE: #{File.basename($PROGRAM_NAME)} DIR"

    results = `rsync -av --exclude '*.DS_Store' #{dir} #{dir}_backup 2>&1` if $?.success? # require "English"; $CHILD_STATUS.success? puts results.grep(/\A#{Regexp.escape(dir)}/) else puts "Error: Couldn't back up #{dir}" # … end
  81. Taming STDERR dir = ARGV.shift or abort "USAGE: #{File.basename($PROGRAM_NAME)} DIR"

    results = `rsync -av --exclude '*.DS_Store' #{dir} #{dir}_backup 2>&1` if $?.success? # require "English"; $CHILD_STATUS.success? puts results.grep(/\A#{Regexp.escape(dir)}/) else puts "Error: Couldn't back up #{dir}" # … end
  82. Proper Shell Escaping

  83. Proper Shell Escaping # escape text to make it useable

    in a shell script as # one “word” (string) def escape_for_shell(str) str.to_s.gsub( /(?=[^a-zA-Z0-9_.\/\-\x7F-\xFF\n])/, '\\' ). gsub( /\n/, "'\n'" ). sub( /^$/, "''" ) end
  84. Tips for Avoiding Errors

  85. Tips for Avoiding Errors ‣Use full paths to programs and

    files whenever possible
  86. Tips for Avoiding Errors ‣Use full paths to programs and

    files whenever possible ‣Send data to STDIN when you can
  87. Tips for Avoiding Errors ‣Use full paths to programs and

    files whenever possible ‣Send data to STDIN when you can ‣If you can’t send it to STDIN, dump the data to a Tempfile and send that path
  88. Tips for Avoiding Errors ‣Use full paths to programs and

    files whenever possible ‣Send data to STDIN when you can ‣If you can’t send it to STDIN, dump the data to a Tempfile and send that path ‣Remember to shell escape any command- line arguments that could contain dangerous characters (even spaces)
  89. Full Control Using popen(), popen3(), and popen4()

  90. Managing Streams

  91. Managing Streams ‣Use popen() to manage STDIN and STDOUT

  92. Managing Streams ‣Use popen() to manage STDIN and STDOUT ‣Use

    popen3() to manage STDIN, STDOUT, and STDERR ‣Use popen4() if you also need the PID
  93. Example: Formatting Prose

  94. Example: Formatting Prose ‣I want to rewrap some prose provided

    by the user
  95. Example: Formatting Prose ‣I want to rewrap some prose provided

    by the user ‣Command-line arguments are not appropriate here ‣Complex shell Escaping ‣Size limit
  96. Example: Formatting Prose ‣I want to rewrap some prose provided

    by the user ‣Command-line arguments are not appropriate here ‣Complex shell Escaping ‣Size limit ‣I need to send the prose to fmt via STDIN
  97. Reading and Writing

  98. Reading and Writing prose = <<END_PROSE Lorem ipsum dolor sit

    amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. END_PROSE ! formatted = IO.popen("fmt -w 30", "r+") do |pipe| # open("| fmt -w 30", "r+") do |pipe| pipe << prose pipe.close_write pipe.read end
  99. Reading and Writing prose = <<END_PROSE Lorem ipsum dolor sit

    amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. END_PROSE ! formatted = IO.popen("fmt -w 30", "r+") do |pipe| # open("| fmt -w 30", "r+") do |pipe| pipe << prose pipe.close_write pipe.read end
  100. Reading and Writing prose = <<END_PROSE Lorem ipsum dolor sit

    amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. END_PROSE ! formatted = IO.popen("fmt -w 30", "r+") do |pipe| # open("| fmt -w 30", "r+") do |pipe| pipe << prose pipe.close_write pipe.read end
  101. Reading and Writing prose = <<END_PROSE Lorem ipsum dolor sit

    amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. END_PROSE ! formatted = IO.popen("fmt -w 30", "r+") do |pipe| # open("| fmt -w 30", "r+") do |pipe| pipe << prose pipe.close_write pipe.read end
  102. Reading and Writing prose = <<END_PROSE Lorem ipsum dolor sit

    amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. END_PROSE ! formatted = IO.popen("fmt -w 30", "r+") do |pipe| # open("| fmt -w 30", "r+") do |pipe| pipe << prose pipe.close_write pipe.read end
  103. Example: A Ruby Session

  104. Example: A Ruby Session ‣I want to run some Ruby

    code
  105. Example: A Ruby Session ‣I want to run some Ruby

    code ‣I don’t want that code to affect my current Ruby process
  106. Example: A Ruby Session ‣I want to run some Ruby

    code ‣I don’t want that code to affect my current Ruby process ‣I may also need to do some special setup, hacking Ruby’s core, before this code is run
  107. Example: A Ruby Session ‣I want to run some Ruby

    code ‣I don’t want that code to affect my current Ruby process ‣I may also need to do some special setup, hacking Ruby’s core, before this code is run ‣I need to format STDOUT and STDERR differently
  108. With Error Handling

  109. With Error Handling require "open3" ! Open3.popen3("ruby") do |stdin, stdout,

    stderr| stdin << %Q{puts "I am a puppet."; oops!()} stdin.close puts "Output:" puts stdout.read puts "Errors:" puts stderr.read end
  110. With Error Handling require "open3" ! Open3.popen3("ruby") do |stdin, stdout,

    stderr| stdin << %Q{puts "I am a puppet."; oops!()} stdin.close puts "Output:" puts stdout.read puts "Errors:" puts stderr.read end
  111. With Error Handling require "open3" ! Open3.popen3("ruby") do |stdin, stdout,

    stderr| stdin << %Q{puts "I am a puppet."; oops!()} stdin.close puts "Output:" puts stdout.read puts "Errors:" puts stderr.read end
  112. With Error Handling require "open3" ! Open3.popen3("ruby") do |stdin, stdout,

    stderr| stdin << %Q{puts "I am a puppet."; oops!()} stdin.close puts "Output:" puts stdout.read puts "Errors:" puts stderr.read end
  113. When you Also Need a PID

  114. When you Also Need a PID ‣Install the POpen4 gem

    ‣Unix version ‣Windows versions
  115. When you Also Need a PID ‣Install the POpen4 gem

    ‣Unix version ‣Windows versions ‣popen4() works like popen3() but it also passes you the PID for the child process ‣The PID is useful for sending the child process signals, possibly to kill the process
  116. “If it’s on the Web, it has an API.” !

    James Britt Using Web Tools
  117. Don’t Forget the Web

  118. Don’t Forget the Web If you need to… Use the

    tool…
  119. Don’t Forget the Web If you need to… Use the

    tool… Read Content open-uri
  120. Don’t Forget the Web If you need to… Use the

    tool… Read Content open-uri Write Form Data Net::HTTP
  121. Don’t Forget the Web If you need to… Use the

    tool… Read Content open-uri Write Form Data Net::HTTP Emulate a Browser Mechanize
  122. Don’t Forget the Web If you need to… Use the

    tool… Read Content open-uri Write Form Data Net::HTTP Emulate a Browser Mechanize Scrape HTML Hpricot
  123. Example: Tracking Ruby

  124. Example: Tracking Ruby ‣I want to download the latest version

    of Ruby as part of a larger automation
  125. Example: Tracking Ruby ‣I want to download the latest version

    of Ruby as part of a larger automation ‣I want to verify the contents of the download
  126. Example: Tracking Ruby ‣I want to download the latest version

    of Ruby as part of a larger automation ‣I want to verify the contents of the download ‣I want to expand the compressed archive
  127. Simple Scraping

  128. Simple Scraping require "open-uri" require "digest/md5" ! require "rubygems" require

    "hpricot" ! dl = Hpricot(open("http://www.ruby-lang.org/en/downloads/")) li = (dl / "div#content" / "ul" / "li").first url = (li / "a").first.attributes["href"] md5 = li.inner_html[/md5:.+?([A-Za-z0-9]{32})/, 1] ! rb = open(url) { |ftp| ftp.read } if Digest::MD5.hexdigest(rb) == md5 IO.popen("tar xvz", "wb") { |tar| tar << rb } else abort "Corrupt download" end
  129. Simple Scraping require "open-uri" require "digest/md5" ! require "rubygems" require

    "hpricot" ! dl = Hpricot(open("http://www.ruby-lang.org/en/downloads/")) li = (dl / "div#content" / "ul" / "li").first url = (li / "a").first.attributes["href"] md5 = li.inner_html[/md5:.+?([A-Za-z0-9]{32})/, 1] ! rb = open(url) { |ftp| ftp.read } if Digest::MD5.hexdigest(rb) == md5 IO.popen("tar xvz", "wb") { |tar| tar << rb } else abort "Corrupt download" end
  130. Simple Scraping require "open-uri" require "digest/md5" ! require "rubygems" require

    "hpricot" ! dl = Hpricot(open("http://www.ruby-lang.org/en/downloads/")) li = (dl / "div#content" / "ul" / "li").first url = (li / "a").first.attributes["href"] md5 = li.inner_html[/md5:.+?([A-Za-z0-9]{32})/, 1] ! rb = open(url) { |ftp| ftp.read } if Digest::MD5.hexdigest(rb) == md5 IO.popen("tar xvz", "wb") { |tar| tar << rb } else abort "Corrupt download" end
  131. Simple Scraping require "open-uri" require "digest/md5" ! require "rubygems" require

    "hpricot" ! dl = Hpricot(open("http://www.ruby-lang.org/en/downloads/")) li = (dl / "div#content" / "ul" / "li").first url = (li / "a").first.attributes["href"] md5 = li.inner_html[/md5:.+?([A-Za-z0-9]{32})/, 1] ! rb = open(url) { |ftp| ftp.read } if Digest::MD5.hexdigest(rb) == md5 IO.popen("tar xvz", "wb") { |tar| tar << rb } else abort "Corrupt download" end
  132. Use Caution

  133. Use Caution ‣These scraping techniques see wider use than talking

    to external processes
  134. Use Caution ‣These scraping techniques see wider use than talking

    to external processes ‣Ironically, they really do seem to be more fragile
  135. Use Caution ‣These scraping techniques see wider use than talking

    to external processes ‣Ironically, they really do seem to be more fragile ‣tips for managing scraping code: ‣Abstract out the scraping code ‣Use more aggressive error handling ‣Make sure the maintenance is worth it
  136. Summary Remain Strong

  137. Pop Quiz

  138. Pop Quiz Out of the box, can Ruby…

  139. Pop Quiz Out of the box, can Ruby… Apply a

    difference algorithm to the contents of two Strings?
  140. Pop Quiz Out of the box, can Ruby… Apply a

    difference algorithm to the contents of two Strings? Efficiently read a file line by line in reverse?
  141. YES!

  142. YES! ‣Don’t be afraid to use your powers

  143. YES! ‣Don’t be afraid to use your powers ‣You will

    literally be able to accomplish anything
  144. String Diff

  145. String Diff require "tempfile" ! class String def diff(other) st

    = Tempfile.new("diff_self") ot = Tempfile.new("diff_other") st << self ot << other [st, ot].each { |t| t.flush } `diff -u #{st.path} #{ot.path}`[/^@.+\z/m] end end ! puts "one\ntwo\n".diff("one\nthree\n") # >> @@ -1,2 +1,2 @@ # >> one # >> -two # >> +three
  146. Reading Backwards

  147. Reading Backwards unless ARGV.size == 1 and File.exist? ARGV.first abort

    "Usage: #{File.basename($PROGRAM_NAME)} FILE" end ! last_five_lines = Array.new ! IO.popen("tail -r #{ARGV.shift}") do |tail| tail.each do |line| last_five_lines << line break if last_five_lines.size == 5 end end last_five_lines.reverse! ! puts last_five_lines
  148. Questions?