Tweet
Tweet

Slide 1

Slide 1 text

Ruby as a Glue Language Claiming Your Super Powers

Slide 2

Slide 2 text

James Edward Gray II

Slide 3

Slide 3 text

James Edward Gray II ‣I run the Ruby Quiz

Slide 4

Slide 4 text

James Edward Gray II ‣I run the Ruby Quiz ‣I wrote some open source libraries ‣FasterCSV ‣HighLine

Slide 5

Slide 5 text

James Edward Gray II ‣I run the Ruby Quiz ‣I wrote some open source libraries ‣FasterCSV ‣HighLine ‣I authored a couple of Pragmatic books with Ruby in them

Slide 6

Slide 6 text

James Edward Gray II ‣I run the Ruby Quiz ‣I wrote some open source libraries ‣FasterCSV ‣HighLine ‣I authored a couple of Pragmatic books with Ruby in them ‣I maintain the Ruby bundle for TextMate

Slide 7

Slide 7 text

No content

Slide 8

Slide 8 text

No content

Slide 9

Slide 9 text

What is Heroes?

Slide 10

Slide 10 text

What is Heroes? ‣A weekly TV show on NBC

Slide 11

Slide 11 text

What is Heroes? ‣A weekly TV show on NBC ‣The premise is that a few ordinary people realize they have super powers

Slide 12

Slide 12 text

Good Programmers are Heroes

Slide 13

Slide 13 text

Good Programmers are Heroes ‣They are seemingly ordinary people

Slide 14

Slide 14 text

Good Programmers are Heroes ‣They are seemingly ordinary people ‣They constantly do what seems impossible ‣They use their super powers

Slide 15

Slide 15 text

Ruby Makes A Great Sidekick

Slide 16

Slide 16 text

Ruby Makes A Great Sidekick ‣Ruby has many powers of her own

Slide 17

Slide 17 text

Ruby Makes A Great Sidekick ‣Ruby has many powers of her own ‣Including the much desired power to borrow the powers of others

Slide 18

Slide 18 text

Ruby Glue Good or bad?

Slide 19

Slide 19 text

Glue Languages

Slide 20

Slide 20 text

Glue Languages ‣A design goal of Perl was to make it a good “glue language” ‣Glue languages are used to join a set of external tools together to get work done

Slide 21

Slide 21 text

Glue Languages ‣A design goal of Perl was to make it a good “glue language” ‣Glue languages are used to join a set of external tools together to get work done ‣Ruby copied this Perlism

Slide 22

Slide 22 text

Evil Experts

Slide 23

Slide 23 text

Evil Experts ‣Multiple books warn programmers away from glue features

Slide 24

Slide 24 text

Evil Experts ‣Multiple books warn programmers away from glue features ‣Experts claim ‣Using these features hurts portability ‣Using these features adds failure points

Slide 25

Slide 25 text

I Have a Super Power

Slide 26

Slide 26 text

I Have a Super Power ‣I’m immune to the word “can’t”

Slide 27

Slide 27 text

I Have a Super Power ‣I’m immune to the word “can’t” ‣We, as an industry, sometimes struggle with that word

Slide 28

Slide 28 text

I Have a Super Power ‣I’m immune to the word “can’t” ‣We, as an industry, sometimes struggle with that word ‣MJD once said: Programming is a young field and when alchemy was as young as we are now, they were still trying to turn lead into gold

Slide 29

Slide 29 text

My Opinion of the Expert Advice

Slide 30

Slide 30 text

BS My Opinion of the Expert Advice

Slide 31

Slide 31 text

We May not Need/ Want Portability

Slide 32

Slide 32 text

We May not Need/ Want Portability ‣If we know where the code will run, there’s no problem ‣TextMate uses Mac OS X glue code ‣Rails applications deployed to a company server have a known platform

Slide 33

Slide 33 text

We May not Need/ Want Portability ‣If we know where the code will run, there’s no problem ‣TextMate uses Mac OS X glue code ‣Rails applications deployed to a company server have a known platform ‣We may be accessing platform specific features like AppleScript, Spotlight, or Plist API’s

Slide 34

Slide 34 text

Libraries Fail Too

Slide 35

Slide 35 text

Libraries Fail Too ‣C extensions can have non-trivial or non-portable installs ‣Dependencies make this even worse

Slide 36

Slide 36 text

Libraries Fail Too ‣C extensions can have non-trivial or non-portable installs ‣Dependencies make this even worse ‣Libraries throw errors you must handle as well

Slide 37

Slide 37 text

Counter Argument: It’s Fast!

Slide 38

Slide 38 text

Counter Argument: It’s Fast! ‣At work, I investigated options for an HTML to PDF conversion job ‣The Good Way: PDF::Writer ‣The Evil Way: wrap `html2ps | ps2pdf`

Slide 39

Slide 39 text

Counter Argument: It’s Fast! ‣At work, I investigated options for an HTML to PDF conversion job ‣The Good Way: PDF::Writer ‣The Evil Way: wrap `html2ps | ps2pdf` ‣I gave each approach three hours of my time ‣I estimated PDF::Writer would take weeks ‣I basically finished the job with glue code

Slide 40

Slide 40 text

Shelling Out Using Backticks

Slide 41

Slide 41 text

Example: A Unique ID

Slide 42

Slide 42 text

Example: A Unique ID ‣A common need

Slide 43

Slide 43 text

Example: A Unique ID ‣A common need ‣Asked a lot on Ruby Talk ‣The last thread included ideas from a lot of smart people

Slide 44

Slide 44 text

Example: A Unique ID ‣A common need ‣Asked a lot on Ruby Talk ‣The last thread included ideas from a lot of smart people ‣There are multiple Libraries for this

Slide 45

Slide 45 text

A UUID from Glue Code

Slide 46

Slide 46 text

A UUID from Glue Code id = `uuidgen`

Slide 47

Slide 47 text

Alternate Syntax

Slide 48

Slide 48 text

Alternate Syntax id = %x{uuidgen} id = %x@uuidgen@

Slide 49

Slide 49 text

Alternate Syntax ‣Use this syntax when you need backticks in your command ‣any symbol can be a delimiter id = %x{uuidgen} id = %x@uuidgen@

Slide 50

Slide 50 text

Alternate Syntax ‣Use this syntax when you need backticks in your command ‣any symbol can be a delimiter ‣You can also use the matching pairs: (…), […], {…}, and <…> ‣These nest properly id = %x{uuidgen} id = %x@uuidgen@

Slide 51

Slide 51 text

No Output Needed Using system()

Slide 52

Slide 52 text

Example: The Pasteboard

Slide 53

Slide 53 text

Example: The Pasteboard ‣I want to put a search string on OS X’s find “pasteboard” (clipboard)

Slide 54

Slide 54 text

Example: The Pasteboard ‣I want to put a search string on OS X’s find “pasteboard” (clipboard) ‣I don’t need any output for this operation

Slide 55

Slide 55 text

Example: The Pasteboard ‣I want to put a search string on OS X’s find “pasteboard” (clipboard) ‣I don’t need any output for this operation ‣I just need to know if the operation succeeded ‣A simple true or false will do

Slide 56

Slide 56 text

Ran or Didn’t Run

Slide 57

Slide 57 text

Ran or Didn’t Run if system "pbcopy -pboard find <<< 'New Search String'" puts "Search string set." else puts "Could not search string." end

Slide 58

Slide 58 text

Shell Expansion

Slide 59

Slide 59 text

Shell Expansion ENV["MY_VAR"] = "Set from Ruby" ! system "echo $MY_VAR" # >> Set from Ruby ! system "echo", "$MY_VAR" # >> $MY_VAR

Slide 60

Slide 60 text

Shell Expansion ENV["MY_VAR"] = "Set from Ruby" ! system "echo $MY_VAR" # >> Set from Ruby ! system "echo", "$MY_VAR" # >> $MY_VAR ‣A single argument goes through shell expansion ‣File glob patterns ‣Environment variables

Slide 61

Slide 61 text

Shell Expansion ENV["MY_VAR"] = "Set from Ruby" ! system "echo $MY_VAR" # >> Set from Ruby ! system "echo", "$MY_VAR" # >> $MY_VAR ‣A single argument goes through shell expansion ‣File glob patterns ‣Environment variables ‣Multiple arguments are passed without going through expansion

Slide 62

Slide 62 text

Handling Errors Mind the Expert Warnings

Slide 63

Slide 63 text

When Trouble strikes

Slide 64

Slide 64 text

When Trouble strikes ‣Remember to handle STDERR

Slide 65

Slide 65 text

When Trouble strikes ‣Remember to handle STDERR ‣Check process exit status

Slide 66

Slide 66 text

When Trouble strikes ‣Remember to handle STDERR ‣Check process exit status ‣Use popen3() when things get complicated

Slide 67

Slide 67 text

Example: Backups

Slide 68

Slide 68 text

Example: Backups ‣I want to backup a directory as part of a larger automation

Slide 69

Slide 69 text

Example: Backups ‣I want to backup a directory as part of a larger automation ‣The rsync program can do what I need

Slide 70

Slide 70 text

Example: Backups ‣I want to backup a directory as part of a larger automation ‣The rsync program can do what I need ‣I need to watch for problems and handle them gracefully ‣Possibly emailing a warning to the user

Slide 71

Slide 71 text

STDERR, The Problem Child

Slide 72

Slide 72 text

STDERR, The Problem Child

Slide 73

Slide 73 text

STDERR, The Problem Child

Slide 74

Slide 74 text

STDERR, The Problem Child

Slide 75

Slide 75 text

STDERR, The Problem Child

Slide 76

Slide 76 text

STDERR, The Problem Child

Slide 77

Slide 77 text

Taming STDERR

Slide 78

Slide 78 text

Taming STDERR dir = ARGV.shift or abort "USAGE: #{File.basename($PROGRAM_NAME)} DIR" results = `rsync -av --exclude '*.DS_Store' #{dir} #{dir}_backup 2>&1` if $?.success? # require "English"; $CHILD_STATUS.success? puts results.grep(/\A#{Regexp.escape(dir)}/) else puts "Error: Couldn't back up #{dir}" # … end

Slide 79

Slide 79 text

Taming STDERR dir = ARGV.shift or abort "USAGE: #{File.basename($PROGRAM_NAME)} DIR" results = `rsync -av --exclude '*.DS_Store' #{dir} #{dir}_backup 2>&1` if $?.success? # require "English"; $CHILD_STATUS.success? puts results.grep(/\A#{Regexp.escape(dir)}/) else puts "Error: Couldn't back up #{dir}" # … end

Slide 80

Slide 80 text

Taming STDERR dir = ARGV.shift or abort "USAGE: #{File.basename($PROGRAM_NAME)} DIR" results = `rsync -av --exclude '*.DS_Store' #{dir} #{dir}_backup 2>&1` if $?.success? # require "English"; $CHILD_STATUS.success? puts results.grep(/\A#{Regexp.escape(dir)}/) else puts "Error: Couldn't back up #{dir}" # … end

Slide 81

Slide 81 text

Taming STDERR dir = ARGV.shift or abort "USAGE: #{File.basename($PROGRAM_NAME)} DIR" results = `rsync -av --exclude '*.DS_Store' #{dir} #{dir}_backup 2>&1` if $?.success? # require "English"; $CHILD_STATUS.success? puts results.grep(/\A#{Regexp.escape(dir)}/) else puts "Error: Couldn't back up #{dir}" # … end

Slide 82

Slide 82 text

Proper Shell Escaping

Slide 83

Slide 83 text

Proper Shell Escaping # escape text to make it useable in a shell script as # one “word” (string) def escape_for_shell(str) str.to_s.gsub( /(?=[^a-zA-Z0-9_.\/\-\x7F-\xFF\n])/, '\\' ). gsub( /\n/, "'\n'" ). sub( /^$/, "''" ) end

Slide 84

Slide 84 text

Tips for Avoiding Errors

Slide 85

Slide 85 text

Tips for Avoiding Errors ‣Use full paths to programs and files whenever possible

Slide 86

Slide 86 text

Tips for Avoiding Errors ‣Use full paths to programs and files whenever possible ‣Send data to STDIN when you can

Slide 87

Slide 87 text

Tips for Avoiding Errors ‣Use full paths to programs and files whenever possible ‣Send data to STDIN when you can ‣If you can’t send it to STDIN, dump the data to a Tempfile and send that path

Slide 88

Slide 88 text

Tips for Avoiding Errors ‣Use full paths to programs and files whenever possible ‣Send data to STDIN when you can ‣If you can’t send it to STDIN, dump the data to a Tempfile and send that path ‣Remember to shell escape any command- line arguments that could contain dangerous characters (even spaces)

Slide 89

Slide 89 text

Full Control Using popen(), popen3(), and popen4()

Slide 90

Slide 90 text

Managing Streams

Slide 91

Slide 91 text

Managing Streams ‣Use popen() to manage STDIN and STDOUT

Slide 92

Slide 92 text

Managing Streams ‣Use popen() to manage STDIN and STDOUT ‣Use popen3() to manage STDIN, STDOUT, and STDERR ‣Use popen4() if you also need the PID

Slide 93

Slide 93 text

Example: Formatting Prose

Slide 94

Slide 94 text

Example: Formatting Prose ‣I want to rewrap some prose provided by the user

Slide 95

Slide 95 text

Example: Formatting Prose ‣I want to rewrap some prose provided by the user ‣Command-line arguments are not appropriate here ‣Complex shell Escaping ‣Size limit

Slide 96

Slide 96 text

Example: Formatting Prose ‣I want to rewrap some prose provided by the user ‣Command-line arguments are not appropriate here ‣Complex shell Escaping ‣Size limit ‣I need to send the prose to fmt via STDIN

Slide 97

Slide 97 text

Reading and Writing

Slide 98

Slide 98 text

Reading and Writing prose = <

Slide 99

Slide 99 text

Reading and Writing prose = <

Slide 100

Slide 100 text

Reading and Writing prose = <

Slide 101

Slide 101 text

Reading and Writing prose = <

Slide 102

Slide 102 text

Reading and Writing prose = <

Slide 103

Slide 103 text

Example: A Ruby Session

Slide 104

Slide 104 text

Example: A Ruby Session ‣I want to run some Ruby code

Slide 105

Slide 105 text

Example: A Ruby Session ‣I want to run some Ruby code ‣I don’t want that code to affect my current Ruby process

Slide 106

Slide 106 text

Example: A Ruby Session ‣I want to run some Ruby code ‣I don’t want that code to affect my current Ruby process ‣I may also need to do some special setup, hacking Ruby’s core, before this code is run

Slide 107

Slide 107 text

Example: A Ruby Session ‣I want to run some Ruby code ‣I don’t want that code to affect my current Ruby process ‣I may also need to do some special setup, hacking Ruby’s core, before this code is run ‣I need to format STDOUT and STDERR differently

Slide 108

Slide 108 text

With Error Handling

Slide 109

Slide 109 text

With Error Handling require "open3" ! Open3.popen3("ruby") do |stdin, stdout, stderr| stdin << %Q{puts "I am a puppet."; oops!()} stdin.close puts "Output:" puts stdout.read puts "Errors:" puts stderr.read end

Slide 110

Slide 110 text

With Error Handling require "open3" ! Open3.popen3("ruby") do |stdin, stdout, stderr| stdin << %Q{puts "I am a puppet."; oops!()} stdin.close puts "Output:" puts stdout.read puts "Errors:" puts stderr.read end

Slide 111

Slide 111 text

With Error Handling require "open3" ! Open3.popen3("ruby") do |stdin, stdout, stderr| stdin << %Q{puts "I am a puppet."; oops!()} stdin.close puts "Output:" puts stdout.read puts "Errors:" puts stderr.read end

Slide 112

Slide 112 text

With Error Handling require "open3" ! Open3.popen3("ruby") do |stdin, stdout, stderr| stdin << %Q{puts "I am a puppet."; oops!()} stdin.close puts "Output:" puts stdout.read puts "Errors:" puts stderr.read end

Slide 113

Slide 113 text

When you Also Need a PID

Slide 114

Slide 114 text

When you Also Need a PID ‣Install the POpen4 gem ‣Unix version ‣Windows versions

Slide 115

Slide 115 text

When you Also Need a PID ‣Install the POpen4 gem ‣Unix version ‣Windows versions ‣popen4() works like popen3() but it also passes you the PID for the child process ‣The PID is useful for sending the child process signals, possibly to kill the process

Slide 116

Slide 116 text

“If it’s on the Web, it has an API.” ! James Britt Using Web Tools

Slide 117

Slide 117 text

Don’t Forget the Web

Slide 118

Slide 118 text

Don’t Forget the Web If you need to… Use the tool…

Slide 119

Slide 119 text

Don’t Forget the Web If you need to… Use the tool… Read Content open-uri

Slide 120

Slide 120 text

Don’t Forget the Web If you need to… Use the tool… Read Content open-uri Write Form Data Net::HTTP

Slide 121

Slide 121 text

Don’t Forget the Web If you need to… Use the tool… Read Content open-uri Write Form Data Net::HTTP Emulate a Browser Mechanize

Slide 122

Slide 122 text

Don’t Forget the Web If you need to… Use the tool… Read Content open-uri Write Form Data Net::HTTP Emulate a Browser Mechanize Scrape HTML Hpricot

Slide 123

Slide 123 text

Example: Tracking Ruby

Slide 124

Slide 124 text

Example: Tracking Ruby ‣I want to download the latest version of Ruby as part of a larger automation

Slide 125

Slide 125 text

Example: Tracking Ruby ‣I want to download the latest version of Ruby as part of a larger automation ‣I want to verify the contents of the download

Slide 126

Slide 126 text

Example: Tracking Ruby ‣I want to download the latest version of Ruby as part of a larger automation ‣I want to verify the contents of the download ‣I want to expand the compressed archive

Slide 127

Slide 127 text

Simple Scraping

Slide 128

Slide 128 text

Simple Scraping require "open-uri" require "digest/md5" ! require "rubygems" require "hpricot" ! dl = Hpricot(open("http://www.ruby-lang.org/en/downloads/")) li = (dl / "div#content" / "ul" / "li").first url = (li / "a").first.attributes["href"] md5 = li.inner_html[/md5:.+?([A-Za-z0-9]{32})/, 1] ! rb = open(url) { |ftp| ftp.read } if Digest::MD5.hexdigest(rb) == md5 IO.popen("tar xvz", "wb") { |tar| tar << rb } else abort "Corrupt download" end

Slide 129

Slide 129 text

Simple Scraping require "open-uri" require "digest/md5" ! require "rubygems" require "hpricot" ! dl = Hpricot(open("http://www.ruby-lang.org/en/downloads/")) li = (dl / "div#content" / "ul" / "li").first url = (li / "a").first.attributes["href"] md5 = li.inner_html[/md5:.+?([A-Za-z0-9]{32})/, 1] ! rb = open(url) { |ftp| ftp.read } if Digest::MD5.hexdigest(rb) == md5 IO.popen("tar xvz", "wb") { |tar| tar << rb } else abort "Corrupt download" end

Slide 130

Slide 130 text

Simple Scraping require "open-uri" require "digest/md5" ! require "rubygems" require "hpricot" ! dl = Hpricot(open("http://www.ruby-lang.org/en/downloads/")) li = (dl / "div#content" / "ul" / "li").first url = (li / "a").first.attributes["href"] md5 = li.inner_html[/md5:.+?([A-Za-z0-9]{32})/, 1] ! rb = open(url) { |ftp| ftp.read } if Digest::MD5.hexdigest(rb) == md5 IO.popen("tar xvz", "wb") { |tar| tar << rb } else abort "Corrupt download" end

Slide 131

Slide 131 text

Simple Scraping require "open-uri" require "digest/md5" ! require "rubygems" require "hpricot" ! dl = Hpricot(open("http://www.ruby-lang.org/en/downloads/")) li = (dl / "div#content" / "ul" / "li").first url = (li / "a").first.attributes["href"] md5 = li.inner_html[/md5:.+?([A-Za-z0-9]{32})/, 1] ! rb = open(url) { |ftp| ftp.read } if Digest::MD5.hexdigest(rb) == md5 IO.popen("tar xvz", "wb") { |tar| tar << rb } else abort "Corrupt download" end

Slide 132

Slide 132 text

Use Caution

Slide 133

Slide 133 text

Use Caution ‣These scraping techniques see wider use than talking to external processes

Slide 134

Slide 134 text

Use Caution ‣These scraping techniques see wider use than talking to external processes ‣Ironically, they really do seem to be more fragile

Slide 135

Slide 135 text

Use Caution ‣These scraping techniques see wider use than talking to external processes ‣Ironically, they really do seem to be more fragile ‣tips for managing scraping code: ‣Abstract out the scraping code ‣Use more aggressive error handling ‣Make sure the maintenance is worth it

Slide 136

Slide 136 text

Summary Remain Strong

Slide 137

Slide 137 text

Pop Quiz

Slide 138

Slide 138 text

Pop Quiz Out of the box, can Ruby…

Slide 139

Slide 139 text

Pop Quiz Out of the box, can Ruby… Apply a difference algorithm to the contents of two Strings?

Slide 140

Slide 140 text

Pop Quiz Out of the box, can Ruby… Apply a difference algorithm to the contents of two Strings? Efficiently read a file line by line in reverse?

Slide 141

Slide 141 text

YES!

Slide 142

Slide 142 text

YES! ‣Don’t be afraid to use your powers

Slide 143

Slide 143 text

YES! ‣Don’t be afraid to use your powers ‣You will literally be able to accomplish anything

Slide 144

Slide 144 text

String Diff

Slide 145

Slide 145 text

String Diff require "tempfile" ! class String def diff(other) st = Tempfile.new("diff_self") ot = Tempfile.new("diff_other") st << self ot << other [st, ot].each { |t| t.flush } `diff -u #{st.path} #{ot.path}`[/^@.+\z/m] end end ! puts "one\ntwo\n".diff("one\nthree\n") # >> @@ -1,2 +1,2 @@ # >> one # >> -two # >> +three

Slide 146

Slide 146 text

Reading Backwards

Slide 147

Slide 147 text

Reading Backwards unless ARGV.size == 1 and File.exist? ARGV.first abort "Usage: #{File.basename($PROGRAM_NAME)} FILE" end ! last_five_lines = Array.new ! IO.popen("tail -r #{ARGV.shift}") do |tail| tail.each do |line| last_five_lines << line break if last_five_lines.size == 5 end end last_five_lines.reverse! ! puts last_five_lines

Slide 148

Slide 148 text

Questions?