Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Command Line Applications in Ruby

keithrbennett
June 28, 2018
370

Command Line Applications in Ruby

Paris.rb Conf 2018 Presentation, including slides skipped during the presentation.

(Note: There is an animated GIF in the presentation (the "number of records processed" slide) that, when converted to PDF, does nothing.)

keithrbennett

June 28, 2018
Tweet

Transcript

  1. A Little Ruby Love There comes a time, When you

    find you need a tool, When a GUI's more trouble than it's worth, The command line is waiting there for you, All it takes is a little Ruby love, Movin' the mouse And then clickin' And then typin' All of that to do just one little thing... Automation We do it for the world, So why don't we do it for ourselves? A CLA (Command Line Application) will solve the problem, No need to fumble or stumble with a mouse And then you will come to find, A whole new peace of mind, As you use your newfound productivity.
  2. About This Talk • Limited to 20 minutes • Natural

    length would be closer to 50 minutes So: • much content will be mentioned only briefly • much source code will be passed over, left for you to view later If something interests you, feel free to: • ask about it during the Q&A • contact me afterwards
  3. airport Is Uncooperative airport often returns no data: … ----------------------------------------------------------------

    Command: /Users/kbennett/bin/airport -s -x Duration: 0.0180 seconds ---------------------------------------------------------------- ---------------------------------------------------------------- Command: /Users/kbennett/bin/airport -s -x Duration: 0.0205 seconds ---------------------------------------------------------------- ---------------------------------------------------------------- Command: /Users/kbennett/bin/airport -s -x <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http:// www.apple.com/DTDs/PropertyList-1.0.dtd"> <plist version="1.0"> … ---------------------------------------------------------------- Command was executed 13 time(s).
  4. airport Is Ambigous Output in human readable mode does not

    differentiate between names with and without leading spaces: SSID BSSID RSSI CHANNEL HT CC SECURITY (auth/unicast/group) freebox_19CDE1 e4:9e:12:19:cd:e2 -92 13 Y -- WPA(PSK/AES,TKIP/TKIP) FreeWifi 68:a3:78:80:fb:c8 -76 2 Y -- NONE FreeWifi_secure 68:a3:78:80:fb:c9 -77 2 Y -- WPA2(802.1x/AES,TKIP/TKIP) freebox_KZLFCE 14:0c:76:f8:9c:4c -86 5 N -- WEP FreeWifi 14:0c:76:f8:9c:4d -86 5 Y -- NONE We cannot even know if the 2 “FreeWifi” entries refer to the same network.
  5. airport Is Archaic It does not support JSON or YAML,

    or even XML in its modern form (see next slide).
  6. Airport’s XML (-x) Mode … <key>SSID_STR</key> <string>NETGEAR65</string> … This solves

    the leading spaces problem, but is overly complex to use. Required XPath to parse it: //key[text() = "SSID_STR"][1]/following-sibling::*[1]'
  7. Idiomatic XML …might look like this instead: <ssid name=“NETGEAR65” />

    //ssid/@name and the XPath would be so much simpler:
  8. wifi_wand Usage Examples • list available networks • list available

    access points • show, clear, or set nameservers • get WiFi detailed status information • show network name (SSID) • show the password for a saved network • show wifi on/off and Internet on/off status
  9. Command Line Interface (CLI) vs. Command Line Application (CLA) “Interface”

    exaggerates the simplicity compared with other apps.
  10. Decision: Use MacOS command line utilities or system calls? •

    using utilities simplifies implementation • no need to write native code • using utilities complicates implementation • need to parse human readable output into objects • text formats may change over time • must deal with utility oddities, e.g. airport often returns no data • locales!
  11. wifi-wand Application Requirements • usable by non-Rubyist Mac users with

    command line expertise • easily installable • provides a shell • installable without additional gems, with caveats: • `pry` is only required if/when running in shell mode • `awesome_print` is optional, fallback is `pretty_print` • support using models without command line execution, i.e. can be used as CLA or not • Support YAML & JSON
  12. Consider Providing a Verbose Mode ➜ ~  wifi-wand na

    Nameservers: 1.1.1.1, 8.8.8.8 ➜ ~  wifi-wand -v na --------------------------------------------------------------- Command: networksetup -getdnsservers Wi-Fi Duration: 0.0527 seconds 1.1.1.1 8.8.8.8 --------------------------------------------------------------- Nameservers: 1.1.1.1, 8.8.8.8 Non-Verbose Mode Verbose Mode
  13. Enable Bypassing the CLI You can use the models in

    your Ruby code without using the CommandLineInterface class. Here is a script ‘public_ip’: #!/usr/bin/env ruby require 'wifi-wand' require 'awesome_print' ap WifiWand::MacOsModel.new.public_ip_address_info When we run it, we get:
  14. Support Multiple Output Formats -o {i,j,k,p,y} outputs data in inspect,

    JSON, pretty JSON, puts, or YAML format when not in shell mode
  15. Support Multiple Output Formats ➜ ~  wifi-wand nameservers #

    default human readable mode Nameservers: 1.1.1.1, 8.8.8.8 ➜ ~  wifi-wand -oi nameservers # 'inspect' mode ["1.1.1.1", "8.8.8.8"] ➜ ~  wifi-wand -oj nameservers # 'JSON' mode ["1.1.1.1","8.8.8.8"] ➜ ~  wifi-wand -ok nameservers # 'Pretty' JSON mode [ "1.1.1.1", "8.8.8.8" ] ➜ ~  wifi-wand -op nameservers # 'puts' mode 1.1.1.1 8.8.8.8 ➜ ~  wifi-wand -oy nameservers # 'YAML' mode --- - 1.1.1.1 - 8.8.8.8
  16. Provide a Shell • no need to type application name

    with every command • returned data as Ruby objects, enabling: • composite commands • custom behavior • storing data in variables/constants for later use
  17. Use ‘pry’ for the shell • full featured REPL •

    one can access other shell commands using the dot ('.') prefix (e.g. ‘.ping google.com’) • Pry commands such as `ls` can be accessed using '%' prefix (e.g. ‘%ls')
  18. Shell Example Use Case: Remove Saved “AIS” Networks [4] pry()>

    ais_nets = pref_nets.grep(/AIS/) => [" AIS SMART Login", ".@ AIS SUPER WiFi", "STARBUCKS_AIS"] [5] pry()> forget(*ais_nets) Password: => [" AIS SMART Login", ".@ AIS SUPER WiFi", “STARBUCKS_AIS"] [6] pry()> pref_nets.grep(/AIS/) => [] # or, using the short command forms, and combining them into one statement: f(*pr.grep(/AIS/))
  19. Provide Shell Convenience Methods You can provide convenience methods not

    directly related to the DSL commands, with both abbreviated and complete names. For example: So that it can be used in the shell like this: def fancy_puts(object) puts fancy_string(object) end alias_method :fp, :fancy_puts
  20. Creating the Gem ➜ temp git:(master) ✗  bundle gem

    foo Creating gem 'foo'... MIT License enabled in config create foo/Gemfile create foo/lib/foo.rb create foo/lib/foo/version.rb create foo/foo.gemspec create foo/Rakefile create foo/README.md create foo/bin/console create foo/bin/setup create foo/.gitignore create foo/.travis.yml create foo/.rspec create foo/spec/spec_helper.rb create foo/spec/foo_spec.rb create foo/LICENSE.txt Initializing git repo in /Users/kbennett/temp/foo As with any gem, the easiest way to create it is with bundler:
  21. The Application’s Entry Point: The Executable You Provide with Your

    Ruby Gem • In your gemspec file, you specify the location of your executable(s): spec.executables = spec.files.grep(%r{^exe/}) { |f| File.basename(f) } # or: spec.executables = [‘wifi-wand’] • The executable can be (and probably should be) a simple wrapper around your other code, e.g.: #!/usr/bin/env ruby require_relative '../lib/wifi-wand/main' WifiWand::Main.new.call
  22. Using Available Tools to Simplify Implementation • pry - gem

    providing richly functional interactive shell (REPL) • awesome_print - gem that outputs simple Ruby objects in a clear, logical, and attractive format • Ruby’s built-in json and yaml support • open - Mac OS command to open a resource identifier with the default application for that resource type, e.g. `open https://www.whatismyip.com/` (Linux: xdg- open, Windows: start)
  23. Output Parsing Example: Detecting the WiFi Port # Identifies the

    (first) wireless network hardware port in the system, e.g. en0 or en1 # This may not detect wifi ports with nonstandard names, such as USB wifi devices. def detect_wifi_port lines = run_os_command("networksetup -listallhardwareports”).split("\n") # Produces something like this: # —————————————————————————————————————————————————————————— # Hardware Port: Wi-Fi # Device: en0 # Ethernet Address: ac:bc:32:b9:a9:9d # # Hardware Port: Bluetooth PAN # Device: en3 # Ethernet Address: ac:bc:32:b9:a9:9e # —————————————————————————————————————————————————————————— wifi_port_line_num = (0...lines.size).detect do |index| /: Wi-Fi$/.match(lines[index]) end if wifi_port_line_num.nil? raise Error.new(%Q{Wifi port (e.g. "en0") not found in output of: } + "networksetup -listallhardwareports”) else lines[wifi_port_line_num + 1].split(': ').last end
  24. The Command class and Its Instances class Command < Struct.new(:min_string,

    :max_string, :action); end def commands @commands_ ||= [ Command.new('a', 'avail_nets', -> (*_options) { cmd_a }), Command.new('ci', 'ci', -> (*_options) { cmd_ci }), Command.new('co', 'connect', -> (*options) { cmd_co(*options) }), Command.new('cy', 'cycle', -> (*_options) { cmd_cy }), Command.new('d', 'disconnect', -> (*_options) { cmd_d }), Command.new('f', 'forget', -> (*options) { cmd_f(*options) }), Command.new('h', 'help', -> (*_options) { cmd_h }), Command.new('i', 'info', -> (*_options) { cmd_i }), Command.new('l', 'ls_avail_nets', -> (*_options) { cmd_l }), Command.new('na', 'nameservers', -> (*options) { cmd_na(*options) }), Command.new('ne', 'network_name', -> (*_options) { cmd_ne }), Command.new('of', 'off', -> (*_options) { cmd_of }), Command.new('on', 'on', -> (*_options) { cmd_on }), Command.new('ro', 'ropen', -> (*options) { cmd_ro(*options) }), Command.new('pa', 'password', -> (*options) { cmd_pa(*options) }), Command.new('pr', 'pref_nets', -> (*_options) { cmd_pr }), Command.new('q', 'quit', -> (*_options) { cmd_q }), Command.new('t', 'till', -> (*options) { cmd_t(*options) }), Command.new('w', 'wifi_on', -> (*_options) { cmd_w }), Command.new('x', 'xit', -> (*_options) { cmd_x })
  25. Commands are Processed by method_missing def method_missing(method_name, *method_args) method_name =

    method_name.to_s action = find_command_action(method_name) if action action.(*method_args) else puts(%Q{"#{method_name}" is not a valid command or option. } \ << 'If you intend for this to be a string literal, ' \ << 'use quotes or %q{}/%Q{}.') end end
  26. The find_command_action method def find_command_action(command_string) result = commands.detect do |cmd|

    cmd.max_string.start_with?(command_string) \ && \ command_string.length >= cmd.min_string.length # e.g. 'c' by itself should not work because # there are ‘ci’, ‘co’, and ‘cy’ commands end result ? result.action : nil end
  27. Formatter Lambda Hash parser.on("-o", "--output_format FORMAT", "Format output data") do

    |v| formatters = { 'i' => ->(object) { object.inspect }, 'j' => ->(object) { object.to_json }, 'k' => ->(object) { JSON.pretty_generate(object) }, 'p' => ->(object) { sio = StringIO.new; sio.puts(object); sio.string }, 'y' => ->(object) { object.to_yaml } } choice = v[0].downcase unless formatters.keys.include?(choice) message = %Q{Output format "#{choice}" not in list of available formats} << " (#{formatters.keys})." puts; puts message; puts raise Error.new(message) end options.post_processor = formatters[choice] end # ------------------------------------------------------ # Used as follows, for example: if options.post_processor puts options.post_processor.(current_nameservers) else ...
  28. Showing Progress on the Console There are many ways to

    do this; one is to use the trick_bag gem’s TextModeStatusUpdater class:
  29. TextModeStatusUpdater Example #!/usr/bin/env ruby require 'trick_bag' count = 0 text_generator

    = -> { "%7d records processed." % count } updater = \ TrickBag::Io::TextModeStatusUpdater.new(text_generator) 1_000_000.times do sleep 0.0000005 # do something real here updater.print if count % 10_000 == 0 count += 1 end puts "\nFinished processing #{count} records."
  30. Tips for Calling Other CLA’s • Redirect stderr to stdout

    (command 2>&1) • Use their exit codes (`$?`, which is threadlocal, not global) • Provide a way for the user to see the commands and their output • Centralize calling the OS in a single method, even if you think you'll never need it. Here’s mine:
  31. run_os_command def run_os_command(command, raise_on_error = true) if @verbose_mode puts CommandOutputFormatter.command_attempt_as_string(command)

    end start_time = Time.now output = `#{command} 2>&1` # join stderr with stdout if @verbose_mode puts "Duration: #{'%.4f' % [Time.now - start_time]} seconds" puts CommandOutputFormatter.command_result_as_string(output) end if $?.exitstatus != 0 && raise_on_error raise OsCommandError.new($?.exitstatus, command, output) end output end
  32. String Formatting with sprintf (%) Integers: "%d" % 3 #

    => "3" "%04d" % 3 # => "0003" "%x" % 256 # => "100" Floating Point Numbers: "%.4f" % (100.0 / 3) # => "33.3333" "%9.4f" % (100.0 / 3) # => " 33.3333” Strings: "%s" % 'foo' # => "foo" "%5s" % 'foo' # => " foo" "%-5s" % 'foo' # => "foo "
  33. Column Formatting with sprintf (%) Example -912.70 loan.to.sh Loan Payable

    to Shareholder 300.00 tr.airfare Travel - Air Fares 117.70 tr.mileage Travel - Mileage Allowance 495.00 tr.perdiem.mi Travel - Per Diem (M & I) ------------ 0.00
  34. The trick_bag Gem • see README at https://github.com/keithrbennett/trick_bag • various

    useful methods, in these categories: Collection Access Enumerables Formatters Functional I/O Metaprogramming Networking Numeric Operators Timing Validations Core Types
  35. Shellwords [8] pry(main)> `export MY_PASSWORD=a b c; echo $MY_PASSWORD` =>

    "a\n" [9] pry(main)> `export MY_PASSWORD=a\\ b\\ c; echo $MY_PASSWORD` => "a b c\n" [10] pry(main)> `export MY_PASSWORD="a b c"; echo $MY_PASSWORD` => "a b c\n" [11] pry(main)> `export MY_PASSWORD='a b c'; echo $MY_PASSWORD` => "a b c\n" [12] pry(main)> require 'shellwords' => false [13] pry(main)> `export MY_PASSWORD=#{Shellwords.escape('a b c')}; echo $MY_PASSWORD` => "a b c\n" [14] pry(main)> backslash = '\\' => "\\" [15] pry(main)> Shellwords.escape(backslash) => "\\\\" [16] pry(main)> Shellwords.escape(‘$') => "\\$"
  36. Chapter 6 Consider Text Files for Your Data Source for

    Small Data I’m not saying we should get rid of data bases, but in some cases…
  37. The Beauty of Text Files (Part 1) • human readable

    data • can be edited with the user’s favorite text editor • can be version controlled with git et al., • providing an audit log with date, author, & content changes • diffs are human readable • printable for archival purposes (practical in small quantities)
  38. The Beauty of Text Files (Part 2) • does not

    require any software (e.g. data base software) other than built-in software available on any OS • more likely to be readable in the distant future than other formats • editing can be faster for power users, no need to move hands away from the keyboard • can be implemented on even the smallest systems, eliminating the need for network connectivity • local or client/server
  39. Text File Data Example: Rock Books Accounting • available as

    the rock_books gem at https://github.com/ keithrbennett/rock_books • still under development, but functional
  40. RockBooks Text File “Schema” • properties, analogous to instance variables

    (preceded by ‘@‘) • repeating data structures • transactions in journals • accounts in the chart of accounts • comments (preceded by ‘#’)
  41. RockBooks Chart of Accounts Input Data File Format (partial excerpt)

    @doc_type: chart_of_accounts @title: Chart of Accounts - 2017 @entity: XYZ Consulting, Inc. # Assets ck.hsbc A HSBC Checking paypal A Paypal accts.rec A Accounts Receivable # Liabilities cc.hsbc.visa L Visa Credit Card loan.to.sh L Loan Payable to Shareholder
  42. RockBooks Check Disbursement Journal Input Data File Format (partial excerpt)

    @doc_type: journal @title: HSBC Checking Disbursements Journal - 2017 @account_code: ck.hsbc @debit_or_credit: debit @short_name: ck.hsbc.disb @date_prefix: 2017- 01-01 -5000.00 own.equity Initial Deposit from Shareholder # To specify further comments about this transaction that will # not be included in reports, specify them as comments # (like this one), that is, lines starting with ‘#’ 01-05 2000.00 cc.hsbc.visa 01-07 -10000.00 sls.cons Invoice #437, Dec. 2016 work, ABC, Inc.
  43. Adding Convenience Methods to Objects Add methods to objects, not

    classes! reports = OpenStruct.new(book_set.all_reports($filter)) # add hash methods for convenience def reports.keys; to_h.keys.map(&:to_s); end def reports.values; to_h.values; end # to access as array, e.g. `a.at(1)` def reports.at(index); self.public_send(keys[index]);
  44. RockBooks Filters In the RockBooks shell you can use a

    filter to include only a subset of all transactions. For example: jan_cowork_filter = all( month(2017, 1), account_code('cowork.fees')) filtered_entries = filter(all_entries, jan_cowork_filter) puts filtered_entries.size 1 puts filtered_entries.first.description New Work City coworking fee for the month of January Receipt: 01/2017-01-02-nwc.pdf
  45. Filters are Lambdas # A simple filter def month(target_year, target_month)

    ->(entry) do entry.date.year == target_year && \ entry.date.month == target_month end end # A compound filter combining multiple other filters def all(*filters) ->(entry) { filters.all? { |filter| filter.(entry) } } end # The method that filters the Enumerable of journal entries def filter(entries, entry_filter) entries.select { |entry| entry_filter.(entry) } end
  46. YAML for (De/)Serialization [9] pry> bs0 = book_set; nil nil

    [10] pry> File.write('bs0.yaml', bs0.to_yaml); nil nil [11] pry> bs1 = YAML.load_file('bs0.yaml'); nil nil [12] pry> bs0 == bs1 true The entire RockBooks data set can be backed up and restored using YAML in the shell:
  47. RockBooks Other Features • 1 step generation of entire report

    suite • interactive shell with data filters • option to specify options in an environment variable • directory specifications with reasonable defaults • output directories are created if nonexistent • use of $filter global
  48. Ruby as a CLA Language • - Distribution can be

    an obstacle to non-Rubyists (unlike, e.g., go) • + (As stated previously) it’s a great DSL language! • + Interpreted, not compiled • + Rich toolset • + In addition to MRI, JRuby can be used. • Can drive JVM code/libraries written in Java, Scala, Clojure, Kotlin, etc. • Can be installed where native code cannot be installed but Java libraries are permitted.
  49. The End Feel free to contact me: Keith Bennett @keithrbennett

    on Github, Twitter, … Fin Command Line Applications & Libraries Mentioned Gem Name Github URL wifi_wand https://github.com/keithrbennett/wifiwand rock_books https://github.com/keithrbennett/ rock_books trick_bag https://github.com/keithrbennett/trick_bag