Command Line Applications in Ruby Keith Bennett @keithrbennett

A Little Ruby Love There comes a time, When you find you need a tool, When a GUI's more trouble than it's worth, The command line is waiting there for you, All it takes is a little Ruby love, Movin' the mouse And then clickin' And then typin' All of that to do just one little thing... Automation We do it for the world, So why don't we do it for ourselves? A CLA (Command Line Application) will solve the problem, No need to fumble or stumble with a mouse And then you will come to find, A whole new peace of mind, As you use your newfound productivity.

About This Talk • Limited to 20 minutes • Natural length would be closer to 50 minutes So: • much content will be mentioned only briefly • much source code will be passed over, left for you to view later If something interests you, feel free to: • ask about it during the Q&A • contact me afterwards

Chapter 1 Why A Command Line Application?

No content

No content

No content

No content

No content

Mac OS WiFi & Networking Command Line Utilities •airport •networksetup •ipconfig •scutil •ping

No content

No content

airport is Secretive Buried 7 Levels Deep Hidden from Your Command Line

airport Is Uncooperative airport often returns no data: … ---------------------------------------------------------------- Command: /Users/kbennett/bin/airport -s -x Duration: 0.0180 seconds ---------------------------------------------------------------- ---------------------------------------------------------------- Command: /Users/kbennett/bin/airport -s -x Duration: 0.0205 seconds ---------------------------------------------------------------- ---------------------------------------------------------------- Command: /Users/kbennett/bin/airport -s -x … ---------------------------------------------------------------- Command was executed 13 time(s).

airport Is Ambigous Output in human readable mode does not differentiate between names with and without leading spaces: SSID BSSID RSSI CHANNEL HT CC SECURITY (auth/unicast/group) freebox_19CDE1 e4:9e:12:19:cd:e2 -92 13 Y -- WPA(PSK/AES,TKIP/TKIP) FreeWifi 68:a3:78:80:fb:c8 -76 2 Y -- NONE FreeWifi_secure 68:a3:78:80:fb:c9 -77 2 Y -- WPA2(802.1x/AES,TKIP/TKIP) freebox_KZLFCE 14:0c:76:f8:9c:4c -86 5 N -- WEP FreeWifi 14:0c:76:f8:9c:4d -86 5 Y -- NONE We cannot even know if the 2 “FreeWifi” entries refer to the same network.

airport Is Archaic It does not support JSON or YAML, or even XML in its modern form (see next slide).

Airport’s XML (-x) Mode … SSID_STR NETGEAR65 … This solves the leading spaces problem, but is overly complex to use. Required XPath to parse it: //key[text() = "SSID_STR"][1]/following-sibling::*[1]'

Idiomatic XML …might look like this instead: //ssid/@name and the XPath would be so much simpler:

No content

wifi_wand Usage Examples • list available networks • list available access points • show, clear, or set nameservers • get WiFi detailed status information • show network name (SSID) • show the password for a saved network • show wifi on/off and Internet on/off status

wifi_wand Installation: gem install wifi_wand Project Page: https://github.com/keithrbennett/wifiwand

Chapter 2 The Term
 “Command Line Application”

Command Line Interface (CLI) vs. Command Line Application (CLA) “Interface” exaggerates the simplicity compared with other apps.

git! magick! ffmpeg! openssl! Complex CLA’s

Chapter 3 Decisions, Decisions

Decision: Perfection vs. Expediency • Reliability • Accuracy • Performance • Operating System Support • Automatability

Decision: Use MacOS command line utilities or system calls? • using utilities simplifies implementation • no need to write native code • using utilities complicates implementation • need to parse human readable output into objects • text formats may change over time • must deal with utility oddities, e.g. airport often returns no data • locales!

Chapter 4 Features

wifi-wand Application Requirements • usable by non-Rubyist Mac users with command line expertise • easily installable • provides a shell • installable without additional gems, with caveats: • `pry` is only required if/when running in shell mode • `awesome_print` is optional, fallback is `pretty_print` • support using models without command line execution, i.e. can be used as CLA or not • Support YAML & JSON

Consider Providing a Verbose Mode ➜ ~  wifi-wand na Nameservers: 1.1.1.1, 8.8.8.8 ➜ ~  wifi-wand -v na --------------------------------------------------------------- Command: networksetup -getdnsservers Wi-Fi Duration: 0.0527 seconds 1.1.1.1 8.8.8.8 --------------------------------------------------------------- Nameservers: 1.1.1.1, 8.8.8.8 Non-Verbose Mode Verbose Mode

Enable Bypassing the CLI You can use the models in your Ruby code without using the CommandLineInterface class. Here is a script ‘public_ip’: #!/usr/bin/env ruby require 'wifi-wand' require 'awesome_print' ap WifiWand::MacOsModel.new.public_ip_address_info When we run it, we get:

Support Multiple Output Formats -o {i,j,k,p,y} outputs data in inspect, JSON, pretty JSON, puts, or YAML format when not in shell mode

Support Multiple Output Formats ➜ ~  wifi-wand nameservers # default human readable mode Nameservers: 1.1.1.1, 8.8.8.8 ➜ ~  wifi-wand -oi nameservers # 'inspect' mode ["1.1.1.1", "8.8.8.8"] ➜ ~  wifi-wand -oj nameservers # 'JSON' mode ["1.1.1.1","8.8.8.8"] ➜ ~  wifi-wand -ok nameservers # 'Pretty' JSON mode [ "1.1.1.1", "8.8.8.8" ] ➜ ~  wifi-wand -op nameservers # 'puts' mode 1.1.1.1 8.8.8.8 ➜ ~  wifi-wand -oy nameservers # 'YAML' mode --- - 1.1.1.1 - 8.8.8.8

Support Both Short & Long Command Names

Provide a Shell • no need to type application name with every command • returned data as Ruby objects, enabling: • composite commands • custom behavior • storing data in variables/constants for later use

Use ‘pry’ for the shell • full featured REPL • one can access other shell commands using the dot ('.') prefix (e.g. ‘.ping google.com’) • Pry commands such as `ls` can be accessed using '%' prefix (e.g. ‘%ls')

Shell Example Use Case: Remove Saved “AIS” Networks [4] pry()> ais_nets = pref_nets.grep(/AIS/) => [" AIS SMART Login", ".@ AIS SUPER WiFi", "STARBUCKS_AIS"] [5] pry()> forget(*ais_nets) Password: => [" AIS SMART Login", ".@ AIS SUPER WiFi", “STARBUCKS_AIS"] [6] pry()> pref_nets.grep(/AIS/) => [] # or, using the short command forms, and combining them into one statement: f(*pr.grep(/AIS/))

Provide Shell Convenience Methods You can provide convenience methods not directly related to the DSL commands, with both abbreviated and complete names. For example: So that it can be used in the shell like this: def fancy_puts(object) puts fancy_string(object) end alias_method :fp, :fancy_puts

Include Version & Project URL in Help Text

Chapter 5 Implementation

Switches and Subcommands

Creating the Gem ➜ temp git:(master) ✗  bundle gem foo Creating gem 'foo'... MIT License enabled in config create foo/Gemfile create foo/lib/foo.rb create foo/lib/foo/version.rb create foo/foo.gemspec create foo/Rakefile create foo/README.md create foo/bin/console create foo/bin/setup create foo/.gitignore create foo/.travis.yml create foo/.rspec create foo/spec/spec_helper.rb create foo/spec/foo_spec.rb create foo/LICENSE.txt Initializing git repo in /Users/kbennett/temp/foo As with any gem, the easiest way to create it is with bundler:

The Application’s Entry Point: The Executable You Provide with Your Ruby Gem

The Application’s Entry Point: The Executable You Provide with Your Ruby Gem • In your gemspec file, you specify the location of your executable(s): spec.executables = spec.files.grep(%r{^exe/}) { |f| File.basename(f) } # or: spec.executables = [‘wifi-wand’] • The executable can be (and probably should be) a simple wrapper around your other code, e.g.: #!/usr/bin/env ruby require_relative '../lib/wifi-wand/main' WifiWand::Main.new.call

Using Available Tools to Simplify Implementation • pry - gem providing richly functional interactive shell (REPL) • awesome_print - gem that outputs simple Ruby objects in a clear, logical, and attractive format • Ruby’s built-in json and yaml support • open - Mac OS command to open a resource identifier with the default application for that resource type, e.g. `open https://www.whatismyip.com/` (Linux: xdg- open, Windows: start)

Output Parsing Example: Detecting the WiFi Port # Identifies the (first) wireless network hardware port in the system, e.g. en0 or en1 # This may not detect wifi ports with nonstandard names, such as USB wifi devices. def detect_wifi_port lines = run_os_command("networksetup -listallhardwareports”).split("\n") # Produces something like this: # —————————————————————————————————————————————————————————— # Hardware Port: Wi-Fi # Device: en0 # Ethernet Address: ac:bc:32:b9:a9:9d # # Hardware Port: Bluetooth PAN # Device: en3 # Ethernet Address: ac:bc:32:b9:a9:9e # —————————————————————————————————————————————————————————— wifi_port_line_num = (0...lines.size).detect do |index| /: Wi-Fi$/.match(lines[index]) end if wifi_port_line_num.nil? raise Error.new(%Q{Wifi port (e.g. "en0") not found in output of: } + "networksetup -listallhardwareports”) else lines[wifi_port_line_num + 1].split(': ').last end

Class Design

The Command class and Its Instances class Command < Struct.new(:min_string, :max_string, :action); end def commands @commands_ ||= [ Command.new('a', 'avail_nets', -> (*_options) { cmd_a }), Command.new('ci', 'ci', -> (*_options) { cmd_ci }), Command.new('co', 'connect', -> (*options) { cmd_co(*options) }), Command.new('cy', 'cycle', -> (*_options) { cmd_cy }), Command.new('d', 'disconnect', -> (*_options) { cmd_d }), Command.new('f', 'forget', -> (*options) { cmd_f(*options) }), Command.new('h', 'help', -> (*_options) { cmd_h }), Command.new('i', 'info', -> (*_options) { cmd_i }), Command.new('l', 'ls_avail_nets', -> (*_options) { cmd_l }), Command.new('na', 'nameservers', -> (*options) { cmd_na(*options) }), Command.new('ne', 'network_name', -> (*_options) { cmd_ne }), Command.new('of', 'off', -> (*_options) { cmd_of }), Command.new('on', 'on', -> (*_options) { cmd_on }), Command.new('ro', 'ropen', -> (*options) { cmd_ro(*options) }), Command.new('pa', 'password', -> (*options) { cmd_pa(*options) }), Command.new('pr', 'pref_nets', -> (*_options) { cmd_pr }), Command.new('q', 'quit', -> (*_options) { cmd_q }), Command.new('t', 'till', -> (*options) { cmd_t(*options) }), Command.new('w', 'wifi_on', -> (*_options) { cmd_w }), Command.new('x', 'xit', -> (*_options) { cmd_x })

Commands are Processed by method_missing def method_missing(method_name, *method_args) method_name = method_name.to_s action = find_command_action(method_name) if action action.(*method_args) else puts(%Q{"#{method_name}" is not a valid command or option. } \ << 'If you intend for this to be a string literal, ' \ << 'use quotes or %q{}/%Q{}.') end end

The find_command_action method def find_command_action(command_string) result = commands.detect do |cmd| cmd.max_string.start_with?(command_string) \ && \ command_string.length >= cmd.min_string.length # e.g. 'c' by itself should not work because # there are ‘ci’, ‘co’, and ‘cy’ commands end result ? result.action : nil end

Formatter Lambda Hash parser.on("-o", "--output_format FORMAT", "Format output data") do |v| formatters = { 'i' => ->(object) { object.inspect }, 'j' => ->(object) { object.to_json }, 'k' => ->(object) { JSON.pretty_generate(object) }, 'p' => ->(object) { sio = StringIO.new; sio.puts(object); sio.string }, 'y' => ->(object) { object.to_yaml } } choice = v[0].downcase unless formatters.keys.include?(choice) message = %Q{Output format "#{choice}" not in list of available formats} << " (#{formatters.keys})." puts; puts message; puts raise Error.new(message) end options.post_processor = formatters[choice] end # ------------------------------------------------------ # Used as follows, for example: if options.post_processor puts options.post_processor.(current_nameservers) else ...

Showing Progress on the Console There are many ways to do this; one is to use the trick_bag gem’s TextModeStatusUpdater class:

TextModeStatusUpdater Example #!/usr/bin/env ruby require 'trick_bag' count = 0 text_generator = -> { "%7d records processed." % count } updater = \ TrickBag::Io::TextModeStatusUpdater.new(text_generator) 1_000_000.times do sleep 0.0000005 # do something real here updater.print if count % 10_000 == 0 count += 1 end puts "\nFinished processing #{count} records."

Tips for Calling Other CLA’s • Redirect stderr to stdout (command 2>&1) • Use their exit codes (`$?`, which is threadlocal, not global) • Provide a way for the user to see the commands and their output • Centralize calling the OS in a single method, even if you think you'll never need it. Here’s mine:

run_os_command def run_os_command(command, raise_on_error = true) if @verbose_mode puts CommandOutputFormatter.command_attempt_as_string(command) end start_time = Time.now output = `#{command} 2>&1` # join stderr with stdout if @verbose_mode puts "Duration: #{'%.4f' % [Time.now - start_time]} seconds" puts CommandOutputFormatter.command_result_as_string(output) end if $?.exitstatus != 0 && raise_on_error raise OsCommandError.new($?.exitstatus, command, output) end output end

String Formatting with sprintf (%) Integers: "%d" % 3 # => "3" "%04d" % 3 # => "0003" "%x" % 256 # => "100" Floating Point Numbers: "%.4f" % (100.0 / 3) # => "33.3333" "%9.4f" % (100.0 / 3) # => " 33.3333” Strings: "%s" % 'foo' # => "foo" "%5s" % 'foo' # => " foo" "%-5s" % 'foo' # => "foo "

Column Formatting with sprintf (%) Example -912.70 loan.to.sh Loan Payable to Shareholder 300.00 tr.airfare Travel - Air Fares 117.70 tr.mileage Travel - Mileage Allowance 495.00 tr.perdiem.mi Travel - Per Diem (M & I) ------------ 0.00

The trick_bag Gem • see README at https://github.com/keithrbennett/trick_bag • various useful methods, in these categories: Collection Access Enumerables Formatters Functional I/O Metaprogramming Networking Numeric Operators Timing Validations Core Types

Shellwords [8] pry(main)> `export MY_PASSWORD=a b c; echo $MY_PASSWORD` => "a\n" [9] pry(main)> `export MY_PASSWORD=a\\ b\\ c; echo $MY_PASSWORD` => "a b c\n" [10] pry(main)> `export MY_PASSWORD="a b c"; echo $MY_PASSWORD` => "a b c\n" [11] pry(main)> `export MY_PASSWORD='a b c'; echo $MY_PASSWORD` => "a b c\n" [12] pry(main)> require 'shellwords' => false [13] pry(main)> `export MY_PASSWORD=#{Shellwords.escape('a b c')}; echo $MY_PASSWORD` => "a b c\n" [14] pry(main)> backslash = '\\' => "\\" [15] pry(main)> Shellwords.escape(backslash) => "\\\\" [16] pry(main)> Shellwords.escape(‘$') => "\\$"

Chapter 6 Consider Text Files for Your Data Source for Small Data I’m not saying we should get rid of data bases, but in some cases…

The Beauty of Text Files (Part 1) • human readable data • can be edited with the user’s favorite text editor • can be version controlled with git et al., • providing an audit log with date, author, & content changes • diffs are human readable • printable for archival purposes (practical in small quantities)

The Beauty of Text Files (Part 2) • does not require any software (e.g. data base software) other than built-in software available on any OS • more likely to be readable in the distant future than other formats • editing can be faster for power users, no need to move hands away from the keyboard • can be implemented on even the smallest systems, eliminating the need for network connectivity • local or client/server

Text File Data Example: Rock Books Accounting • available as the rock_books gem at https://github.com/ keithrbennett/rock_books • still under development, but functional

RockBooks Text File “Schema” • properties, analogous to instance variables (preceded by ‘@‘) • repeating data structures • transactions in journals • accounts in the chart of accounts • comments (preceded by ‘#’)

RockBooks Chart of Accounts Input Data File Format (partial excerpt) @doc_type: chart_of_accounts @title: Chart of Accounts - 2017 @entity: XYZ Consulting, Inc. # Assets ck.hsbc A HSBC Checking paypal A Paypal accts.rec A Accounts Receivable # Liabilities cc.hsbc.visa L Visa Credit Card loan.to.sh L Loan Payable to Shareholder

RockBooks Check Disbursement Journal Input Data File Format (partial excerpt) @doc_type: journal @title: HSBC Checking Disbursements Journal - 2017 @account_code: ck.hsbc @debit_or_credit: debit @short_name: ck.hsbc.disb @date_prefix: 2017- 01-01 -5000.00 own.equity Initial Deposit from Shareholder # To specify further comments about this transaction that will # not be included in reports, specify them as comments # (like this one), that is, lines starting with ‘#’ 01-05 2000.00 cc.hsbc.visa 01-07 -10000.00 sls.cons Invoice #437, Dec. 2016 work, ABC, Inc.

Adding Convenience Methods to Objects Add methods to objects, not classes! reports = OpenStruct.new(book_set.all_reports($filter)) # add hash methods for convenience def reports.keys; to_h.keys.map(&:to_s); end def reports.values; to_h.values; end # to access as array, e.g. `a.at(1)` def reports.at(index); self.public_send(keys[index]);

RockBooks Filters In the RockBooks shell you can use a filter to include only a subset of all transactions. For example: jan_cowork_filter = all( month(2017, 1), account_code('cowork.fees')) filtered_entries = filter(all_entries, jan_cowork_filter) puts filtered_entries.size 1 puts filtered_entries.first.description New Work City coworking fee for the month of January Receipt: 01/2017-01-02-nwc.pdf

Filters are Lambdas # A simple filter def month(target_year, target_month) ->(entry) do entry.date.year == target_year && \ entry.date.month == target_month end end # A compound filter combining multiple other filters def all(*filters) ->(entry) { filters.all? { |filter| filter.(entry) } } end # The method that filters the Enumerable of journal entries def filter(entries, entry_filter) entries.select { |entry| entry_filter.(entry) } end

YAML for (De/)Serialization [9] pry> bs0 = book_set; nil nil [10] pry> File.write('bs0.yaml', bs0.to_yaml); nil nil [11] pry> bs1 = YAML.load_file('bs0.yaml'); nil nil [12] pry> bs0 == bs1 true The entire RockBooks data set can be backed up and restored using YAML in the shell:

RockBooks Other Features • 1 step generation of entire report suite • interactive shell with data filters • option to specify options in an environment variable • directory specifications with reasonable defaults • output directories are created if nonexistent • use of $filter global

Chapter 7 Conclusion

Ruby as a DSL-Friendly Language • optional parentheses • method_missing

Ruby as a CLA Language • - Distribution can be an obstacle to non-Rubyists (unlike, e.g., go) • + (As stated previously) it’s a great DSL language! • + Interpreted, not compiled • + Rich toolset • + In addition to MRI, JRuby can be used. • Can drive JVM code/libraries written in Java, Scala, Clojure, Kotlin, etc. • Can be installed where native code cannot be installed but Java libraries are permitted.

The End Feel free to contact me: Keith Bennett @keithrbennett on Github, Twitter, … Fin Command Line Applications & Libraries Mentioned Gem Name Github URL wifi_wand https://github.com/keithrbennett/wifiwand rock_books https://github.com/keithrbennett/ rock_books trick_bag https://github.com/keithrbennett/trick_bag