Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Command Line Applications in Ruby

keithrbennett
June 28, 2018
310

Command Line Applications in Ruby

Paris.rb Conf 2018 Presentation, including slides skipped during the presentation.

(Note: There is an animated GIF in the presentation (the "number of records processed" slide) that, when converted to PDF, does nothing.)

keithrbennett

June 28, 2018
Tweet

Transcript

  1. Command Line
    Applications
    in Ruby
    Keith Bennett
    @keithrbennett

    View Slide

  2. A Little Ruby Love
    There comes a time,
    When you find you need a tool,
    When a GUI's more trouble than it's worth,
    The command line is waiting there for you,
    All it takes is a little Ruby love,
    Movin' the mouse
    And then clickin'
    And then typin'
    All of that to do just one little thing...
    Automation
    We do it for the world,
    So why don't we do it for ourselves?
    A CLA (Command Line Application)
    will solve the problem,
    No need to fumble or stumble with a mouse
    And then you will come to find,
    A whole new peace of mind,
    As you use your newfound productivity.

    View Slide

  3. About This Talk
    • Limited to 20 minutes

    • Natural length would be closer to 50 minutes

    So:

    • much content will be mentioned only briefly

    • much source code will be passed over, left for you to view later

    If something interests you, feel free to:

    • ask about it during the Q&A

    • contact me afterwards

    View Slide

  4. Chapter 1
    Why A Command Line
    Application?

    View Slide

  5. View Slide

  6. View Slide

  7. View Slide

  8. View Slide

  9. View Slide

  10. Mac OS WiFi & Networking
    Command Line Utilities
    •airport

    •networksetup

    •ipconfig

    •scutil

    •ping

    View Slide

  11. View Slide

  12. View Slide

  13. airport is Secretive
    Buried 7 Levels Deep
    Hidden from Your Command Line

    View Slide

  14. airport Is Uncooperative
    airport often returns no data:

    ----------------------------------------------------------------
    Command: /Users/kbennett/bin/airport -s -x
    Duration: 0.0180 seconds
    ----------------------------------------------------------------
    ----------------------------------------------------------------
    Command: /Users/kbennett/bin/airport -s -x
    Duration: 0.0205 seconds
    ----------------------------------------------------------------
    ----------------------------------------------------------------
    Command: /Users/kbennett/bin/airport -s -x




    ----------------------------------------------------------------
    Command was executed 13 time(s).

    View Slide

  15. airport Is Ambigous
    Output in human readable mode does not
    differentiate between names with and without
    leading spaces:
    SSID BSSID RSSI CHANNEL HT CC SECURITY (auth/unicast/group)
    freebox_19CDE1 e4:9e:12:19:cd:e2 -92 13 Y -- WPA(PSK/AES,TKIP/TKIP)
    FreeWifi 68:a3:78:80:fb:c8 -76 2 Y -- NONE
    FreeWifi_secure 68:a3:78:80:fb:c9 -77 2 Y -- WPA2(802.1x/AES,TKIP/TKIP)
    freebox_KZLFCE 14:0c:76:f8:9c:4c -86 5 N -- WEP
    FreeWifi 14:0c:76:f8:9c:4d -86 5 Y -- NONE
    We cannot even know if the 2 “FreeWifi” entries

    refer to the same network.

    View Slide

  16. airport Is Archaic
    It does not support JSON or
    YAML, or even XML in its
    modern form (see next slide).

    View Slide

  17. Airport’s XML (-x) Mode

    SSID_STR
    NETGEAR65

    This solves the leading spaces problem, but is
    overly complex to use.

    Required XPath to parse it:
    //key[text() = "SSID_STR"][1]/following-sibling::*[1]'

    View Slide

  18. Idiomatic XML
    …might look like this instead:

    //ssid/@name
    and the XPath would be so much simpler:

    View Slide

  19. View Slide

  20. wifi_wand Usage Examples
    • list available networks

    • list available access points

    • show, clear, or set nameservers

    • get WiFi detailed status information

    • show network name (SSID)

    • show the password for a saved network

    • show wifi on/off and Internet on/off status

    View Slide

  21. wifi_wand
    Installation:
    gem install wifi_wand
    Project Page:
    https://github.com/keithrbennett/wifiwand

    View Slide

  22. Chapter 2
    The Term

    “Command Line
    Application”

    View Slide

  23. Command Line Interface (CLI)

    vs.

    Command Line Application (CLA)

    “Interface” exaggerates the simplicity
    compared with other apps.

    View Slide

  24. git!
    magick!
    ffmpeg!
    openssl!
    Complex CLA’s

    View Slide

  25. Chapter 3
    Decisions, Decisions

    View Slide

  26. Decision:
    Perfection vs. Expediency
    • Reliability

    • Accuracy

    • Performance

    • Operating System Support

    • Automatability

    View Slide

  27. Decision:
    Use MacOS command line utilities or
    system calls?
    • using utilities simplifies implementation

    • no need to write native code

    • using utilities complicates implementation

    • need to parse human readable output into objects

    • text formats may change over time

    • must deal with utility oddities, e.g. airport often
    returns no data

    • locales!

    View Slide

  28. Chapter 4
    Features

    View Slide

  29. wifi-wand Application
    Requirements
    • usable by non-Rubyist Mac users with command line
    expertise

    • easily installable

    • provides a shell

    • installable without additional gems, with caveats:

    • `pry` is only required if/when running in shell mode

    • `awesome_print` is optional, fallback is `pretty_print`

    • support using models without command line execution, i.e.
    can be used as CLA or not

    • Support YAML & JSON

    View Slide

  30. Consider Providing a Verbose Mode
    ➜ ~  wifi-wand na
    Nameservers: 1.1.1.1, 8.8.8.8
    ➜ ~  wifi-wand -v na
    ---------------------------------------------------------------
    Command: networksetup -getdnsservers Wi-Fi
    Duration: 0.0527 seconds
    1.1.1.1
    8.8.8.8
    ---------------------------------------------------------------
    Nameservers: 1.1.1.1, 8.8.8.8
    Non-Verbose Mode
    Verbose Mode

    View Slide

  31. Enable Bypassing the CLI
    You can use the models in your Ruby code without using the
    CommandLineInterface class. Here is a script ‘public_ip’:
    #!/usr/bin/env ruby
    require 'wifi-wand'
    require 'awesome_print'
    ap WifiWand::MacOsModel.new.public_ip_address_info
    When we run it, we get:

    View Slide

  32. Support Multiple Output
    Formats
    -o {i,j,k,p,y}
    outputs data in inspect, JSON, pretty
    JSON, puts, or YAML format when not
    in shell mode

    View Slide

  33. Support Multiple Output
    Formats
    ➜ ~  wifi-wand nameservers # default human readable mode
    Nameservers: 1.1.1.1, 8.8.8.8
    ➜ ~  wifi-wand -oi nameservers # 'inspect' mode
    ["1.1.1.1", "8.8.8.8"]
    ➜ ~  wifi-wand -oj nameservers # 'JSON' mode
    ["1.1.1.1","8.8.8.8"]
    ➜ ~  wifi-wand -ok nameservers # 'Pretty' JSON mode
    [
    "1.1.1.1",
    "8.8.8.8"
    ]
    ➜ ~  wifi-wand -op nameservers # 'puts' mode
    1.1.1.1
    8.8.8.8
    ➜ ~  wifi-wand -oy nameservers # 'YAML' mode
    ---
    - 1.1.1.1
    - 8.8.8.8

    View Slide

  34. Support Both Short & Long
    Command Names

    View Slide

  35. Provide a Shell
    • no need to type application name with every command

    • returned data as Ruby objects, enabling:

    • composite commands

    • custom behavior

    • storing data in variables/constants for later use

    View Slide

  36. Use ‘pry’ for the shell
    • full featured REPL

    • one can access other shell commands using the dot ('.')
    prefix (e.g. ‘.ping google.com’)

    • Pry commands such as `ls` can be accessed using '%'
    prefix (e.g. ‘%ls')

    View Slide

  37. Shell Example
    Use Case: Remove Saved “AIS” Networks
    [4] pry()> ais_nets = pref_nets.grep(/AIS/)
    => [" AIS SMART Login", ".@ AIS SUPER WiFi", "STARBUCKS_AIS"]
    [5] pry()> forget(*ais_nets)
    Password:
    => [" AIS SMART Login", ".@ AIS SUPER WiFi", “STARBUCKS_AIS"]
    [6] pry()> pref_nets.grep(/AIS/)
    => []
    # or, using the short command forms, and combining them into one
    statement:
    f(*pr.grep(/AIS/))

    View Slide

  38. Provide Shell Convenience
    Methods
    You can provide convenience methods not directly related to the DSL commands,
    with both abbreviated and complete names. For example:
    So that it can be used in the shell like this:
    def fancy_puts(object)
    puts fancy_string(object)
    end
    alias_method :fp, :fancy_puts

    View Slide

  39. Include Version & Project URL in Help Text

    View Slide

  40. Chapter 5
    Implementation

    View Slide

  41. Switches and Subcommands

    View Slide

  42. Creating the Gem
    ➜ temp git:(master) ✗  bundle gem foo
    Creating gem 'foo'...
    MIT License enabled in config
    create foo/Gemfile
    create foo/lib/foo.rb
    create foo/lib/foo/version.rb
    create foo/foo.gemspec
    create foo/Rakefile
    create foo/README.md
    create foo/bin/console
    create foo/bin/setup
    create foo/.gitignore
    create foo/.travis.yml
    create foo/.rspec
    create foo/spec/spec_helper.rb
    create foo/spec/foo_spec.rb
    create foo/LICENSE.txt
    Initializing git repo in /Users/kbennett/temp/foo
    As with any gem, the easiest way to create it is with bundler:

    View Slide

  43. The Application’s Entry Point:
    The Executable You Provide with Your
    Ruby Gem

    View Slide

  44. The Application’s Entry Point:
    The Executable You Provide with Your
    Ruby Gem
    • In your gemspec file, you specify the location of your
    executable(s):



    spec.executables = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
    # or:
    spec.executables = [‘wifi-wand’]
    • The executable can be (and probably should be) a simple
    wrapper around your other code, e.g.:

    #!/usr/bin/env ruby
    require_relative '../lib/wifi-wand/main'
    WifiWand::Main.new.call

    View Slide

  45. Using Available Tools to
    Simplify Implementation
    • pry - gem providing richly functional interactive shell
    (REPL)

    • awesome_print - gem that outputs simple Ruby objects
    in a clear, logical, and attractive format

    • Ruby’s built-in json and yaml support

    • open - Mac OS command to open a resource identifier
    with the default application for that resource type, e.g.
    `open https://www.whatismyip.com/` (Linux: xdg-
    open, Windows: start)

    View Slide

  46. Output Parsing Example:
    Detecting the WiFi Port
    # Identifies the (first) wireless network hardware port in the system, e.g. en0 or en1
    # This may not detect wifi ports with nonstandard names, such as USB wifi devices.
    def detect_wifi_port
    lines = run_os_command("networksetup -listallhardwareports”).split("\n")
    # Produces something like this:
    # ——————————————————————————————————————————————————————————
    # Hardware Port: Wi-Fi
    # Device: en0
    # Ethernet Address: ac:bc:32:b9:a9:9d
    #
    # Hardware Port: Bluetooth PAN
    # Device: en3
    # Ethernet Address: ac:bc:32:b9:a9:9e
    # ——————————————————————————————————————————————————————————
    wifi_port_line_num = (0...lines.size).detect do |index|
    /: Wi-Fi$/.match(lines[index])
    end
    if wifi_port_line_num.nil?
    raise Error.new(%Q{Wifi port (e.g. "en0") not found in output of: } +
    "networksetup -listallhardwareports”)
    else
    lines[wifi_port_line_num + 1].split(': ').last
    end

    View Slide

  47. Class Design

    View Slide

  48. The Command class and
    Its Instances
    class Command < Struct.new(:min_string, :max_string, :action); end
    def commands
    @commands_ ||= [
    Command.new('a', 'avail_nets', -> (*_options) { cmd_a }),
    Command.new('ci', 'ci', -> (*_options) { cmd_ci }),
    Command.new('co', 'connect', -> (*options) { cmd_co(*options) }),
    Command.new('cy', 'cycle', -> (*_options) { cmd_cy }),
    Command.new('d', 'disconnect', -> (*_options) { cmd_d }),
    Command.new('f', 'forget', -> (*options) { cmd_f(*options) }),
    Command.new('h', 'help', -> (*_options) { cmd_h }),
    Command.new('i', 'info', -> (*_options) { cmd_i }),
    Command.new('l', 'ls_avail_nets', -> (*_options) { cmd_l }),
    Command.new('na', 'nameservers', -> (*options) { cmd_na(*options) }),
    Command.new('ne', 'network_name', -> (*_options) { cmd_ne }),
    Command.new('of', 'off', -> (*_options) { cmd_of }),
    Command.new('on', 'on', -> (*_options) { cmd_on }),
    Command.new('ro', 'ropen', -> (*options) { cmd_ro(*options) }),
    Command.new('pa', 'password', -> (*options) { cmd_pa(*options) }),
    Command.new('pr', 'pref_nets', -> (*_options) { cmd_pr }),
    Command.new('q', 'quit', -> (*_options) { cmd_q }),
    Command.new('t', 'till', -> (*options) { cmd_t(*options) }),
    Command.new('w', 'wifi_on', -> (*_options) { cmd_w }),
    Command.new('x', 'xit', -> (*_options) { cmd_x })

    View Slide

  49. Commands are Processed by
    method_missing
    def method_missing(method_name, *method_args)
    method_name = method_name.to_s
    action = find_command_action(method_name)
    if action
    action.(*method_args)
    else
    puts(%Q{"#{method_name}" is not a valid command or option. } \
    << 'If you intend for this to be a string literal, ' \
    << 'use quotes or %q{}/%Q{}.')
    end
    end

    View Slide

  50. The find_command_action
    method
    def find_command_action(command_string)
    result = commands.detect do |cmd|
    cmd.max_string.start_with?(command_string) \
    && \
    command_string.length >= cmd.min_string.length
    # e.g. 'c' by itself should not work because
    # there are ‘ci’, ‘co’, and ‘cy’ commands
    end
    result ? result.action : nil
    end

    View Slide

  51. Formatter Lambda Hash
    parser.on("-o", "--output_format FORMAT", "Format output data") do |v|
    formatters = {
    'i' => ->(object) { object.inspect },
    'j' => ->(object) { object.to_json },
    'k' => ->(object) { JSON.pretty_generate(object) },
    'p' => ->(object) { sio = StringIO.new; sio.puts(object); sio.string },
    'y' => ->(object) { object.to_yaml }
    }
    choice = v[0].downcase
    unless formatters.keys.include?(choice)
    message = %Q{Output format "#{choice}" not in list of available formats} <<
    " (#{formatters.keys})."
    puts; puts message; puts
    raise Error.new(message)
    end
    options.post_processor = formatters[choice]
    end
    # ------------------------------------------------------
    # Used as follows, for example:
    if options.post_processor
    puts options.post_processor.(current_nameservers)
    else
    ...

    View Slide

  52. Showing Progress
    on the Console
    There are many ways to do this; one is to use the trick_bag
    gem’s TextModeStatusUpdater class:

    View Slide

  53. TextModeStatusUpdater
    Example
    #!/usr/bin/env ruby
    require 'trick_bag'
    count = 0
    text_generator = -> { "%7d records processed." % count }
    updater = \
    TrickBag::Io::TextModeStatusUpdater.new(text_generator)
    1_000_000.times do
    sleep 0.0000005 # do something real here
    updater.print if count % 10_000 == 0
    count += 1
    end
    puts "\nFinished processing #{count} records."

    View Slide

  54. Tips for Calling Other CLA’s
    • Redirect stderr to stdout (command 2>&1)

    • Use their exit codes (`$?`, which is threadlocal, not global)

    • Provide a way for the user to see the commands and
    their output

    • Centralize calling the OS in a single method, even if you
    think you'll never need it. Here’s mine:

    View Slide

  55. run_os_command
    def run_os_command(command, raise_on_error = true)
    if @verbose_mode
    puts CommandOutputFormatter.command_attempt_as_string(command)
    end
    start_time = Time.now
    output = `#{command} 2>&1` # join stderr with stdout
    if @verbose_mode
    puts "Duration: #{'%.4f' % [Time.now - start_time]} seconds"
    puts CommandOutputFormatter.command_result_as_string(output)
    end
    if $?.exitstatus != 0 && raise_on_error
    raise OsCommandError.new($?.exitstatus, command, output)
    end
    output
    end

    View Slide

  56. String Formatting
    with sprintf (%)
    Integers:
    "%d" % 3 # => "3"
    "%04d" % 3 # => "0003"
    "%x" % 256 # => "100"
    Floating Point Numbers:
    "%.4f" % (100.0 / 3) # => "33.3333"
    "%9.4f" % (100.0 / 3) # => " 33.3333”
    Strings:
    "%s" % 'foo' # => "foo"
    "%5s" % 'foo' # => " foo"
    "%-5s" % 'foo' # => "foo "

    View Slide

  57. Column Formatting with
    sprintf (%) Example
    -912.70 loan.to.sh Loan Payable to Shareholder
    300.00 tr.airfare Travel - Air Fares
    117.70 tr.mileage Travel - Mileage Allowance
    495.00 tr.perdiem.mi Travel - Per Diem (M & I)
    ------------
    0.00

    View Slide

  58. The trick_bag Gem
    • see README at https://github.com/keithrbennett/trick_bag

    • various useful methods, in these categories:

    Collection Access Enumerables Formatters
    Functional I/O Metaprogramming
    Networking Numeric Operators
    Timing Validations Core Types

    View Slide

  59. Shellwords
    [8] pry(main)> `export MY_PASSWORD=a b c; echo $MY_PASSWORD`
    => "a\n"
    [9] pry(main)> `export MY_PASSWORD=a\\ b\\ c; echo $MY_PASSWORD`
    => "a b c\n"
    [10] pry(main)> `export MY_PASSWORD="a b c"; echo $MY_PASSWORD`
    => "a b c\n"
    [11] pry(main)> `export MY_PASSWORD='a b c'; echo $MY_PASSWORD`
    => "a b c\n"
    [12] pry(main)> require 'shellwords'
    => false
    [13] pry(main)> `export MY_PASSWORD=#{Shellwords.escape('a b c')}; echo
    $MY_PASSWORD`
    => "a b c\n"
    [14] pry(main)> backslash = '\\'
    => "\\"
    [15] pry(main)> Shellwords.escape(backslash)
    => "\\\\"
    [16] pry(main)> Shellwords.escape(‘$')
    => "\\$"

    View Slide

  60. Chapter 6
    Consider Text Files for
    Your Data Source
    for Small Data
    I’m not saying we should get rid of data bases, but in some cases…

    View Slide

  61. The Beauty of Text Files
    (Part 1)
    • human readable data

    • can be edited with the user’s favorite text editor

    • can be version controlled with git et al.,

    • providing an audit log with date, author, & content
    changes

    • diffs are human readable

    • printable for archival purposes (practical in small quantities)

    View Slide

  62. The Beauty of Text Files
    (Part 2)
    • does not require any software (e.g. data base software) other
    than built-in software available on any OS

    • more likely to be readable in the distant future than other
    formats

    • editing can be faster for power users, no need to move hands
    away from the keyboard

    • can be implemented on even the smallest systems,
    eliminating the need for network connectivity

    • local or client/server

    View Slide

  63. Text File Data Example:
    Rock Books Accounting
    • available as the rock_books gem at https://github.com/
    keithrbennett/rock_books

    • still under development, but functional

    View Slide

  64. RockBooks Text File
    “Schema”
    • properties, analogous to instance variables (preceded by
    ‘@‘)

    • repeating data structures

    • transactions in journals

    • accounts in the chart of accounts

    • comments (preceded by ‘#’)

    View Slide

  65. RockBooks Chart of Accounts
    Input Data File Format
    (partial excerpt)
    @doc_type: chart_of_accounts
    @title: Chart of Accounts - 2017
    @entity: XYZ Consulting, Inc.
    # Assets
    ck.hsbc A HSBC Checking
    paypal A Paypal
    accts.rec A Accounts Receivable
    # Liabilities
    cc.hsbc.visa L Visa Credit Card
    loan.to.sh L Loan Payable to Shareholder

    View Slide

  66. RockBooks
    Check Disbursement Journal
    Input Data File Format
    (partial excerpt)
    @doc_type: journal
    @title: HSBC Checking Disbursements Journal - 2017
    @account_code: ck.hsbc
    @debit_or_credit: debit
    @short_name: ck.hsbc.disb
    @date_prefix: 2017-
    01-01 -5000.00 own.equity
    Initial Deposit from Shareholder
    # To specify further comments about this transaction that will
    # not be included in reports, specify them as comments
    # (like this one), that is, lines starting with ‘#’
    01-05 2000.00 cc.hsbc.visa
    01-07 -10000.00 sls.cons
    Invoice #437, Dec. 2016 work, ABC, Inc.

    View Slide

  67. Adding Convenience
    Methods to Objects
    Add methods to objects, not classes!

    reports = OpenStruct.new(book_set.all_reports($filter))
    # add hash methods for convenience
    def reports.keys; to_h.keys.map(&:to_s); end
    def reports.values; to_h.values; end
    # to access as array, e.g. `a.at(1)`
    def reports.at(index); self.public_send(keys[index]);

    View Slide

  68. RockBooks Filters
    In the RockBooks shell you can use a filter to include only a subset
    of all transactions. For example:

    jan_cowork_filter = all(
    month(2017, 1),
    account_code('cowork.fees'))
    filtered_entries = filter(all_entries, jan_cowork_filter)
    puts filtered_entries.size
    1
    puts filtered_entries.first.description
    New Work City coworking fee for the month of January
    Receipt: 01/2017-01-02-nwc.pdf

    View Slide

  69. Filters are Lambdas
    # A simple filter
    def month(target_year, target_month)
    ->(entry) do
    entry.date.year == target_year && \
    entry.date.month == target_month
    end
    end
    # A compound filter combining multiple other filters
    def all(*filters)
    ->(entry) { filters.all? { |filter| filter.(entry) } }
    end
    # The method that filters the Enumerable of journal entries
    def filter(entries, entry_filter)
    entries.select { |entry| entry_filter.(entry) }
    end

    View Slide

  70. YAML for (De/)Serialization
    [9] pry> bs0 = book_set; nil
    nil
    [10] pry> File.write('bs0.yaml', bs0.to_yaml); nil
    nil
    [11] pry> bs1 = YAML.load_file('bs0.yaml'); nil
    nil
    [12] pry> bs0 == bs1
    true
    The entire RockBooks data set can be backed up
    and restored using YAML in the shell:

    View Slide

  71. RockBooks
    Other Features
    • 1 step generation of entire report suite

    • interactive shell with data filters

    • option to specify options in an environment variable

    • directory specifications with reasonable defaults

    • output directories are created if nonexistent

    • use of $filter global

    View Slide

  72. Chapter 7
    Conclusion

    View Slide

  73. Ruby as a DSL-Friendly
    Language
    • optional parentheses

    • method_missing

    View Slide

  74. Ruby as a CLA Language
    • - Distribution can be an obstacle to non-Rubyists (unlike, e.g., go)

    • + (As stated previously) it’s a great DSL language!

    • + Interpreted, not compiled

    • + Rich toolset

    • + In addition to MRI, JRuby can be used.

    • Can drive JVM code/libraries written in Java, Scala, Clojure, Kotlin,
    etc.

    • Can be installed where native code cannot be installed but Java
    libraries are permitted.

    View Slide

  75. The End
    Feel free to contact me:

    Keith Bennett
    @keithrbennett on Github, Twitter, …

    Fin
    Command Line Applications & Libraries Mentioned
    Gem Name Github URL
    wifi_wand https://github.com/keithrbennett/wifiwand
    rock_books
    https://github.com/keithrbennett/
    rock_books
    trick_bag https://github.com/keithrbennett/trick_bag

    View Slide