Writing a modern CLI tool

Talk I gave at work about my experience writing a modern CLI tool and why I have chosen Go for writing it.

Transcript

1. Date Writing a modern CLI tool. Henrique Vicente 27th July, 2016
2. Originally I/O happened via a physically connected system console. Then…
4. From Time-sharing to today’s iTerm ✤ Type on the terminal
 
 cat ~/Downloads/behind-the-firewall.gif | imgcat
5. IBM XT ✤ First computer coming with standard hard- drive
6. Classic hypothetical program 1. Takes an input 2. Does some magic 3. Writes to output
7. POSIX real world program ✤ Standard in / out streams (stdin, stdout, stderr) ✤ Environment variables ✤ Working directory ✤ Arguments ✤ Exit code ✤ parent process, process id, process group,
8. What you want ✤ Static binaries ✤ Easy and secure distribution channel ✤ Easy to program ✤ Garbage collection ✤ High-level programming language ✤ Modern, clean, safe (type safety) ✤ Avoid the hype (unless if its arguments win) ✤ Lightweight startup for short-lived processes ✤ Great performance for long-lived daemons ✤ Reliable tests ✤ Culture ✤ No unbundled or dynamically linked dependencies ✤ No interpretation or virtual machines ✤ Trustworthy dependency managing
9. stdout and stderr unexpected merging issues
10. stdout and stderr unexpected merging issues ✤ ordering sort (e.g., verifying what is printed on tests) ✤ "updating" output (i.e., progress bar) modifying what you get
12. What about the children?
13. Child processes ✤ Zombies, orphans… ✤ Quite hard to work with
 reapers, actually.
14. From “gh” to “we”
16. Maintenance hell ✤ No tests ✤ Node.js 0.8 was still borning, API changes ✤ dumb npm packages… left-pad ¯\_(ツ)_/¯ ✤ dependency hell ✤ callback hell: no Promise() ✤ At certain point [I thought?] people hated me
18. Choosing a language for WeDeploy’s CLI
19. Build one pet project before committing to an unfamiliar language or ecosystem ✤ I wrote picel: is a light-weight, blazing fast* REST-ful micro service for image processing with a lean API https://github.com/henvic/picel on my own time
 
 After feeling comfortable with it, I sold the idea of building the we CLI tool
 
 
 * this statement isn't guaranteed.
20. Static executable binaries = less headaches ✤ No dependency hell ✤ No virtual machines ✤ Implication: disadvantage for interpreted languages
21. Why not C ✤ No protection against programmer mistakes ✤ No code coverage suite for high-level programming ✤ Pointers are too low level (memory based, not object based) ✤ Supporting different platforms is a hell.
22. Why not Bourne shell ✤ Environment ✤ No code coverage tool ✤ External dependencies hell (curl, etc) ✤ Performance ✤ OS X stuck in outdated 9 years old bash (shame on you, GNU GPLv3)
23. Why not JavaScript (ES6 + Node.JS) ✤ Almost good enough, but… ✤ Environment hell ✤ V8 virtual machine is light-weight, but not light enough ✤ requiring files is not cheap ✤ Dependency hell ✤ Can't trust npm today (and probably not in anything less than 5-years)
24. Why not Java, Scala, or… ✤ JVM ✤ Performance ✤ Lack of culture for this use-case ✤ Not trivial to try new things (Java 8 is way better, actually)
25. Why not Python, Ruby, or interpreted ones… ✤ Almost good enough, but… ✤ Environment hell ✤ Versioning issues (less dramatic than Node) ✤ Less files to require than npm ecosystem, still some… ✤ Dependency hell ✤ Why bother with runtimes?
26. LLVM is a killer compiler infrastructure… Why not Swift? ✤ A really serious competitor to Go on this area, but…
 
 Ecosystem is missing, hence couldn't even consider.
27. Go for-the-win ✤ Emerging, proven language by Google ✤ docker, Kubernetes, and many other great tools are written in Go ✤ ______ is written in Go ✤ Sexy syntax, simple API ✤ Garbage collected, small static binaries
28. Programming in Go vs. Node.js ✤ Static binaries vs. Environment nightmare ✤ Lean garbage collector vs. Virtual machine weight ✤ Almost no experience vs. Quite experienced ✤ Reliable, above average quality dependencies vs. lots of dependencies ✤ Powerful, above average quality built-in dependencies vs. fewer ✤ Compiled, better performance vs. interpreted, high startup cost for CLI tools ✤ Go routines easier concurrency support vs. Promises, events
29. Infrastructure & tests ✤ Complexity tests (unchecked errors, CRAP, etc) ✤ Missing documentation ✤ Unit tests ✤ Integration tests (simulated environment) ✤ Functional tests (real environments, can destroy things)
30. Configuration ✤ .ini format >> JSON, JSON 5, XML, etc.
32. Commands, subcommands, flags, persistent flags, suggestions, help command auto generation… ✤ You want all that: use a framework ✤ Don't forget about autocompletion
34. Autocompletion ✤ several levels of completeness and trade-offs ✤ from easy to very complex… ✤ zsh's autocomplete is very powerful:
 http://zsh.sourceforge.net/Doc/Release/Completion-System.html ✤ asking yourself: support autocomplete on what shells?
 zsh? bash? fish?
35. Simple autocompletion for bash [and zsh trough plugin] ✤ Add hidden "we autocomplete" command ✤ Traverses all commands and flags, filtering by arguments ✤ Prints out an output command or flag per line ✤ Uses custom function (next slide) to add dead simple (but limited) autocomplete functionality ✤ Works on bash natively ✤ Doesn't work on fish ✤ Works on zsh with plugin (bash_complete for zsh) we autocomplete script loads
37. ANSI escape code (colors, bold, italic…) ✤ You don't have to use rainbows. 256-color and even 24 bit true color support is possible nowadays on some shells, but your user loses control and you don't want it ✤ Remember you don't have exact control over the color palette your users use for ANSI colors. Just expect green to look green and red to look red. Not #ff0000 or #cc0000 or whatever you use ✤ Don't use fancy experimental features such as opening images on the terminal unless you like death threats (or have an experimental use case)
38. “oh, it compiles for Windows. So, I am going to setup a SSH server on a Windows 10 vm and all should work”… Except… ✤ Good luck with setting up a nice SSH setup on Windows with keys and everything.
 95 > 10 ⇒ Windows 95 > Windows 10 ✤ Good luck with autocomplete (kidding, just forget about') ✤ Wait, where are my colors? What are these weird characters?
 "fix": —no-color by default ✤ 10+ consoles: cmd, PowerShell, PowerCmd, cygwin, bash, …
 All inconsistent with each other (or it wouldn't be Windows) ✤ new bash support solves everything (just kidding - it's a lie) ✤ Everything can go goes wrong
39. How to distribute? ✤ curl | bash issues: security, errors, etc. ✤ need solid, replicable, release procedures to avoid breaking production (which you don't have control, probably) ✤ need auto-update strategy that doesn't suck
40. Release procedure ✤ git status should report directory is clean ✤ check for unchecked errors ✤ check for code issues and defects ✤ run tests (but functional) ✤ run tests (but functional) on a docker instance ✤ run functional tests on OS X, Linux, and Windows virtual machines ✤ ask for a release tag ✤ edit release tag message ✤ publish (if for the stable channel, publish on unstable actually) ✤ push tag to GitHub ✤ download release and run functional tests on OS X, Linux, and Windows virtual machines ✤ if for the stable channel, publish (promote) release [from the unstable] to the stable channel ✤ if for the stable channel, download release and run functional tests on OS X, Linux, and Windows virtual machines to report any abnormalitiesa
41. Questions?
42. Images ripped from ✤ https://en.wikipedia.org/wiki/System_console ✤ https://commons.wikimedia.org/wiki/File:IBM_1620.jpg ✤ https://commons.wikimedia.org/wiki/File:ASR-33_at_CHM.agr.jpg ✤ https://commons.wikimedia.org/wiki/File:Ibm_px_xt_color.jpg ✤ https://www.youtube.com/watch?v=RybNI0KB1bg ✤ https://en.wikipedia.org/wiki/File:Graveling.jpg ✤ http://deadlikeme.wikia.com/wiki/Dead_Like_Me_Wiki ✤ https://lastlog.de/blog/posts/nodejs_on_nixos_status.html ✤ http://maxogden.com/nested-dependencies.html