Get Power From Command Line

It's the deck for a modular course at NYCU. The goal of this course is to help the students and engineers who have focused on programming or data science to catch up the command line skills.

This deck covers an introduction of command line, TSV, CSV, JSON processing (using sed, awk, jq, etc.), file, process, container management (using Docker), system monitoring, SSH, SSH tunneling, and developing on command line.

This deck is not going to help you become a professional system administrator, but it will give you a wonderful head start on becoming a superb command line user!

Transcript

1. Mosky Get Power From Command Line The Command Line Skills You Missed
3. Financial Freedom
4. Power Freedom
5. Mosky Architect, Pinkoi • Data Science With Python • Statistical Regression With Python • Hypothesis Testing With Python • Practicing Python 3 • The mediums. • The PyCons, etc.
6. 1. Introduction to command line 2. Text processing: e.g., CSV, JSON, parallelizing. 3. System management: e.g., fi les, processes, monitoring. 4. Work on a remote machine: e.g., SSH, SSH tunneling. 5. Developing on CLI Outline
7. What commands? • POSIX • Portable Operating System Interface • IEEE Std 1003.1-2008 • List of Unix commands – Wikipedia • Ubuntu 22.04 LTS Server • curl, gawk, git, htop, less, lsof, man - db, rsync, strace, tmux, vim, ssh, etc. • The de facto standard • tree, ripgrep, jq, etc. • The modern alternatives • bat (cat), fd - f i nd ( fi nd), etc.
8. How to use this deck? • Practice makes perfect: • Try every single command in this deck, and 
 downloading the deck as a PDF may make it easier. • Be careful of the spaces and newlines after paste. • Visit the links, but don't read; scan them to index in your brain.
9. Introduction to command line
10. “Command Notes” • Memorize all the commands? No. • You may try to note them, and make them searchable. • Inputs & Outputs. • Tags. • Like →
11. Get your terminal emulator • iTerm for macOS • See also: Mosky’s Preferences in macOS § iTerm and search “Bash 5” in the doc! • Windows Terminal • GNOME Terminal / Konsole
12. Connect to ... • macOS with GNU/Linux commands • Install the GNU/Linux commands on macOS • WSL • Windows Subsystem for Linux • Install Linux on Windows with WSL • Ubuntu • Or,
13. The container • Get Docker • $ docker pull ubuntu:22.04 
 $ docker network create ubuntu - n 
 $ docker run - dit - p 2222 : 22 - p 8000 : 8000 - - network ubuntu - n - - name ubuntu - c ubuntu:22.04 
 $ docker exec - it ubuntu - c bash • ubuntu - c# yes | unminimize && apt install curl gawk git htop less lsof man - db rsync strace tmux vim ssh - y
14. The container • Get Docker • $ docker pull ubuntu:22.04 
 $ docker network create ubuntu - n 
 $ docker run - dit - p 2222 : 22 - p 8000 : 8000 - - network ubuntu - n - - name ubuntu - c ubuntu:22.04 
 $ docker exec - it ubuntu - c bash • ubuntu - c# yes | unminimize && apt install curl gawk git htop less lsof man - db rsync strace tmux vim ssh - y $ hints as a normal user. ubuntu - c# hints as a superuser and in the Ubuntu container. They are default in the Ubuntu Server, but not in the container image.
15. Checkpoint: Log into the container • You shall see → • Hints: • $ docker stop ubuntu - c; docker rm ubuntu - c # to start over. • $ docker start ubuntu - c 
 $ docker exec - it ubuntu - c bash # after reboot if any.
16. Read the manual • $ man man 
 # Try `\ - k` <n> <u> 
 # and <h> <q> `\ - k` • $ man - k ls • $ ls - - help | less # <q> to exit. • $ ls - - help | grep ' - l' • $ help for • $ type man • $ type ls • $ type for • Google: “man yes” • explainshell.com • Bash Reference Manual
17. Read the manual • $ man man 
 # Try `\ - k` <n> <u> 
 # and <h> <q> `\ - k` • $ man - k ls • $ ls - - help | less # <q> to exit. • $ ls - - help | grep ' - l' • $ help for • $ type man • $ type ls • $ type for • Google: “man yes” • explainshell.com • Bash Reference Manual Two spaces.
18. Edit commands like a ninja • Bash Keystrokes – Mosky’s Notes • $ ec<Tab>Hello, World! • $ <Up><C-w>Ctrl-W! • $ <Up><M-C-h>H! • $ <Up><C-h>? • Also works in Python shell and many other places. • See also: • “CTRL-p CTRL-q” to detach – Docker Docs • macOS Shortcuts – Mosky’s Notes • Windows Shortcuts – Mosky’s Notes
19. Edit commands like a ninja • Bash Keystrokes – Mosky’s Notes • $ ec<Tab>Hello, World! • $ <Up><C-w>Ctrl-W! • $ <Up><M-C-h>H! • $ <Up><C-h>? • Also works in Python shell and many other places. • See also: • “CTRL-p CTRL-q” to detach – Docker Docs • macOS Shortcuts – Mosky’s Notes • Windows Shortcuts – Mosky’s Notes Or <M-Rubout>. The <Rubout> is <Delete> or <Backspace>. Or <C-p>, if not from the docker. “Space” vs. “Alphanumeric” The <M> is 
 <Opt> or <Alt>.
20. Checkpoint: “Hey, Ctrl-K!” • $ echo Hello, World! • $ <Up><your keystrokes> 
 Hey, Ctrl-K! • Hints: • <C-a>, <M-f>, <C-k>
21. Quoting • $ echo Hello, World! # 2 arguments. • $ echo 'Hello, World!' # 1 argument. • $ echo Hello, $HOSTNAME! • $ echo "Hello, $HOSTNAME!" • $ echo "Hello, \$HOSTNAME!" • $ echo 'Hello, $'HOSTNAME! • $ echo $'Hello,\nWorld' # ANSI-C Quoting
22. Pipe, AND, OR • $ echo 'Hello, World!' | grep Hello • $ echo 'Hello, World!' | grep Hello && echo Found! • $ echo 'Hello, World!' | grep Hey && echo Found! • $ echo 'Hello, World!' | grep Hey || echo Not found. • $ echo 'Hello, World!' | grep Hello && echo Found! || echo Not found. • $ echo 'Hello, World!' | grep Hey && echo Found! || echo Not found.
23. Checkpoint: Explain the echo by yourself • explainshell.com • $ echo 'Hello, World!' | grep Hello && echo Found! || echo Not found.
24. Loops • $ cd ~ # cd: change directory • ~$ touch a.txt b.txt c.txt • $ for name in *.txt; do echo mv $name ${name/.txt/.csv}; done 
 # It's a dry run. • $ for name in *.txt; do echo mv $name ${name/.txt/.csv}; done • $ while true; do date; docker container top ubuntu - c; echo; sleep 3; done • $ until docker start ubuntu - c; do sleep 3; done
25. Redirections fi le descriptor:name = 0:stdin, 1:stdout, 2:stderr • $ cd ~ • ~$ echo Hello, File! >hello.txt • $ cat hello.txt • $ cat hello.txt | grep Hello • $ grep <hello.txt Hello • $ grep <hello.txt Hello >out.txt • $ grep <hello.txt Hello >>out.txt # appends to out.txt.
26. • $ grep <hello.txt - - bad Hello >out.txt # still prints errors. • $ grep <hello.txt - - bad Hello >out.txt 2>err.txt • $ grep <hello.txt - - bad Hello >out.txt 2>&1 • $ grep <hello.txt - - bad Hello &>both.txt • $ grep <hello.txt - - bad Hello 2>/dev/null
27. • $ cat <<HEREDOC | grep paste 
 > Hello, Here Doc! 
 > It's useful to paste something. 
 > HEREDOC • $ cat <<<'Hello, Here String!' • $ python3 <<<'print(1234 * * 1234)' 
 # C's unsigned long is 0 to 18446744073709551615, btw. 
 # Useful!
28. Expansions & Substitutions • $ echo x.{txt,csv} • $ echo ~ • $ echo ~root # or any username. • $ cd ~ • ~$ touch a.{txt,csv} • $ echo a.* • $ echo x.* # - > x.* • $ echo ?.* • $ echo [abcx].txt • $ echo [abcx].csv
29. • $ echo $name • $ echo ${name} • $ echo ${name/.txt/.csv} • $ echo `echo 'Hello, Echo!'` • $ echo $(echo 'Hello, Echo!') • $ cat <(echo 'Hello, Cat!') • See also: • Bash scripting cheatsheet § Parameter expansions • Shell Style Guide § Variable expansion
30. Environments & Parameters • Bash scans its own 
 environment variables and creates shell parameters. • $ echo $PATH • $ A=1 env | grep A= # - > A=1 • $ echo $A # A is empty. • $ A=2 • $ echo $A # - > 2 • $ env | grep A= # - > <empty> • $ export A • $ # export A=20 • $ env | grep A= # - > A=2 • $ A=3 echo $A # - > 2 • $ # = = = $ A=3 echo 2 • $ export - n A && env | grep A=
31. Job Control • $ vi a.txt # and <C-z>. • $ vi b.txt # and <C-z>. • $ jobs • $ fg # <CTRL-G> shows f i lename. • $ fg - # like Cmd-Tab or Alt-Tab. • $ fg 1 # `:q` to exit vim. • $ kill %2 • $ sleep 3 && echo Hi! & # = = = • $ bash - c 'sleep 3 && echo Hi!' 
 # and <C-z>. • $ bg
32. Aliases • $ alias ll • $ type ll • $ ls - - help | grep ' -[alF]' • $ alias hi=history • $ hi 10 • $ alias . . = 'cd . . ' • $ cd /usr/bin • usr: User System Resources • $ . .
33. History • <Up>, <Down>; <C-p>, <C-n> • <C-r>, <C-s>: search history. • $ <a long long command> #TAG • $ <C-r>#TAG • $ stty - ixon # <C-s> needs it. • <C-c>: discard the editing. • $ history 10 • $ history - a • $ history - n • $ history - - help | grep ' -[an]'
34. Checkpoint: Hack the which • $ which echo 
 /usr/bin/echo • $ ??? which echo 
 bash: which: No such f i le or directory • Hints: • which searches the PATH for the executable fi le.
35. Text processing
37. TSV Create TSV fi le using cat • $ cd • ~$ cat <<EOS >cmds.tsv 
 > <Open https: / / en.wikipedia.org/wiki/List_of_Unix_commands, 
 copy the table, and paste here.> 
 > EOS
38. TSV Create TSV fi le using vim • $ cd • ~$ vi cmds.tsv • (vim) i # goes the insert mode. • (vim) <Open https: / / en.wikipedia.org/wiki/List_of_Unix_commands, 
 copy the table, and paste here.> • (vim) <Esc> # backs to the normal mode. • (vim) :x<CR> # goes the command - line mode, write and quit.
39. • If any garbled text: • ubuntu - c$ LANG=C.UTF-8 bash • or, • (host)$ docker exec - it ubuntu - c env LANG=C.UTF-8 bash
40. • $ wc - l cmds.tsv # wc: word count • $ cat cmds.tsv • $ head cmds.tsv • $ tail cmds.tsv • $ grep <cmds.tsv cat 
 # = = = grep cat cmds.tsv • grep: g/re/p in the ed command, globally search for a regular expression and print matching lines. • $ grep <cmds.tsv '^.cat' • $ grep <cmds.tsv '^.cat' - o
41. TSV is literally everywhere.
42. Checkpoint: Copy the head and paste to a sheet • A sheet in Google Sheets, Excel, LibreO ff i ce Calc, like → • Hints: • Just like how did you paste the table from the browser to the terminal. • Copy the output on the terminal and paste it to the sheet.
43. TSV (cont.) • $ seq 1 3 • $ seq 1 3 | column • $ seq 1 3 | column | hexdump - c • $ seq 1 3 | column - t
44. • $ head <cmds.tsv | column - s '<C-v><TAB>' - t • $ column - - help | grep ' -[st]' • $ head <cmds.tsv -1 
 # = = = $ head - n 1 cmds.tsv • $ head <cmds.tsv -1 | sed 's/(Option code)//' • sed: stream editor • $ sed <cmds.tsv - n - e 1p - e /^.cat/p • $ sed - - help | grep ' -[ne]'
45. • $ cut <cmds.tsv - f 2 | head • $ cut <cmds.tsv - f 2 | tail +2 | head • $ cut <cmds.tsv - f 2 | tail +2 | sort • $ cut <cmds.tsv - f 2 | tail +2 | sort | uniq • $ cut <cmds.tsv - f 2 | tail +2 | sort | uniq - c • $ cut <cmds.tsv - f 2 | tail +2 | sort | uniq - c | sort - n
46. • $ awk <cmds.tsv -F $'\t' '{ print $2 }' | head • $ awk <cmds.tsv -F $'\t' '$5 ~ /BSD/' | column - s $'\t' - t • $ awk <cmds.tsv -F $'\t' '$5 ~ /BSD/ { counts[$2]++ } END { for (cat in counts) print counts[cat] "\t" cat }' • $ awk <cmds.tsv -F $'\t' '$5 ~ /BSD/ { counts[$2]++ } END { for (cat in counts) print counts[cat] "\t" cat }' | sort - n
47. • See also: • The GNU Awk User’s Guide • 6.3.2.2 Comparison Operators • 6.1.1.3 Regular Expression Constants • 8.1.5 Scanning All Elements of an Array
48. How many mandatory text processing does POSIX have? Checkpoint: • $ ??? <cmds.tsv ??? | wc - l 
 ??? • Hints: • Either grep or awk is a good choice. • Try = = and & & if you choose awk. • “Text processing” • “Mandatory”
49. TSV (cont.) • $ cut <cmds.tsv - f 2 | head • $ awk <cmds.tsv '{ print $2 }' | head • $ vimdiff <(cut <cmds.tsv - f 2 | head) <(awk <cmds.tsv '{ print $2 }' | head) 
 # `:qa` to exit. • $ awk <cmds.tsv -F $'\t' '{ print $2 }' | head
50. • $ cut <cmds.tsv - f 2 | head >a • $ awk <cmds.tsv '{ print $2 }' | head >b • $ vimdiff a b • $ diff a b • $ diff a b - y # using the side by side format. • $ diff a b - u # using the unif i ed format. • $ diff - - help | grep ' -[yu]'
51. • $ sudo apt install bat - y # or, • # apt install bat - y • $ alias bat=batcat • $ diff a b | bat # <q> to exit.
52. Interact with the system clipboards • (macOS)$ pbpaste | tail -1 | pbcopy • The way I make slides with highlighted codes: • $ pbpaste | highlight - u utf-8 -S python - s solarized - light - k Monaco -K 28 -O rtf | pbcopy # and then paste into a slide. • (X11)$ xclip - o | tail -1 | xclip • (WSL)$ powershell.exe Get-Clipboard | tail -1 | clip.exe • See also: vor0nwe/clip – Gist for a neat clip script in WSL.
53. cURL • $ curl https://google.com • $ curl - v https://google.com • $ curl -L https://google.com
55. JSON with jq • $ sudo apt install jq - y # or, • # apt install jq - y • jq is like sed for JSON. • $ cd • ~$ curl https://raw.githubusercontent.com/5etools - mirror-1/5etools - mirror-1.github.io/ master/data/bestiary/bestiary - mm.json >monsters.json # or, • ~$ curl https://raw.githubusercontent.com/hazmole/TheGiddyLimit.github.io/master/ data/bestiary/bestiary - mm.json >monsters_zh_tw.json • $ wc - l monsters.json # - > 70811
56. • $ jq <monsters.json • $ jq <monsters.json | less • $ jq <monsters.json -C | less -R • $ grep <monsters.json dragon - - color=always | less -R • $ jq <monsters.json | bat - l json • $ bat monsters.json • $ vi monsters.json # `:q` to exit.
57. • $ jq <monsters.json '.monster[] | .name' | head • $ jq <monsters.json '.monster[] | select(.type == "dragon") | .name' | head • $ jq <monsters.json '.monster[] | select(.type == "dragon") | [.hp.average, .name, .type, .page]' | head
58. • $ jq <monsters.json '.monster[] | select(.type == "dragon") | [.hp.average, .name, .type, .page] | @tsv' | head • $ jq <monsters.json '.monster[] | select(.type == "dragon") | [.hp.average, .name, .type, .page] | @tsv' - r | head • $ jq <monsters.json '["HP", "Name", "Type", "Page"], (.monster[] | select(.type == "dragon") | [.hp.average, .name, .type, .page]) | @tsv' - r | head • $ jq <monsters.json '["HP", "Name", "Type", "Page"], (.monster[] | select(.type == "dragon") | [.hp.average, .name, .type, .page]) | @tsv' - r >dragons.tsv
59. • $ sort <dragons.tsv # Sort by HP. • $ sort <dragons.tsv - k 2 - - debug | head # Try to sort by Name. • $ sort <dragons.tsv - k 2,2 - - debug | head • $ sort <dragons.tsv - t $'\t' - k 2,2 - - debug | head
60. What is the first undead monster in the book? Checkpoint: • $ jq ??? | tail -1 
 ??? • Hints: • “undead” • “page”
61. JSON with gron • $ sudo apt install gron - y # or, • # apt install gron - y • $ jq <monsters.json '.monster[] | select(.type == "dragon") | .name' | head • $ jq <monsters.json '[ .monster[] | select(.type == "dragon") ]' >dragons.json • $ bat dragons.json • $ gron dragons.json • $ gron dragons.json | grep -E 'json\[[0-9]+\]\.name =' • $ gron dragons.json | grep -E 'json\[[0-9]+\]\.name =' | gron - u
62. . * [a - z] ^$ ? + | Fixed string . * [a - z] ^$ ? + | $ grep -F BRE . * [a - z] ^$ ? + | $ grep 
 $ grep -G ERE . * [a - z] ^$ ? + | $ grep -E Glob . * [a - z] ^$ ? + | Fixed string, BRE, ERE, vs. glob
63. Test the regex types • $ echo 'i++' | grep -F '.+' - o # f i xed string (not regex) 
 <empty> • $ echo 'i++' | grep -G '.+' - o # basic regular expression (BRE) 
 i + • $ echo 'i++' | grep -E '.+' - o # extended regular expression (ERE) 
 i + +
64. • See also: • Regex cheatsheet for BRE, ERE, Vim, Python's re, PCRE (Perl). • regex101: a regex debugger for Python, JavaScript, Java, Golang, PHP. • jq Manual • tomnomnom/gron – GitHub • yq for YAML.
65. Checkpoint: The regex type of less may be? • BRE-like ... ERE-like ... or not regex? • Hints: • How to put i + + into less? • How to search something in less?
66. Bulk editing • $ cp dragons.tsv cats.tsv • $ cp dragons.tsv cats_2.tsv • $ head cats * .tsv • $ sed s/Dragon/Cat/ cats * .tsv | less • $ sed - e s/Dragon/Cat/ - e s/dragon/cat/ cats * .tsv | less • $ sed - e s/Dragon/Cat/g - e s/dragon/cat/g cats * .tsv | less • $ sed - e s/Dragon/Cat/g - e s/dragon/cat/g cats * .tsv - i
67. CSV with CSVKit • $ sudo apt install csvkit - y # or, • # apt install csvkit - y • $ jq <monsters.json '["HP", "Name", "Type", "Page"], (.monster[] | select(.type == "dragon") | [.hp.average, .name, .type, .page]) | @tsv' - r >dragons.tsv • $ jq <monsters.json '["HP", "Name", "Type", "Page"], (.monster[] | select(.type == "dragon") | [.hp.average, .name, .type, .page]) | @csv' - r >dragons.csv • $ bat dragons.csv
68. • $ csvcut <dragons.csv - c Name,HP • $ csvsort <dragons.csv - c HP,Page • $ csvstat <dragons.csv - c HP,Name,Type • $ csvlook <dragons.csv # - > a Markdown table • $ csvgrep <dragons.csv - c Name - m 'Adult Black Dragon' • $ csvsql <dragons.csv - - query "select avg(HP) from stdin where Name like 'Adult%'"
69. • See also: • Tutorial – csvkit • 1.4. in2csv: the Excel killer • 3.3. csvsql and sql2csv: ultimate power
70. Parallelizing • $ time sleep 1 • $ seq 3 | xargs - i echo sleep {} # xargs: extended arguments • $ seq 3 | xargs - i echo sleep 1 • $ time seq 3 | xargs - i sleep 1 # takes 3 seconds. • $ time seq 3 | xargs - i -P 3 sleep 1 # takes 1 second. • $ man xargs | less +'/ -[iI]'
71. System management
72. File management • $ cd • ~$ mkdir mydir && cd mydir • ~/mydir$ touch a b .h • $ mkdir - p x/y/z • $ cp b c • $ rm b • $ ls • $ ll • $ type ll • $ ll - t 
 # List and sort by time. • $ ll -S - h 
 # List and sort by size.
73. • $ sudo apt install tree - y # or, • # apt install tree - y • $ cd ~/mydir • ~/mydir$ ls • $ tree • $ tree - a # List all f i les. • $ tree -L 1 # List one level deep. • $ tree - f # Print full paths.
74. • $ cd • ~$ ln - s mydir mydir_link # Make a symbolic link. • $ ls mydir_link • $ ll mydir_link • $ cp - r mydir mydir_backup # Copy recursively. • $ ll mydir_backup
75. • $ tar czf mydir.tar.gz mydir • $ rmdir mydir 
 # - > failed . . . not empty. • $ rmdir mydir/x/y/z 
 # OK! • $ rm - r mydir/ • $ tar xzf mydir.tar.gz • $ tree • $ du - h - d 1 # du: disk usage • $ df - h # df: disk free
76. Describe one difference between cp - r and cp - a Checkpoint: • Read and search the manual! • Hints: • $ man cp # `/` to search; `n` to next; `h` to get help. • $ cp - - help | grep . . .
77. Terminology EN program process thread concurrency parallelism TW 程式 程序 執⾏緒 並⾏性 平⾏性 CN 程序 进程 线程 并发性 并⾏性
78. Process management • $ ps # ps: process status • $ ps a # a: includes other users. • $ ps ax # x: includes other ttys or no tty's. 
 # tty: teletypewriter • $ ps axu # u: user - oriented format • $ ps f # f: "forest" • $ man ps | grep -E ' (VSZ|RSS|TTY|STAT|START|TIME) '
79. • $ vi test.txt # and <C-z>. • $ ps u | grep vi • $ ps u | grep [v]i • $ ps u | sed - n - e 1p - e /[v]i/p • $ ps axu | grep [v]i | 
 awk '{ rss += $6 } END { printf "%.4g MB\n", rss/1024 }' • (macOS)$ ps axu | grep [C]hrome | 
 awk '{ rss += $6 } END { printf "%.4g MB\n", rss/1024 }'
80. • $ vi test.txt # and <C-z>. • $ pgrep v # - > <pid> • $ pgrep vi # - > <pid> • $ pgrep test # - > <empty> • $ pgrep - f test # - > <pid> • $ pgrep - f test - l • $ pgrep - - help | grep ' -[fl]' • $ # kill <pid> • $ pkill - f test # -SIGTERM • $ fg • $ vi test.txt # and <C-z>. • $ pkill - f test -SIGKILL • See also: • Signal (IPC) – Wikipedia
81. Monitoring • $ sudo apt install iotop iftop - y # or, # if: interface • # apt install iotop iftop - y • $ free - h • $ man free | grep -E ' (total|used|free|shared|buff/cache|available)' -A 1 • $ vi test.txt # and <C-z>. • $ top • $ htop • Select a process, and <F9> (kill)
82. • $ cd • ~$ python3 - m http.server &> my.log 
 # <C-c> to interrupt. • Open http://localhost:8000/. • Open another terminal: • # iftop # <q> to exit. • $ tail - f ~/my.log • $ less +F ~/my.log 
 # Try <C-c> ` - i` `/http` <F>. • $ watch tail -10 ~/my.log
83. • The commands which are useful but not relevant to containers: • $ sudo iotop • Important system logs: • $ sudo vi /var/log/syslog • $ sudo vi /var/log/auth.log
84. Checkpoint: Monitor your HTTP server by less • Try to get three new logs. • Hints: • Click on your browser!
85. Monitoring (cont.) • $ python3 - c 'print(1234)' • $ strace python3 - c 'print(1234)' • $ sudo strace - p <pid> • $ ps x • $ pgrep bash • $ tty • $ pgrep bash - t pts/1 
 # pts: pseudo terminal slave • $ lsof - p `pgrep bash - t pts/1` 
 # lsof: list open f i les • $ lsof /usr/bin/bash • $ man lsof | less - i +/^examples • $ htop • Select a process, and: • <s> (strace) • <l> (lsof)
86. • $ sudo apt install sysstat - y # or, • # apt install sysstat - y • $ sar 1 
 # Report CPU stats per second. • $ sar 
 # Report today's CPU stats. • $ sar - f /var/log/sysstat/sa10 • $ sar - f /var/log/sysstat/sa10 
 - s 12 : 00 - e 15 : 00 • $ man sar | grep 'Report ' -B 1 • $ sar - f /var/log/sysstat/sa10 
 - s 12 : 00 - e 15 : 00 - r - - human 
 # Report memory stats. • See also: • sysstat/sysstat – GitHub • Amazon CloudWatch • Grafana
87. Container management • OCI: Open Container Initiative • Docker fi le: part of OCI spec. • image: container image • Why? • Once you pack your app as a container image, it can be run anywhere seamlessly.
88. • (host)$ docker container commit ubuntu - c ubuntu - cli - power 
 # = = = $ docker container commit . . . • $ docker image ls 
 # = = = $ docker images
89. • $ docker image save ubuntu - cli - power >ubuntu - cli - power.tar • $ docker image rm ubuntu - cli - power 
 # = = = $ docker rmi ubuntu - cli - power • $ docker image ls • $ docker image load <ubuntu - cli - power.tar • $ # docker image (save|load) = = = docker image (save|load) • $ docker image ls
90. • $ docker container create - - name ls - expert ubuntu - cli - power man ls • $ docker container ls - a # = = = $ docker ps - a • $ docker container start ls - expert • $ docker container ls - a # - > Exited (0) 4 seconds ago • $ docker container start - a ls - expert # Start and attach. • $ docker container rm ls - expert • $ # docker container (create|start|stop|rm) = = = 
 docker container (create|start|stop|rm)
91. • $ docker container cp ubuntu - c:/root/cmds.tsv ~/cmds.tsv 
 # = = = $ docker cp . . . • $ docker container cp ~/cmds.tsv ubuntu - c:/root/cmds_2.tsv • $ docker container run - dit - v ~/container:/root - - name volume - c ubuntu - cli - power 
 # = = = $ docker run . . .
92. Work on a remote machine
93. Run programs after disconnect • $ tmux # tmux: terminal multiplexer • (tmux)$ while true; do date; sleep 1; done # and then <C-b><d>. • $ tmux ls # ls: the alias of list - sessions. • $ tmux a # a: attach • (tmux) <C-b><?> # List key bindings; <q> to exit. • (tmux) <C-c><C-d> • $ tmux ls
94. • The way I start the server of JupyterLab: • $ tmux new - s jupyter - lab - d 'cd && jupyter lab - - no - browser'
95. The local key pair • (local)$ ll ~/.ssh/id_rsa * • If nothing there: • (local)$ ssh - keygen • (local)$ cat ~/.ssh/id_rsa.pub # pub: public key • id_rsa is the private key which proves who you are. • Do not share id_rsa – the private key – to anyone. • Simply generate a new pair when you have a little doubt about the old pair.
96. Add a user in the remote host • (remote)# new_user=<new username> • (remote)# useradd - m - s /bin/bash $new_user • (remote)# su $new_user • <new user>$ mkdir ~/.ssh 
 <new user>$ cat <<EOS >~/.ssh/authorized_keys 
 > <paste the content of id_rsa.pub> 
 > EOS 
 <new user>$ exit • (remote)# service ssh start
97. Connect to a remote host • (local)$ ssh ssh://<user>@localhost:2222 
 # = = = $ ssh <user>@localhost - p 2222 
 # $ ssh <host> # if port is 22 and usernames are the same. • If there is a warning and you're sure you changed something in the remote: • $ sed <line no>d ~/.ssh/known_hosts - i
98. ssh -A: Allow the remote to use the local key pair • (local) $ ssh ssh://<user>@localhost:2222 • (remote)$ ssh ubuntu - c 
 # . . . password: • (local) $ ssh ssh://<user>@localhost:2222 -A • (remote)$ ssh ubuntu - c # OK • The ubuntu-c works since the docker run --network enables the DNS resolution between containers. • See also: Consul, for the similar functionality without Docker.
99. ssh -J: Jump to a host in a private network • (local)$ ssh ubuntu - c 
 # - > Could not resolve hostname ubuntu - c • (local)$ ssh -J ssh://<user>@localhost:2222 ubuntu - c • -J is securer than -A; use -J when applicable.
100. Transfer files between local and remote • (remote)$ cp cmds.tsv ~<user> • (local) $ rsync - e 'ssh ssh://<user>@localhost:2222' :~/cmds.tsv ~ 
 # $ rsync <user>@<host > : ~/cmds.tsv ~ # if port is 22. • (local) $ rsync - e 'ssh ssh://<user>@localhost:2222' ~/cmds.tsv :~/cmds_2.tsv • Deprecating scp – LWN.net, but scp is still a common way nowadays: • (local)$ scp scp://<user>@localhost:2222/~/cmds.tsv ~ 
 # $ scp <user>@<host > : ~/cmds.tsv ~ # if port is 22. • (local)$ scp ~/cmds.tsv scp://<user>@localhost:2222/~/cmds_2.tsv
101. Mount a remote directory • (local)$ sudo apt install sshfs # or macFUSE for macOS. • (local)$ mkdir ~/sshfs_container • (local)$ sshfs <user>@localhost:/home/<user> - p 2222 ~/sshfs_container • (local)$ ls ~/sshfs_container • (local)$ umount ~/sshfs_container
102. ssh -L: Forward a local port to a remote port • As known as “SSH tunneling”. • (remote)$ python3 - m http.server 1234 # <C-c> to interrupt. • (local)$ ssh ssh://<user>@localhost:2222 -L 4321:ubuntu - c:1234 -Nf • Open http://localhost:4321/. • localhost:4321 
 → localhost:2222 → ubuntu-c:22 
 → ubuntu-c:1234 → http.server 1234 • (local)$ pgrep - f 4321 • (local)$ # pkill - f 4321
103. Access the HTTP server by http://localhost:8765/ Checkpoint: • Open http://localhost:8765/ and there should be a page. • Hints: • Use ssh -L.
104. ssh -D: Start a proxy server • (local)$ ssh ssh://<user>@localhost:2222 -D 1080 -Nf • (remote)# iftop • Con fi gure the system proxy or Proxy SwitchyOmega for Chrome. • (remote)# python3 - m http.server 80 • Open http://ubuntu-c/. • For accessing the services in a private network by private DNS and standard ports! • If the server has another public IP, try Google “my ip”. • $ man ssh | grep ' -[LDNf]' -A 1
105. System proxy for macOS
106. Proxy SwitchyOmega for Chrome
108. • See also: • A simple ~/.ssh/con fi g for macOS • A simple Bash con fi g for both macOS and Linux
109. Developing on CLI
110. Edit a config • $ cp ~/.bashrc ~/backup_bashrc • $ echo 'alias bat=batcat' >>~/.bashrc • Re-enter the shell to take e ff ect. • $ grep batcat ~/.bashrc - n • $ sed <line no>d ~/.bashrc - i • $ grep batcat ~/.bashrc - n
111. • $ vi ~/.bashrc • (vim) <C-End> # <Fn-Ctrl-Right> • (vim) i<Right><CR> # or `a<CR>` • (vim) alias bat=batcat # literally • (vim) <Esc> # backs to Normal • (vim) :x<CR> • $ grep batcat ~/.bashrc - n • $ vi ~/.bashrc • (vim) /cat<CR> • (vim) n . . . • (vim) dd • (vim) :x<CR> • $ grep batcat ~/.bashrc - n
112. The mini-cheatsheet of Vim 1. / n • Search; Next. 2. i <Esc> :x • Go Insert; Back to Normal; Write and quit. 3. dd • Delete a line. • It's all you need to know now.
 Of course, the more you know, the more e ff i ciency you'll get. 💪
113. Spell checking • $ echo 'Oops. Does it spll like this?'<C-x><C-e> • (vim) :set spell<CR> • (vim) /sp<CR> • (vim) z= # List spelling suggestions. • (vim) 1<CR> • (vim) :x<CR>
114. • See also: • $ vimtutor # Learn Vim in 30 mins! • (vim) :help dd, for example. • Vim Keystrokes – Mosky’s Notes • Vim’s Search Tips – Mosky’s Notes • Vim’s Replace Tips – Mosky’s Notes • Mosky's Vim Con fi gs
115. Checkpoint: Add your bat • (host)$ docker exec - it ubuntu - c bash • ubuntu - c# bat ~/.bashrc # It should work immediately. • Hints: • You may use echo to add. • Use grep to con fi rm. • Re-enter the shell to take e ff ect.
116. Developing Slak: Collect data from Slack like a pro. ⚡ 1. System Tools: the common tools which are supposed to be installed in the system. • git: the version control system. • pip: the Python package manager. • pipenv: one of Python package environment managers. • The Docker (containers) provides OS-level isolation; Pipenv provides Python interpreter and package–level isolation. 2. Project Tools: the tools of this project, good to be managed within the project by, for example, pipenv. • ipdb: a simple yet beautiful debugger. • pytest: the testing framework. 3. Personal Tools: the tools of a developer, may be installed in one's system. • flake8: the static analyzer (linter). • mypy: the static type checker. • black: one of the formatters.
117. • $ sudo apt install pip - y # or, # System Tool • # apt install pip - y • $ pip install pipenv # System Tool • $ cd • ~$ git clone - - depth 1 https://github.com/moskytw/slak.git
118. • ~$ cd slak • ~/slak$ bat Pipf i le • $ pipenv sync - - dev • $ slak • $ python slak.py • $ for cmd in python python3 slak; do type $cmd; done • $ pipenv shell • (slak)$ slak • $ python slak.py • $ for cmd in python python3 slak; do type $cmd; done
119. • (slak)$ python - m ipdb slak.py # Project Tool, `q` to exit. • $ pytest # Project Tool • $ pytest - - cov • $ pytest - - cov - - cov - report html • $ cd htmlcov • ~/slak/htmlcov$ python - m http.server # <C-c> to interrupt. • Open http://localhost:8000/. • $ exit # or <C-d> to exit the shell of pipenv.
120. • (my mac)$ for cmd in flake8 mypy black xxx; do type $cmd; done 
 flake8 is hashed (/usr/local/bin/flake8) 
 mypy is aliased to `\mypy - - ignore - missing - imports' 
 black is aliased to `black - - line - length 79 - - skip - string - normalization' 
 - bash: type: xxx: not found • $ for cmd in flake8 mypy black xxx; do type $cmd >/dev/null; done 
 - bash: type: xxx: not found
121. • $ cd ~/slak • ~/slak$ flake8 slak.py • $ mypy slak.py • $ black slak.py
122. • The di ff erent languages have di ff erent toolsets; however, the concepts and the categories are similar, e.g.: • $ vi /etc/nginx/sites - enabled/your - site.conf • $ nginx - tc /etc/nginx/sites - enabled/your - site.conf 
 # Test the conf i g f i le.
123. • See also: • git - the simple guide: “no deep shit ;)” • Git Documentation • Cheat Sheets • Pro Git, the book • Reference Manual • Mosky’s Pipenv Notes
124. In a large project • It's all about locate the path and line number: • f i nd/fd: locate by path keywords. • grep/rg: locate by content keywords. • ctags: locate by pre-built “tags”. • Language Server: a server keeps analyzing your code in background.
125. • $ cd /usr/lib/python3.10 • /usr/lib/python3.10$ f i nd - name ' * server * ' • $ sudo apt install fd - f i nd - y # or, • # apt install fd - f i nd - y • $ alias fd=fdf i nd • $ fd server # fd: f i nd
126. • $ cd /usr/lib/python3.10 • /usr/lib/python3.10$ grep - r 'Serving HTTP' - n • $ sudo apt install ripgrep - y # or, • # apt install ripgrep - y • $ rg 'Serving HTTP' # rg: ripgrep
127. • $ f i nd - name 'server.py' - exec grep 'Serving HTTP' - n {} + • $ fd '^server.py$' -X rg 'Serving HTTP' {} • $ man f i nd | less +'/ - exec ' • $ man fd | less +'/ -[Xx]'
128. • Ctags & Language Server: • $ sudo apt install universal - ctags 
 $ ctags -R 
 $ vim +'help tags | only' • A 10 MB–level solution. • neoclide/coc.nvim – GitHub for the Language Server solution. • A GB-level solution.
129. The end • But hope it's the start of your journey of getting the CLI power. • Having fun is still the key, 
 as always. • May the man guide you!
130. Image Credits • “The Power”
 https://www.facebook.com/nixcraft/photos/ meme-based-upon-iamnotanartist_-comic- for-linuxunix-users/3443131449033542/ • “The Terminal”
 https://en.wikipedia.org/wiki/ Computer_terminal#/media/ File:DEC_VT100_terminal_transparent.png • “What Is a Container?”
 https://www.docker.com/resources/what- container/ • “The Cat”
 https://www.facebook.com/nixcraft/posts/ linux-explained-for-new-users-including-cat- command-explained-httpswwwcybercitib/ 3034767979869893/ • “The Dragon”
 https://unsplash.com/photos/F6920BvzrZE • “The Miniatures”
 https://unsplash.com/photos/Q4Honp3Pyqs • “The Banshee Appears”
 https://en.wikipedia.org/wiki/Banshee#/ media/File:The_Banshee_Appears_(1862).jpg • “The Earth”
 https://unsplash.com/photos/Q1p7bh3SHj8