Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Get Power From Command Line

Mosky Liu
October 28, 2022

Get Power From Command Line

It's the deck for a modular course at NYCU. The goal of this course is to help the students and engineers who have focused on programming or data science to catch up the command line skills.

This deck covers an introduction of command line, TSV, CSV, JSON processing (using sed, awk, jq, etc.), file, process, container management (using Docker), system monitoring, SSH, SSH tunneling, and developing on command line.

This deck is not going to help you become a professional system administrator, but it will give you a wonderful head start on becoming a superb command line user!

Mosky Liu

October 28, 2022
Tweet

More Decks by Mosky Liu

Other Decks in Programming

Transcript

  1. Mosky
    Get Power From Command Line
    The Command Line Skills You Missed

    View Slide

  2. View Slide

  3. Financial Freedom

    View Slide

  4. Power Freedom

    View Slide

  5. Mosky
    Architect, Pinkoi
    • Data Science With Python

    • Statistical Regression With
    Python

    • Hypothesis Testing With Python

    • Practicing Python 3

    • The mediums.

    • The PyCons, etc.

    View Slide

  6. 1. Introduction to command line

    2. Text processing: e.g., CSV, JSON, parallelizing.

    3. System management: e.g.,
    fi
    les, processes, monitoring.

    4. Work on a remote machine: e.g., SSH, SSH tunneling.

    5. Developing on CLI
    Outline

    View Slide

  7. What commands?
    • POSIX

    • Portable Operating System Interface

    • IEEE Std 1003.1-2008

    • List of Unix commands – Wikipedia

    • Ubuntu 22.04 LTS Server

    • curl, gawk, git, htop, less,
    lsof, man
    -
    db, rsync, strace,
    tmux, vim, ssh, etc.

    • The de facto standard

    • tree, ripgrep, jq, etc.

    • The modern alternatives

    • bat (cat), fd
    -
    f
    i
    nd (
    fi
    nd), etc.

    View Slide

  8. How to use this deck?
    • Practice makes perfect:

    • Try every single command in this deck, and 

    downloading the deck as a PDF may make it easier.

    • Be careful of the spaces and newlines after paste.

    • Visit the links, but don't read; scan them to index in your brain.

    View Slide

  9. Introduction to command line

    View Slide

  10. “Command Notes”
    • Memorize all the commands?
    No.

    • You may try to note them, and
    make them searchable.

    • Inputs & Outputs.

    • Tags.

    • Like →

    View Slide

  11. Get your terminal emulator
    • iTerm for macOS

    • See also: Mosky’s Preferences
    in macOS § iTerm and search
    “Bash 5” in the doc!

    • Windows Terminal
    • GNOME Terminal / Konsole

    View Slide

  12. Connect to ...
    • macOS with GNU/Linux commands

    • Install the GNU/Linux commands on macOS

    • WSL

    • Windows Subsystem for Linux

    • Install Linux on Windows with WSL

    • Ubuntu

    • Or,

    View Slide

  13. The container
    • Get Docker

    • $ docker pull ubuntu:22.04

    $ docker network create ubuntu
    -
    n

    $ docker run
    -
    dit
    -
    p 2222
    :
    22
    -
    p
    8000
    :
    8000
    - -
    network ubuntu
    -
    n
    - -
    name ubuntu
    -
    c ubuntu:22.04

    $ docker exec
    -
    it ubuntu
    -
    c bash


    • ubuntu
    -
    c# yes | unminimize &&
    apt install curl gawk git htop
    less lsof man
    -
    db rsync strace
    tmux vim ssh
    -
    y

    View Slide

  14. The container
    • Get Docker

    • $ docker pull ubuntu:22.04

    $ docker network create ubuntu
    -
    n

    $ docker run
    -
    dit
    -
    p 2222
    :
    22
    -
    p
    8000
    :
    8000
    - -
    network ubuntu
    -
    n
    - -
    name ubuntu
    -
    c ubuntu:22.04

    $ docker exec
    -
    it ubuntu
    -
    c bash


    • ubuntu
    -
    c# yes | unminimize &&
    apt install curl gawk git htop
    less lsof man
    -
    db rsync strace
    tmux vim ssh
    -
    y
    $ hints as a normal
    user.
    ubuntu
    -
    c# hints
    as a superuser and
    in the Ubuntu
    container.
    They are default in
    the Ubuntu Server,
    but not in the
    container image.

    View Slide

  15. Checkpoint: Log into the container
    • You shall see →

    • Hints:

    • $ docker stop ubuntu
    -
    c;
    docker rm ubuntu
    -
    c # to
    start over.


    • $ docker start ubuntu
    -
    c

    $ docker exec
    -
    it ubuntu
    -
    c
    bash # after reboot if any.


    View Slide

  16. Read the manual
    • $ man man

    # Try `\
    -
    k`

    # and `\
    -
    k`


    • $ man
    -
    k ls


    • $ ls
    - -
    help | less # to exit.


    • $ ls
    - -
    help | grep '
    -
    l'


    • $ help for


    • $ type man


    • $ type ls


    • $ type for


    • Google: “man yes”

    • explainshell.com

    • Bash Reference Manual

    View Slide

  17. Read the manual
    • $ man man

    # Try `\
    -
    k`

    # and `\
    -
    k`


    • $ man
    -
    k ls


    • $ ls
    - -
    help | less # to exit.


    • $ ls
    - -
    help | grep '
    -
    l'


    • $ help for


    • $ type man


    • $ type ls


    • $ type for


    • Google: “man yes”

    • explainshell.com

    • Bash Reference Manual
    Two spaces.

    View Slide

  18. Edit commands like a ninja
    • Bash Keystrokes – Mosky’s Notes

    • $ ecHello, World!


    • $ Ctrl-W!


    • $ H!


    • $ ?


    • Also works in Python shell and
    many other places.

    • See also:

    • “CTRL-p CTRL-q” to detach –
    Docker Docs

    • macOS Shortcuts – Mosky’s
    Notes

    • Windows Shortcuts – Mosky’s
    Notes

    View Slide

  19. Edit commands like a ninja
    • Bash Keystrokes – Mosky’s Notes

    • $ ecHello, World!


    • $ Ctrl-W!


    • $ H!


    • $ ?


    • Also works in Python shell and
    many other places.

    • See also:

    • “CTRL-p CTRL-q” to detach –
    Docker Docs

    • macOS Shortcuts – Mosky’s
    Notes

    • Windows Shortcuts – Mosky’s
    Notes
    Or .
    The is
    or
    .
    Or , if not
    from the docker.
    “Space” vs.
    “Alphanumeric”
    The is

    or .

    View Slide

  20. Checkpoint: “Hey, Ctrl-K!”
    • $ echo Hello, World!


    • $

    Hey, Ctrl-K!


    • Hints:

    • , ,

    View Slide

  21. Quoting
    • $ echo Hello, World! # 2 arguments.


    • $ echo 'Hello, World!' # 1 argument.


    • $ echo Hello, $HOSTNAME!


    • $ echo "Hello, $HOSTNAME!"


    • $ echo "Hello, \$HOSTNAME!"


    • $ echo 'Hello, $'HOSTNAME!


    • $ echo $'Hello,\nWorld' # ANSI-C Quoting

    View Slide

  22. Pipe, AND, OR
    • $ echo 'Hello, World!' | grep Hello


    • $ echo 'Hello, World!' | grep Hello && echo Found!


    • $ echo 'Hello, World!' | grep Hey && echo Found!


    • $ echo 'Hello, World!' | grep Hey || echo Not found.


    • $ echo 'Hello, World!' | grep Hello && echo Found! || echo Not found.


    • $ echo 'Hello, World!' | grep Hey && echo Found! || echo Not found.

    View Slide

  23. Checkpoint: Explain the echo by yourself
    • explainshell.com

    • $ echo 'Hello, World!' | grep Hello && echo Found! || echo Not found.

    View Slide

  24. Loops
    • $ cd ~ # cd: change directory


    • ~$ touch a.txt b.txt c.txt


    • $ for name in *.txt; do echo mv $name ${name/.txt/.csv}; done

    # It's a dry run.


    • $ for name in *.txt; do echo mv $name ${name/.txt/.csv}; done


    • $ while true; do date; docker container top ubuntu
    -
    c; echo; sleep 3; done


    • $ until docker start ubuntu
    -
    c; do sleep 3; done

    View Slide

  25. Redirections
    fi
    le descriptor:name = 0:stdin, 1:stdout, 2:stderr
    • $ cd ~


    • ~$ echo Hello, File! >hello.txt


    • $ cat hello.txt


    • $ cat hello.txt | grep Hello


    • $ grep

    • $ grep out.txt


    • $ grep >out.txt # appends to out.txt.

    View Slide

  26. • $ grep - -
    bad Hello >out.txt # still prints errors.


    • $ grep - -
    bad Hello >out.txt 2>err.txt


    • $ grep - -
    bad Hello >out.txt 2>&1


    • $ grep - -
    bad Hello &>both.txt


    • $ grep - -
    bad Hello 2>/dev/null

    View Slide

  27. • $ cat <

    > Hello, Here Doc!

    > It's useful to paste something.

    > HEREDOC


    • $ cat <<<'Hello, Here String!'


    • $ python3 <<<'print(1234
    * *
    1234)'

    # C's unsigned long is 0 to 18446744073709551615, btw.

    # Useful!

    View Slide

  28. Expansions & Substitutions
    • $ echo x.{txt,csv}


    • $ echo ~


    • $ echo ~root # or any username.


    • $ cd ~


    • ~$ touch a.{txt,csv}


    • $ echo a.*


    • $ echo x.* #
    - >
    x.*


    • $ echo ?.*


    • $ echo [abcx].txt


    • $ echo [abcx].csv

    View Slide

  29. • $ echo $name


    • $ echo ${name}


    • $ echo ${name/.txt/.csv}


    • $ echo `echo 'Hello, Echo!'`


    • $ echo $(echo 'Hello, Echo!')


    • $ cat <(echo 'Hello, Cat!')


    • See also:

    • Bash scripting cheatsheet §
    Parameter expansions

    • Shell Style Guide § Variable
    expansion

    View Slide

  30. Environments & Parameters
    • Bash scans its own 

    environment variables and creates
    shell parameters.

    • $ echo $PATH


    • $ A=1 env | grep A= #
    - >
    A=1


    • $ echo $A # A is empty.


    • $ A=2


    • $ echo $A #
    - >
    2


    • $ env | grep A= #
    - >



    • $ export A


    • $ # export A=20


    • $ env | grep A= #
    - >
    A=2


    • $ A=3 echo $A #
    - >
    2


    • $ #
    = = =
    $ A=3 echo 2


    • $ export
    -
    n A && env | grep A=

    View Slide

  31. Job Control
    • $ vi a.txt # and .


    • $ vi b.txt # and .


    • $ jobs


    • $ fg # shows f
    i
    lename.


    • $ fg - # like Cmd-Tab or Alt-Tab.


    • $ fg 1 # `:q` to exit vim.


    • $ kill %2


    • $ sleep 3 && echo Hi! & #
    = = =

    • $ bash
    -
    c 'sleep 3 && echo Hi!'

    # and .


    • $ bg

    View Slide

  32. Aliases
    • $ alias ll


    • $ type ll


    • $ ls
    - -
    help | grep ' -[alF]'


    • $ alias hi=history


    • $ hi 10


    • $ alias
    . . =
    'cd
    . .
    '


    • $ cd /usr/bin


    • usr: User System Resources

    • $
    . .

    View Slide

  33. History
    • , ; ,

    • , : search history.

    • $ #TAG


    • $ #TAG


    • $ stty
    -
    ixon # needs it.


    • : discard the editing.

    • $ history 10


    • $ history
    -
    a


    • $ history
    -
    n


    • $ history
    - -
    help | grep ' -[an]'

    View Slide

  34. Checkpoint: Hack the which
    • $ which echo

    /usr/bin/echo


    • $ ??? which echo

    bash: which: No such f
    i
    le or directory


    • Hints:

    • which searches the PATH for the executable
    fi
    le.

    View Slide

  35. Text processing

    View Slide

  36. View Slide

  37. TSV
    Create TSV
    fi
    le using cat
    • $ cd


    • ~$ cat <cmds.tsv

    > / /
    en.wikipedia.org/wiki/List_of_Unix_commands,

    copy the table, and paste here.>

    > EOS

    View Slide

  38. TSV
    Create TSV
    fi
    le using vim
    • $ cd


    • ~$ vi cmds.tsv


    • (vim) i # goes the insert mode.


    • (vim) / /
    en.wikipedia.org/wiki/List_of_Unix_commands,

    copy the table, and paste here.>


    • (vim) # backs to the normal mode.


    • (vim) :x # goes the command
    -
    line mode, write and quit.

    View Slide

  39. • If any garbled text:

    • ubuntu
    -
    c$ LANG=C.UTF-8 bash


    • or,

    • (host)$ docker exec
    -
    it ubuntu
    -
    c env LANG=C.UTF-8 bash

    View Slide

  40. • $ wc
    -
    l cmds.tsv # wc: word count


    • $ cat cmds.tsv


    • $ head cmds.tsv


    • $ tail cmds.tsv


    • $ grep 

    #
    = = =
    grep cat cmds.tsv


    • grep: g/re/p in the ed
    command, globally search for a
    regular expression and print
    matching lines.

    • $ grep

    • $ grep -
    o

    View Slide

  41. TSV is literally everywhere.

    View Slide

  42. Checkpoint: Copy the head and paste to a sheet
    • A sheet in Google Sheets, Excel,
    LibreO
    ff i
    ce Calc, like →

    • Hints:

    • Just like how did you paste the
    table from the browser to the
    terminal.

    • Copy the output on the
    terminal and paste it to the
    sheet.

    View Slide

  43. TSV (cont.)
    • $ seq 1 3


    • $ seq 1 3 | column


    • $ seq 1 3 | column | hexdump
    -
    c


    • $ seq 1 3 | column
    -
    t

    View Slide

  44. • $ head -
    s ''
    -
    t


    • $ column
    - -
    help | grep ' -[st]'


    • $ head 

    #
    = = =
    $ head
    -
    n 1 cmds.tsv


    • $ head

    • sed: stream editor

    • $ sed -
    n
    -
    e 1p
    -
    e /^.cat/p


    • $ sed
    - -
    help | grep ' -[ne]'

    View Slide

  45. • $ cut -
    f 2 | head


    • $ cut -
    f 2 | tail +2 | head


    • $ cut -
    f 2 | tail +2 | sort


    • $ cut -
    f 2 | tail +2 | sort | uniq


    • $ cut -
    f 2 | tail +2 | sort | uniq
    -
    c


    • $ cut -
    f 2 | tail +2 | sort | uniq
    -
    c | sort
    -
    n

    View Slide

  46. • $ awk

    • $ awk -
    s $'\t'
    -
    t


    • $ awk counts) print counts[cat] "\t" cat }'


    • $ awk counts) print counts[cat] "\t" cat }' | sort
    -
    n

    View Slide

  47. • See also:

    • The GNU Awk User’s Guide

    • 6.3.2.2 Comparison Operators

    • 6.1.1.3 Regular Expression Constants

    • 8.1.5 Scanning All Elements of an Array

    View Slide

  48. How many mandatory text processing does POSIX have?
    Checkpoint:
    • $ ??? -
    l

    ???


    • Hints:

    • Either grep or awk is a good choice.

    • Try
    = =
    and
    & &
    if you choose awk.

    • “Text processing”

    • “Mandatory”

    View Slide

  49. TSV (cont.)
    • $ cut -
    f 2 | head


    • $ awk

    • $ vimdiff <(cut -
    f 2 | head) <(awk head)

    # `:qa` to exit.


    • $ awk

    View Slide

  50. • $ cut -
    f 2 | head >a


    • $ awk b


    • $ vimdiff a b


    • $ diff a b


    • $ diff a b
    -
    y # using the side by side format.


    • $ diff a b
    -
    u # using the unif
    i
    ed format.


    • $ diff
    - -
    help | grep ' -[yu]'

    View Slide

  51. • $ sudo apt install bat
    -
    y # or,


    • # apt install bat
    -
    y


    • $ alias bat=batcat


    • $ diff a b | bat # to exit.

    View Slide

  52. Interact with the system clipboards
    • (macOS)$ pbpaste | tail -1 | pbcopy


    • The way I make slides with highlighted codes:

    • $ pbpaste | highlight
    -
    u utf-8 -S python
    -
    s solarized
    -
    light
    -
    k Monaco
    -K 28 -O rtf | pbcopy # and then paste into a slide.


    • (X11)$ xclip
    -
    o | tail -1 | xclip


    • (WSL)$ powershell.exe Get-Clipboard | tail -1 | clip.exe


    • See also: vor0nwe/clip – Gist for a neat clip script in WSL.

    View Slide

  53. cURL
    • $ curl https://google.com


    • $ curl
    -
    v https://google.com


    • $ curl -L https://google.com

    View Slide

  54. View Slide

  55. JSON with jq
    • $ sudo apt install jq
    -
    y # or,


    • # apt install jq
    -
    y


    • jq is like sed for JSON.

    • $ cd


    • ~$ curl https://raw.githubusercontent.com/5etools
    -
    mirror-1/5etools
    -
    mirror-1.github.io/
    master/data/bestiary/bestiary
    -
    mm.json >monsters.json # or,


    • ~$ curl https://raw.githubusercontent.com/hazmole/TheGiddyLimit.github.io/master/
    data/bestiary/bestiary
    -
    mm.json >monsters_zh_tw.json


    • $ wc
    -
    l monsters.json #
    - >
    70811

    View Slide

  56. • $ jq

    • $ jq

    • $ jq

    • $ grep - -
    color=always | less -R


    • $ jq -
    l json


    • $ bat monsters.json


    • $ vi monsters.json # `:q` to exit.

    View Slide

  57. • $ jq

    • $ jq

    • $ jq [.hp.average, .name, .type, .page]' | head

    View Slide

  58. • $ jq [.hp.average, .name, .type, .page] | @tsv' | head


    • $ jq [.hp.average, .name, .type, .page] | @tsv'
    -
    r | head


    • $ jq select(.type == "dragon") | [.hp.average, .name, .type, .page]) | @tsv'
    -
    r |
    head


    • $ jq select(.type == "dragon") | [.hp.average, .name, .type, .page]) | @tsv'
    -
    r
    >dragons.tsv

    View Slide

  59. • $ sort

    • $ sort -
    k 2
    - -
    debug | head # Try to sort by Name.


    • $ sort -
    k 2,2
    - -
    debug | head


    • $ sort -
    t $'\t'
    -
    k 2,2
    - -
    debug | head

    View Slide

  60. What is the first undead monster in the book?
    Checkpoint:
    • $ jq ??? | tail -1

    ???


    • Hints:

    • “undead”

    • “page”

    View Slide

  61. JSON with gron
    • $ sudo apt install gron
    -
    y # or,


    • # apt install gron
    -
    y


    • $ jq

    • $ jq dragons.json


    • $ bat dragons.json


    • $ gron dragons.json


    • $ gron dragons.json | grep -E 'json\[[0-9]+\]\.name ='


    • $ gron dragons.json | grep -E 'json\[[0-9]+\]\.name =' | gron
    -
    u

    View Slide

  62. . * [a
    -
    z] ^$ ? + |
    Fixed string . * [a
    -
    z] ^$ ? + | $ grep -F
    BRE . * [a
    -
    z] ^$ ? + |
    $ grep

    $ grep -G
    ERE . * [a
    -
    z] ^$ ? + | $ grep -E
    Glob . * [a
    -
    z] ^$ ? + |
    Fixed string, BRE, ERE, vs. glob

    View Slide

  63. Test the regex types
    • $ echo 'i++' | grep -F '.+'
    -
    o # f
    i
    xed string (not regex)




    • $ echo 'i++' | grep -G '.+'
    -
    o # basic regular expression (BRE)

    i
    +

    • $ echo 'i++' | grep -E '.+'
    -
    o # extended regular expression (ERE)

    i
    + +

    View Slide

  64. • See also:

    • Regex cheatsheet for BRE, ERE, Vim, Python's re, PCRE (Perl).

    • regex101: a regex debugger for Python, JavaScript, Java, Golang, PHP.

    • jq Manual

    • tomnomnom/gron – GitHub

    • yq for YAML.

    View Slide

  65. Checkpoint: The regex type of less may be?
    • BRE-like ... ERE-like ... or not regex?

    • Hints:

    • How to put i
    + +
    into less?

    • How to search something in less?

    View Slide

  66. Bulk editing
    • $ cp dragons.tsv cats.tsv


    • $ cp dragons.tsv cats_2.tsv


    • $ head cats
    *
    .tsv


    • $ sed s/Dragon/Cat/ cats
    *
    .tsv | less


    • $ sed
    -
    e s/Dragon/Cat/
    -
    e s/dragon/cat/ cats
    *
    .tsv | less


    • $ sed
    -
    e s/Dragon/Cat/g
    -
    e s/dragon/cat/g cats
    *
    .tsv | less


    • $ sed
    -
    e s/Dragon/Cat/g
    -
    e s/dragon/cat/g cats
    *
    .tsv
    -
    i

    View Slide

  67. CSV with CSVKit
    • $ sudo apt install csvkit
    -
    y # or,


    • # apt install csvkit
    -
    y


    • $ jq select(.type == "dragon") | [.hp.average, .name, .type, .page]) | @tsv'
    -
    r
    >dragons.tsv


    • $ jq select(.type == "dragon") | [.hp.average, .name, .type, .page]) | @csv'
    -
    r
    >dragons.csv


    • $ bat dragons.csv

    View Slide

  68. • $ csvcut -
    c Name,HP


    • $ csvsort -
    c HP,Page


    • $ csvstat -
    c HP,Name,Type


    • $ csvlook - >
    a Markdown table


    • $ csvgrep -
    c Name
    -
    m 'Adult Black Dragon'


    • $ csvsql - -
    query "select avg(HP) from stdin where Name like
    'Adult%'"

    View Slide

  69. • See also:

    • Tutorial – csvkit

    • 1.4. in2csv: the Excel killer

    • 3.3. csvsql and sql2csv: ultimate power

    View Slide

  70. Parallelizing
    • $ time sleep 1


    • $ seq 3 | xargs
    -
    i echo sleep {} # xargs: extended arguments


    • $ seq 3 | xargs
    -
    i echo sleep 1


    • $ time seq 3 | xargs
    -
    i sleep 1 # takes 3 seconds.


    • $ time seq 3 | xargs
    -
    i -P 3 sleep 1 # takes 1 second.


    • $ man xargs | less +'/ -[iI]'

    View Slide

  71. System management

    View Slide

  72. File management
    • $ cd


    • ~$ mkdir mydir && cd mydir


    • ~/mydir$ touch a b .h


    • $ mkdir
    -
    p x/y/z


    • $ cp b c


    • $ rm b


    • $ ls


    • $ ll


    • $ type ll


    • $ ll
    -
    t

    # List and sort by time.


    • $ ll -S
    -
    h

    # List and sort by size.

    View Slide

  73. • $ sudo apt install tree
    -
    y # or,


    • # apt install tree
    -
    y


    • $ cd ~/mydir


    • ~/mydir$ ls


    • $ tree


    • $ tree
    -
    a # List all f
    i
    les.


    • $ tree -L 1 # List one level deep.


    • $ tree
    -
    f # Print full paths.

    View Slide

  74. • $ cd


    • ~$ ln
    -
    s mydir mydir_link # Make a symbolic link.


    • $ ls mydir_link


    • $ ll mydir_link


    • $ cp
    -
    r mydir mydir_backup # Copy recursively.


    • $ ll mydir_backup

    View Slide

  75. • $ tar czf mydir.tar.gz mydir


    • $ rmdir mydir

    #
    - >
    failed
    . . .
    not empty.


    • $ rmdir mydir/x/y/z

    # OK!


    • $ rm
    -
    r mydir/


    • $ tar xzf mydir.tar.gz


    • $ tree


    • $ du
    -
    h
    -
    d 1 # du: disk usage


    • $ df
    -
    h # df: disk free

    View Slide

  76. Describe one difference between cp
    -
    r and cp
    -
    a
    Checkpoint:
    • Read and search the manual!

    • Hints:

    • $ man cp # `/` to search; `n` to next; `h` to get help.


    • $ cp
    - -
    help | grep
    . . .

    View Slide

  77. Terminology
    EN program process thread concurrency parallelism
    TW 程式 程序 執⾏緒 並⾏性 平⾏性
    CN 程序 进程 线程 并发性 并⾏性

    View Slide

  78. Process management
    • $ ps # ps: process status


    • $ ps a # a: includes other users.


    • $ ps ax # x: includes other ttys or no tty's.

    # tty: teletypewriter


    • $ ps axu # u: user
    -
    oriented format


    • $ ps f # f: "forest"


    • $ man ps | grep -E ' (VSZ|RSS|TTY|STAT|START|TIME) '

    View Slide

  79. • $ vi test.txt # and .


    • $ ps u | grep vi


    • $ ps u | grep [v]i


    • $ ps u | sed
    -
    n
    -
    e 1p
    -
    e /[v]i/p


    • $ ps axu | grep [v]i |

    awk '{ rss += $6 } END { printf "%.4g MB\n", rss/1024 }'


    • (macOS)$ ps axu | grep [C]hrome |

    awk '{ rss += $6 } END { printf "%.4g MB\n", rss/1024 }'

    View Slide

  80. • $ vi test.txt # and .


    • $ pgrep v #
    - >



    • $ pgrep vi #
    - >



    • $ pgrep test #
    - >



    • $ pgrep
    -
    f test #
    - >



    • $ pgrep
    -
    f test
    -
    l


    • $ pgrep
    - -
    help | grep ' -[fl]'


    • $ # kill


    • $ pkill
    -
    f test # -SIGTERM


    • $ fg


    • $ vi test.txt # and .


    • $ pkill
    -
    f test -SIGKILL


    • See also:

    • Signal (IPC) – Wikipedia

    View Slide

  81. Monitoring
    • $ sudo apt install iotop iftop
    -
    y # or, # if: interface


    • # apt install iotop iftop
    -
    y


    • $ free
    -
    h


    • $ man free | grep -E ' (total|used|free|shared|buff/cache|available)' -A 1


    • $ vi test.txt # and .


    • $ top


    • $ htop


    • Select a process, and (kill)

    View Slide

  82. • $ cd


    • ~$ python3
    -
    m http.server &> my.log

    # to interrupt.


    • Open http://localhost:8000/.

    • Open another terminal:

    • # iftop # to exit.


    • $ tail
    -
    f ~/my.log


    • $ less +F ~/my.log

    # Try `
    -
    i` `/http` .


    • $ watch tail -10 ~/my.log

    View Slide

  83. • The commands which are useful but not relevant to containers:

    • $ sudo iotop


    • Important system logs:

    • $ sudo vi /var/log/syslog


    • $ sudo vi /var/log/auth.log

    View Slide

  84. Checkpoint: Monitor your HTTP server by less
    • Try to get three new logs.

    • Hints:

    • Click on your browser!

    View Slide

  85. Monitoring (cont.)
    • $ python3
    -
    c 'print(1234)'


    • $ strace python3
    -
    c 'print(1234)'


    • $ sudo strace
    -
    p


    • $ ps x


    • $ pgrep bash


    • $ tty


    • $ pgrep bash
    -
    t pts/1

    # pts: pseudo terminal slave


    • $ lsof
    -
    p `pgrep bash
    -
    t pts/1`

    # lsof: list open f
    i
    les


    • $ lsof /usr/bin/bash


    • $ man lsof | less
    -
    i +/^examples


    • $ htop


    • Select a process, and:

    • (strace)

    • (lsof)

    View Slide

  86. • $ sudo apt install sysstat
    -
    y # or,


    • # apt install sysstat
    -
    y


    • $ sar 1

    # Report CPU stats per second.


    • $ sar

    # Report today's CPU stats.


    • $ sar
    -
    f /var/log/sysstat/sa10


    • $ sar
    -
    f /var/log/sysstat/sa10

    -
    s 12
    :
    00
    -
    e 15
    :
    00


    • $ man sar | grep 'Report ' -B 1


    • $ sar
    -
    f /var/log/sysstat/sa10

    -
    s 12
    :
    00
    -
    e 15
    :
    00
    -
    r
    - -
    human

    # Report memory stats.


    • See also:

    • sysstat/sysstat – GitHub

    • Amazon CloudWatch

    • Grafana

    View Slide

  87. Container management
    • OCI: Open Container Initiative

    • Docker
    fi
    le: part of OCI spec.

    • image: container image

    • Why?
    • Once you pack your app as a
    container image, it can be run
    anywhere seamlessly.

    View Slide

  88. • (host)$ docker container commit ubuntu
    -
    c ubuntu
    -
    cli
    -
    power

    #
    = = =
    $ docker container commit
    . . .

    • $ docker image ls

    #
    = = =
    $ docker images

    View Slide

  89. • $ docker image save ubuntu
    -
    cli
    -
    power >ubuntu
    -
    cli
    -
    power.tar


    • $ docker image rm ubuntu
    -
    cli
    -
    power

    #
    = = =
    $ docker rmi ubuntu
    -
    cli
    -
    power


    • $ docker image ls


    • $ docker image load -
    cli
    -
    power.tar


    • $ # docker image (save|load)
    = = =
    docker image (save|load)


    • $ docker image ls

    View Slide

  90. • $ docker container create
    - -
    name ls
    -
    expert ubuntu
    -
    cli
    -
    power man ls


    • $ docker container ls
    -
    a #
    = = =
    $ docker ps
    -
    a


    • $ docker container start ls
    -
    expert


    • $ docker container ls
    -
    a #
    - >
    Exited (0) 4 seconds ago


    • $ docker container start
    -
    a ls
    -
    expert # Start and attach.


    • $ docker container rm ls
    -
    expert


    • $ # docker container (create|start|stop|rm)
    = = = 

    docker container (create|start|stop|rm)

    View Slide

  91. • $ docker container cp ubuntu
    -
    c:/root/cmds.tsv ~/cmds.tsv

    #
    = = =
    $ docker cp
    . . .

    • $ docker container cp ~/cmds.tsv ubuntu
    -
    c:/root/cmds_2.tsv


    • $ docker container run
    -
    dit
    -
    v ~/container:/root
    - -
    name volume
    -
    c ubuntu
    -
    cli
    -
    power

    #
    = = =
    $ docker run
    . . .

    View Slide

  92. Work on a remote machine

    View Slide

  93. Run programs after disconnect
    • $ tmux # tmux: terminal multiplexer


    • (tmux)$ while true; do date; sleep 1; done # and then .


    • $ tmux ls # ls: the alias of list
    -
    sessions.


    • $ tmux a # a: attach


    • (tmux) > # List key bindings; to exit.


    • (tmux)


    • $ tmux ls

    View Slide

  94. • The way I start the server of JupyterLab:

    • $ tmux new
    -
    s jupyter
    -
    lab
    -
    d 'cd && jupyter lab
    - -
    no
    -
    browser'

    View Slide

  95. The local key pair
    • (local)$ ll ~/.ssh/id_rsa
    *

    • If nothing there:

    • (local)$ ssh
    -
    keygen


    • (local)$ cat ~/.ssh/id_rsa.pub # pub: public key


    • id_rsa is the private key which proves who you are.

    • Do not share id_rsa – the private key – to anyone.

    • Simply generate a new pair when you have a little doubt about the old pair.

    View Slide

  96. Add a user in the remote host
    • (remote)# new_user=


    • (remote)# useradd
    -
    m
    -
    s /bin/bash $new_user


    • (remote)# su $new_user


    • $ mkdir ~/.ssh

    $ cat <~/.ssh/authorized_keys

    >

    > EOS

    $ exit


    • (remote)# service ssh start

    View Slide

  97. Connect to a remote host
    • (local)$ ssh ssh://@localhost:2222

    #
    = = =
    $ ssh @localhost
    -
    p 2222

    # $ ssh # if port is 22 and usernames are the same.


    • If there is a warning and you're sure you changed something in the remote:

    • $ sed d ~/.ssh/known_hosts
    -
    i

    View Slide

  98. ssh -A: Allow the remote to use the local key pair
    • (local) $ ssh ssh://@localhost:2222


    • (remote)$ ssh ubuntu
    -
    c

    #
    . . .
    password:


    • (local) $ ssh ssh://@localhost:2222 -A


    • (remote)$ ssh ubuntu
    -
    c # OK


    • The ubuntu-c works since the docker run --network enables the DNS
    resolution between containers.

    • See also: Consul, for the similar functionality without Docker.

    View Slide

  99. ssh -J: Jump to a host in a private network
    • (local)$ ssh ubuntu
    -
    c

    #
    - >
    Could not resolve hostname ubuntu
    -
    c


    • (local)$ ssh -J ssh://@localhost:2222 ubuntu
    -
    c


    • -J is securer than -A; use -J when applicable.

    View Slide

  100. Transfer files between local and remote
    • (remote)$ cp cmds.tsv ~


    • (local) $ rsync
    -
    e 'ssh ssh://@localhost:2222' :~/cmds.tsv ~

    # $ rsync @> :
    ~/cmds.tsv ~ # if port is 22.


    • (local) $ rsync
    -
    e 'ssh ssh://@localhost:2222' ~/cmds.tsv :~/cmds_2.tsv


    • Deprecating scp – LWN.net, but scp is still a common way nowadays:

    • (local)$ scp scp://@localhost:2222/~/cmds.tsv ~

    # $ scp @> :
    ~/cmds.tsv ~ # if port is 22.


    • (local)$ scp ~/cmds.tsv scp://@localhost:2222/~/cmds_2.tsv

    View Slide

  101. Mount a remote directory
    • (local)$ sudo apt install sshfs # or macFUSE for macOS.


    • (local)$ mkdir ~/sshfs_container


    • (local)$ sshfs @localhost:/home/
    -
    p 2222 ~/sshfs_container


    • (local)$ ls ~/sshfs_container


    • (local)$ umount ~/sshfs_container

    View Slide

  102. ssh -L: Forward a local port to a remote port
    • As known as “SSH tunneling”.

    • (remote)$ python3
    -
    m http.server 1234 # to interrupt.


    • (local)$ ssh ssh://@localhost:2222 -L 4321:ubuntu
    -
    c:1234 -Nf


    • Open http://localhost:4321/.

    • localhost:4321 

    → localhost:2222 → ubuntu-c:22 

    → ubuntu-c:1234 → http.server 1234

    • (local)$ pgrep
    -
    f 4321


    • (local)$ # pkill
    -
    f 4321

    View Slide

  103. Access the HTTP server by http://localhost:8765/
    Checkpoint:
    • Open http://localhost:8765/ and there should be a page.

    • Hints:

    • Use ssh -L.

    View Slide

  104. ssh -D: Start a proxy server
    • (local)$ ssh ssh://@localhost:2222 -D 1080 -Nf


    • (remote)# iftop


    • Con
    fi
    gure the system proxy or Proxy SwitchyOmega for Chrome.

    • (remote)# python3
    -
    m http.server 80


    • Open http://ubuntu-c/.

    • For accessing the services in a private network by private DNS and standard ports!

    • If the server has another public IP, try Google “my ip”.

    • $ man ssh | grep ' -[LDNf]' -A 1

    View Slide

  105. System proxy for macOS

    View Slide

  106. Proxy SwitchyOmega for Chrome

    View Slide

  107. View Slide

  108. • See also:

    • A simple ~/.ssh/con
    fi
    g for macOS

    • A simple Bash con
    fi
    g for both macOS and Linux

    View Slide

  109. Developing on CLI

    View Slide

  110. Edit a config
    • $ cp ~/.bashrc ~/backup_bashrc


    • $ echo 'alias bat=batcat' >>~/.bashrc


    • Re-enter the shell to take e
    ff
    ect.

    • $ grep batcat ~/.bashrc
    -
    n


    • $ sed d ~/.bashrc
    -
    i


    • $ grep batcat ~/.bashrc
    -
    n

    View Slide

  111. • $ vi ~/.bashrc


    • (vim) #


    • (vim) i # or `a`


    • (vim) alias bat=batcat # literally


    • (vim) # backs to Normal


    • (vim) :x


    • $ grep batcat ~/.bashrc
    -
    n


    • $ vi ~/.bashrc


    • (vim) /cat


    • (vim) n
    . . .

    • (vim) dd


    • (vim) :x


    • $ grep batcat ~/.bashrc
    -
    n

    View Slide

  112. The mini-cheatsheet of Vim
    1. / n


    • Search; Next.

    2. i :x


    • Go Insert; Back to Normal; Write and quit.

    3. dd


    • Delete a line.

    • It's all you need to know now.

    Of course, the more you know, the more e
    ff
    i
    ciency you'll get. 💪

    View Slide

  113. Spell checking
    • $ echo 'Oops. Does it spll like this?'


    • (vim) :set spell


    • (vim) /sp


    • (vim) z= # List spelling suggestions.


    • (vim) 1


    • (vim) :x

    View Slide

  114. • See also:

    • $ vimtutor # Learn Vim in 30 mins!


    • (vim) :help dd, for example.

    • Vim Keystrokes – Mosky’s Notes

    • Vim’s Search Tips – Mosky’s Notes

    • Vim’s Replace Tips – Mosky’s Notes

    • Mosky's Vim Con
    fi
    gs

    View Slide

  115. Checkpoint: Add your bat
    • (host)$ docker exec
    -
    it ubuntu
    -
    c bash


    • ubuntu
    -
    c# bat ~/.bashrc # It should work immediately.


    • Hints:

    • You may use echo to add.

    • Use grep to con
    fi
    rm.

    • Re-enter the shell to take e
    ff
    ect.

    View Slide

  116. Developing
    Slak: Collect data from Slack like a pro. ⚡
    1. System Tools: the common tools which are
    supposed to be installed in the system.

    • git: the version control system.

    • pip: the Python package manager.

    • pipenv: one of Python package environment
    managers.

    • The Docker (containers) provides OS-level
    isolation; Pipenv provides Python
    interpreter and package–level isolation.

    2. Project Tools: the tools of this project, good to
    be managed within the project by, for example,
    pipenv.

    • ipdb: a simple yet beautiful debugger.

    • pytest: the testing framework.

    3. Personal Tools: the tools of a developer, may be
    installed in one's system.

    • flake8: the static analyzer (linter).

    • mypy: the static type checker.

    • black: one of the formatters.

    View Slide

  117. • $ sudo apt install pip
    -
    y # or, # System Tool


    • # apt install pip
    -
    y


    • $ pip install pipenv # System Tool


    • $ cd


    • ~$ git clone
    - -
    depth 1 https://github.com/moskytw/slak.git

    View Slide

  118. • ~$ cd slak


    • ~/slak$ bat Pipf
    i
    le


    • $ pipenv sync
    - -
    dev


    • $ slak


    • $ python slak.py


    • $ for cmd in python python3 slak;
    do type $cmd; done


    • $ pipenv shell


    • (slak)$ slak


    • $ python slak.py


    • $ for cmd in python python3 slak;
    do type $cmd; done

    View Slide

  119. • (slak)$ python
    -
    m ipdb slak.py # Project Tool, `q` to exit.


    • $ pytest # Project Tool


    • $ pytest
    - -
    cov


    • $ pytest
    - -
    cov
    - -
    cov
    -
    report html


    • $ cd htmlcov


    • ~/slak/htmlcov$ python
    -
    m http.server # to interrupt.


    • Open http://localhost:8000/.

    • $ exit # or to exit the shell of pipenv.

    View Slide

  120. • (my mac)$ for cmd in flake8 mypy black xxx; do type $cmd; done

    flake8 is hashed (/usr/local/bin/flake8)

    mypy is aliased to `\mypy
    - -
    ignore
    -
    missing
    -
    imports'

    black is aliased to `black
    - -
    line
    -
    length 79
    - -
    skip
    -
    string
    -
    normalization'

    -
    bash: type: xxx: not found


    • $ for cmd in flake8 mypy black xxx; do type $cmd >/dev/null; done

    -
    bash: type: xxx: not found

    View Slide

  121. • $ cd ~/slak


    • ~/slak$ flake8 slak.py


    • $ mypy slak.py


    • $ black slak.py

    View Slide

  122. • The di
    ff
    erent languages have di
    ff
    erent toolsets; however, the concepts and
    the categories are similar, e.g.:

    • $ vi /etc/nginx/sites
    -
    enabled/your
    -
    site.conf


    • $ nginx
    -
    tc /etc/nginx/sites
    -
    enabled/your
    -
    site.conf

    # Test the conf
    i
    g f
    i
    le.

    View Slide

  123. • See also:

    • git - the simple guide: “no deep shit ;)”

    • Git Documentation

    • Cheat Sheets

    • Pro Git, the book

    • Reference Manual

    • Mosky’s Pipenv Notes

    View Slide

  124. In a large project
    • It's all about locate the path and line number:

    • f
    i
    nd/fd: locate by path keywords.

    • grep/rg: locate by content keywords.

    • ctags: locate by pre-built “tags”.

    • Language Server: a server keeps analyzing your code in background.

    View Slide

  125. • $ cd /usr/lib/python3.10


    • /usr/lib/python3.10$ f
    i
    nd
    -
    name '
    *
    server
    *
    '


    • $ sudo apt install fd
    -
    f
    i
    nd
    -
    y # or,


    • # apt install fd
    -
    f
    i
    nd
    -
    y


    • $ alias fd=fdf
    i
    nd


    • $ fd server # fd: f
    i
    nd

    View Slide

  126. • $ cd /usr/lib/python3.10


    • /usr/lib/python3.10$ grep
    -
    r 'Serving HTTP'
    -
    n


    • $ sudo apt install ripgrep
    -
    y # or,


    • # apt install ripgrep
    -
    y


    • $ rg 'Serving HTTP' # rg: ripgrep

    View Slide

  127. • $ f
    i
    nd
    -
    name 'server.py'
    -
    exec grep 'Serving HTTP'
    -
    n {} +


    • $ fd '^server.py$' -X rg 'Serving HTTP' {}


    • $ man f
    i
    nd | less +'/
    -
    exec '


    • $ man fd | less +'/ -[Xx]'

    View Slide

  128. • Ctags & Language Server:

    • $ sudo apt install universal
    -
    ctags

    $ ctags -R

    $ vim +'help tags | only'


    • A 10 MB–level solution.

    • neoclide/coc.nvim – GitHub for the Language Server solution.

    • A GB-level solution.

    View Slide

  129. The end
    • But hope it's the start of your
    journey of getting the CLI power.

    • Having fun is still the key, 

    as always.

    • May the man guide you!

    View Slide

  130. Image Credits
    • “The Power”

    https://www.facebook.com/nixcraft/photos/
    meme-based-upon-iamnotanartist_-comic-
    for-linuxunix-users/3443131449033542/

    • “The Terminal”

    https://en.wikipedia.org/wiki/
    Computer_terminal#/media/
    File:DEC_VT100_terminal_transparent.png

    • “What Is a Container?”

    https://www.docker.com/resources/what-
    container/

    • “The Cat”

    https://www.facebook.com/nixcraft/posts/
    linux-explained-for-new-users-including-cat-
    command-explained-httpswwwcybercitib/
    3034767979869893/

    • “The Dragon”

    https://unsplash.com/photos/F6920BvzrZE

    • “The Miniatures”

    https://unsplash.com/photos/Q4Honp3Pyqs

    • “The Banshee Appears”

    https://en.wikipedia.org/wiki/Banshee#/
    media/File:The_Banshee_Appears_(1862).jpg

    • “The Earth”

    https://unsplash.com/photos/Q1p7bh3SHj8

    View Slide