Slide 1
Slide 1 text
Mosky
Get Power From Command Line
The Command Line Skills You Missed
Slide 2
Slide 2 text
No content
Slide 3
Slide 3 text
Financial Freedom
Slide 4
Slide 4 text
Power Freedom
Slide 5
Slide 5 text
Mosky
Architect, Pinkoi
• Data Science With Python
• Statistical Regression With
Python
• Hypothesis Testing With Python
• Practicing Python 3
• The mediums.
• The PyCons, etc.
Slide 6
Slide 6 text
1. Introduction to command line
2. Text processing: e.g., CSV, JSON, parallelizing.
3. System management: e.g.,
fi
les, processes, monitoring.
4. Work on a remote machine: e.g., SSH, SSH tunneling.
5. Developing on CLI
Outline
Slide 7
Slide 7 text
What commands?
• POSIX
• Portable Operating System Interface
• IEEE Std 1003.1-2008
• List of Unix commands – Wikipedia
• Ubuntu 22.04 LTS Server
• curl, gawk, git, htop, less,
lsof, man
-
db, rsync, strace,
tmux, vim, ssh, etc.
• The de facto standard
• tree, ripgrep, jq, etc.
• The modern alternatives
• bat (cat), fd
-
f
i
nd (
fi
nd), etc.
Slide 8
Slide 8 text
How to use this deck?
• Practice makes perfect:
• Try every single command in this deck, and
downloading the deck as a PDF may make it easier.
• Be careful of the spaces and newlines after paste.
• Visit the links, but don't read; scan them to index in your brain.
Slide 9
Slide 9 text
Introduction to command line
Slide 10
Slide 10 text
“Command Notes”
• Memorize all the commands?
No.
• You may try to note them, and
make them searchable.
• Inputs & Outputs.
• Tags.
• Like →
Slide 11
Slide 11 text
Get your terminal emulator
• iTerm for macOS
• See also: Mosky’s Preferences
in macOS § iTerm and search
“Bash 5” in the doc!
• Windows Terminal
• GNOME Terminal / Konsole
Slide 12
Slide 12 text
Connect to ...
• macOS with GNU/Linux commands
• Install the GNU/Linux commands on macOS
• WSL
• Windows Subsystem for Linux
• Install Linux on Windows with WSL
• Ubuntu
• Or,
Slide 13
Slide 13 text
The container
• Get Docker
• $ docker pull ubuntu:22.04
$ docker network create ubuntu
-
n
$ docker run
-
dit
-
p 2222
:
22
-
p
8000
:
8000
- -
network ubuntu
-
n
- -
name ubuntu
-
c ubuntu:22.04
$ docker exec
-
it ubuntu
-
c bash
• ubuntu
-
c# yes | unminimize &&
apt install curl gawk git htop
less lsof man
-
db rsync strace
tmux vim ssh
-
y
Slide 14
Slide 14 text
The container
• Get Docker
• $ docker pull ubuntu:22.04
$ docker network create ubuntu
-
n
$ docker run
-
dit
-
p 2222
:
22
-
p
8000
:
8000
- -
network ubuntu
-
n
- -
name ubuntu
-
c ubuntu:22.04
$ docker exec
-
it ubuntu
-
c bash
• ubuntu
-
c# yes | unminimize &&
apt install curl gawk git htop
less lsof man
-
db rsync strace
tmux vim ssh
-
y
$ hints as a normal
user.
ubuntu
-
c# hints
as a superuser and
in the Ubuntu
container.
They are default in
the Ubuntu Server,
but not in the
container image.
Slide 15
Slide 15 text
Checkpoint: Log into the container
• You shall see →
• Hints:
• $ docker stop ubuntu
-
c;
docker rm ubuntu
-
c # to
start over.
• $ docker start ubuntu
-
c
$ docker exec
-
it ubuntu
-
c
bash # after reboot if any.
Slide 16
Slide 16 text
Read the manual
• $ man man
# Try `\
-
k`
# and `\
-
k`
• $ man
-
k ls
• $ ls
- -
help | less # to exit.
• $ ls
- -
help | grep '
-
l'
• $ help for
• $ type man
• $ type ls
• $ type for
• Google: “man yes”
• explainshell.com
• Bash Reference Manual
Slide 17
Slide 17 text
Read the manual
• $ man man
# Try `\
-
k`
# and `\
-
k`
• $ man
-
k ls
• $ ls
- -
help | less # to exit.
• $ ls
- -
help | grep '
-
l'
• $ help for
• $ type man
• $ type ls
• $ type for
• Google: “man yes”
• explainshell.com
• Bash Reference Manual
Two spaces.
Slide 18
Slide 18 text
Edit commands like a ninja
• Bash Keystrokes – Mosky’s Notes
• $ ecHello, World!
• $ Ctrl-W!
• $ H!
• $ ?
• Also works in Python shell and
many other places.
• See also:
• “CTRL-p CTRL-q” to detach –
Docker Docs
• macOS Shortcuts – Mosky’s
Notes
• Windows Shortcuts – Mosky’s
Notes
Slide 19
Slide 19 text
Edit commands like a ninja
• Bash Keystrokes – Mosky’s Notes
• $ ecHello, World!
• $ Ctrl-W!
• $ H!
• $ ?
• Also works in Python shell and
many other places.
• See also:
• “CTRL-p CTRL-q” to detach –
Docker Docs
• macOS Shortcuts – Mosky’s
Notes
• Windows Shortcuts – Mosky’s
Notes
Or .
The is
or
.
Or , if not
from the docker.
“Space” vs.
“Alphanumeric”
The is
or .
Slide 20
Slide 20 text
Checkpoint: “Hey, Ctrl-K!”
• $ echo Hello, World!
• $
Hey, Ctrl-K!
• Hints:
• , ,
Slide 21
Slide 21 text
Quoting
• $ echo Hello, World! # 2 arguments.
• $ echo 'Hello, World!' # 1 argument.
• $ echo Hello, $HOSTNAME!
• $ echo "Hello, $HOSTNAME!"
• $ echo "Hello, \$HOSTNAME!"
• $ echo 'Hello, $'HOSTNAME!
• $ echo $'Hello,\nWorld' # ANSI-C Quoting
Slide 22
Slide 22 text
Pipe, AND, OR
• $ echo 'Hello, World!' | grep Hello
• $ echo 'Hello, World!' | grep Hello && echo Found!
• $ echo 'Hello, World!' | grep Hey && echo Found!
• $ echo 'Hello, World!' | grep Hey || echo Not found.
• $ echo 'Hello, World!' | grep Hello && echo Found! || echo Not found.
• $ echo 'Hello, World!' | grep Hey && echo Found! || echo Not found.
Slide 23
Slide 23 text
Checkpoint: Explain the echo by yourself
• explainshell.com
• $ echo 'Hello, World!' | grep Hello && echo Found! || echo Not found.
Slide 24
Slide 24 text
Loops
• $ cd ~ # cd: change directory
• ~$ touch a.txt b.txt c.txt
• $ for name in *.txt; do echo mv $name ${name/.txt/.csv}; done
# It's a dry run.
• $ for name in *.txt; do echo mv $name ${name/.txt/.csv}; done
• $ while true; do date; docker container top ubuntu
-
c; echo; sleep 3; done
• $ until docker start ubuntu
-
c; do sleep 3; done
Slide 25
Slide 25 text
Redirections
fi
le descriptor:name = 0:stdin, 1:stdout, 2:stderr
• $ cd ~
• ~$ echo Hello, File! >hello.txt
• $ cat hello.txt
• $ cat hello.txt | grep Hello
• $ grep out.txt
• $ grep >out.txt # appends to out.txt.
Slide 26
Slide 26 text
• $ grep out.txt # still prints errors.
• $ grep out.txt 2>err.txt
• $ grep out.txt 2>&1
• $ grep both.txt
• $ grep /dev/null
Slide 27
Slide 27 text
• $ cat < Hello, Here Doc!
> It's useful to paste something.
> HEREDOC
• $ cat <<<'Hello, Here String!'
• $ python3 <<<'print(1234
* *
1234)'
# C's unsigned long is 0 to 18446744073709551615, btw.
# Useful!
Slide 28
Slide 28 text
Expansions & Substitutions
• $ echo x.{txt,csv}
• $ echo ~
• $ echo ~root # or any username.
• $ cd ~
• ~$ touch a.{txt,csv}
• $ echo a.*
• $ echo x.* #
- >
x.*
• $ echo ?.*
• $ echo [abcx].txt
• $ echo [abcx].csv
Slide 29
Slide 29 text
• $ echo $name
• $ echo ${name}
• $ echo ${name/.txt/.csv}
• $ echo `echo 'Hello, Echo!'`
• $ echo $(echo 'Hello, Echo!')
• $ cat <(echo 'Hello, Cat!')
• See also:
• Bash scripting cheatsheet §
Parameter expansions
• Shell Style Guide § Variable
expansion
Slide 30
Slide 30 text
Environments & Parameters
• Bash scans its own
environment variables and creates
shell parameters.
• $ echo $PATH
• $ A=1 env | grep A= #
- >
A=1
• $ echo $A # A is empty.
• $ A=2
• $ echo $A #
- >
2
• $ env | grep A= #
- >
• $ export A
• $ # export A=20
• $ env | grep A= #
- >
A=2
• $ A=3 echo $A #
- >
2
• $ #
= = =
$ A=3 echo 2
• $ export
-
n A && env | grep A=
Slide 31
Slide 31 text
Job Control
• $ vi a.txt # and .
• $ vi b.txt # and .
• $ jobs
• $ fg # shows f
i
lename.
• $ fg - # like Cmd-Tab or Alt-Tab.
• $ fg 1 # `:q` to exit vim.
• $ kill %2
• $ sleep 3 && echo Hi! & #
= = =
• $ bash
-
c 'sleep 3 && echo Hi!'
# and .
• $ bg
Slide 32
Slide 32 text
Aliases
• $ alias ll
• $ type ll
• $ ls
- -
help | grep ' -[alF]'
• $ alias hi=history
• $ hi 10
• $ alias
. . =
'cd
. .
'
• $ cd /usr/bin
• usr: User System Resources
• $
. .
Slide 33
Slide 33 text
History
• , ; ,
• , : search history.
• $ #TAG
• $ #TAG
• $ stty
-
ixon # needs it.
• : discard the editing.
• $ history 10
• $ history
-
a
• $ history
-
n
• $ history
- -
help | grep ' -[an]'
Slide 34
Slide 34 text
Checkpoint: Hack the which
• $ which echo
/usr/bin/echo
• $ ??? which echo
bash: which: No such f
i
le or directory
• Hints:
• which searches the PATH for the executable
fi
le.
Slide 35
Slide 35 text
Text processing
Slide 36
Slide 36 text
No content
Slide 37
Slide 37 text
TSV
Create TSV
fi
le using cat
• $ cd
• ~$ cat <cmds.tsv
>
> EOS
Slide 38
Slide 38 text
TSV
Create TSV
fi
le using vim
• $ cd
• ~$ vi cmds.tsv
• (vim) i # goes the insert mode.
• (vim)
• (vim) # backs to the normal mode.
• (vim) :x # goes the command
-
line mode, write and quit.
Slide 39
Slide 39 text
• If any garbled text:
• ubuntu
-
c$ LANG=C.UTF-8 bash
• or,
• (host)$ docker exec
-
it ubuntu
-
c env LANG=C.UTF-8 bash
Slide 40
Slide 40 text
• $ wc
-
l cmds.tsv # wc: word count
• $ cat cmds.tsv
• $ head cmds.tsv
• $ tail cmds.tsv
• $ grep
Slide 41
Slide 41 text
TSV is literally everywhere.
Slide 42
Slide 42 text
Checkpoint: Copy the head and paste to a sheet
• A sheet in Google Sheets, Excel,
LibreO
ff i
ce Calc, like →
• Hints:
• Just like how did you paste the
table from the browser to the
terminal.
• Copy the output on the
terminal and paste it to the
sheet.
Slide 43
Slide 43 text
TSV (cont.)
• $ seq 1 3
• $ seq 1 3 | column
• $ seq 1 3 | column | hexdump
-
c
• $ seq 1 3 | column
-
t
Slide 44
Slide 44 text
• $ head '
-
t
• $ column
- -
help | grep ' -[st]'
• $ head
Slide 45
Slide 45 text
• $ cut
Slide 46
Slide 46 text
• $ awk
Slide 47
Slide 47 text
• See also:
• The GNU Awk User’s Guide
• 6.3.2.2 Comparison Operators
• 6.1.1.3 Regular Expression Constants
• 8.1.5 Scanning All Elements of an Array
Slide 48
Slide 48 text
How many mandatory text processing does POSIX have?
Checkpoint:
• $ ???
Slide 49
Slide 49 text
TSV (cont.)
• $ cut
Slide 50
Slide 50 text
• $ cut a
• $ awk b
• $ vimdiff a b
• $ diff a b
• $ diff a b
-
y # using the side by side format.
• $ diff a b
-
u # using the unif
i
ed format.
• $ diff
- -
help | grep ' -[yu]'
Slide 51
Slide 51 text
• $ sudo apt install bat
-
y # or,
• # apt install bat
-
y
• $ alias bat=batcat
• $ diff a b | bat # to exit.
Slide 52
Slide 52 text
Interact with the system clipboards
• (macOS)$ pbpaste | tail -1 | pbcopy
• The way I make slides with highlighted codes:
• $ pbpaste | highlight
-
u utf-8 -S python
-
s solarized
-
light
-
k Monaco
-K 28 -O rtf | pbcopy # and then paste into a slide.
• (X11)$ xclip
-
o | tail -1 | xclip
• (WSL)$ powershell.exe Get-Clipboard | tail -1 | clip.exe
• See also: vor0nwe/clip – Gist for a neat clip script in WSL.
Slide 53
Slide 53 text
cURL
• $ curl https://google.com
• $ curl
-
v https://google.com
• $ curl -L https://google.com
Slide 54
Slide 54 text
No content
Slide 55
Slide 55 text
JSON with jq
• $ sudo apt install jq
-
y # or,
• # apt install jq
-
y
• jq is like sed for JSON.
• $ cd
• ~$ curl https://raw.githubusercontent.com/5etools
-
mirror-1/5etools
-
mirror-1.github.io/
master/data/bestiary/bestiary
-
mm.json >monsters.json # or,
• ~$ curl https://raw.githubusercontent.com/hazmole/TheGiddyLimit.github.io/master/
data/bestiary/bestiary
-
mm.json >monsters_zh_tw.json
• $ wc
-
l monsters.json #
- >
70811
Slide 56
Slide 56 text
• $ jq
Slide 57
Slide 57 text
• $ jq
Slide 58
Slide 58 text
• $ jq dragons.tsv
Slide 59
Slide 59 text
• $ sort
Slide 60
Slide 60 text
What is the first undead monster in the book?
Checkpoint:
• $ jq ??? | tail -1
???
• Hints:
• “undead”
• “page”
Slide 61
Slide 61 text
JSON with gron
• $ sudo apt install gron
-
y # or,
• # apt install gron
-
y
• $ jq dragons.json
• $ bat dragons.json
• $ gron dragons.json
• $ gron dragons.json | grep -E 'json\[[0-9]+\]\.name ='
• $ gron dragons.json | grep -E 'json\[[0-9]+\]\.name =' | gron
-
u
Slide 62
Slide 62 text
. * [a
-
z] ^$ ? + |
Fixed string . * [a
-
z] ^$ ? + | $ grep -F
BRE . * [a
-
z] ^$ ? + |
$ grep
$ grep -G
ERE . * [a
-
z] ^$ ? + | $ grep -E
Glob . * [a
-
z] ^$ ? + |
Fixed string, BRE, ERE, vs. glob
Slide 63
Slide 63 text
Test the regex types
• $ echo 'i++' | grep -F '.+'
-
o # f
i
xed string (not regex)
• $ echo 'i++' | grep -G '.+'
-
o # basic regular expression (BRE)
i
+
• $ echo 'i++' | grep -E '.+'
-
o # extended regular expression (ERE)
i
+ +
Slide 64
Slide 64 text
• See also:
• Regex cheatsheet for BRE, ERE, Vim, Python's re, PCRE (Perl).
• regex101: a regex debugger for Python, JavaScript, Java, Golang, PHP.
• jq Manual
• tomnomnom/gron – GitHub
• yq for YAML.
Slide 65
Slide 65 text
Checkpoint: The regex type of less may be?
• BRE-like ... ERE-like ... or not regex?
• Hints:
• How to put i
+ +
into less?
• How to search something in less?
Slide 66
Slide 66 text
Bulk editing
• $ cp dragons.tsv cats.tsv
• $ cp dragons.tsv cats_2.tsv
• $ head cats
*
.tsv
• $ sed s/Dragon/Cat/ cats
*
.tsv | less
• $ sed
-
e s/Dragon/Cat/
-
e s/dragon/cat/ cats
*
.tsv | less
• $ sed
-
e s/Dragon/Cat/g
-
e s/dragon/cat/g cats
*
.tsv | less
• $ sed
-
e s/Dragon/Cat/g
-
e s/dragon/cat/g cats
*
.tsv
-
i
Slide 67
Slide 67 text
CSV with CSVKit
• $ sudo apt install csvkit
-
y # or,
• # apt install csvkit
-
y
• $ jq dragons.tsv
• $ jq dragons.csv
• $ bat dragons.csv
Slide 68
Slide 68 text
• $ csvcut
a Markdown table
• $ csvgrep
Slide 69
Slide 69 text
• See also:
• Tutorial – csvkit
• 1.4. in2csv: the Excel killer
• 3.3. csvsql and sql2csv: ultimate power
Slide 70
Slide 70 text
Parallelizing
• $ time sleep 1
• $ seq 3 | xargs
-
i echo sleep {} # xargs: extended arguments
• $ seq 3 | xargs
-
i echo sleep 1
• $ time seq 3 | xargs
-
i sleep 1 # takes 3 seconds.
• $ time seq 3 | xargs
-
i -P 3 sleep 1 # takes 1 second.
• $ man xargs | less +'/ -[iI]'
Slide 71
Slide 71 text
System management
Slide 72
Slide 72 text
File management
• $ cd
• ~$ mkdir mydir && cd mydir
• ~/mydir$ touch a b .h
• $ mkdir
-
p x/y/z
• $ cp b c
• $ rm b
• $ ls
• $ ll
• $ type ll
• $ ll
-
t
# List and sort by time.
• $ ll -S
-
h
# List and sort by size.
Slide 73
Slide 73 text
• $ sudo apt install tree
-
y # or,
• # apt install tree
-
y
• $ cd ~/mydir
• ~/mydir$ ls
• $ tree
• $ tree
-
a # List all f
i
les.
• $ tree -L 1 # List one level deep.
• $ tree
-
f # Print full paths.
Slide 74
Slide 74 text
• $ cd
• ~$ ln
-
s mydir mydir_link # Make a symbolic link.
• $ ls mydir_link
• $ ll mydir_link
• $ cp
-
r mydir mydir_backup # Copy recursively.
• $ ll mydir_backup
Slide 75
Slide 75 text
• $ tar czf mydir.tar.gz mydir
• $ rmdir mydir
#
- >
failed
. . .
not empty.
• $ rmdir mydir/x/y/z
# OK!
• $ rm
-
r mydir/
• $ tar xzf mydir.tar.gz
• $ tree
• $ du
-
h
-
d 1 # du: disk usage
• $ df
-
h # df: disk free
Slide 76
Slide 76 text
Describe one difference between cp
-
r and cp
-
a
Checkpoint:
• Read and search the manual!
• Hints:
• $ man cp # `/` to search; `n` to next; `h` to get help.
• $ cp
- -
help | grep
. . .
Slide 77
Slide 77 text
Terminology
EN program process thread concurrency parallelism
TW 程式 程序 執⾏緒 並⾏性 平⾏性
CN 程序 进程 线程 并发性 并⾏性
Slide 78
Slide 78 text
Process management
• $ ps # ps: process status
• $ ps a # a: includes other users.
• $ ps ax # x: includes other ttys or no tty's.
# tty: teletypewriter
• $ ps axu # u: user
-
oriented format
• $ ps f # f: "forest"
• $ man ps | grep -E ' (VSZ|RSS|TTY|STAT|START|TIME) '
Slide 79
Slide 79 text
• $ vi test.txt # and .
• $ ps u | grep vi
• $ ps u | grep [v]i
• $ ps u | sed
-
n
-
e 1p
-
e /[v]i/p
• $ ps axu | grep [v]i |
awk '{ rss += $6 } END { printf "%.4g MB\n", rss/1024 }'
• (macOS)$ ps axu | grep [C]hrome |
awk '{ rss += $6 } END { printf "%.4g MB\n", rss/1024 }'
Slide 80
Slide 80 text
• $ vi test.txt # and .
• $ pgrep v #
- >
• $ pgrep vi #
- >
• $ pgrep test #
- >
• $ pgrep
-
f test #
- >
• $ pgrep
-
f test
-
l
• $ pgrep
- -
help | grep ' -[fl]'
• $ # kill
• $ pkill
-
f test # -SIGTERM
• $ fg
• $ vi test.txt # and .
• $ pkill
-
f test -SIGKILL
• See also:
• Signal (IPC) – Wikipedia
Slide 81
Slide 81 text
Monitoring
• $ sudo apt install iotop iftop
-
y # or, # if: interface
• # apt install iotop iftop
-
y
• $ free
-
h
• $ man free | grep -E ' (total|used|free|shared|buff/cache|available)' -A 1
• $ vi test.txt # and .
• $ top
• $ htop
• Select a process, and (kill)
Slide 82
Slide 82 text
• $ cd
• ~$ python3
-
m http.server &> my.log
# to interrupt.
• Open http://localhost:8000/.
• Open another terminal:
• # iftop # to exit.
• $ tail
-
f ~/my.log
• $ less +F ~/my.log
# Try `
-
i` `/http` .
• $ watch tail -10 ~/my.log
Slide 83
Slide 83 text
• The commands which are useful but not relevant to containers:
• $ sudo iotop
• Important system logs:
• $ sudo vi /var/log/syslog
• $ sudo vi /var/log/auth.log
Slide 84
Slide 84 text
Checkpoint: Monitor your HTTP server by less
• Try to get three new logs.
• Hints:
• Click on your browser!
Slide 85
Slide 85 text
Monitoring (cont.)
• $ python3
-
c 'print(1234)'
• $ strace python3
-
c 'print(1234)'
• $ sudo strace
-
p
• $ ps x
• $ pgrep bash
• $ tty
• $ pgrep bash
-
t pts/1
# pts: pseudo terminal slave
• $ lsof
-
p `pgrep bash
-
t pts/1`
# lsof: list open f
i
les
• $ lsof /usr/bin/bash
• $ man lsof | less
-
i +/^examples
• $ htop
• Select a process, and:
• (strace)
• (lsof)
Slide 86
Slide 86 text
• $ sudo apt install sysstat
-
y # or,
• # apt install sysstat
-
y
• $ sar 1
# Report CPU stats per second.
• $ sar
# Report today's CPU stats.
• $ sar
-
f /var/log/sysstat/sa10
• $ sar
-
f /var/log/sysstat/sa10
-
s 12
:
00
-
e 15
:
00
• $ man sar | grep 'Report ' -B 1
• $ sar
-
f /var/log/sysstat/sa10
-
s 12
:
00
-
e 15
:
00
-
r
- -
human
# Report memory stats.
• See also:
• sysstat/sysstat – GitHub
• Amazon CloudWatch
• Grafana
Slide 87
Slide 87 text
Container management
• OCI: Open Container Initiative
• Docker
fi
le: part of OCI spec.
• image: container image
• Why?
• Once you pack your app as a
container image, it can be run
anywhere seamlessly.
Slide 88
Slide 88 text
• (host)$ docker container commit ubuntu
-
c ubuntu
-
cli
-
power
#
= = =
$ docker container commit
. . .
• $ docker image ls
#
= = =
$ docker images
Slide 89
Slide 89 text
• $ docker image save ubuntu
-
cli
-
power >ubuntu
-
cli
-
power.tar
• $ docker image rm ubuntu
-
cli
-
power
#
= = =
$ docker rmi ubuntu
-
cli
-
power
• $ docker image ls
• $ docker image load
Slide 90
Slide 90 text
• $ docker container create
- -
name ls
-
expert ubuntu
-
cli
-
power man ls
• $ docker container ls
-
a #
= = =
$ docker ps
-
a
• $ docker container start ls
-
expert
• $ docker container ls
-
a #
- >
Exited (0) 4 seconds ago
• $ docker container start
-
a ls
-
expert # Start and attach.
• $ docker container rm ls
-
expert
• $ # docker container (create|start|stop|rm)
= = =
docker container (create|start|stop|rm)
Slide 91
Slide 91 text
• $ docker container cp ubuntu
-
c:/root/cmds.tsv ~/cmds.tsv
#
= = =
$ docker cp
. . .
• $ docker container cp ~/cmds.tsv ubuntu
-
c:/root/cmds_2.tsv
• $ docker container run
-
dit
-
v ~/container:/root
- -
name volume
-
c ubuntu
-
cli
-
power
#
= = =
$ docker run
. . .
Slide 92
Slide 92 text
Work on a remote machine
Slide 93
Slide 93 text
Run programs after disconnect
• $ tmux # tmux: terminal multiplexer
• (tmux)$ while true; do date; sleep 1; done # and then .
• $ tmux ls # ls: the alias of list
-
sessions.
• $ tmux a # a: attach
• (tmux) > # List key bindings; to exit.
• (tmux)
• $ tmux ls
Slide 94
Slide 94 text
• The way I start the server of JupyterLab:
• $ tmux new
-
s jupyter
-
lab
-
d 'cd && jupyter lab
- -
no
-
browser'
Slide 95
Slide 95 text
The local key pair
• (local)$ ll ~/.ssh/id_rsa
*
• If nothing there:
• (local)$ ssh
-
keygen
• (local)$ cat ~/.ssh/id_rsa.pub # pub: public key
• id_rsa is the private key which proves who you are.
• Do not share id_rsa – the private key – to anyone.
• Simply generate a new pair when you have a little doubt about the old pair.
Slide 96
Slide 96 text
Add a user in the remote host
• (remote)# new_user=
• (remote)# useradd
-
m
-
s /bin/bash $new_user
• (remote)# su $new_user
• $ mkdir ~/.ssh
$ cat <~/.ssh/authorized_keys
>
> EOS
$ exit
• (remote)# service ssh start
Slide 97
Slide 97 text
Connect to a remote host
• (local)$ ssh ssh://@localhost:2222
#
= = =
$ ssh @localhost
-
p 2222
# $ ssh # if port is 22 and usernames are the same.
• If there is a warning and you're sure you changed something in the remote:
• $ sed d ~/.ssh/known_hosts
-
i
Slide 98
Slide 98 text
ssh -A: Allow the remote to use the local key pair
• (local) $ ssh ssh://@localhost:2222
• (remote)$ ssh ubuntu
-
c
#
. . .
password:
• (local) $ ssh ssh://@localhost:2222 -A
• (remote)$ ssh ubuntu
-
c # OK
• The ubuntu-c works since the docker run --network enables the DNS
resolution between containers.
• See also: Consul, for the similar functionality without Docker.
Slide 99
Slide 99 text
ssh -J: Jump to a host in a private network
• (local)$ ssh ubuntu
-
c
#
- >
Could not resolve hostname ubuntu
-
c
• (local)$ ssh -J ssh://@localhost:2222 ubuntu
-
c
• -J is securer than -A; use -J when applicable.
Slide 100
Slide 100 text
Transfer files between local and remote
• (remote)$ cp cmds.tsv ~
• (local) $ rsync
-
e 'ssh ssh://@localhost:2222' :~/cmds.tsv ~
# $ rsync @ :
~/cmds.tsv ~ # if port is 22.
• (local) $ rsync
-
e 'ssh ssh://@localhost:2222' ~/cmds.tsv :~/cmds_2.tsv
• Deprecating scp – LWN.net, but scp is still a common way nowadays:
• (local)$ scp scp://@localhost:2222/~/cmds.tsv ~
# $ scp @ :
~/cmds.tsv ~ # if port is 22.
• (local)$ scp ~/cmds.tsv scp://@localhost:2222/~/cmds_2.tsv
Slide 101
Slide 101 text
Mount a remote directory
• (local)$ sudo apt install sshfs # or macFUSE for macOS.
• (local)$ mkdir ~/sshfs_container
• (local)$ sshfs @localhost:/home/
-
p 2222 ~/sshfs_container
• (local)$ ls ~/sshfs_container
• (local)$ umount ~/sshfs_container
Slide 102
Slide 102 text
ssh -L: Forward a local port to a remote port
• As known as “SSH tunneling”.
• (remote)$ python3
-
m http.server 1234 # to interrupt.
• (local)$ ssh ssh://@localhost:2222 -L 4321:ubuntu
-
c:1234 -Nf
• Open http://localhost:4321/.
• localhost:4321
→ localhost:2222 → ubuntu-c:22
→ ubuntu-c:1234 → http.server 1234
• (local)$ pgrep
-
f 4321
• (local)$ # pkill
-
f 4321
Slide 103
Slide 103 text
Access the HTTP server by http://localhost:8765/
Checkpoint:
• Open http://localhost:8765/ and there should be a page.
• Hints:
• Use ssh -L.
Slide 104
Slide 104 text
ssh -D: Start a proxy server
• (local)$ ssh ssh://@localhost:2222 -D 1080 -Nf
• (remote)# iftop
• Con
fi
gure the system proxy or Proxy SwitchyOmega for Chrome.
• (remote)# python3
-
m http.server 80
• Open http://ubuntu-c/.
• For accessing the services in a private network by private DNS and standard ports!
• If the server has another public IP, try Google “my ip”.
• $ man ssh | grep ' -[LDNf]' -A 1
Slide 105
Slide 105 text
System proxy for macOS
Slide 106
Slide 106 text
Proxy SwitchyOmega for Chrome
Slide 107
Slide 107 text
No content
Slide 108
Slide 108 text
• See also:
• A simple ~/.ssh/con
fi
g for macOS
• A simple Bash con
fi
g for both macOS and Linux
Slide 109
Slide 109 text
Developing on CLI
Slide 110
Slide 110 text
Edit a config
• $ cp ~/.bashrc ~/backup_bashrc
• $ echo 'alias bat=batcat' >>~/.bashrc
• Re-enter the shell to take e
ff
ect.
• $ grep batcat ~/.bashrc
-
n
• $ sed d ~/.bashrc
-
i
• $ grep batcat ~/.bashrc
-
n
Slide 111
Slide 111 text
• $ vi ~/.bashrc
• (vim) #
• (vim) i # or `a`
• (vim) alias bat=batcat # literally
• (vim) # backs to Normal
• (vim) :x
• $ grep batcat ~/.bashrc
-
n
• $ vi ~/.bashrc
• (vim) /cat
• (vim) n
. . .
• (vim) dd
• (vim) :x
• $ grep batcat ~/.bashrc
-
n
Slide 112
Slide 112 text
The mini-cheatsheet of Vim
1. / n
• Search; Next.
2. i :x
• Go Insert; Back to Normal; Write and quit.
3. dd
• Delete a line.
• It's all you need to know now.
Of course, the more you know, the more e
ff
i
ciency you'll get. 💪
Slide 113
Slide 113 text
Spell checking
• $ echo 'Oops. Does it spll like this?'
• (vim) :set spell
• (vim) /sp
• (vim) z= # List spelling suggestions.
• (vim) 1
• (vim) :x
Slide 114
Slide 114 text
• See also:
• $ vimtutor # Learn Vim in 30 mins!
• (vim) :help dd, for example.
• Vim Keystrokes – Mosky’s Notes
• Vim’s Search Tips – Mosky’s Notes
• Vim’s Replace Tips – Mosky’s Notes
• Mosky's Vim Con
fi
gs
Slide 115
Slide 115 text
Checkpoint: Add your bat
• (host)$ docker exec
-
it ubuntu
-
c bash
• ubuntu
-
c# bat ~/.bashrc # It should work immediately.
• Hints:
• You may use echo to add.
• Use grep to con
fi
rm.
• Re-enter the shell to take e
ff
ect.
Slide 116
Slide 116 text
Developing
Slak: Collect data from Slack like a pro. ⚡
1. System Tools: the common tools which are
supposed to be installed in the system.
• git: the version control system.
• pip: the Python package manager.
• pipenv: one of Python package environment
managers.
• The Docker (containers) provides OS-level
isolation; Pipenv provides Python
interpreter and package–level isolation.
2. Project Tools: the tools of this project, good to
be managed within the project by, for example,
pipenv.
• ipdb: a simple yet beautiful debugger.
• pytest: the testing framework.
3. Personal Tools: the tools of a developer, may be
installed in one's system.
• flake8: the static analyzer (linter).
• mypy: the static type checker.
• black: one of the formatters.
Slide 117
Slide 117 text
• $ sudo apt install pip
-
y # or, # System Tool
• # apt install pip
-
y
• $ pip install pipenv # System Tool
• $ cd
• ~$ git clone
- -
depth 1 https://github.com/moskytw/slak.git
Slide 118
Slide 118 text
• ~$ cd slak
• ~/slak$ bat Pipf
i
le
• $ pipenv sync
- -
dev
• $ slak
• $ python slak.py
• $ for cmd in python python3 slak;
do type $cmd; done
• $ pipenv shell
• (slak)$ slak
• $ python slak.py
• $ for cmd in python python3 slak;
do type $cmd; done
Slide 119
Slide 119 text
• (slak)$ python
-
m ipdb slak.py # Project Tool, `q` to exit.
• $ pytest # Project Tool
• $ pytest
- -
cov
• $ pytest
- -
cov
- -
cov
-
report html
• $ cd htmlcov
• ~/slak/htmlcov$ python
-
m http.server # to interrupt.
• Open http://localhost:8000/.
• $ exit # or to exit the shell of pipenv.
Slide 120
Slide 120 text
• (my mac)$ for cmd in flake8 mypy black xxx; do type $cmd; done
flake8 is hashed (/usr/local/bin/flake8)
mypy is aliased to `\mypy
- -
ignore
-
missing
-
imports'
black is aliased to `black
- -
line
-
length 79
- -
skip
-
string
-
normalization'
-
bash: type: xxx: not found
• $ for cmd in flake8 mypy black xxx; do type $cmd >/dev/null; done
-
bash: type: xxx: not found
Slide 121
Slide 121 text
• $ cd ~/slak
• ~/slak$ flake8 slak.py
• $ mypy slak.py
• $ black slak.py
Slide 122
Slide 122 text
• The di
ff
erent languages have di
ff
erent toolsets; however, the concepts and
the categories are similar, e.g.:
• $ vi /etc/nginx/sites
-
enabled/your
-
site.conf
• $ nginx
-
tc /etc/nginx/sites
-
enabled/your
-
site.conf
# Test the conf
i
g f
i
le.
Slide 123
Slide 123 text
• See also:
• git - the simple guide: “no deep shit ;)”
• Git Documentation
• Cheat Sheets
• Pro Git, the book
• Reference Manual
• Mosky’s Pipenv Notes
Slide 124
Slide 124 text
In a large project
• It's all about locate the path and line number:
• f
i
nd/fd: locate by path keywords.
• grep/rg: locate by content keywords.
• ctags: locate by pre-built “tags”.
• Language Server: a server keeps analyzing your code in background.
Slide 125
Slide 125 text
• $ cd /usr/lib/python3.10
• /usr/lib/python3.10$ f
i
nd
-
name '
*
server
*
'
• $ sudo apt install fd
-
f
i
nd
-
y # or,
• # apt install fd
-
f
i
nd
-
y
• $ alias fd=fdf
i
nd
• $ fd server # fd: f
i
nd
Slide 126
Slide 126 text
• $ cd /usr/lib/python3.10
• /usr/lib/python3.10$ grep
-
r 'Serving HTTP'
-
n
• $ sudo apt install ripgrep
-
y # or,
• # apt install ripgrep
-
y
• $ rg 'Serving HTTP' # rg: ripgrep
Slide 127
Slide 127 text
• $ f
i
nd
-
name 'server.py'
-
exec grep 'Serving HTTP'
-
n {} +
• $ fd '^server.py$' -X rg 'Serving HTTP' {}
• $ man f
i
nd | less +'/
-
exec '
• $ man fd | less +'/ -[Xx]'
Slide 128
Slide 128 text
• Ctags & Language Server:
• $ sudo apt install universal
-
ctags
$ ctags -R
$ vim +'help tags | only'
• A 10 MB–level solution.
• neoclide/coc.nvim – GitHub for the Language Server solution.
• A GB-level solution.
Slide 129
Slide 129 text
The end
• But hope it's the start of your
journey of getting the CLI power.
• Having fun is still the key,
as always.
• May the man guide you!
Slide 130
Slide 130 text
Image Credits
• “The Power”
https://www.facebook.com/nixcraft/photos/
meme-based-upon-iamnotanartist_-comic-
for-linuxunix-users/3443131449033542/
• “The Terminal”
https://en.wikipedia.org/wiki/
Computer_terminal#/media/
File:DEC_VT100_terminal_transparent.png
• “What Is a Container?”
https://www.docker.com/resources/what-
container/
• “The Cat”
https://www.facebook.com/nixcraft/posts/
linux-explained-for-new-users-including-cat-
command-explained-httpswwwcybercitib/
3034767979869893/
• “The Dragon”
https://unsplash.com/photos/F6920BvzrZE
• “The Miniatures”
https://unsplash.com/photos/Q4Honp3Pyqs
• “The Banshee Appears”
https://en.wikipedia.org/wiki/Banshee#/
media/File:The_Banshee_Appears_(1862).jpg
• “The Earth”
https://unsplash.com/photos/Q1p7bh3SHj8