Get Power From Command Line

Mosky
Get Power From Command Line
The Command Line Skills You Missed

View Slide

View Slide

Financial Freedom

View Slide

Power Freedom

View Slide

Mosky
Architect, Pinkoi
• Data Science With Python

• Statistical Regression With
Python

• Hypothesis Testing With Python

• Practicing Python 3

• The mediums.

• The PyCons, etc.

View Slide

1. Introduction to command line

2. Text processing: e.g., CSV, JSON, parallelizing.

3. System management: e.g.,
fi
les, processes, monitoring.

4. Work on a remote machine: e.g., SSH, SSH tunneling.

5. Developing on CLI
Outline

View Slide

What commands?
• POSIX

• Portable Operating System Interface

• IEEE Std 1003.1-2008

• List of Unix commands – Wikipedia

• Ubuntu 22.04 LTS Server

• curl, gawk, git, htop, less,
lsof, man
-
db, rsync, strace,
tmux, vim, ssh, etc.

• The de facto standard

• tree, ripgrep, jq, etc.

• The modern alternatives

• bat (cat), fd
-
f
i
nd (
fi
nd), etc.

View Slide

How to use this deck?
• Practice makes perfect:

• Try every single command in this deck, and  
downloading the deck as a PDF may make it easier.

• Be careful of the spaces and newlines after paste.

• Visit the links, but don't read; scan them to index in your brain.

View Slide

Introduction to command line

View Slide

“Command Notes”
• Memorize all the commands?
No.

• You may try to note them, and
make them searchable.

• Inputs & Outputs.

• Tags.

• Like →

View Slide

Get your terminal emulator
• iTerm for macOS

• See also: Mosky’s Preferences
in macOS § iTerm and search
“Bash 5” in the doc!

• Windows Terminal
• GNOME Terminal / Konsole

View Slide

Connect to ...
• macOS with GNU/Linux commands

• Install the GNU/Linux commands on macOS

• WSL

• Windows Subsystem for Linux

• Install Linux on Windows with WSL

• Ubuntu

• Or,

View Slide

The container
• Get Docker

• $ docker pull ubuntu:22.04
 
$ docker network create ubuntu
-
n
 
$ docker run
-
dit
-
p 2222
:
22
-
p
8000
:
8000
- -
network ubuntu
-
n
- -
name ubuntu
-
c ubuntu:22.04
 
$ docker exec
-
it ubuntu
-
c bash

• ubuntu
-
c# yes | unminimize &&
apt install curl gawk git htop
less lsof man
-
db rsync strace
tmux vim ssh
-
y

View Slide

The container
• Get Docker

• $ docker pull ubuntu:22.04
 
$ docker network create ubuntu
-
n
 
$ docker run
-
dit
-
p 2222
:
22
-
p
8000
:
8000
- -
network ubuntu
-
n
- -
name ubuntu
-
c ubuntu:22.04
 
$ docker exec
-
it ubuntu
-
c bash

• ubuntu
-
c# yes | unminimize &&
apt install curl gawk git htop
less lsof man
-
db rsync strace
tmux vim ssh
-
y
$ hints as a normal
user.
ubuntu
-
c# hints
as a superuser and
in the Ubuntu
container.
They are default in
the Ubuntu Server,
but not in the
container image.

View Slide

Checkpoint: Log into the container
• You shall see →

• Hints:

• $ docker stop ubuntu
-
c;
docker rm ubuntu
-
c # to
start over.

• $ docker start ubuntu
-
c
 
$ docker exec
-
it ubuntu
-
c
bash # after reboot if any.

View Slide

Read the manual
• $ man man
 
# Try `\
-
k`
 
# and `\
-
k`

• $ man
-
k ls

• $ ls
- -
help | less # to exit.

• $ ls
- -
help | grep '
-
l'

• $ help for

• $ type man

• $ type ls

• $ type for

• Google: “man yes”

• explainshell.com

• Bash Reference Manual

View Slide

Read the manual
• $ man man
 
# Try `\
-
k`
 
# and `\
-
k`

• $ man
-
k ls

• $ ls
- -
help | less # to exit.

• $ ls
- -
help | grep '
-
l'

• $ help for

• $ type man

• $ type ls

• $ type for

• Google: “man yes”


• Bash Reference Manual
Two spaces.

View Slide

Edit commands like a ninja
• Bash Keystrokes – Mosky’s Notes

• $ ecHello, World!

• $ Ctrl-W!

• $ H!

• $ ?

• Also works in Python shell and
many other places.

• See also:

• “CTRL-p CTRL-q” to detach –
Docker Docs

• macOS Shortcuts – Mosky’s
Notes

• Windows Shortcuts – Mosky’s
Notes

View Slide

Edit commands like a ninja
• Bash Keystrokes – Mosky’s Notes

• $ ecHello, World!

• $ Ctrl-W!

• $ H!

• $ ?

• Also works in Python shell and
many other places.

• See also:

• “CTRL-p CTRL-q” to detach –
Docker Docs

• macOS Shortcuts – Mosky’s
Notes

• Windows Shortcuts – Mosky’s
Notes
Or .
The is
or
.
Or , if not
from the docker.
“Space” vs.
“Alphanumeric”
The is
 
or .

View Slide

Checkpoint: “Hey, Ctrl-K!”
• $ echo Hello, World!

• $
 
Hey, Ctrl-K!

• Hints:

• , ,

View Slide

Quoting
• $ echo Hello, World! # 2 arguments.

• $ echo 'Hello, World!' # 1 argument.

• $ echo Hello, $HOSTNAME!

• $ echo "Hello, $HOSTNAME!"

• $ echo "Hello, \$HOSTNAME!"

• $ echo 'Hello, $'HOSTNAME!

• $ echo $'Hello,\nWorld' # ANSI-C Quoting

View Slide

Pipe, AND, OR
• $ echo 'Hello, World!' | grep Hello

• $ echo 'Hello, World!' | grep Hello && echo Found!

• $ echo 'Hello, World!' | grep Hey && echo Found!

• $ echo 'Hello, World!' | grep Hey || echo Not found.

• $ echo 'Hello, World!' | grep Hello && echo Found! || echo Not found.

• $ echo 'Hello, World!' | grep Hey && echo Found! || echo Not found.

View Slide

Checkpoint: Explain the echo by yourself

• $ echo 'Hello, World!' | grep Hello && echo Found! || echo Not found.

View Slide

Loops
• $ cd ~ # cd: change directory

• ~$ touch a.txt b.txt c.txt

• $ for name in *.txt; do echo mv $name ${name/.txt/.csv}; done
 
# It's a dry run.

• $ for name in *.txt; do echo mv $name ${name/.txt/.csv}; done

• $ while true; do date; docker container top ubuntu
-
c; echo; sleep 3; done

• $ until docker start ubuntu
-
c; do sleep 3; done

View Slide

Redirections
fi
le descriptor:name = 0:stdin, 1:stdout, 2:stderr
• $ cd ~

• ~$ echo Hello, File! >hello.txt

• $ cat hello.txt

• $ cat hello.txt | grep Hello

• $ grep

• $ grep out.txt

• $ grep >out.txt # appends to out.txt.

View Slide

• $ grep - -
bad Hello >out.txt # still prints errors.

• $ grep - -
bad Hello >out.txt 2>err.txt

• $ grep - -
bad Hello >out.txt 2>&1

• $ grep - -
bad Hello &>both.txt

• $ grep - -
bad Hello 2>/dev/null

View Slide

• $ cat < 
> Hello, Here Doc!
 
> It's useful to paste something.
 
> HEREDOC

• $ cat <<<'Hello, Here String!'

• $ python3 <<<'print(1234
* *
1234)'
 
# C's unsigned long is 0 to 18446744073709551615, btw.
 
# Useful!

View Slide

Expansions & Substitutions
• $ echo x.{txt,csv}

• $ echo ~

• $ echo ~root # or any username.

• $ cd ~

• ~$ touch a.{txt,csv}

• $ echo a.*

• $ echo x.* #
- >
x.*

• $ echo ?.*

• $ echo [abcx].txt

• $ echo [abcx].csv

View Slide

• $ echo $name

• $ echo ${name}

• $ echo ${name/.txt/.csv}

• $ echo `echo 'Hello, Echo!'`

• $ echo $(echo 'Hello, Echo!')

• $ cat <(echo 'Hello, Cat!')

• See also:

• Bash scripting cheatsheet §
Parameter expansions

• Shell Style Guide § Variable
expansion

View Slide

Environments & Parameters
• Bash scans its own  
environment variables and creates
shell parameters.

• $ echo $PATH

• $ A=1 env | grep A= #
- >
A=1

• $ echo $A # A is empty.

• $ A=2

• $ echo $A #
- >
2

• $ env | grep A= #
- >

• $ export A

• $ # export A=20

• $ env | grep A= #
- >
A=2

• $ A=3 echo $A #
- >
2

• $ #
= = =
$ A=3 echo 2

• $ export
-
n A && env | grep A=

View Slide

Job Control
• $ vi a.txt # and .

• $ vi b.txt # and .

• $ jobs

• $ fg # shows f
i
lename.

• $ fg - # like Cmd-Tab or Alt-Tab.

• $ fg 1 # `:q` to exit vim.

• $ kill %2

• $ sleep 3 && echo Hi! & #
= = =

• $ bash
-
c 'sleep 3 && echo Hi!'
 
# and .

• $ bg

View Slide

Aliases
• $ alias ll

• $ type ll

• $ ls
- -
help | grep ' -[alF]'

• $ alias hi=history

• $ hi 10

• $ alias
. . =
'cd
. .
'

• $ cd /usr/bin

• usr: User System Resources

• $
. .

View Slide

History
• , ; ,

• , : search history.

• $ #TAG

• $ #TAG

• $ stty
-
ixon # needs it.

• : discard the editing.

• $ history 10

• $ history
-
a

• $ history
-
n

• $ history
- -
help | grep ' -[an]'

View Slide

Checkpoint: Hack the which
• $ which echo
 
/usr/bin/echo

• $ ??? which echo
 
bash: which: No such f
i
le or directory

• Hints:

• which searches the PATH for the executable
fi
le.

View Slide

Text processing

View Slide

View Slide

TSV
Create TSV
fi
le using cat
• $ cd

• ~$ cat <cmds.tsv
 
> / /
en.wikipedia.org/wiki/List_of_Unix_commands,
 
copy the table, and paste here.>
 
> EOS

View Slide

TSV
Create TSV
fi
le using vim
• $ cd

• ~$ vi cmds.tsv

• (vim) i # goes the insert mode.

• (vim) / /
en.wikipedia.org/wiki/List_of_Unix_commands,
 
copy the table, and paste here.>

• (vim) # backs to the normal mode.

• (vim) :x # goes the command
-
line mode, write and quit.

View Slide

• If any garbled text:

• ubuntu
-
c$ LANG=C.UTF-8 bash

• or,

• (host)$ docker exec
-
it ubuntu
-
c env LANG=C.UTF-8 bash

View Slide

• $ wc
-
l cmds.tsv # wc: word count

• $ cat cmds.tsv

• $ head cmds.tsv

• $ tail cmds.tsv

• $ grep  
#
= = =
grep cat cmds.tsv

• grep: g/re/p in the ed
command, globally search for a
regular expression and print
matching lines.

• $ grep

• $ grep -
o

View Slide

TSV is literally everywhere.

View Slide

Checkpoint: Copy the head and paste to a sheet
• A sheet in Google Sheets, Excel,
LibreO
ff i
ce Calc, like →

• Hints:

• Just like how did you paste the
table from the browser to the
terminal.

• Copy the output on the
terminal and paste it to the
sheet.

View Slide

TSV (cont.)
• $ seq 1 3

• $ seq 1 3 | column

• $ seq 1 3 | column | hexdump
-
c

• $ seq 1 3 | column
-
t

View Slide

• $ head -
s ''
-
t

• $ column
- -
help | grep ' -[st]'

• $ head  
#
= = =
$ head
-
n 1 cmds.tsv

• $ head

• sed: stream editor

• $ sed -
n
-
e 1p
-
e /^.cat/p

• $ sed
- -
help | grep ' -[ne]'

View Slide

• $ awk

• $ awk -
s $'\t'
-
t

• $ awk counts) print counts[cat] "\t" cat }'

• $ awk counts) print counts[cat] "\t" cat }' | sort
-
n

View Slide

• See also:

• The GNU Awk User’s Guide

• 6.3.2.2 Comparison Operators

• 6.1.1.3 Regular Expression Constants

• 8.1.5 Scanning All Elements of an Array

View Slide

How many mandatory text processing does POSIX have?
Checkpoint:
• $ ??? -
l
 
???

• Hints:

• Either grep or awk is a good choice.

• Try
= =
and
& &
if you choose awk.

• “Text processing”

• “Mandatory”

View Slide

TSV (cont.)
• $ cut -
f 2 | head

• $ awk

• $ vimdiff <(cut -
f 2 | head) <(awk head)
 
# `:qa` to exit.

• $ awk

View Slide

• $ cut -
f 2 | head >a

• $ awk b

• $ vimdiff a b

• $ diff a b

• $ diff a b
-
y # using the side by side format.

• $ diff a b
-
u # using the unif
i
ed format.

• $ diff
- -
help | grep ' -[yu]'

View Slide

• $ sudo apt install bat
-
y # or,

• # apt install bat
-
y

• $ alias bat=batcat

• $ diff a b | bat # to exit.

View Slide

Interact with the system clipboards
• (macOS)$ pbpaste | tail -1 | pbcopy

• The way I make slides with highlighted codes:

• $ pbpaste | highlight
-
u utf-8 -S python
-
s solarized
-
light
-
k Monaco
-K 28 -O rtf | pbcopy # and then paste into a slide.

• (X11)$ xclip
-
o | tail -1 | xclip

• (WSL)$ powershell.exe Get-Clipboard | tail -1 | clip.exe

• See also: vor0nwe/clip – Gist for a neat clip script in WSL.

View Slide

cURL
• $ curl https://google.com

• $ curl
-
v https://google.com

• $ curl -L https://google.com

View Slide

View Slide

JSON with jq
• $ sudo apt install jq
-
y # or,

• # apt install jq
-
y

• jq is like sed for JSON.

• $ cd

• ~$ curl https://raw.githubusercontent.com/5etools
-
mirror-1/5etools
-
mirror-1.github.io/
master/data/bestiary/bestiary
-
mm.json >monsters.json # or,

• ~$ curl https://raw.githubusercontent.com/hazmole/TheGiddyLimit.github.io/master/
data/bestiary/bestiary
-
mm.json >monsters_zh_tw.json

• $ wc
-
l monsters.json #
- >
70811

View Slide

• $ jq

• $ jq

• $ jq

• $ grep - -
color=always | less -R

• $ jq -
l json

• $ bat monsters.json

• $ vi monsters.json # `:q` to exit.

View Slide

• $ jq

• $ jq

• $ jq [.hp.average, .name, .type, .page]' | head

View Slide

• $ sort

• $ sort -
k 2
- -
debug | head # Try to sort by Name.

• $ sort -
k 2,2
- -
debug | head

• $ sort -
t $'\t'
-
k 2,2
- -
debug | head

View Slide

What is the first undead monster in the book?
Checkpoint:
• $ jq ??? | tail -1
 
???

• Hints:

• “undead”

• “page”

View Slide

JSON with gron
• $ sudo apt install gron
-
y # or,

• # apt install gron
-
y

• $ jq

• $ jq dragons.json

• $ bat dragons.json

• $ gron dragons.json

• $ gron dragons.json | grep -E 'json\[[0-9]+\]\.name ='

• $ gron dragons.json | grep -E 'json\[[0-9]+\]\.name =' | gron
-
u

View Slide

Test the regex types
• $ echo 'i++' | grep -F '.+'
-
o # f
i
xed string (not regex)
 

• $ echo 'i++' | grep -G '.+'
-
o # basic regular expression (BRE)
 
i
+

• $ echo 'i++' | grep -E '.+'
-
o # extended regular expression (ERE)
 
i
+ +

View Slide

• See also:

• Regex cheatsheet for BRE, ERE, Vim, Python's re, PCRE (Perl).

• regex101: a regex debugger for Python, JavaScript, Java, Golang, PHP.

• jq Manual

• tomnomnom/gron – GitHub

• yq for YAML.

View Slide

Checkpoint: The regex type of less may be?
• BRE-like ... ERE-like ... or not regex?

• Hints:

• How to put i
+ +
into less?

• How to search something in less?

View Slide

Bulk editing
• $ cp dragons.tsv cats.tsv

• $ cp dragons.tsv cats_2.tsv

• $ head cats
*
.tsv

• $ sed s/Dragon/Cat/ cats
*
.tsv | less

• $ sed
-
e s/Dragon/Cat/
-
e s/dragon/cat/ cats
*
.tsv | less

• $ sed
-
e s/Dragon/Cat/g
-
e s/dragon/cat/g cats
*
.tsv | less

• $ sed
-
e s/Dragon/Cat/g
-
e s/dragon/cat/g cats
*
.tsv
-
i

View Slide

CSV with CSVKit
• $ sudo apt install csvkit
-
y # or,

• # apt install csvkit
-
y

-
r
>dragons.tsv

• $ jq select(.type == "dragon") | [.hp.average, .name, .type, .page]) | @csv'
-
r
>dragons.csv

• $ bat dragons.csv

View Slide

• $ csvcut -
c Name,HP

• $ csvsort -
c HP,Page

• $ csvstat -
c HP,Name,Type

• $ csvlook - >
a Markdown table

• $ csvgrep -
c Name
-
m 'Adult Black Dragon'

• $ csvsql - -
query "select avg(HP) from stdin where Name like
'Adult%'"

View Slide

• See also:

• Tutorial – csvkit

• 1.4. in2csv: the Excel killer

• 3.3. csvsql and sql2csv: ultimate power

View Slide

Parallelizing
• $ time sleep 1

• $ seq 3 | xargs
-
i echo sleep {} # xargs: extended arguments

• $ seq 3 | xargs
-
i echo sleep 1

• $ time seq 3 | xargs
-
i sleep 1 # takes 3 seconds.

• $ time seq 3 | xargs
-
i -P 3 sleep 1 # takes 1 second.

• $ man xargs | less +'/ -[iI]'

View Slide

System management

View Slide

File management
• $ cd

• ~$ mkdir mydir && cd mydir

• ~/mydir$ touch a b .h

• $ mkdir
-
p x/y/z

• $ cp b c

• $ rm b

• $ ls

• $ ll

• $ type ll

• $ ll
-
t
 
# List and sort by time.

• $ ll -S
-
h
 
# List and sort by size.

View Slide

• $ sudo apt install tree
-
y # or,

• # apt install tree
-
y

• $ cd ~/mydir

• ~/mydir$ ls

• $ tree

• $ tree
-
a # List all f
i
les.

• $ tree -L 1 # List one level deep.

• $ tree
-
f # Print full paths.

View Slide

• $ cd

• ~$ ln
-
s mydir mydir_link # Make a symbolic link.

• $ ls mydir_link

• $ ll mydir_link

• $ cp
-
r mydir mydir_backup # Copy recursively.

• $ ll mydir_backup

View Slide

• $ tar czf mydir.tar.gz mydir

• $ rmdir mydir
 
#
- >
failed
. . .
not empty.

• $ rmdir mydir/x/y/z
 
# OK!

• $ rm
-
r mydir/

• $ tar xzf mydir.tar.gz

• $ tree

• $ du
-
h
-
d 1 # du: disk usage

• $ df
-
h # df: disk free

View Slide

Describe one difference between cp
-
r and cp
-
a
Checkpoint:
• Read and search the manual!

• Hints:

• $ man cp # `/` to search; `n` to next; `h` to get help.

• $ cp
- -
help | grep
. . .

View Slide

Terminology
EN program process thread concurrency parallelism
TW 程式程序執⾏緒並⾏性平⾏性
CN 程序进程线程并发性并⾏性

View Slide

• $ vi test.txt # and .

• $ ps u | grep vi

• $ ps u | grep [v]i

• $ ps u | sed
-
n
-
e 1p
-
e /[v]i/p

• $ ps axu | grep [v]i |
 
awk '{ rss += $6 } END { printf "%.4g MB\n", rss/1024 }'

• (macOS)$ ps axu | grep [C]hrome |
 
awk '{ rss += $6 } END { printf "%.4g MB\n", rss/1024 }'

View Slide


• $ pgrep v #
- >

• $ pgrep vi #
- >

• $ pgrep test #
- >

• $ pgrep
-
f test #
- >

• $ pgrep
-
f test
-
l

• $ pgrep
- -
help | grep ' -[fl]'

• $ # kill

• $ pkill
-
f test # -SIGTERM

• $ fg


• $ pkill
-
f test -SIGKILL

• See also:

• Signal (IPC) – Wikipedia

View Slide

• $ cd

• ~$ python3
-
m http.server &> my.log
 
# to interrupt.

• Open http://localhost:8000/.

• Open another terminal:

• # iftop # to exit.

• $ tail
-
f ~/my.log

• $ less +F ~/my.log
 
# Try `
-
i` `/http` .

• $ watch tail -10 ~/my.log

View Slide

• The commands which are useful but not relevant to containers:

• $ sudo iotop

• Important system logs:

• $ sudo vi /var/log/syslog

• $ sudo vi /var/log/auth.log

View Slide

Checkpoint: Monitor your HTTP server by less
• Try to get three new logs.

• Hints:

• Click on your browser!

View Slide

Monitoring (cont.)
• $ python3
-
c 'print(1234)'

• $ strace python3
-
c 'print(1234)'

• $ sudo strace
-
p

• $ ps x

• $ pgrep bash

• $ tty

• $ pgrep bash
-
t pts/1
 
# pts: pseudo terminal slave

• $ lsof
-
p `pgrep bash
-
t pts/1`
 
# lsof: list open f
i
les

• $ lsof /usr/bin/bash

• $ man lsof | less
-
i +/^examples

• $ htop

• Select a process, and:

• (strace)

• (lsof)

View Slide

• $ sudo apt install sysstat
-
y # or,

• # apt install sysstat
-
y

• $ sar 1
 
# Report CPU stats per second.

• $ sar
 
# Report today's CPU stats.

• $ sar
-
f /var/log/sysstat/sa10

• $ sar
-
 
-
s 12
:
00
-
e 15
:
00

• $ man sar | grep 'Report ' -B 1

• $ sar
-
 
-
s 12
:
00
-
e 15
:
00
-
r
- -
human
 
# Report memory stats.

• See also:

• sysstat/sysstat – GitHub

• Amazon CloudWatch

• Grafana

View Slide

Container management
• OCI: Open Container Initiative

• Docker
fi
le: part of OCI spec.

• image: container image

• Why?
• Once you pack your app as a
container image, it can be run
anywhere seamlessly.

View Slide

• (host)$ docker container commit ubuntu
-
c ubuntu
-
cli
-
power
 
#
= = =
$ docker container commit
. . .

• $ docker image ls
 
#
= = =
$ docker images

View Slide

• $ docker image save ubuntu
-
cli
-
power >ubuntu
-
cli
-
power.tar

• $ docker image rm ubuntu
-
cli
-
power
 
#
= = =
$ docker rmi ubuntu
-
cli
-
power


• $ docker image load -
cli
-
power.tar

• $ # docker image (save|load)
= = =
docker image (save|load)


View Slide

• $ docker container create
- -
name ls
-
expert ubuntu
-
cli
-
power man ls

• $ docker container ls
-
a #
= = =
$ docker ps
-
a

• $ docker container start ls
-
expert

• $ docker container ls
-
a #
- >
Exited (0) 4 seconds ago

• $ docker container start
-
a ls
-
expert # Start and attach.

• $ docker container rm ls
-
expert

• $ # docker container (create|start|stop|rm)
= = =  
docker container (create|start|stop|rm)

View Slide

• $ docker container cp ubuntu
-
c:/root/cmds.tsv ~/cmds.tsv
 
#
= = =
$ docker cp
. . .

• $ docker container cp ~/cmds.tsv ubuntu
-
c:/root/cmds_2.tsv

• $ docker container run
-
dit
-
v ~/container:/root
- -
name volume
-
c ubuntu
-
cli
-
power
 
#
= = =
$ docker run
. . .

View Slide

Work on a remote machine

View Slide

Run programs after disconnect
• $ tmux # tmux: terminal multiplexer

• (tmux)$ while true; do date; sleep 1; done # and then .

• $ tmux ls # ls: the alias of list
-
sessions.

• $ tmux a # a: attach

• (tmux) > # List key bindings; to exit.

• (tmux)

• $ tmux ls

View Slide

• The way I start the server of JupyterLab:

• $ tmux new
-
s jupyter
-
lab
-
d 'cd && jupyter lab
- -
no
-
browser'

View Slide

The local key pair
• (local)$ ll ~/.ssh/id_rsa
*

• If nothing there:

• (local)$ ssh
-
keygen

• (local)$ cat ~/.ssh/id_rsa.pub # pub: public key

• id_rsa is the private key which proves who you are.

• Do not share id_rsa – the private key – to anyone.

• Simply generate a new pair when you have a little doubt about the old pair.

View Slide

Add a user in the remote host
• (remote)# new_user=

• (remote)# useradd
-
m
-
s /bin/bash $new_user

• (remote)# su $new_user

• $ mkdir ~/.ssh
 
$ cat <~/.ssh/authorized_keys
 
>
 
> EOS
 
$ exit

• (remote)# service ssh start

View Slide

Connect to a remote host
• (local)$ ssh ssh://@localhost:2222
 
#
= = =
$ ssh @localhost
-
p 2222
 
# $ ssh # if port is 22 and usernames are the same.

• If there is a warning and you're sure you changed something in the remote:

• $ sed d ~/.ssh/known_hosts
-
i

View Slide

ssh -A: Allow the remote to use the local key pair
• (local) $ ssh ssh://@localhost:2222

• (remote)$ ssh ubuntu
-
c
 
#
. . .
password:

• (local) $ ssh ssh://@localhost:2222 -A

• (remote)$ ssh ubuntu
-
c # OK

• The ubuntu-c works since the docker run --network enables the DNS
resolution between containers.

• See also: Consul, for the similar functionality without Docker.

View Slide

ssh -J: Jump to a host in a private network
• (local)$ ssh ubuntu
-
c
 
#
- >
Could not resolve hostname ubuntu
-
c

• (local)$ ssh -J ssh://@localhost:2222 ubuntu
-
c

• -J is securer than -A; use -J when applicable.

View Slide

Transfer files between local and remote
• (remote)$ cp cmds.tsv ~

• (local) $ rsync
-
e 'ssh ssh://@localhost:2222' :~/cmds.tsv ~
 
# $ rsync @> :
~/cmds.tsv ~ # if port is 22.

• (local) $ rsync
-
e 'ssh ssh://@localhost:2222' ~/cmds.tsv :~/cmds_2.tsv

• Deprecating scp – LWN.net, but scp is still a common way nowadays:

• (local)$ scp scp://@localhost:2222/~/cmds.tsv ~
 
# $ scp @> :
~/cmds.tsv ~ # if port is 22.

• (local)$ scp ~/cmds.tsv scp://@localhost:2222/~/cmds_2.tsv

View Slide

Mount a remote directory
• (local)$ sudo apt install sshfs # or macFUSE for macOS.

• (local)$ mkdir ~/sshfs_container

• (local)$ sshfs @localhost:/home/
-
p 2222 ~/sshfs_container

• (local)$ ls ~/sshfs_container

• (local)$ umount ~/sshfs_container

View Slide

ssh -L: Forward a local port to a remote port
• As known as “SSH tunneling”.

• (remote)$ python3
-
m http.server 1234 # to interrupt.

• (local)$ ssh ssh://@localhost:2222 -L 4321:ubuntu
-
c:1234 -Nf


• localhost:4321  
→ localhost:2222 → ubuntu-c:22  
→ ubuntu-c:1234 → http.server 1234

• (local)$ pgrep
-
f 4321

• (local)$ # pkill
-
f 4321

View Slide

Access the HTTP server by http://localhost:8765/
Checkpoint:
• Open http://localhost:8765/ and there should be a page.

• Hints:

• Use ssh -L.

View Slide

ssh -D: Start a proxy server
• (local)$ ssh ssh://@localhost:2222 -D 1080 -Nf

• (remote)# iftop

• Con
fi
gure the system proxy or Proxy SwitchyOmega for Chrome.

• (remote)# python3
-
m http.server 80

• Open http://ubuntu-c/.

• For accessing the services in a private network by private DNS and standard ports!

• If the server has another public IP, try Google “my ip”.

• $ man ssh | grep ' -[LDNf]' -A 1

View Slide

System proxy for macOS

View Slide

Proxy SwitchyOmega for Chrome

View Slide

View Slide

• See also:

• A simple ~/.ssh/con
fi
g for macOS

• A simple Bash con
fi
g for both macOS and Linux

View Slide

Developing on CLI

View Slide

Edit a config
• $ cp ~/.bashrc ~/backup_bashrc

• $ echo 'alias bat=batcat' >>~/.bashrc

• Re-enter the shell to take e
ff
ect.

• $ grep batcat ~/.bashrc
-
n

• $ sed d ~/.bashrc
-
i

-
n

View Slide

• $ vi ~/.bashrc

• (vim) #

• (vim) i # or `a`

• (vim) alias bat=batcat # literally

• (vim) # backs to Normal

• (vim) :x

-
n

• $ vi ~/.bashrc

• (vim) /cat

• (vim) n
. . .

• (vim) dd

• (vim) :x

-
n

View Slide

The mini-cheatsheet of Vim
1. / n

• Search; Next.

2. i :x

• Go Insert; Back to Normal; Write and quit.

3. dd

• Delete a line.

• It's all you need to know now. 
Of course, the more you know, the more e
ff
i
ciency you'll get. 💪

View Slide

Spell checking
• $ echo 'Oops. Does it spll like this?'

• (vim) :set spell

• (vim) /sp

• (vim) z= # List spelling suggestions.

• (vim) 1

• (vim) :x

View Slide

• See also:

• $ vimtutor # Learn Vim in 30 mins!

• (vim) :help dd, for example.

• Vim Keystrokes – Mosky’s Notes

• Vim’s Search Tips – Mosky’s Notes

• Vim’s Replace Tips – Mosky’s Notes

• Mosky's Vim Con
fi
gs

View Slide

Checkpoint: Add your bat
• (host)$ docker exec
-
it ubuntu
-
c bash

• ubuntu
-
c# bat ~/.bashrc # It should work immediately.

• Hints:

• You may use echo to add.

• Use grep to con
fi
rm.

• Re-enter the shell to take e
ff
ect.

View Slide

Developing
Slak: Collect data from Slack like a pro. ⚡
1. System Tools: the common tools which are
supposed to be installed in the system.

• git: the version control system.

• pip: the Python package manager.

• pipenv: one of Python package environment
managers.

• The Docker (containers) provides OS-level
isolation; Pipenv provides Python
interpreter and package–level isolation.

2. Project Tools: the tools of this project, good to
be managed within the project by, for example,
pipenv.

• ipdb: a simple yet beautiful debugger.

• pytest: the testing framework.

3. Personal Tools: the tools of a developer, may be
installed in one's system.

• flake8: the static analyzer (linter).

• mypy: the static type checker.

• black: one of the formatters.

View Slide

• $ sudo apt install pip
-
y # or, # System Tool

• # apt install pip
-
y

• $ pip install pipenv # System Tool

• $ cd

• ~$ git clone
- -
depth 1 https://github.com/moskytw/slak.git

View Slide

• ~$ cd slak

• ~/slak$ bat Pipf
i
le

• $ pipenv sync
- -
dev

• $ slak

• $ python slak.py

• $ for cmd in python python3 slak;
do type $cmd; done

• $ pipenv shell

• (slak)$ slak

• $ python slak.py

• $ for cmd in python python3 slak;
do type $cmd; done

View Slide

• (slak)$ python
-
m ipdb slak.py # Project Tool, `q` to exit.

• $ pytest # Project Tool

• $ pytest
- -
cov

• $ pytest
- -
cov
- -
cov
-
report html

• $ cd htmlcov

• ~/slak/htmlcov$ python
-
m http.server # to interrupt.


• $ exit # or to exit the shell of pipenv.

View Slide

• (my mac)$ for cmd in flake8 mypy black xxx; do type $cmd; done
 
flake8 is hashed (/usr/local/bin/flake8)
 
mypy is aliased to `\mypy
- -
ignore
-
missing
-
imports'
 
black is aliased to `black
- -
line
-
length 79
- -
skip
-
string
-
normalization'
 
-
bash: type: xxx: not found

• $ for cmd in flake8 mypy black xxx; do type $cmd >/dev/null; done
 
-
bash: type: xxx: not found

View Slide

• $ cd ~/slak

• ~/slak$ flake8 slak.py

• $ mypy slak.py

• $ black slak.py

View Slide

• The di
ff
erent languages have di
ff
erent toolsets; however, the concepts and
the categories are similar, e.g.:

• $ vi /etc/nginx/sites
-
enabled/your
-
site.conf

• $ nginx
-
tc /etc/nginx/sites
-
enabled/your
-
site.conf
 
# Test the conf
i
g f
i
le.

View Slide

• See also:

• git - the simple guide: “no deep shit ;)”

• Git Documentation

• Cheat Sheets

• Pro Git, the book

• Reference Manual

• Mosky’s Pipenv Notes

View Slide

In a large project
• It's all about locate the path and line number:

• f
i
nd/fd: locate by path keywords.

• grep/rg: locate by content keywords.

• ctags: locate by pre-built “tags”.

• Language Server: a server keeps analyzing your code in background.

View Slide

• $ cd /usr/lib/python3.10

• /usr/lib/python3.10$ f
i
nd
-
name '
*
server
*
'

• $ sudo apt install fd
-
f
i
nd
-
y # or,

• # apt install fd
-
f
i
nd
-
y

• $ alias fd=fdf
i
nd

• $ fd server # fd: f
i
nd

View Slide

• $ cd /usr/lib/python3.10

• /usr/lib/python3.10$ grep
-
r 'Serving HTTP'
-
n

• $ sudo apt install ripgrep
-
y # or,

• # apt install ripgrep
-
y

• $ rg 'Serving HTTP' # rg: ripgrep

View Slide

• $ f
i
nd
-
name 'server.py'
-
exec grep 'Serving HTTP'
-
n {} +

• $ fd '^server.py$' -X rg 'Serving HTTP' {}

• $ man f
i
nd | less +'/
-
exec '

• $ man fd | less +'/ -[Xx]'

View Slide

• Ctags & Language Server:

• $ sudo apt install universal
-
ctags
 
$ ctags -R
 
$ vim +'help tags | only'

• A 10 MB–level solution.

• neoclide/coc.nvim – GitHub for the Language Server solution.

• A GB-level solution.

View Slide

The end
• But hope it's the start of your
journey of getting the CLI power.

• Having fun is still the key,  
as always.

• May the man guide you!

View Slide

Image Credits
• “The Power” 
https://www.facebook.com/nixcraft/photos/
meme-based-upon-iamnotanartist_-comic-
for-linuxunix-users/3443131449033542/

• “The Terminal” 
https://en.wikipedia.org/wiki/
Computer_terminal#/media/
File:DEC_VT100_terminal_transparent.png

• “What Is a Container?” 
https://www.docker.com/resources/what-
container/

• “The Cat” 
https://www.facebook.com/nixcraft/posts/
linux-explained-for-new-users-including-cat-
command-explained-httpswwwcybercitib/
3034767979869893/

• “The Dragon” 
https://unsplash.com/photos/F6920BvzrZE

• “The Miniatures” 
https://unsplash.com/photos/Q4Honp3Pyqs

• “The Banshee Appears” 
https://en.wikipedia.org/wiki/Banshee#/
media/File:The_Banshee_Appears_(1862).jpg

• “The Earth” 
https://unsplash.com/photos/Q1p7bh3SHj8

View Slide

Get Power From Command Line

Get Power From Command Line

Mosky Liu

More Decks by Mosky Liu

Other Decks in Programming

Featured

Transcript