Shell Scripting Lessons from a year administering remote machines 

Basic Concepts 

Shell Anatomy 101 • time php test.php 2>&1 | grep $NEEDLE > results.txt Shell Builtin Executables Arguments Redirection Pipe Environment Variable output builtin: a regular command, except provided by your shell instead of an executable executables: a file in your path marked as executable arguments: you all know what this means Redirection: Swapping around the standard streams 

Process Pipes, a review Process stdin stdout stderr 2 1 

2>&1 Process stdin stdout stderr 2 1 

Shell Anatomy 101 • time php test.php 2>&1 | grep $NEEDLE > results.txt Shell Builtin Executables Arguments Redirection Pipe Environment Variable output Pipe: Special case of redirection, connects stdout to stdin of the other side Environment variable: a variable that gets the value substituted by the shell 

Tools to Know About 

man • read the manual page for a given command • $ man grep info also exists, but that means using emacs-like bindings… ugh! 

cat • Puts the contents of a file onto stdout • $ cat “hello.txt”
 Hello, world 

head/tail • Puts the beginning or end of a file on stdout • great for dealing with logs, especially with -f • $ tail -f /var/log/system.log
 Hello, world 

ps • list information about processes • $ ps axu | grep grep
 bnicholas 4521 0.0 0.0 2453264 692 s000 S+ 10:26PM 0:00.00 grep grep 

kill/killall • send a signal to processes by pid or name • $ kill -SIGUSR1 2048 • $ killall perl 

sort • Takes lines from stdin and sorts them on stdout • $ cat file.txt | sort • $ cat numbers.txt | sort -n 

uniq • Removes duplicate lines from stdin to stdout • $ cat file.txt | sort | uniq • $ cat ~/history | sort | uniq -c | sort -n 

grep • Search for a regular expression • can be used on stdin or files • $ cat /var/log/product.log | grep -i “error” 

tr • “trim” text, replacing characters with others • $ cat sentence.txt | tr ‘ ‘ ‘\n’
 My
 name
 is
 ben. 

sed • stream editor: a mini text editor • Similar to vim find+replace syntax, but so much more too • $ echo “hello world!” | sed -e “s/ w.*d/bn.u/“
 hello bn.u! 

find • A file finder with extra • $ find . -name “*.orig” -print - delete • $ find . -type f —mtime -1 -print This will find all the files ending in .orig (merge files from git), print them out, and then delete them exec also allows for running any command selectors let you pick files very concisely 

awk • a small programming language for dealing with columns • ps aux | grep grep | awk ‘{print $2}’
 6845 So much more complicated and powerful than I have time for I’ve only scratched the surface, but it’s really helpful for certain things this example prints the pid of any grep processes 

Control Flow 

conditionals • if [ $VAR = “foo” ]
 then
 echo “bar”
 else
 echo “baz”
 fi explain test, and how dumb it can be sometimes -eq for integer equality, -gt -lt -ne… 

Better conditionals • Use && and || when possible • cat a.txt | grep “hello” && echo “world” 

For loops • for FILE in `ls`
 do
 cat $FILE
 done 

SSH The magic glue that holds things together 

Key-based auth • More secure and convenient than passwords • ssh-copy-id @ • Last time you’ll need to type a password • Also applies to git, scp, sftp, and others 

~/.ssh/config • Set up usernames across computers • Keep connections alive longer • Set up tunnels tunnels: local ports actually go through to the remote server good things like accessing servers that are firewall’d out 

Great Tricks/Tools 

autocomplete stop typing and hit tab. 95% of the time it’s gonna do what you want. trust your shell. 

fish If you’re still using bash, stop it, install fish, make all of your config 100000x easier autocomplete by default, web config, syntax highlighting and checking, sane scripting, small config files (instead of one monolithic _THING_), man page completion 

ack http://beyondgrep.com it’s faster grep that ignores the files you don’t want. Simple as that. 

tmux If you’re working on a remote machine, this will save you so many times (worked on a laptop, connection on and off, long running tasks that never died) 

vim This is going to start a war, I know, but knowing a command line editor helps a lot vim is the one of the most powerful, nano, pico, emacs all work, but why? 

dsh Elliot and James swear by it, I don’t know if it’s worth the config time, but it depends on how you work 

Some example problems