Crash Course on Unix Tools and Scripting Stephen McDowell September 12th, 2016 Cornell University 

Table of contents 1. UNIX vs Windows 2. Basic Navigational Commands 3. File and Folder Manipulation 4. Flags & Command Clarifaction 5. Working with Files 6. Assorted Commands 7. Chaining Commands 1 

Notation Commands will be shown on slides using teletype text. Introducing new commands some-command [opt1] [opt2] New commands will be introduced in block boxes like this one, sometimes including common flags or warnings. To execute some-command, just type its name into the shell and press return / enter. When displaying commands in code blocks, the >>> sequence indicates a new command being entered. >>> first-command output of first-command (where applicable) >>> second-command output of second-command (where applicable) 2 

UNIX vs Windows 

The Unix Filesystem • Unlike Windows, UNIX has a single global ”root” directory (instead of a root directory for each disk or volume). • The root directory is just / • All files and directories are case sensitive. • hello.txt != hElLo.TxT • Directories are separated by / instead of \ in Unix. • UNIX: /home/sven/lemurs • Windows: E:\Documents\lemurs • Hidden files and folders begin with a ”.” • e.g. .git/ (a hidden directory). • Example: my home directory. • Of course, there are many more differences. This should be enough to get you started. 3 

Personal Files Your personal files are in your home directory (and its subdirectories), which is usually* located at Linux Mac /home/username /Users/username There is also a built-in alias for it: ~ For example, the Desktop for the user sven is located at Linux Mac /home/sven/Desktop /Users/sven/Desktop ~/Desktop ~/Desktop 4 

Basic Navigational Commands 

Where am I? Most shells default to using the current path in their prompt. If not, you can find out where you are with Print working directory pwd - Prints the ”full” path of the current directory. - Handy on minimalist systems when you get lost. - Can be used in scripts. Note that if you have a path with symbolic links, you need to use the -P flag. 5 

What’s here? Knowing where you are is useful, but understanding what else is there is too... The list command ls - Lists directory contents (including subdirectories). - Works like the dir command in Windows. - The -l flag lists detailed file / directory information (we’ll learn more about flags later). - Use -a to list hidden files. 6 

Ok lets go! Moving around is as easy as Changing directories cd [directory name] - Changes directory to [directory name]. - If not given a destination defaults to the user’s home directory. - You can specify both absolute and relative paths. - If you do not specify a directory, the ~ (home) directory is assumed. • Absolute paths start at / (the global root). • e.g. cd /home/sven/Desktop • Relative paths start at the current directory. • e.g. cd Desktop, if you were already at /home/sven 7 

Relative Path Shortcuts Shortcuts ~ current user’s home directory . the current directory (this is actually useful...) .. the parent directory of the current directory - for cd command, return to previous working directory An example: starting in /usr/local/src >>> cd # now at /home/sven >>> cd - # now at /usr/local/src >>> cd .. # now at /usr/local 8 

File and Folder Manipulation 

Creating a new File The easiest way to create an empty file is using touch touch [flags] - Adjusts the timestamp of the specified file. - With no flags uses the current date and time. - If the file does not exist, touch creates it. File extensions (.txt, .c, .py, etc) often don’t matter in Unix. Using touch to create a file results in a blank plain-text file (so you don’t necessarily have to hadd .txt to it). 9 

Creating a new Directory No magic here... Make directory mkdir [flags] <...> - Can use relative or absolute paths. - a.k.a. you are not restricted to making directories in the current directory only. - Need to specify at least one directory name. - Can specify multiple, separated by spaces. - The -p flag is commonly used in scripts: create parent directories if needed. - The mkdir command will fail if a parent directory does not exist and -p not specified. - By default, the mkdir command fails if you give it a directory that already exists. 10 

File Deletion Warning: once you delete a file (from the command line) there is no easy way to recover the file. Remove File rm [flags] - Removes the file . - Remove multiple files with wildcards (more on this later). - Remove every file in the current directory: rm * - Remove every .jpg file in the current directory: rm *.jpg - Prompt before deletion: rm -i 11 

Deleting Directories By default, rm cannot remove directories. Instead we use... Remove directory rmdir [flags] - Removes an empty directory. - Throws an error if the directory is not empty. - You are encouraged to use this command: failing on non-empty can and will save you! To delete a directory and all its subdirectories, we pass rm the flag -r (for recursive), e.g. rm -r /home/sven/oldstuff 12 

Copy That! Copy cp [flags] - Copies from one location to another. - To copy multiple files, use wildcards (such as *). - To copy a complete directory: cp -r 13 

Move it! Unlike the cp command, the move command automatically recurses for directories. Move mv [flags] - Moves a file or directory from one place to another. - Also used for renaming, just move from to . - E.g. mv badFolderName correctName 14 

Recap ls list directory contents cd change directory pwd print working directory rm remove file rmdir remove directory cp copy file mv move file 15 

Flags & Command Clarifaction 

Flags and Options • Most commands take flags and optional arguments. • These come in two general forms: • Switches (no argument required), and • Argument specifiers (for lack of a better name). • When specifying flags for a given command, keep in mind: • Flags modify the behavior of the command / how it executes. • Some flags take precedence over others, and some flags you specify can implicitly pass additional flags to the command. 16 

Flags and Options: Formats A flag that is • One letter is specified with a single dash (-a). • More than one letter is specified with two dashes (--all). • The reason is because of how switches can be combined (next page). 17 

Flags and Options: Switches Switches take no arguments, and can be specified in a couple of different ways. Switches are usually one letter, and multiple letter switches usually have a one letter alias (the ls command has --all aliased to -a). • One option: • ls -a • ls --all • Two options: • ls -l -Q • Two options: • ls -lQ • Applied from left to right: • rm -fi ⇒ prompts • rm -if ⇒ does not prompt 18 

Flags and Options: Argument Specifiers These flags expect an input, and you will encounter two general kinds. • The --argument="value" format, where the = and quotes are needed if value is more than one word. • Yes: ls --hide="Desktop" ~/ • Yes: ls --hide=Desktop ~/ • one word, no quotes necessary • No: ls --hide = "Desktop" ~/ • spaces by the = will be misinterpreted (it used = as the hide value...) • The --argument value format, with a space after the argument. Quote rules same as above. • ls --hide "Desktop" ~/ • ls --hide Desktop ~/ Note: The example I gave you was using the same --hide in both formats, but not all commands will accept both. Advise --argument="value" format for higher success rates. 19 

Flags and Options: Conventions, Warnings Generally, you should always specify the flags before the arguments. In this example, the flag is -l and ~/Desktop/ is the argument. • ls -l ~/Desktop/ and ls ~/Desktop/ -l both work • there exist scenarios in which flags after arguments do not get processed There is a special sequence -- that signals the end of the options. I will use another flag to demonstrate: • ls -l -a ~/Desktop/ ⇒ executes as expected • ls -l -- -a ~/Desktop/ ⇒ only used -l • ”ls: cannot access -a: No such file or directory” • -a was treated as an argument, and there is no -a directory (for me) 20 

Flags and Options: Conventions, Warnings (cont) The special sequence -- that signals the end of the options is often most useful if you need to do something special. Suppose I wanted to make the folder -a on my Desktop. >>> cd ~/Desktop # for demonstration purpose >>> mkdir -a # fails: invalid option -- 'a' >>> mkdir -- -a # success! (ls to confirm) >>> rmdir -a # fails: invalid option -- 'a' >>> rmdir -- -a # success! (ls to confirm) This trick can be useful in many scenarios, and generally arises when you need to work with special characters of some sort. 21 

Your new best friend How do I know what the flags / options for all of these commands are? The manual command man - Loads the manual (manpage) for the specified command. - Unlike google, manpages are system-specific. - Usually very comprehensive. Sometimes too comprehensive. - Type / to search. - The n key jumps through the search results. Search example on next page if that was confusing. Intended for side-by-side follow-along. 22 

Man oh man >>> man man # you now have the manual loaded >>> /useful # type /useful, then hit enter ############# [first result highlighted] >>> n # followed by enter ############# [next result highlighted] Note that there are subtle differences between options on different systems. For example, ls -B: • BSD/OSX: Force printing of non-printable characters in file names as \xxx, where xxx is the numeric value of the character in octal. • Fedora, Ubuntu: do not list implied entries ending with ~ • In these OS’s, files ending with ~ are temporary backup files that certain programs (e.g. some text-editors) 23 

Working with Files 

Reading Files Without Opening Concatenate cat - Prints the contents of the file to the terminal window cat - Prints file1 first, then file2. more more - Scroll through one page at a time. - Program exits when end is reached. less less - Scroll pages or lines (mouse wheel, space bar, and arrows). - Program does not exit when end is reached. 24 

Beginning and End Long files can be a pain with the previous tools. Head and Tail of Input head -[numlines] tail -[numlines] - Prints the first / last numlines of the file. - Default is 10 lines. 25 

Not Really a File...YET You can talk to yourself in the terminal too! Echo echo - Prints the input string to the standard output (the terminal). - We will soon learn how to use echo to put things into files, append to files, etc. 26 

Assorted Commands 

Before we can Chain... ...we need some more interesting tools to chain together! 27 

Counting Word Count wc [options] -l: count the number of lines. -w: count the number of words. -m: count the number of characters. -c: count the number of bytes. Great for things like: • revelling in the number of lines you have programmed. • analyzing the verbosity of your personal statement. • showing people how cool you are. 28 

Sorting Sort sort [options] - Default: sort by the ASCII code (roughly alphabetical) for the whole line. - Use -r to reverse the order. - Use -n to sort by numerical order. - Use -u to remove duplicates. >>> cat peeps.txt Manson, Charles Bundy, Ted Bundy, Jed Nevs, Sven Nevs, Sven >>> sort -r peeps.txt Nevs, Sven Nevs, Sven Manson, Charles Bundy, Ted Bundy, Jed >>> sort -ru peeps.txt Nevs, Sven Manson, Charles Bundy, Ted Bundy, Jed # only 1 Nevs, Sven 29 

Advanced Sorting • The sort command is quite powerful, for example you can do: >>> sort -n -k 2 -t "," • Sorts the file numerically by using the second column, separating by a comma as the delimiter instead of a space. • Read the man page! >>> cat numbers.txt 02,there 04,how 01,hi 06,you 03,bob 05,are >>> sort -n -k 2 -t "," numbers.txt 01,hi 02,there 03,bob 04,how 05,are 06,you 30 

Special Snowflakes Unique uniq [options] - No flags: discards all but one of successive identical lines. - Use -c to prints the number of successive identical lines next to each line. 31 

Search and Replace Translate tr [options] [set2] - Translate or delete characters. - Sets are strings of characters. - By default, searches for strings matching set1 and replaces them with set2. - You can use POSIX and custom-defined sets (we’ll get there soon!). • The tr command only works with streams. • Examples to come after we learn about chaining commands in the next section. 32 

Chaining Commands 

Your Environment and Variables • There are various environment variables defined in your environment. They are almost always all capital letters. • You obtain their value by dereferencing them with a $. >>> echo $PWD # present working directory >>> echo $OLDPWD # print previous working directory >>> printenv # print all environment variables • When you execute commands, they have something called an ”exit code”. • The exit code of the last command executed is stored in the $? environment variable. 33 

What is Defined? • The environment: • env: displays all environment variables. • unsetenv : remove an environment variable. • The local variables: • set: displays all shell / local variables. • unset : remove a shell variable. • We’ll cover these a little more when we talk about customizing your terminal shell. 34 

Exit Codes • There are various exit codes, here are a few examples: >>> super_awesome_command bash: super_awesome_command: command not found... >>> echo $? 127 >>> echo "What is the exit code we want?" >>> echo $? 0 • The success code we want is actually 0. Refer to [1] for some more examples. • Remember that cat /dev/urandom trickery? You will have to ctrl+c to kill it, what would the exit code be? 35 

Executing Multiple Commands in a Row With exit codes, we can define some simple rules to chain commands together: • Always execute: >>> cmd1; cmd2 # exec cmd1 first, then cmd2 • Execute conditioned upon exit code: >>> cmd1 && cmd2 # exec cmd2 only if cmd1 returned 0 >>> cmd1 || cmd2 # exec cmd2 only if cmd1 returned NOT 0 • Kind of backwards, in terms of what means continue for and, but that was likely easier to implement since there is only one 0 and many not 0’s. 36 

Piping Commands Bash scripting is all about combining simple commands together to do more powerful things. This is accomplished using the ”pipe” character. Piping | - Passes the output from command1 to be the input of command2. - Works for heaps of programs that take input and provide output to the terminal. 37 

Some Piping Examples Piping along... >>> ls -al /bin | less - Allows you to scroll through the long list of programs in /bin >>> history | tail -20 | head -10 - Displays the 10th - 19th previous commands from the previous session. >>> echo * | tr ' ' '\n' - Replaces all spaces characters with new lines. - Execute just echo * to see the difference. 38 

Redirection To redirect input / output streams, you can use one of >, >>, <, or <<. • To redirect standard output, use the > operator. • command > file • To redirect standard input, use the < operator. • command < file • To redirect standard error, use the > operator and specify the stream number 2. • command 2> file • Combine streams together by using 2>&1 syntax. • This says: send standard error to where standard output is going. • Useful for debugging / catching error messages... • ...or ignoring them (you will often see that sent to /dev/null). 39 

Redirection Example • Bash processes I/O redirection from left to right, allowing us to do fun things like this: Magic tr -cd '0-9' < test1.txt > test2.txt - Deletes everything but the numbers from test1.txt, then store them in test2.txt. - CAUTION: do not ever use the same file as output that was input. - Example: tr -cd '0-9' < original.txt > original.txt - You will lose all your data, you cannot read and write this way. • Piping and Redirection are quite sophisticated, please refer to the Wikipedia page in [2]. 40 

References I [1] T. L. D. Project. Exit codes with special meanings. http://tldp.org/LDP/abs/html/exitcodes.html. [2] Wikipedia. Redirection (computing). https://en.wikipedia.org/wiki/Redirection_ %28computing%29. 41