Share
Embed
Crash Bash
Presentation courtesy of Stephen McDowell.
A lightning speed crash course on bash / functioning in a Unix terminal. An amalgamation of 3-4 lectures of material from http://cs2043-sp16.github.io/
Presented at SSW: https://cornell-ssw.github.io/meetings/2016-09-12
CUSSW Hosted
September 12, 2016
Transcript
49 slides
-
Crash Course on Unix Tools and Scripting Stephen McDowell September
12th, 2016 Cornell University -
Table of contents 1. UNIX vs Windows 2. Basic Navigational
Commands 3. File and Folder Manipulation 4. Flags & Command Clarifaction 5. Working with Files 6. Assorted Commands 7. Chaining Commands 1 -
Notation Commands will be shown on slides using teletype text.
Introducing new commands some-command [opt1] [opt2] New commands will be introduced in block boxes like this one, sometimes including common flags or warnings. To execute some-command, just type its name into the shell and press return / enter. When displaying commands in code blocks, the >>> sequence indicates a new command being entered. >>> first-command output of first-command (where applicable) >>> second-command output of second-command (where applicable) 2 -
The Unix Filesystem • Unlike Windows, UNIX has a single
global ”root” directory (instead of a root directory for each disk or volume). • The root directory is just / • All files and directories are case sensitive. • hello.txt != hElLo.TxT • Directories are separated by / instead of \ in Unix. • UNIX: /home/sven/lemurs • Windows: E:\Documents\lemurs • Hidden files and folders begin with a ”.” • e.g. .git/ (a hidden directory). • Example: my home directory. • Of course, there are many more differences. This should be enough to get you started. 3 -
Personal Files Your personal files are in your home directory
(and its subdirectories), which is usually* located at Linux Mac /home/username /Users/username There is also a built-in alias for it: ~ For example, the Desktop for the user sven is located at Linux Mac /home/sven/Desktop /Users/sven/Desktop ~/Desktop ~/Desktop 4 -
Where am I? Most shells default to using the current
path in their prompt. If not, you can find out where you are with Print working directory pwd - Prints the ”full” path of the current directory. - Handy on minimalist systems when you get lost. - Can be used in scripts. Note that if you have a path with symbolic links, you need to use the -P flag. 5 -
What’s here? Knowing where you are is useful, but understanding
what else is there is too... The list command ls - Lists directory contents (including subdirectories). - Works like the dir command in Windows. - The -l flag lists detailed file / directory information (we’ll learn more about flags later). - Use -a to list hidden files. 6 -
Ok lets go! Moving around is as easy as Changing
directories cd [directory name] - Changes directory to [directory name]. - If not given a destination defaults to the user’s home directory. - You can specify both absolute and relative paths. - If you do not specify a directory, the ~ (home) directory is assumed. • Absolute paths start at / (the global root). • e.g. cd /home/sven/Desktop • Relative paths start at the current directory. • e.g. cd Desktop, if you were already at /home/sven 7 -
Relative Path Shortcuts Shortcuts ~ current user’s home directory .
the current directory (this is actually useful...) .. the parent directory of the current directory - for cd command, return to previous working directory An example: starting in /usr/local/src >>> cd # now at /home/sven >>> cd - # now at /usr/local/src >>> cd .. # now at /usr/local 8 -
Creating a new File The easiest way to create an
empty file is using touch touch [flags] <file> - Adjusts the timestamp of the specified file. - With no flags uses the current date and time. - If the file does not exist, touch creates it. File extensions (.txt, .c, .py, etc) often don’t matter in Unix. Using touch to create a file results in a blank plain-text file (so you don’t necessarily have to hadd .txt to it). 9 -
Creating a new Directory No magic here... Make directory mkdir
[flags] <directory1> <directory2> <...> - Can use relative or absolute paths. - a.k.a. you are not restricted to making directories in the current directory only. - Need to specify at least one directory name. - Can specify multiple, separated by spaces. - The -p flag is commonly used in scripts: create parent directories if needed. - The mkdir command will fail if a parent directory does not exist and -p not specified. - By default, the mkdir command fails if you give it a directory that already exists. 10 -
File Deletion Warning: once you delete a file (from the
command line) there is no easy way to recover the file. Remove File rm [flags] <filename> - Removes the file <filename>. - Remove multiple files with wildcards (more on this later). - Remove every file in the current directory: rm * - Remove every .jpg file in the current directory: rm *.jpg - Prompt before deletion: rm -i <filename> 11 -
Deleting Directories By default, rm cannot remove directories. Instead we
use... Remove directory rmdir [flags] <directory> - Removes an empty directory. - Throws an error if the directory is not empty. - You are encouraged to use this command: failing on non-empty can and will save you! To delete a directory and all its subdirectories, we pass rm the flag -r (for recursive), e.g. rm -r /home/sven/oldstuff 12 -
Copy That! Copy cp [flags] <file> <destination> - Copies from
one location to another. - To copy multiple files, use wildcards (such as *). - To copy a complete directory: cp -r <src> <dest> 13 -
Move it! Unlike the cp command, the move command automatically
recurses for directories. Move mv [flags] <source> <destination> - Moves a file or directory from one place to another. - Also used for renaming, just move from <oldname> to <newname>. - E.g. mv badFolderName correctName 14 -
Recap ls list directory contents cd change directory pwd print
working directory rm remove file rmdir remove directory cp copy file mv move file 15 -
Flags and Options • Most commands take flags and optional
arguments. • These come in two general forms: • Switches (no argument required), and • Argument specifiers (for lack of a better name). • When specifying flags for a given command, keep in mind: • Flags modify the behavior of the command / how it executes. • Some flags take precedence over others, and some flags you specify can implicitly pass additional flags to the command. 16 -
Flags and Options: Formats A flag that is • One
letter is specified with a single dash (-a). • More than one letter is specified with two dashes (--all). • The reason is because of how switches can be combined (next page). 17 -
Flags and Options: Switches Switches take no arguments, and can
be specified in a couple of different ways. Switches are usually one letter, and multiple letter switches usually have a one letter alias (the ls command has --all aliased to -a). • One option: • ls -a • ls --all • Two options: • ls -l -Q • Two options: • ls -lQ • Applied from left to right: • rm -fi <file> ⇒ prompts • rm -if <file> ⇒ does not prompt 18 -
Flags and Options: Argument Specifiers These flags expect an input,
and you will encounter two general kinds. • The --argument="value" format, where the = and quotes are needed if value is more than one word. • Yes: ls --hide="Desktop" ~/ • Yes: ls --hide=Desktop ~/ • one word, no quotes necessary • No: ls --hide = "Desktop" ~/ • spaces by the = will be misinterpreted (it used = as the hide value...) • The --argument value format, with a space after the argument. Quote rules same as above. • ls --hide "Desktop" ~/ • ls --hide Desktop ~/ Note: The example I gave you was using the same --hide in both formats, but not all commands will accept both. Advise --argument="value" format for higher success rates. 19 -
Flags and Options: Conventions, Warnings Generally, you should always specify
the flags before the arguments. In this example, the flag is -l and ~/Desktop/ is the argument. • ls -l ~/Desktop/ and ls ~/Desktop/ -l both work • there exist scenarios in which flags after arguments do not get processed There is a special sequence -- that signals the end of the options. I will use another flag to demonstrate: • ls -l -a ~/Desktop/ ⇒ executes as expected • ls -l -- -a ~/Desktop/ ⇒ only used -l • ”ls: cannot access -a: No such file or directory” • -a was treated as an argument, and there is no -a directory (for me) 20 -
Flags and Options: Conventions, Warnings (cont) The special sequence --
that signals the end of the options is often most useful if you need to do something special. Suppose I wanted to make the folder -a on my Desktop. >>> cd ~/Desktop # for demonstration purpose >>> mkdir -a # fails: invalid option -- 'a' >>> mkdir -- -a # success! (ls to confirm) >>> rmdir -a # fails: invalid option -- 'a' >>> rmdir -- -a # success! (ls to confirm) This trick can be useful in many scenarios, and generally arises when you need to work with special characters of some sort. 21 -
Your new best friend How do I know what the
flags / options for all of these commands are? The manual command man <command_name> - Loads the manual (manpage) for the specified command. - Unlike google, manpages are system-specific. - Usually very comprehensive. Sometimes too comprehensive. - Type /<keyword> to search. - The n key jumps through the search results. Search example on next page if that was confusing. Intended for side-by-side follow-along. 22 -
Man oh man >>> man man # you now have
the manual loaded >>> /useful # type /useful, then hit enter ############# [first result highlighted] >>> n # followed by enter ############# [next result highlighted] Note that there are subtle differences between options on different systems. For example, ls -B: • BSD/OSX: Force printing of non-printable characters in file names as \xxx, where xxx is the numeric value of the character in octal. • Fedora, Ubuntu: do not list implied entries ending with ~ • In these OS’s, files ending with ~ are temporary backup files that certain programs (e.g. some text-editors) 23 -
Reading Files Without Opening Concatenate cat <filename> - Prints the
contents of the file to the terminal window cat <file1> <file2> - Prints file1 first, then file2. more more <filename> - Scroll through one page at a time. - Program exits when end is reached. less less <filename> - Scroll pages or lines (mouse wheel, space bar, and arrows). - Program does not exit when end is reached. 24 -
Beginning and End Long files can be a pain with
the previous tools. Head and Tail of Input head -[numlines] <filename> tail -[numlines] <filename> - Prints the first / last numlines of the file. - Default is 10 lines. 25 -
Not Really a File...YET You can talk to yourself in
the terminal too! Echo echo <text> - Prints the input string to the standard output (the terminal). - We will soon learn how to use echo to put things into files, append to files, etc. 26 -
Before we can Chain... ...we need some more interesting tools
to chain together! 27 -
Counting Word Count wc [options] <file> -l: count the number
of lines. -w: count the number of words. -m: count the number of characters. -c: count the number of bytes. Great for things like: • revelling in the number of lines you have programmed. • analyzing the verbosity of your personal statement. • showing people how cool you are. 28 -
Sorting Sort sort [options] <file> - Default: sort by the
ASCII code (roughly alphabetical) for the whole line. - Use -r to reverse the order. - Use -n to sort by numerical order. - Use -u to remove duplicates. >>> cat peeps.txt Manson, Charles Bundy, Ted Bundy, Jed Nevs, Sven Nevs, Sven >>> sort -r peeps.txt Nevs, Sven Nevs, Sven Manson, Charles Bundy, Ted Bundy, Jed >>> sort -ru peeps.txt Nevs, Sven Manson, Charles Bundy, Ted Bundy, Jed # only 1 Nevs, Sven 29 -
Advanced Sorting • The sort command is quite powerful, for
example you can do: >>> sort -n -k 2 -t "," <filename> • Sorts the file numerically by using the second column, separating by a comma as the delimiter instead of a space. • Read the man page! >>> cat numbers.txt 02,there 04,how 01,hi 06,you 03,bob 05,are >>> sort -n -k 2 -t "," numbers.txt 01,hi 02,there 03,bob 04,how 05,are 06,you 30 -
Special Snowflakes Unique uniq [options] <file> - No flags: discards
all but one of successive identical lines. - Use -c to prints the number of successive identical lines next to each line. 31 -
Search and Replace Translate tr [options] <set1> [set2] - Translate
or delete characters. - Sets are strings of characters. - By default, searches for strings matching set1 and replaces them with set2. - You can use POSIX and custom-defined sets (we’ll get there soon!). • The tr command only works with streams. • Examples to come after we learn about chaining commands in the next section. 32 -
Your Environment and Variables • There are various environment variables
defined in your environment. They are almost always all capital letters. • You obtain their value by dereferencing them with a $. >>> echo $PWD # present working directory >>> echo $OLDPWD # print previous working directory >>> printenv # print all environment variables • When you execute commands, they have something called an ”exit code”. • The exit code of the last command executed is stored in the $? environment variable. 33 -
What is Defined? • The environment: • env: displays all
environment variables. • unsetenv <name>: remove an environment variable. • The local variables: • set: displays all shell / local variables. • unset <name>: remove a shell variable. • We’ll cover these a little more when we talk about customizing your terminal shell. 34 -
Exit Codes • There are various exit codes, here are
a few examples: >>> super_awesome_command bash: super_awesome_command: command not found... >>> echo $? 127 >>> echo "What is the exit code we want?" >>> echo $? 0 • The success code we want is actually 0. Refer to [1] for some more examples. • Remember that cat /dev/urandom trickery? You will have to ctrl+c to kill it, what would the exit code be? 35 -
Executing Multiple Commands in a Row With exit codes, we
can define some simple rules to chain commands together: • Always execute: >>> cmd1; cmd2 # exec cmd1 first, then cmd2 • Execute conditioned upon exit code: >>> cmd1 && cmd2 # exec cmd2 only if cmd1 returned 0 >>> cmd1 || cmd2 # exec cmd2 only if cmd1 returned NOT 0 • Kind of backwards, in terms of what means continue for and, but that was likely easier to implement since there is only one 0 and many not 0’s. 36 -
Piping Commands Bash scripting is all about combining simple commands
together to do more powerful things. This is accomplished using the ”pipe” character. Piping <command1> | <command2> - Passes the output from command1 to be the input of command2. - Works for heaps of programs that take input and provide output to the terminal. 37 -
Some Piping Examples Piping along... >>> ls -al /bin |
less - Allows you to scroll through the long list of programs in /bin >>> history | tail -20 | head -10 - Displays the 10th - 19th previous commands from the previous session. >>> echo * | tr ' ' '\n' - Replaces all spaces characters with new lines. - Execute just echo * to see the difference. 38 -
Redirection To redirect input / output streams, you can use
one of >, >>, <, or <<. • To redirect standard output, use the > operator. • command > file • To redirect standard input, use the < operator. • command < file • To redirect standard error, use the > operator and specify the stream number 2. • command 2> file • Combine streams together by using 2>&1 syntax. • This says: send standard error to where standard output is going. • Useful for debugging / catching error messages... • ...or ignoring them (you will often see that sent to /dev/null). 39 -
Redirection Example • Bash processes I/O redirection from left to
right, allowing us to do fun things like this: Magic tr -cd '0-9' < test1.txt > test2.txt - Deletes everything but the numbers from test1.txt, then store them in test2.txt. - CAUTION: do not ever use the same file as output that was input. - Example: tr -cd '0-9' < original.txt > original.txt - You will lose all your data, you cannot read and write this way. • Piping and Redirection are quite sophisticated, please refer to the Wikipedia page in [2]. 40 -
References I [1] T. L. D. Project. Exit codes with
special meanings. http://tldp.org/LDP/abs/html/exitcodes.html. [2] Wikipedia. Redirection (computing). https://en.wikipedia.org/wiki/Redirection_ %28computing%29. 41