A Bash Primer

Given as a presentation for Academic Software Plus.

Transcript

1. Bash Primer Saturday, April 6, 13
2. What is Bash Scripting language for *nix Shell Descendent of sh A loosely coupled collection of command line tools Saturday, April 6, 13 Bash is a scripting language but more importantly it is a Shell and all of the tools you have installed in your shell. As long as a tool has a cli and behaves properly with stdin, stdout, stderror it can play with Bash. Check shell options with the ‘shopt` builtin
3. Ways to run code From a file set to executable (+x) From command line eval : executes a string as code exec : starts another process, exiting the current one source : forces a script to run in the current process Saturday, April 6, 13 you can call bash directly or provide a shebang line The behavior of source will be relevant in a moment. When we want to export variables
4. Bash Environment /etc/profile ~/.bash_profile interactive shell non -interactive shell Saturday, April 6, 13
5. Shell Expansion Command line is split in to tokens EAch token is then resolved 8 types of expansion (Brace, Tilde, Parameter, Command substitution, Arithmetic expansion, Word splitting, File name expansion, Process substitution) Saturday, April 6, 13 $
6. Command Lifecycle Syntax Analysis Brace expansion {} (not ${}) Tilde Expansion Command substitution , Arithmetic Expansion, parameter expansion ($) Word Splitting Filename expansion Quote Removal Saturday, April 6, 13 brace expansion must have unquoted braces and at least one comma We use tilde expansion all the time Arithmetic expansion can also use single square brackets $[]
7. Command Substitution Allows the output of a command to replace the command itself $() vs ‘’ Recommended to use $(comand) Saturday, April 6, 13 Bash will execute the command and replace with the stdout from the command sans newlines $() is recommended over ``
8. Word Splitting The shell scans results of parameter, command, and arithmetic expansion for word splitting IFS (internal field separator) <space> <tab> <Newline> only occurs when results are not double quoted Saturday, April 6, 13 IFS holds the characters that bash things are word boundaries Encountering a word boundary causes bash to split the statement and interpret each word as a separate command or parameter example cp $file $target # if either of thise variables have spaces in the file name this will not behave as you expect You can get around this by double quoting the variables
9. File Expansion Last pass before command execution Scans for *, ? and [ If any exist the word is considered a pattern Pattern is replaced with a list of file names in the current or stated directory that match that pattern Saturday, April 6, 13 * matches any string for ** if followed by / (**/) will match only directories and subdirectories ? matches any single character [matches any of the enclosed characters] if you set the extglob shell option You can also do pattern lists test if it is on using ‘shopt’ ? (list) matches zero or one of the patterns * (list) matches zero or more + (list) matches one or more @(list) matches one !(anything not in the patterns)
10. Variables Globals ( default status, use env or printenv to see them) local (define with keyword local) Use $ to retreve value but not assignment Case sensitive and capitalized by convention lowercase often used for locals Saturday, April 6, 13 hyphens are reserved
11. Variables and Subshells A shell is an instance of the command processor Every script runs in a subshell (child of the command terminal) You can force a subshell with () for parallelization Saturday, April 6, 13
12. export keyword By default variables are local to the current shell and child processes (sub shells) will not be aware of them export tells the current shell that the variable is to be treated as an environment variable accessible by all subshells Saturday, April 6, 13 A subshell is a separate instance of the command processor, these can be launched by When you export a variable a copy is made in each subshell. You cannot simply export from a scripted subshell back out to the parent. If you source a script it is not
13. Reserved Variables There is a fairly long list of reserved variables Some of them can be pretty useful to know about (FUNCNAME, GROUPS, etc..) $*, $@, $#, $?, #-, #!, $0, #_ Saturday, April 6, 13 $* : Expands to the positional parameters, starting from one. When the expansion occurs within double quotes, it expands to a single word with the value of each parameter separated by the first character of the IFS special variable. $@ : Expands to the positional parameters, starting from one. When the expansion occurs within double quotes, each parameter expands to a separate word. $# : Expands to the number of positional parameters in decimal. $? : Expands to the exit status of the most recently executed foreground pipeline. $- : A hyphen expands to the current option flags as specified upon invocation, by the set built-in command, or those set by the shell itself (such as the -i). $$ : Expands to the process ID of the shell. $! : Expands to the process ID of the most recently executed background (asynchronous) command. $0 : Expands to the name of the shell or shell script. $_ : The underscore variable is set at shell startup and contains the absolute file name of the shell or script being executed as passed in the argument list. Subsequently, it expands to the last argument to the previous command, after expansion. It is also set to the full pathname of each command executed and placed in the environment exported to that command. When checking mail, this parameter holds the name of the mail file.
14. Strings “ ” and ‘ ’ are not the same. Variable expansion only occurs in double quotes Singe quotes inside of double quotes are escaped automatically and vice versa Literals do not require quotes (unless they have meta characters in them) String Operators are provided via ${} Be careful with spaces Saturday, April 6, 13 str1=abcd str2=”abcd abd” #String Operators are provided via ${} ${#str} #return the lenght of the string Concatenation ${str}otherstring String Length ${#string} Substring Extraction ${string:position} ${string:position:length} Substring Removal substring is actually a pattern with ( ${string#substring} # delete shortest match of substring from string ${string##substring} #delete longest match of substring from string ${string%substring} #delete shortest possible match from the right ${string%%substring} #delete longest possible match from the right Substring Replacement ${string/abz/xyz} Variable Defaults ${foo:-bar} #return bar if foo is null or undefined
15. Integers 3 and “3” are basically the same, type is decided by context Special operators for arithmetic, $(()) indicates arithmetic inside Integers can be compared with -eq (= is for literals) Saturday, April 6, 13 # $(()) indicates arithmetic to be performed one=”1” two=2 $(( $one + $two + 3 ))
16. Constants declare -rx will set a variable as read only and exported Saturday, April 6, 13
17. Arrays declare -a Ary=(foo bar baz) ary[expr] = foo Gaps are allowed Ary=([0]=”foo”,[21]=”bar”,[19]=baz) files=(~/MyPics/*.jpg) Saturday, April 6, 13 echo ${Ary[@]} prints the whole array #print the whole array ${#Ary[@]} will give you the length of the array just as it does a string Extraction ${Ary[2]:0:4} # return the first four elments (from index 0 to 4) of the item at index 2 Find and Replace ${Ary[@]/string/pattern} #returns the adjusted array after substitution Nullify and index or an entire array unset Ary[expr] #delete at index unset Ary #delete entire array Concatenation Ary3=("${Ary1[@]}"  "${Ary2[@]}")
18. I/O Redirection stdin(0), stderr(2), stdout(1) Use > for redirection & Indicates you want a reference to the output location Saturday, April 6, 13 program 2> errors.log #redirect standard out to errors program 2>&1 #redirect standard error to wherever standard out is going
19. Parameter Passing command line and function input is parsed as positional parameters $1,$2... Functions do not explicitly take defined parameters Functions cannot return a value to the caller (cannot generally be used as a parameter) Saturday, April 6, 13
20. Functions Can only return integers to represent status return 0 for success anything else is failure STatus is stored in $? Common approach to return values for function is to use globals (yuck) Saturday, April 6, 13
21. Control Structures Bash supports looping and conditional constructs Some of the syntax may seem a bit odd Saturday, April 6, 13
22. ! If [ ] ... then ... else ... fi = is test operator for equivalence when there is a space before the LHS !=, < , >, -z(is null), -n(not null),- f(file exist) ... []is any expression full construct if..then.. elif..then..else.. fi Saturday, April 6, 13 if [ “$foo” = “$bar” ]; then #then is put on this line for convenience but is separate expr echo It isn’t else echo It is fi #Does a file exist if[ -f $FILENAME ] [ -z $foo] # is foo null [ -n $foo} # is foo non null [ $foo -eq $bar ]
23. If Rules Always spaces between brackets and expressions [ <gimme space> ] test,[],[[]] expression ends line ( ; ) Quote string variables Can use boolean expressions Saturday, April 6, 13 Quoting string vars as a matter of practices will help guard from errors with space in your variables if [$foo -ge 3 -a $foo -lt 10]; -a, -o for and and or in single bracket syntax
24. While and Until while [expr]; do <code> done until [expr]; do <code> done Saturday, April 6, 13 var1=0 until[ $var1 -eq 10]; do var1=$(( $var1 + 1 )) done
25. For For VAR in LIST do <CODE> done LIST can be an array, subrange of array, a space separated list,ranges,seq Supports 3-expression (c style) format Can use ‘break’ to exit the loop at any time and ‘continue’ to skip iteration Saturday, April 6, 13 for VAR in {0..5} # use a range for VAR in {0..10..2} #user range but count by 2 for (( i=0; i<=5; i++)) #c style 3 expression syntax for (( ; ; ; )) #will loop infinitely for file in /etc/* #go through all of the files in that directory for i in “$(Ary[@])” #loop over the array, the quote tell bash it is a single list of things
26. Alternative ‘IF’ syntax [] vs [[]] vs test vs (()) Unless you require posix support use [[]] or (()) Saturday, April 6, 13 [] is the same as using the built in ‘test’ command ‘if test $foo -ge 3; then #test if foo is greater or equal to 3 This explains why bash often uses -ge, -eq etc.. becuase they are command line options for the test command [[]] is the newer preferred syntax for if statements you can use ‘globbing’ (*) or *[aA]lpha* etc.. You can also omit quotes around variables with this version because word splitting is prevented but it is still a good habit to maintain Also there are differences in expansion of file names if [ -a *.sh ]; then # returns true if there is a single file with a .sh extension but will error if there is more than one file because *.sh is automatically expanded if [[ -a *.sh ]]; then # this will work because [[]] does not expand file names (()) syntax is for arithmetic (( $num <= 5 )) and supports normal operators like == < > >= etc.. get list of conditions here http://www.linuxtutorialblog.com/post/tutorial-conditions-in- bash-scripting-if-statements
27. Files [-r file] #test is readable [-f file ] # test file exists command > file Saturday, April 6, 13
28. Interactive Behavior `read’ command Saturday, April 6, 13 read a b # assigned the variables $a and $b via prompting the user
29. Beyond Bash Bash is really a small language and a large ecosystem of tools that run in the shell Any shell script or binary can be integrated in to your bash scripts Saturday, April 6, 13
30. rsync Synchronize remote and local directroes rsync -auvHP foo@bar.com:/remote/ filepath/ local/filepath/ / is important Saturday, April 6, 13
31. Pipe firstcommand | secondcommand Allows you to chain stdout to stdin evaluated from left to right Human centipede was actually a Bash Allegory Saturday, April 6, 13 examples: du -h | sort > diskusage.text #pipe disk usage through sort and then direct to file ps ax | grep processname | grep -v grep | awk ‘{print $1}’ #list all process ides for given process name ls -C | more #view directories in tidy form -C forces multi column
32. Awk Yet another programming language Beyond the scope of this presentation but it is worth mentioning c-like input language Basic usage is processing column oriented text data Saturday, April 6, 13 awk < file ‘{print $2 }’ # on every line print the second column awk < file ‘{ print $2, $3}’ # print column 2 and 3 ps aux | grep process_name | grep -v grep
33. Grep Regular expression matching grep ‘word’ filename grep ‘str1 str2’ filename command | grep ‘string’ Saturday, April 6, 13 grep -i “foo” filename # case insenstive search for the word foo in filename grep -r “bar” /filedir # recursively search a directory of files grep -w # match words only egrep -w ‘word1|word2’ /path/to/file #match on two words grep -c # get number of times a pattern has been matched grep -v #invert (not match)
34. SSH / SCP ssh foo@bar.com ‘command to run’ scp -c foo@bar.com:/path/to/remote.file / path/to/local.file pipe data over the network Saturday, April 6, 13
35. Event Designators Use with the history command !! runs last command !-2 runs next to last command !499 runs command 499 in the history Saturday, April 6, 13 history | grep processname | awk ‘{print $1}’ | !$($1)
36. One line File copy tar zcf - stuff | ssh foo@bar ‘tar zxf -’ ssh foo@bar ‘tar zcf - stuff’ | tar zxf - Saturday, April 6, 13 First case tars the file and then sends it to users home directory and untars second one does the opposite. We would use this for moving lots of files because scp would open up a different connection for each file and take forever. Although I tend to prefer rsync for moving lots of files, this approach can sometimes be faster
37. Run local script remotely ssh user@server ‘bash -s’ < script.sh Will display the output locally Saturday, April 6, 13 Basically you are directly the content of file script.sh into the bash command on the remote system. We see wayne doing this kind of stuff all the time
38. A Web Server ? #!/bin/bash # web.sh -- http://localhost:3000/hello?world RESP=/tmp/webresp [ -p $RESP ] || mkfifo $RESP #Loop Forever while true ; do ( cat $RESP ) | nc -l -p 9000 | ( REQ=`while read L && [ " " "<" "$L" ] ; do echo "$L" ; done` echo "[`date '+%Y-%m-%d %H:%M:%S'`] $REQ" | head -1 cat >$RESP <<EOF HTTP/1.0 200 OK Cache-Control: private Content-Type: text/plain Server: bash/2.0 Connection: Close Content-Length: ${#REQ} $REQ EOF ) done http://paulbuchheit.blogspot.com/2007/04/webserver-in-bash.html Saturday, April 6, 13 #!/bin/bash RESP=/tmp/webresp [ -p $RESP ] || mkfifo $RESP while true ; do ( cat $RESP ) | nc -l -p 9000 | ( REQ=`while read L && [ " " "<" "$L" ] ; do echo "$L" ; done` echo "[`date '+%Y-%m-%d %H:%M:%S'`] $REQ" | head -1 cat >$RESP <<EOF HTTP/1.0 200 OK Cache-Control: private Content-Type: text/plain Server: bash/2.0 Connection: Close Content-Length: ${#REQ} $REQ EOF ) done
39. Crack Root on Mysql for i in `seq 1 1000`; do mysql -u root -- password=bad -h 127.0.0.1 2>/dev/null; done effects all version before 6/11/2012 on certain platforms Saturday, April 6, 13 On Saturday afternoon Sergei Golubchik posted to the oss-sec mailing list about a recently patched security flaw (CVE-2012-2122) in the MySQL and MariaDB database servers. This flaw was rooted in an assumption that the memcmp() function would always return a value within the range -127 to 127 (signed character). On some platforms and with certain optimizations enabled, this routine can return values outside of this range, eventually causing the code that compares a hashed password to sometimes return true even when the wrong password is specified. Since the authentication protocol generates a different hash each time this comparison is done, there is a 1 in 256 chance that ANY password would be accepted for authentication. In short, if you try to authenticate to a MySQL server affected by this flaw, there is a chance it will accept your password even if the wrong one was supplied. The following one-liner in bash will provide access to an affected MySQL server as the root user account, without actually knowing the password. • Ubuntu Linux 64-bit ( 10.04, 10.10, 11.04, 11.10, 12.04 ) ( via many including @michealc ) • OpenSuSE 12.1 64-bit MySQL 5.5.23-log ( via @michealc ) • Debian Unstable 64-bit 5.5.23-2 ( via @derickr ) • Fedora ( via hexed and confirmed by Red Hat ) • Arch Linux (unspecified version)
40. Cool Resources http://www.bashoneliners.com/ http://mywiki.wooledge.org/ BashPitfalls Saturday, April 6, 13