awk

simple introduction of awk

Transcript

1. 姜博 <jiang.bo.hit@gmail.com>
2.  历史  命令行  一些概念  脚本  示例 CSCI 330 - The UNIX System 2
3.  始祖 : Aho, Weinberger, and Kernighan  AWK is a programming language designed for processing text-based data, either in files or data streams. ——wikipedia  版本  awk, nawk, mawk, pgawk, …  GNU awk: gawk 3
4.  可以用来 :  转换数据文件  产生格式化报告 4
6.  awk [options] ‘script’ file(s)  awk [options] –f scriptfile file(s) Options: -F to change input field separator -f to name script file 6
7.  构成 : pattern {action}  缺 pattern ，对所有行执行 action  缺 action ，打印所有匹配行  两个都没有？——逗我玩呢！ Example: awk '/for/' testfile awk ‘{print}’ testfile CSCI 330 - The UNIX System 7
8.  awk 运行过程 :  读取：按行浏览文件（ Line by line ）  切分：将每个输入行切分为多个域（ Field ）  匹配：根据 pattern 进行行或域匹配  执行：对匹配行执行操作 CSCI 330 - The UNIX System 8
9.  域 (Field): 每一行的基础单元  分隔符 (field separator):  default field separator is whitespace  记录 (record): is the collection of fields in a line  A data file is made up of records CSCI 330 - The UNIX System 9
10. CSCI 330 - The UNIX System 10
11.  AWK 支持两类 buffer: record and field  field buffer:  当前记录中的每一个 field  表示 : $1, $2, …  record buffer :  $0 表示整条记录 11 我是 zero
12. FS Field separator (default=whitespace) RS Record separator (default=\n) NF Number of fields in current record NR Number of the current record OFS Output field separator (default=space) ORS Output record separator (default=\n) 12
13.  awk 脚本包含三个主要部分：  # 为注释表示符 13
16.  match  entire input record regular expression enclosed by ‘/’s  explicit pattern-matching expressions ~ (match), !~ (not match)  expression operators  算术  关系  逻辑 16
17. OperatorMeaning Example + Add x + y - Subtract x – y * Multiply x * y / Divide x / y % Modulus x % y ^ Exponential x ^ y Example: % awk '$3 * $4 > 500 {print $0}' file 17
18. Operator Meaning Example < Less than x < y < = Less than or equal x < = y == Equal to x == y != Not equal to x != y > Greater than x > y > = Greater than or equal to x > = y 18
19. Operator Meaning Example && Logical AND a && b || Logical OR a || b ! NOT ! a Examples: % awk '($2 > 5) && ($2 <= 15) {print $0}' file % awk '$3 == 100 || $4 > 50' file CSCI 330 - The UNIX System 19
20.  匹配连续输入行中的一个区域 语法 : pattern1 , pattern2 {action}  pattern 可以为任何 simple pattern  pattern1 开始执行 action  pattern2 结束执行 action 20
23.  计算并返回值 23
24.  任意数量  三种类型：数字，字符串，数组  变量名字母开头，尾随数字，字母，下划 线  弱类型：在第一次引用时就一直存在，不 需声明  所有变量被初始化为空字符串 24
25. = assign result of right-hand-side expression to left-hand-side variable ++ Add 1 to variable -- Subtract 1 from variable += Assign result of addition -= Assign result of subtraction *= Assign result of multiplication /= Assign result of division %= Assign result of modulo ^= Assign result of exponentiation 25
26. print 直接输出 printf 格式化输出 (C printf) sprintf 格式化字符串 (C sprintf) ，返回值 CSCI 330 - The UNIX System 26
27.  Print output goes to standard output unless redirected via: > “file” >> “file” | “command”  will open file or command only once  subsequent redirections append to already open stream CSCI 330 - The UNIX System 27
28. 语法 : printf(format-string, var1, var2, …)  works like C printf  each format specifier in “format-string” requires argument of matching type CSCI 330 - The UNIX System 28
29. %d, %i decimal integer %c single character %s string of characters %f floating point number %o octal number %x hexadecimal number %e scientific floating point notation %% the letter “%” CSCI 330 - The UNIX System 29
30. CSCI 330 - The UNIX System 30 Given: x = ‘A’, y = 15, z = 2.3, and $1 = Bob Smith Printf Format Specifier What it Does %c printf("The character is %c \n", x) output: The character is A %d printf("The boy is %d years old \n", y) output: The boy is 15 years old %s printf("My name is %s \n", $1) output: My name is Bob Smith %f printf("z is %5.3f \n", z) output: z is 2.300
31.  between “%” and letter %10s %7d %10.4f %-20s  meaning:  width of field, field is printed right justified  precision: number of digits after decimal point CSCI 330 - The UNIX System 31
32. Syntax: sprintf(format-string, var1, var2, …)  Works like printf, but does not produce output  Instead it returns formatted string Example: { text = sprintf("1: %d – 2: %d", $1, $2) print text } CSCI 330 - The UNIX System 32
33. tolower(string) toupper(string) 转化字符大小写，非字母字符不变 Ex:  批量转换文件名大小写：  ls -l | awk '{printf("mv %s %s\n", $8, tolower($8))|"sh"}' CSCI 330 - The UNIX System 33
34. split(string, array, fieldsep)  divides string into pieces separated by fieldsep, and stores the pieces in array  if the fieldsep is omitted, the value of FS is used. Example: split("auto-da-fe", a, "-")  sets the contents of the array a as follows: a[1] = "auto" a[2] = "da" CSCI 330 - The UNIX System 34
35.  一维  索引可以是数字或字符串  无需声明 CSCI 330 - The UNIX System 35
36. Syntax: arrayName[index] = value Examples: list[1] = "one" list[2] = "three" list["other"] = "oh my !" CSCI 330 - The UNIX System 36
37.  The delete function can be used to delete an element from an array. Format: delete array_name [index] Example: delete deptSales["supplies"] CSCI 330 - The UNIX System 37
38. 条件  if-else 循环  for ▪ with counter ▪ with array index  while  do-while  also: break, continue CSCI 330 - The UNIX System 38
39. Syntax: if (conditional expression) statement-1 else statement-2 Example: if ( NR < 3 ) print $2 else print $3 CSCI 330 - The UNIX System 39
40. Syntax: for (initialization; limit-test; update) statement Example: for (i = 1; i <= NR; i++) { total += $i count++ } CSCI 330 - The UNIX System 40
41. Syntax: for (var in array) statement Example: for (x in deptSales) { print x, deptSales[x] } CSCI 330 - The UNIX System 41
42. Syntax: while (logical expression) statement Example: i = 1 while (i <= NF) { print i, $i i++ } CSCI 330 - The UNIX System 42
43. Syntax: do statement while (condition) statement is executed at least once, even if condition is false at the beginning Example: i = 1 do { print $0 i++ } while (i <= 10) CSCI 330 - The UNIX System 43
44.  break exits loop  continue skips rest of current iteration, continues with next iteration CSCI 330 - The UNIX System 44
45. for (x = 0; x < 20; x++) { if ( array[x] > 100) continue printf "%d ", x if ( array[x] < 0 ) break } CSCI 330 - The UNIX System 45
46. 1. 生成 SQL 2. 生成文档 CSCI 330 - The UNIX System 46