Flint axes in the information age

Slides from my talk at BT DevCon9, 13 Feb 2015.

The final commands built up in the worked examples were:

# Find distribution of number of files changed by git commits
git log --oneline | cut -d' ' -f1 | while read sha ; do git show --stat $sha | grep -c [-+]$ ; done | sort -n | uniq -c

# Count references to each class in project
awk '/^ *class / { print $2 } /Struct.new/ { print $1 }' app/**/*.rb | while read class ; do echo -n $class " " ; cat app/**/*.rb | grep -c "\b$class\b" ; done | awk '$2 > 1 { print $2 - 1 " " $1 }' | sort -rn

Transcript

1. Flint axes in the information age Kerry Buckley ※ DevCon9
2. What am I talking about?
3. Why?
4. Weather munger Dy MxT MnT AvT HDDay AvDP 1HrP TPcpn WxType PDir AvSp Dir MxS SkyC MxR MnR AvSLP 1 88 59 74 53.8 0.00 F 280 9.6 270 17 1.6 93 23 1004.5 2 79 63 71 46.5 0.00 330 8.7 340 23 3.3 70 28 1004.5 3 77 55 66 39.6 0.00 350 5.0 350 9 2.8 59 24 1016.8 4 77 59 68 51.1 0.00 110 9.1 130 12 8.6 62 40 1021.1 5 90 66 78 68.3 0.00 TFH 220 8.3 260 12 6.9 84 55 1014.4 6 81 61 71 63.7 0.00 RFH 030 6.2 030 13 9.7 93 60 1012.7 7 73 57 65 53.0 0.00 RF 050 9.5 050 17 5.3 90 48 1021.8 8 75 54 65 50.0 0.00 FH 160 4.2 150 10 2.6 93 41 1026.3 9 86 32* 59 6 61.5 0.00 240 7.6 220 12 6.0 78 46 1018.6 10 84 64 74 57.5 0.00 F 210 6.6 050 9 3.4 84 40 1019.0 11 91 59 75 66.3 0.00 H 250 7.1 230 12 2.5 93 45 1012.6 12 88 73 81 68.7 0.00 RTH 250 8.1 270 21 7.9 94 51 1007.0 13 70 59 65 55.0 0.00 H 150 3.0 150 8 10.0 83 59 1012.6 14 61 59 60 5 55.9 0.00 RF 060 6.7 080 9 10.0 93 87 1008.6 15 64 55 60 5 54.9 0.00 F 040 4.3 200 7 9.6 96 70 1006.1 16 79 59 69 56.7 0.00 F 250 7.6 240 21 7.8 87 44 1007.0 17 81 57 69 51.7 0.00 T 260 9.1 270 29* 5.2 90 34 1012.5 18 82 52 67 52.6 0.00 230 4.0 190 12 5.0 93 34 1021.3 19 81 61 71 58.9 0.00 H 250 5.2 230 12 5.3 87 44 1028.5 20 84 57 71 58.9 0.00 FH 150 6.3 160 13 3.6 90 43 1032.5 21 86 59 73 57.7 0.00 F 240 6.1 250 12 1.0 87 35 1030.7 22 90 64 77 61.1 0.00 H 250 6.4 230 9 0.2 78 38 1026.4 23 90 68 79 63.1 0.00 H 240 8.3 230 12 0.2 68 42 1021.3 24 90 77 84 67.5 0.00 H 350 8.5 010 14 6.9 74 48 1018.2 25 90 72 81 61.3 0.00 190 4.9 230 9 5.6 81 29 1019.6 26 97* 64 81 70.4 0.00 H 050 5.1 200 12 4.0 107 45 1014.9
5. Weather munger Dy MxT MnT AvT HDDay AvDP 1 1 88 59 74 53.8 2 79 63 71 46.5 3 77 55 66 39.6 4 77 59 68 51.1 5 90 66 78 68.3 6 81 61 71 63.7 7 73 57 65 53.0 8 75 54 65 50.0
6. Weather munger require "weather_data_reader" require "smallest_temperature_range_finder" require "day_number_printer" class WeatherMunger def initialize(file_path, reader: WeatherDataReader.new, finder: SmallestTemperatureRangeFinder.new, printer: DayNumberPrinter.new) @file_path = file_path @reader = reader @finder = finder @printer = printer end def call records = @reader.call @file_path record = @finder.call records @printer.call record end end
7. Weather munger WeatherRecord = Struct.new :day, :min_temp, :max_temp do def temperature_range max_temp - min_temp end end
8. Weather munger require "weather_record" class WeatherDataReader DATA_PATTERN = \ /^\s*(?<day>\d+)\s+(?<max_temp>\d+)\*?\s+(?<min_temp>\d+)/ def call path File.read(path).lines.select {|line| line =~ DATA_PATTERN }.map {|line| fields = line.match(DATA_PATTERN) WeatherRecord.new(fields[:day].to_i, fields[:min_temp].to_i, fields[:max_temp].to_i) } end end
9. Weather munger require "weather_record" class SmallestTemperatureRangeFinder def call records records.inject do |result, record| if record.temperature_range < result.temperature_range record else result end end end end
10. Weather munger require "weather_record" class DayNumberPrinter def call record puts record.day end end
11. Weather munger awk '/^ +[0-9]/ && (min == "" || $2 - $3 < min) {day = $1; min = $2 - $3} END {print day}' weather.dat
12. Command-line tools can be 235x faster than your Hadoop cluster
13. Unix commands ❖ Work with file streams ❖ Expect and output plain text ❖ Avoid verbosity ❖ Separate errors from output ❖ Exit code: 0 means success
14. Unix commands +---------+ | +---> Output Input +---+ Command | | +---> Errors +---------+
15. Unix commands +---------+ | +---> stdout stdin +---+ Command | | +---> stderr +---------+
16. Unix commands | Ceci n’est pas un pipe
17. Unix commands | +------+ | +------+ | +--->+ | +---> +--+ Cmd1 | +---+ Cmd2 | | +-> | | +-> +------+ | +------+ |
18. Useful commands
19. cat
20. cat _ \`*-. ) _`-. . : `. . : _ ' \ ; *` _. `*-._ `-.-' `-. ; ` `. :. . \ . \ . : .-' . ' `+.; ; ' : : ' | ; ;-. ; ' : :`-: _.`* ; [bug] .*' / .*' ; .*`- +' `*' `*-* `*-* `*-*'
21. cat $ $ cat /etc/paths
22. cat $ $ cat /etc/paths /usr/local/bin /usr/bin /bin /usr/sbin /sbin $
23. cat $ $ cat /etc/paths /etc/manpaths
24. cat $ $ cat /etc/paths /etc/manpaths /usr/local/bin /usr/bin /bin /usr/sbin /sbin /usr/share/man /usr/local/share/man $
25. cat $ $ cat
26. cat $ $ cat Hello DevCon Hello DevCon
27. cat $ $ cat Hello DevCon Hello DevCon I'm reading lines from stdin I'm reading lines from stdin
28. cat $ $ cat Hello DevCon Hello DevCon I'm reading lines from stdin I'm reading lines from stdin And writing them to stdout And writing them to stdout
29. cat $ $ cat Hello DevCon Hello DevCon I'm reading lines from stdin I'm reading lines from stdin And writing them to stdout And writing them to stdout ^D $
30. cat cat cat file cat file1 file2 file3 cat file* cat <file1 <file1 cat cat file1 file2 >file3 cat file 2>/dev/null cat file1 >file2 2>&1 cat file1 &>file2
31. grep $ $ grep '^d.v.l...r$' /usr/ share/dict/words
32. grep $ $ grep '^d.v.l...r$' /usr/ share/dict/words developer $
33. grep $ $ cat /usr/share/dict/words | grep '^d.v.l...r$'
34. grep $ $ cat /usr/share/dict/words | grep '^d.v.l...r$' developer $
35. grep $ $ cat /usr/share/dict/words | grep '^d.v.l...r$' developer $ UUOC!
36. sed $ $ cat file
37. sed $ $ cat file Hello world foo=42 print "foo is " + foo goodbye $
38. sed $ $ sed 's/foo/bar/g' file
39. sed $ $ sed 's/foo/bar/g' file Hello world bar=42 print "bar is " + bar Goodbye $
40. sed $ $ sed '2,3d' file
41. sed $ $ sed '2,3d' file Hello world Goodbye $
42. sed $ $ sed '/^Good/d' file
43. sed $ $ sed '/^Good/d' file Hello world foo=42 print "foo is " + foo $
44. sed $ $ sed '/^print/ i\ # ' file
45. sed $ $ sed '/^print/ i\ # ' file Hello world foo=42 # print "foo is " + foo Goodbye $
46. sed $ $ sed '$ a\ END OF FILE ' file
47. sed $ $ sed '$ a\ END OF FILE ' file Hello world foo=42 print "foo is " + foo Goodbye END OF FILE $
48. $ $ find app -name '*_controller.rb' find
49. $ $ find app -name '*_controller.rb' find app/controllers/application_controller.rb app/controllers/sessions_controller.rb app/controllers/users_controller.rb $
50. find $ $ find app/assets -type f
51. find $ $ find app/assets -type f app/assets/images/.keep app/assets/images/kitten.jpg app/assets/javascripts/application.js app/assets/stylesheets/application.css.scss $
52. find $ $ find spec -type d
53. find $ $ find spec -type d spec spec/controllers spec/mailers spec/models spec/presenters spec/services spec/support $
54. xargs $ $ find spec -type d | xargs
55. xargs $ $ find spec -type d | xargs spec spec/controllers spec/ mailers spec/models spec/ presenters spec/services spec/ support $
56. xargs $ $ find spec -type d | xargs du -ks
57. xargs $ $ find spec -type d | xargs du -ks 84 spec 16 spec/controllers 4 spec/mailers 16 spec/models 4 spec/presenters 28 spec/services 8 spec/support $
58. sort $ $ find spec -type d | xargs du -ks | sort -rn
59. sort $ $ find spec -type d | xargs du -ks | sort -rn 84 spec 28 spec/services 16 spec/models 16 spec/controllers 8 spec/support 4 spec/presenters 4 spec/mailers $
60. head $ $ find spec -type d | xargs du -ks | sort -rn | head -2
61. head $ $ find spec -type d | xargs du -ks | sort -rn | head -2 84 spec 28 spec/services $
62. tail $ $ find spec -type d | xargs du -ks | sort -rn | tail -2
63. tail $ $ find spec -type d | xargs du -ks | sort -rn | tail -2 4 spec/presenters 4 spec/mailers $
64. cut $ $ cat /etc/passwd nobody:*:-2:-2:Unprivileged User:/var/ empty:/usr/bin/false root:*:0:0:System Administrator:/var/ root:/bin/sh daemon:*:1:1:System Services:/var/root:/ usr/bin/false _uucp:*:4:4:Unix to Unix Copy Protocol:/ var/spool/uucp:/usr/sbin/uucico _taskgated:*:13:13:Task Gate Daemon:/var/ empty:/usr/bin/false ... $
65. cut $ $ cut -d: -f1,3 /etc/passwd
66. cut $ $ cut -d: -f1,3 /etc/passwd nobody:-2 root:0 daemon:1 _uucp:4 _taskgated:13 ... $
67. awk $ $ grep -v '^#' /etc/passwd | cut -d: -f1,3
68. awk $ $ grep -v '^#' /etc/passwd | cut -d: -f1,3 nobody:-2 root:0 daemon:1 _uucp:4 _taskgated:13 ... $
69. awk $ $ awk 'BEGIN { FS = ":" } /^[^#]/ { print $1 ":" $3 }' /etc/passwd
70. awk $ $ awk 'BEGIN { FS = ":" } /^[^#]/ { print $1 ":" $3 }' /etc/passwd nobody:-2 root:0 daemon:1 _uucp:4 _taskgated:13 ... $
71. BEGIN { FS = ":" } /^[^#]/ { print $1 ":" $3 } awk
72. BEGIN { FS = ":" } /^[^#]/ { print $1 ":" $3 } awk
73. awk awk '/^ +[0-9]/ && (min == "" || $2 - $3 < min) {day = $1; min = $2 - $3} END {print day}' weather.dat
74. awk /^ +[0-9]/ && (min == "" || $2 - $3 < min) {day = $1; min = $2 - $3} END {print day}
75. awk /^ +[0-9]/ && (min == "" || $2 - $3 < min) { day = $1 min = $2 - $3 } END { print day }
76. Worked examples