Command-line Filters: Reprised

Starting with the bare basics of command-line pipes and redirection, I further cover the standard "Unix filter" form of code, and the benefits of nudging old code into this form.

These techniques are my bread-and-butter for exploratory programming and data analysis, and include a few of my best tricks.

Slides will cover variants for (Raku|Perl|Python) and (Mac|Windows|Unix).

Filters are the basic tools of command-line data munging. grep head cut sort uniq wc diff, pipes + redirects, intro + advanced tips. Why stop there? *You* can write your *own*. Perl, Raku, & Python have strong support for writing filters. Existing code often works better when refactored as a filter.

Transcript

1. Command-line Filters: Time to Shine
 The Perl and Raku Conference
 
 2023-07-12
2. Command-line Filters: Time to Shine PLEASE DOWNLOAD THESE SLIDES! 
 http://speakerdeck.com/util
 <bruce.gray@acm.org> <<< >>> >>> <<<
3. /me
 
 Bruce Gray

4. /me
 
 Bruce Gray
 
 'Util'
5. Command-line Filters: Time to Shine
6. Order of Battle • • • • •
7. Order of Battle • What are fi lters? • • • •
8. Order of Battle • What are fi lters? • How to use fi lters? < | > >> • • •
9. Order of Battle • What are fi lters? • How to use fi lters? < | > >> • Really? • •
10. Order of Battle • What are fi lters? • How to use fi lters? < | > >> • Really! • •
11. Order of Battle • What are fi lters? • How to use fi lters? < | > >> • Really! • HTF? •
12. Order of Battle • What are fi lters? • How to use fi lters? < | > >> • Really! • HTF? • Now You!
13. Rationale • 
 • 
 •
14. Rationale • PyCharm IDE: Python
 • 
 •
15. Rationale • PyCharm IDE: Python
 • RStudio: R-Lang
 •
16. Rationale • PyCharm IDE: Python
 • RStudio: R-Lang
 • Jupyter Notebook (JupyterLab?): Everything
17. 2023-07-12
18. Tedious,
 and Well Covered Elsewhere • Getting a command line
 (Terminal, CMD, xterm...) • Current directory
 (and Windows "Drive Letter") • Text Editor
 (not Word Processor!) • wsl --install
19. Very Useful,
 but Not Covered This Year • Command-line Perl • -e and -E • -n and -p • -a and -F • -l and -0777 • BEGIN and END
20. perl -0777 -wnE 's{\rMSH}{\nMSH}g; s{\r\z}{\n}; my @L = split "\n";
 s{\A.+\r(OBR)}{$1} for @L; s{\r.+\z}{} for @L; tr{|}{\t} for @L; for (@L) {my @F = split "\t"; $ARGV =~ /lab_(\d\d\d\d)-/ or warn; say "$1\t$F[20]"}' lab_*.txt > ../assessions.txt
21. Redirect, Pipe • seq 6 12 • seq 6 12 > a.txt • seq 6 12 >> a.txt • less a.txt • less < a.txt • seq 6 12 | less
22. Redirect, Pipe • seq 6 12 • seq 6 12 > a.txt • seq 6 12 >> a.txt • less a.txt • less < a.txt • seq 6 12 | less
23. Redirect, Pipe • seq 6 12 • seq 6 12 > a.txt • seq 6 12 >> a.txt • less a.txt • less < a.txt • seq 6 12 | less
24. Redirect, Pipe • seq 6 12 • seq 6 12 > a.txt • seq 6 12 >> a.txt • less a.txt • less < a.txt • seq 6 12 | less
25. Redirect, Pipe • seq 6 12 • seq 6 12 > a.txt • seq 6 12 >> a.txt • less a.txt • less < a.txt • seq 6 12 | less
26. $ seq 6 12 6 7 8 9 10 11 12
27. $ seq 6 12 > a.a
28. $ seq 6 12 > a.a
29. $ seq 6 12 > a.a $ grep 1 a.a
30. $ seq 6 12 > a.a $ grep 1 a.a
31. $ seq 6 12 > a.a $ grep 1 a.a
32. $ seq 6 12 > a.a $ grep 1 a.a 10 11 12
33. $ seq 6 12 > a.a $ grep 1 a.a 10 11 12 $ grep 1 <a.a
34. $ seq 6 12 > a.a $ grep 1 a.a 10 11 12 $ grep 1 <a.a $ cat a.a | grep 1
35. $ seq 6 12 > a.a $ grep 1 a.a 10 11 12 $ grep 1 <a.a $ cat a.a | grep 1
36. $ seq 6 12 > a.a $ grep 1 a.a 10 11 12 $ grep 1 <a.a $ cat a.a | grep 1
37. $ seq 6 12 > a.a $ grep 1 a.a 10 11 12 $ grep 1 <a.a $ cat a.a | grep 1 $ seq 6 12 | grep 1
38. $ seq 1 999 | less
40. $ seq 1 999 | e
42. seq 1 999 | e
43. | e ??? seq 1 999 | e
44. | e ??? Really? seq 1 999 | e
45. | e ??? Really? seq 1 999 | e
46. | e ??? Really! seq 1 999 | e
47. $ seq 6 12 > a.a $ seq 6 12 >> a.a
48. Really!
49. https://csvkit.readthedocs.io/ • csvcut • csvgrep • csvclean • csvjoin • csvsort • csvstack • csvformat • csvjson • csvlook • csvpy • csvsql • csvstat • in2csv • sql2csv
50. https://csvkit.readthedocs.io/ • csvcut • csvgrep • csvclean • csvjoin • csvsort • csvstack • csvformat • csvjson • csvlook • csvpy • csvsql • csvstat • in2csv • sql2csv
51. csvcut -c 5,13 input.csv csvcut -c name,post_code input.csv
52. csvcut -c 5,13 input.csv | md5 csvcut -c name,post_code input.csv | md5
53. my_prog > a.txt (edit)
54. my_prog > a.txt (edit) my_prog > b.txt; diff -u {a,b}.txt | wc
55. my_prog > a.txt (edit) my_prog > b.txt; diff -u {a,b}.txt | wc 0
56. my_prog > a.txt (edit) my_prog > b.txt; diff -u {a,b}.txt | wc 0 On Windows, you may need `a.txt b.txt` instead of `{a,b}.txt`
57. my_prog > a.txt (edit) my_prog > b.txt; diff -u {a,b}.txt | wc my_prog > b.txt; diff -u {a,b}.txt >
 ab.diff; wc ab.diff
58. my_prog > a.txt (edit) my_prog > b.txt; diff -u {a,b}.txt | wc my_prog > b.txt; diff -u {a,b}.txt >
 ab.diff; wc ab.diff e ab.diff
59. my_prog > a.txt (edit) my_prog > b.txt; diff -u {a,b}.txt | wc my_prog > b.txt; diff -u {a,b}.txt >
 ab.diff; wc ab.diff e ab.diff mv b.txt a.txt
60. | sort | uniq -c | sort -nr
61. | sort | uniq -c | sort -nr
62. | sort | uniq -c | sort -nr
63. | sort | uniq -c | sort -nr
64. | sort | uniq -c | sort -nr
65. THAT SENTENCE HAD TOO MANY SYLLABLES!
 
 APOLOGIZE!
66. THAT SENTENCE HAD TOO MANY SYLLABLES!
 
 APOLOGIZE!
67. THAT SENTENCE HAD TOO MANY SYLLABLES!
 
 APOLOGIZE!
68. cd perl-5.36.0
69. grep -Rh '#!' . | sort | uniq -c | sort -nr | head -14
70. grep -Rh '#!' . | sort | uniq -c | sort -nr | head -14
71. grep -Rh '#!' . | sort | uniq -c | sort -nr | head -14
72. grep -Rh '#!' . | sort | uniq -c | sort -nr | head -14
73. grep -Rh '#!' . | sort | uniq -c | sort -nr | head -14
74. grep -Rh '#!' . | sort | uniq -c | sort -nr | head -14
75. grep -Rh '#!' . | sort | uniq -c | sort -nr | head -14 556 #!./perl 322 #!/usr/bin/perl -w 300 #!/usr/bin/perl 196 #!perl 178 #!./perl -w 126 #!perl -w 37 #!/bin/sh 31 #!/usr/local/bin/perl 24 #!/pro/bin/perl 20 #!/usr/bin/perl -Tw 19 #!/usr/bin/perl 15 #! /bin/sh 13 #!/usr/bin/perl -wT 13 #!/usr/bin/env perl
76. grep -Rh '#!' . | sort | uniq -c | sort -nr | head -14 556 #!./perl 322 #!/usr/bin/perl -w 300 #!/usr/bin/perl 196 #!perl 178 #!./perl -w 126 #!perl -w 37 #!/bin/sh 31 #!/usr/local/bin/perl 24 #!/pro/bin/perl 20 #!/usr/bin/perl -Tw 19 #!/usr/bin/perl 15 #! /bin/sh 13 #!/usr/bin/perl -wT 13 #!/usr/bin/env perl
77. csvcut -C 1 v_o.csv | sort | uniq -c | sort -nr | e
78. perl -wpE 'if ($. != 1) {s/^"\d+",// or die}' v_o.csv | sort | uniq -c | sort -nr | e
79. __END__ Adapted from original one-liner: perl -wpE 'if ($. != 1) {s/^"\d+",// or die}' v_o.csv | sort | uniq -c | sort -nr
80. How it works • open my $fh, '<', 'my fi le.txt'; • Magic cookie • • • •
82. How it works • open my $fh, '<', 'my fi le.txt'; • Magic cookie • 0,1,2 • STDIN, STDOUT, STDERR • $*IN, $*OUT, $*ERR • Sane defaults, invention of STDERR
83. Peter Piper Processed a Plethora of
 Programs
84. Peter Piper Processed a Plethora of
 Programs, in Parallel !
85. Peter Piper Processed a Plethora of
 Programs, in Parallel ! seq
86. Peter Piper Processed a Plethora of
 Programs, in Parallel ! seq 1000000000
87. Peter Piper Processed a Plethora of
 Programs, in Parallel ! seq 1000000000 | cut -c 1
88. Peter Piper Processed a Plethora of
 Programs, in Parallel ! seq 1000000000 | cut -c 1 | uniq
89. Peter Piper Processed a Plethora of
 Programs, in Parallel ! seq 1000000000 | cut -c 1 | uniq
90. Peter Piper Processed a Plethora of
 Programs, in Parallel ! seq 1000000000 | cut -c 1 | uniq
91. Now You!
92. use autodie; my $input_path = './name1_to_edit_by_hand.txt'; my $output_path = './name2_to_edit_by_hand.txt'; open my $fh_in, '<', $input_path; open my $fh_out, '>', $output_path; while ( my $line = <$fh_in> ) { chomp $line; my $output = do_something_with($line); say {$out_fh} $output; }
93. use autodie; my $input_path = './name1_to_edit_by_hand.txt'; my $output_path = './name2_to_edit_by_hand.txt'; open my $fh_in, '<', $input_path; open my $fh_out, '>', $output_path; while ( my $line = <$fh_in> ) { chomp $line; my $output = do_something_with($line); say {$out_fh} $output; }
94. use autodie; my $input_path = './name1_to_edit_by_hand.txt'; my $output_path = './name2_to_edit_by_hand.txt'; open my $fh_in, '<', $input_path; open my $fh_out, '>', $output_path; while ( my $line = <$fh_in> ) { chomp $line; my $output = do_something_with($line); say {$out_fh} $output; }
95. while (<>) { chomp; say do_something_with($_); } prog ./name1_to_edit_by_hand.txt >./name2_to_edit_by_hand.txt
96. #!/usr/bin/env python3 import fileinput for line in fileinput.input(): line = line.rstrip('\r\n') print(">>", line, "<<") https://docs.python.org/3/library/fileinput.html #!/usr/bin/env perl use 5.36.0; while (<>) { chomp; say ">> $_ <<"; } https://perldoc.perl.org/perlop#I/O-Operators #!/usr/bin/env raku use v6; for lines() { say ">> $_ <<"; } https://docs.raku.org/type/IO::CatHandle
97. #!/usr/bin/env python3 import fileinput for line in fileinput.input(): line = line.rstrip('\r\n') print(">>", line, "<<") https://docs.python.org/3/library/fileinput.html #!/usr/bin/env perl use 5.36.0; while (<>) { chomp; say ">> $_ <<"; } https://perldoc.perl.org/perlop#I/O-Operators #!/usr/bin/env raku use v6; for lines() { say ">> $_ <<"; } https://docs.raku.org/type/IO::CatHandle
98. #!/usr/bin/env python3 import fileinput for line in fileinput.input(): line = line.rstrip('\r\n') print(">>", line, "<<") https://docs.python.org/3/library/fileinput.html #!/usr/bin/env perl use 5.36.0; while (<>) { chomp; say ">> $_ <<"; } https://perldoc.perl.org/perlop#I/O-Operators #!/usr/bin/env raku use v6; for lines() { say ">> $_ <<"; } https://docs.raku.org/type/IO::CatHandle
99. #!/usr/bin/env python3 import fileinput for line in fileinput.input(): line = line.rstrip('\r\n') print(">>", line, "<<") https://docs.python.org/3/library/ fi leinput.html #!/usr/bin/env perl use 5.36.0; while (<>) { chomp; say ">> $_ <<"; } https://perldoc.perl.org/perlop#I/O-Operators #!/usr/bin/env raku use v6; for lines() { say ">> $_ <<"; } https://docs.raku.org/type/IO::CatHandle
100. # XXX Remove after testing !!! my $default_file = 'C:/path/to/trimmed.dat'; push @ARGV, $default_file if not @ARGV;
101. # XXX Remove after testing !!! my $default_file = 'C:/path/to/trimmed.dat'; push @ARGV, $default_file if not @ARGV; @ARGV ||= $default_file;
102. # XXX Remove after testing !!! my $default_file = 'C:/path/to/trimmed.dat'; push @ARGV, $default_file if not @ARGV; @ARGV ||= $default_file; # Fails sometimes
103. Further Reading • https://utcc.utoronto.ca/~cks/space/blog/python/ ProgramFilterVsWrapper
 Programs as wrappers versus fi lters of other programs • http://catb.org/~esr/writings/taoup/html/ ch01s06.html
 Basics of the Unix Philosophy
104. Q & A
105. Command-line Filters: Time to Shine PLEASE DOWNLOAD THESE SLIDES! 
 http://speakerdeck.com/util
 <bruce.gray@acm.org> <<< >>> >>> <<<
106. Much Thanks
 to You All!
107. Copyrights
108. Copyright Information: Images and Video • Camelia • © 2009 by Larry Wall
 http://github.com/perl6/mu/raw/master/misc/camelia.txt • Yellow Laundry Ticket • © (assumed) TC Dry Clean Supply of Houston, TX
 Used without permission, even though we are *in* Houston!
 https://drycleansupply.com/product-category/invoice-tag- thermal-ribbon/laundry-ticket-tag/ •
109. Copyright Information: This Talk This work is licensed under a Creative Commons Attribution 4.0 International License. CC BY https://creativecommons.org/licenses/by/4.0/ (email me for the original Apple Keynote .key fi le)
110. History • v 0.98 2022-06-02
 Presented incomplete version to Atlanta Perlmongers
 • v 1.00 2022-06-24
 Presented fi nal version to The Perl and Raku Conference in Houston, TX, USA
 • v 1.01 2023-07-12
 Reprised at The Perl and Raku Conference in Toronto, Ontario, CA