Command-line Filters: Time to Shine

Transcript

1. Command-line Filters: Time to Shine
 The Perl and Raku Conference
 
 2022-06-24
2. Command-line Filters: Time to Shine PLEASE DOWNLOAD THESE SLIDES! 
 http://speakerdeck.com/util
 <bruce.gray@acm.org> <<< >>> >>> <<<
3. /me
 
 Bruce Gray

4. /me
 
 Bruce Gray
 
 'Util'
5. Command-line Filters: Time to Shine
6. Order of Battle • • • • •
7. Order of Battle • What are filters? • • • •
8. Order of Battle • What are filters? • How to use filters? < | > >> • • •
9. Order of Battle • What are filters? • How to use filters? < | > >> • Really? • •
10. Order of Battle • What are filters? • How to use filters? < | > >> • Really! • •
11. Order of Battle • What are filters? • How to use filters? < | > >> • Really! • HTF? •
12. Order of Battle • What are filters? • How to use filters? < | > >> • Really! • HTF? • Now You!
13. Rationale • 
 • 
 •
14. Rationale • PyCharm IDE: Python
 • 
 •
15. Rationale • PyCharm IDE: Python
 • RStudio: R-Lang
 •
16. Rationale • PyCharm IDE: Python
 • RStudio: R-Lang
 • Jupyter Notebook (JupyterLab?): Everything
17. Tedious,
 and Well Covered Elsewhere • Getting a command line
 (Terminal, CMD, xterm...) • Current directory
 (and Windows "Drive Letter") • Text Editor
 (not Word Processor!) • wsl --install
18. Very Useful,
 but Not Covered This Year • Command-line Perl • -e and -E • -n and -p • -a and -F • -l and -0777 • BEGIN and END
19. perl -0777 -wnE 's{\rMSH}{\nMSH}g; s{\r\z}{\n}; my @L = split "\n";
 s{\A.+\r(OBR)}{$1} for @L; s{\r.+\z}{} for @L; tr{|}{\t} for @L; for (@L) {my @F = split "\t"; $ARGV =~ /lab_(\d\d\d\d)-/ or warn; say "$1\t$F[20]"}' lab_*.txt > ../assessions.txt
20. Loosely Related • CLI : Command Line Interface • git status, git commit... • ls [-ABCFGHLOPRSTUW@abcdefghiklmnopqrstuwx1]
21. Redirect, Pipe • seq 6 12 • seq 6 12 > a.txt • seq 6 12 >> a.txt • less a.txt • less < a.txt • seq 6 12 | less
22. Redirect, Pipe • seq 6 12 • seq 6 12 > a.txt • seq 6 12 >> a.txt • less a.txt • less < a.txt • seq 6 12 | less
23. Redirect, Pipe • seq 6 12 • seq 6 12 > a.txt • seq 6 12 >> a.txt • less a.txt • less < a.txt • seq 6 12 | less
24. Redirect, Pipe • seq 6 12 • seq 6 12 > a.txt • seq 6 12 >> a.txt • less a.txt • less < a.txt • seq 6 12 | less
25. Redirect, Pipe • seq 6 12 • seq 6 12 > a.txt • seq 6 12 >> a.txt • less a.txt • less < a.txt • seq 6 12 | less
26. $ seq 6 12 6 7 8 9 10 11 12
27. $ seq 6 12 > a.a
28. $ seq 6 12 > a.a
29. $ seq 6 12 > a.a $ grep 1 a.a
30. $ seq 6 12 > a.a $ grep 1 a.a
31. $ seq 6 12 > a.a $ grep 1 a.a
32. $ seq 6 12 > a.a $ grep 1 a.a 10 11 12
33. $ seq 6 12 > a.a $ grep 1 a.a 10 11 12 $ grep 1 <a.a
34. $ seq 6 12 > a.a $ grep 1 a.a 10 11 12 $ grep 1 <a.a $ cat a.a | grep 1
35. $ seq 6 12 > a.a $ grep 1 a.a 10 11 12 $ grep 1 <a.a $ cat a.a | grep 1
36. $ seq 6 12 > a.a $ grep 1 a.a 10 11 12 $ grep 1 <a.a $ cat a.a | grep 1
37. $ seq 6 12 > a.a $ grep 1 a.a 10 11 12 $ grep 1 <a.a $ cat a.a | grep 1 $ seq 6 12 | grep 1
38. $ seq 1 999 | less
40. $ seq 1 999 | m
42. seq 1 999 | m
43. | m ??? seq 1 999 | m
44. | m ??? Really? seq 1 999 | m
45. | m ??? Really? seq 1 999 | m
46. $ seq 6 12 > a.a $ seq 6 12 >> a.a
47. Really!
48. https://csvkit.readthedocs.io/ • csvcut • csvgrep • csvclean • csvjoin • csvsort • csvstack • csvformat • csvjson • csvlook • csvpy • csvsql • csvstat • in2csv • sql2csv
49. csvcut -c 5,13 input.csv csvcut -c name,post_code input.csv
50. csvcut -c 5,13 input.csv | md5 csvcut -c name,post_code input.csv | md5
51. my_prog > a.txt (edit)
52. my_prog > a.txt (edit) my_prog > b.txt; diff -u {a,b}.txt | wc
53. my_prog > a.txt (edit) my_prog > b.txt; diff -u {a,b}.txt | wc 0
54. my_prog > a.txt (edit) my_prog > b.txt; diff -u {a,b}.txt | wc my_prog > b.txt; diff -u {a,b}.txt >
 ab.diff; wc ab.diff
55. my_prog > a.txt (edit) my_prog > b.txt; diff -u {a,b}.txt | wc my_prog > b.txt; diff -u {a,b}.txt >
 ab.diff; wc ab.diff m ab.diff
56. my_prog > a.txt (edit) my_prog > b.txt; diff -u {a,b}.txt | wc my_prog > b.txt; diff -u {a,b}.txt >
 ab.diff; wc ab.diff m ab.diff mv b.txt a.txt
57. | sort | uniq -c | sort -nr
58. | sort | uniq -c | sort -nr
59. | sort | uniq -c | sort -nr
60. | sort | uniq -c | sort -nr
61. | sort | uniq -c | sort -nr
62. THAT SENTENCE HAD TOO MANY SYLLABLES!
 
 APOLOGIZE!
63. THAT SENTENCE HAD TOO MANY SYLLABLES!
 
 APOLOGIZE!
64. THAT SENTENCE HAD TOO MANY SYLLABLES!
 
 APOLOGIZE!
65. cd perl-5.36.0
66. grep -Rh '#!' . | sort | uniq -c | sort -nr | head -14
67. grep -Rh '#!' . | sort | uniq -c | sort -nr | head -14
68. grep -Rh '#!' . | sort | uniq -c | sort -nr | head -14
69. grep -Rh '#!' . | sort | uniq -c | sort -nr | head -14
70. grep -Rh '#!' . | sort | uniq -c | sort -nr | head -14
71. grep -Rh '#!' . | sort | uniq -c | sort -nr | head -14
72. grep -Rh '#!' . | sort | uniq -c | sort -nr | head -14 556 #!./perl 322 #!/usr/bin/perl -w 300 #!/usr/bin/perl 196 #!perl 178 #!./perl -w 126 #!perl -w 37 #!/bin/sh 31 #!/usr/local/bin/perl 24 #!/pro/bin/perl 20 #!/usr/bin/perl -Tw 19 #!/usr/bin/perl 15 #! /bin/sh 13 #!/usr/bin/perl -wT 13 #!/usr/bin/env perl
73. grep -Rh '#!' . | sort | uniq -c | sort -nr | head -14 556 #!./perl 322 #!/usr/bin/perl -w 300 #!/usr/bin/perl 196 #!perl 178 #!./perl -w 126 #!perl -w 37 #!/bin/sh 31 #!/usr/local/bin/perl 24 #!/pro/bin/perl 20 #!/usr/bin/perl -Tw 19 #!/usr/bin/perl 15 #! /bin/sh 13 #!/usr/bin/perl -wT 13 #!/usr/bin/env perl
74. perl -wpE
 'if ($. != 1) {s/^"\d+",// or die}' v_o.csv | sort | uniq -c | sort -nr | m
75. csvcut -C 1 v_o.csv | sort | uniq -c | sort -nr | m
76. __END__ Adapted from original one-liner: perl -wpE 'if ($. != 1) {s/^"\d+",// or die}' v_o.csv | sort | uniq -c | sort -nr
77. How it works • open my $fh, '<', 'myfile.txt'; • Magic cookie • • • •
79. How it works • open my $fh, '<', 'myfile.txt'; • Magic cookie • 0,1,2 • STDIN, STDOUT, STDERR • $*IN, $*OUT, $*ERR • Sane defaults, invention of STDERR
80. Peter Piper Processed a Plethora of
 Programs
81. Peter Piper Processed a Plethora of
 Programs, in Parallel !
82. Peter Piper Processed a Plethora of
 Programs, in Parallel ! seq
83. Peter Piper Processed a Plethora of
 Programs, in Parallel ! seq 1000000000
84. Peter Piper Processed a Plethora of
 Programs, in Parallel ! seq 1000000000 | cut -c 1
85. Peter Piper Processed a Plethora of
 Programs, in Parallel ! seq 1000000000 | cut -c 1 | uniq
86. Peter Piper Processed a Plethora of
 Programs, in Parallel ! seq 1000000000 | cut -c 1 | uniq
87. Now You!
88. use autodie; my $input_path = './name1_to_edit_by_hand.txt'; my $output_path = './name2_to_edit_by_hand.txt'; open my $fh_in, '<', $input_path; open my $fh_out, '>', $output_path; while ( my $line = <$fh_in> ) { chomp $line; my $output = do_something_with($line); say {$out_fh} $output; }
89. use autodie; my $input_path = './name1_to_edit_by_hand.txt'; my $output_path = './name2_to_edit_by_hand.txt'; open my $fh_in, '<', $input_path; open my $fh_out, '>', $output_path; while ( my $line = <$fh_in> ) { chomp $line; my $output = do_something_with($line); say {$out_fh} $output; }
90. use autodie; my $input_path = './name1_to_edit_by_hand.txt'; my $output_path = './name2_to_edit_by_hand.txt'; open my $fh_in, '<', $input_path; open my $fh_out, '>', $output_path; while ( my $line = <$fh_in> ) { chomp $line; my $output = do_something_with($line); say {$out_fh} $output; }
91. while (<>) { chomp; say do_something_with($_); } prog ./name1_to_edit_by_hand.txt >./name2_to_edit_by_hand.txt
92. #!/usr/bin/env python3 import fileinput for line in fileinput.input(): line = line.rstrip('\r\n') print(">>", line, "<<") https://docs.python.org/3/library/fileinput.html #!/usr/bin/env perl use 5.36.0; while (<>) { chomp; say ">> $_ <<"; } https://perldoc.perl.org/perlop#I/O-Operators #!/usr/bin/env raku use v6; for lines() { say ">> $_ <<"; } https://docs.raku.org/type/IO::CatHandle
93. #!/usr/bin/env python3 import fileinput for line in fileinput.input(): line = line.rstrip('\r\n') print(">>", line, "<<") https://docs.python.org/3/library/fileinput.html #!/usr/bin/env perl use 5.36.0; while (<>) { chomp; say ">> $_ <<"; } https://perldoc.perl.org/perlop#I/O-Operators #!/usr/bin/env raku use v6; for lines() { say ">> $_ <<"; } https://docs.raku.org/type/IO::CatHandle
94. #!/usr/bin/env python3 import fileinput for line in fileinput.input(): line = line.rstrip('\r\n') print(">>", line, "<<") https://docs.python.org/3/library/fileinput.html #!/usr/bin/env perl use 5.36.0; while (<>) { chomp; say ">> $_ <<"; } https://perldoc.perl.org/perlop#I/O-Operators #!/usr/bin/env raku use v6; for lines() { say ">> $_ <<"; } https://docs.raku.org/type/IO::CatHandle
95. #!/usr/bin/env python3 import fileinput for line in fileinput.input(): line = line.rstrip('\r\n') print(">>", line, "<<") https://docs.python.org/3/library/fileinput.html #!/usr/bin/env perl use 5.36.0; while (<>) { chomp; say ">> $_ <<"; } https://perldoc.perl.org/perlop#I/O-Operators #!/usr/bin/env raku use v6; for lines() { say ">> $_ <<"; } https://docs.raku.org/type/IO::CatHandle
96. # XXX Remove after testing !!! my $default_file = 'C:/path/to/trimmed.dat'; push @ARGV, $default_file if not @ARGV;
97. # XXX Remove after testing !!! my $default_file = 'C:/path/to/trimmed.dat'; push @ARGV, $default_file if not @ARGV; @ARGV ||= $default_file;
98. # XXX Remove after testing !!! my $default_file = 'C:/path/to/trimmed.dat'; push @ARGV, $default_file if not @ARGV; @ARGV ||= $default_file; # Fails sometimes
99. Further Reading • https://utcc.utoronto.ca/~cks/space/blog/python/ ProgramFilterVsWrapper
 Programs as wrappers versus filters of other programs • http://catb.org/~esr/writings/taoup/html/ ch01s06.html
 Basics of the Unix Philosophy
100. Sched • https://utcc.utoronto.ca/~cks/space/blog/python/ ProgramFilterVsWrapper
 Programs as wrappers versus filters of other programs • http://catb.org/~esr/writings/taoup/html/ ch01s06.html
 Basics of the Unix Philosophy
101. Q & A
102. Command-line Filters: Time to Shine PLEASE DOWNLOAD THESE SLIDES! 
 http://speakerdeck.com/util
 <bruce.gray@acm.org> <<< >>> >>> <<<
103. Much Thanks
 to You All!
104. Copyrights
105. Copyright Information: Images and Video • Camelia • © 2009 by Larry Wall
 http://github.com/perl6/mu/raw/master/misc/camelia.txt • Yellow Laundry Ticket • © (assumed) TC Dry Clean Supply of Houston, TX
 Used without permission, even though we are *in* Houston!
 https://drycleansupply.com/product-category/invoice-tag- thermal-ribbon/laundry-ticket-tag/ •
106. Copyright Information: This Talk This work is licensed under a Creative Commons Attribution 4.0 International License. CC BY https://creativecommons.org/licenses/by/4.0/ (email me for the original Apple Keynote .key file)
107. History • v 0.98 2022-06-02
 Presented incomplete version to Atlanta Perlmongers
 • v 1.00 2022-06-24
 Presented final version to The Perl and Raku Conference in Houston, TX, USA