Refactoring_and_Readability_-_CRHS_-_Perl_Conference_20200624.pdf

Refactoring and Readability:  Crouching Regex, Hidden Structures  Perl & Raku
Conference in the Cloud    2020-06-24

Refactoring and Readability 1.5:  The Pre-Sequel http://speakerdeck.com/util    <[email protected]> <<<
>>> >>> <<<

Refactoring and Readability 2:  Crouching Regex, Hidden Structures

Refactoring and Readability 1.5 :  The Pre-Sequel

/me    Bruce Gray    'Util'

Perl 6 ==> Raku

Face Oﬀ ( Rubin's Vase )

Fun for the whole family • Regular Expressions (Regex) •
Multi-dimensional Data Structures • Trees •

Fun for the whole family • Regular Expressions (Regex) •
Multi-dimensional Data Structures • Trees • Q & A + Bonus!!!

Sinkholes  in the  space/code continuum or, how to you decide
what to learn, when you cannot judge its value until *after* you learn it?

Last time, on Dragonball Z

Modes Fix Add Refactor

  Forces Surrounding Code Code Easy to write Standard form
Ease of change Easy to read Performance Boundaries of Responsibility

Regular Expressions  Regex

–Jamie Zawinski “Some people, when confronted with a problem, think 
`I know, I'll use regular expressions.`  Now they have two problems.”

File Globbing Regex . Literal dot Any single character ?
Any single character Quantifier: Once or none * Any string without '/' Quantifier: Zero or more + Quantifier: One or more {2,4} Alternation: '2' or '4' Quantifier: Two to four times (2|4) Alternation: '2' or '4'

ack https://beyondgrep.com/ ag https://github.com/ggreer/the_silver_searcher rg https://blog.burntsushi.net/ripgrep/

$ find . -not -type d ./SS_20200503/Screen Shot 2020-05-01 at
4.45.30 PM.png ./SS_20200503/Screen Shot 2020-04-28 at 1.29.14 PM.png ./SS_20200503/Screen Shot 2020-04-24 at 8.08.38 PM.png ./SS_20200503/Screen Shot 2020-04-28 at 2.35.18 PM.png ./SS_20200527/Screen Shot 2020-05-03 at 7.40.14 PM.png …

./SS_20200503/Screen Shot 2020-05-01 at 4.45.30 PM.png ./SS_20200503/Screen Shot 2020-04-28 at
1.29.14 PM.png ./SS_20200503/Screen Shot 2020-04-24 at 8.08.38 PM.png ./SS_20200503/Screen Shot 2020-04-28 at 2.35.18 PM.png ./SS_20200527/Screen Shot 2020-05-03 at 7.40.14 PM.png …

./SS_20200503/Screen Shot 2020-05-01 at 4.45.30 PM.png

Screen Shot 2020-05-01 at 4.45.30 PM.png

Screen Shot 2020-05-01 at 4.45.30 PM.png ^ ^ ^

Screen Shot 2020-05-01 at 4.45.30 PM.png

Screen Shot 2020-\d5-01 at 4.45.30 PM.png

Screen Shot 2020-\d\d-01 at 4.45.30 PM.png

Screen Shot 2020-\d\d-\d1 at 4.45.30 PM.png

Screen Shot 2020-\d\d-\d\d at 4.45.30 PM.png

Screen Shot 2020-\d\d-\d\d at \d.45.30 PM.png

Screen Shot 2020-\d\d-\d\d at \d\.45.30 PM.png

Screen Shot 2020-\d\d-\d\d at \d\.\d5.30 PM.png

Screen Shot 2020-\d\d-\d\d at \d\.\d\d.30 PM.png

Screen Shot 2020-\d\d-\d\d at \d\.\d\d\.30 PM.png

Screen Shot 2020-\d\d-\d\d at \d\.\d\d\.\d0 PM.png

Screen Shot 2020-\d\d-\d\d at \d\.\d\d\.\d\d PM.png

Screen Shot 2020-\d\d-\d\d at \d\.\d\d\.\d\d PM\.png

Screen Shot 2020-\d\d-\d\d at \d\.\d\d\.\d\d PM\.png$

'Screen Shot 2020-\d\d-\d\d at \d\.\d\d\.\d\d PM\.png$'

$ find . -not -type d | ack -v \

'Screen Shot 2020-\d\d-\d\d at \d\.\d\d\.\d\d PM\.png$ '

'Screen Shot 2020-\d\d-\d\d at \d\.\d\d\.\d\d PM\.png$' ./SS_20190828/Screen Shot 2019-08-10 at 9.28.52 PM.png ./SS_20190828/Screen Shot 2019-08-15 at 8.59.19 PM.png …

'Screen Shot 20\d\d-\d\d-\d\d at \d\.\d\d\.\d\d PM\.png$'

'Screen Shot 20\d\d-\d\d-\d\d at \d\.\d\d\.\d\d PM\.png$' ./SS_20200503/Screen Shot 2020-04-27 at 7.14.42 AM.png ./SS_20190828/Screen Shot 2019-08-21 at 8.04.16 AM.png …

'Screen Shot 20\d\d-\d\d-\d\d at \d\.\d\d\.\d\d [AP]M\.png$' ./SS_20200503/Screen Shot 2020-04-27 at 7.14.42 AM.png ./SS_20190828/Screen Shot 2019-08-21 at 8.04.16 AM.png …

'Screen Shot 20\d\d-\d\d-\d\d at \d\.\d\d\.\d\d [AP]M\.png$'

'Screen Shot 20\d\d-\d\d-\d\d at \d\.\d\d\.\d\d [AP]M\.png$' ./SS_20200503/Screen Shot 2020-04-29 at 11.52.35 PM.png ./SS_20190828/Screen Shot 2019-07-31 at 10.14.57 AM.png

'Screen Shot 20\d\d-\d\d-\d\d at \d{}\.\d\d\.\d\d [AP]M\.png$' ./SS_20200503/Screen Shot 2020-04-29 at 11.52.35 PM.png ./SS_20190828/Screen Shot 2019-07-31 at 10.14.57 AM.png

'Screen Shot 20\d\d-\d\d-\d\d at \d{1,2}\.\d\d\.\d\d [AP]M\.png$' ./SS_20200503/Screen Shot 2020-04-29 at 11.52.35 PM.png ./SS_20190828/Screen Shot 2019-07-31 at 10.14.57 AM.png

'Screen Shot 20\d\d-\d\d-\d\d at \d{1,2}\.\d\d\.\d\d [AP]M\.png$'

'Screen Shot 20\d\d-\d\d-\d\d at \d{1,2}\.\d\d\.\d\d [AP]M\.png$' ./SS_20200111/color_wheel_2.numbers ./SS_20200111/color_wheel_1.numbers ./SS_20200131/RenFlorenceWalk.mp3 ./SS_20200131/spa-barcelona-city.mp3 ./SS_20200131/six_flags_20190112.txt ./SS_20190716/c_04.png ./SS_20190716/c_77_edge.png

https://xkcd.com/208/

Wait, forgot to escape a space. Wheeeeee[taptaptap]eeeeee.

http://regex101.com/

https://regexper.com/

'Screen Shot 20\d\d-\d\d-\d\d at \d{1,2}\.\d\d\.\d\d [AP]M\.png$' ./SS_20200111/color_wheel_2.numbers ./SS_20200111/color_wheel_1.numbers ./SS_20200131/RenFlorenceWalk.mp3 ./SS_20200131/spa-barcelona-city.mp3 ./SS_20200131/six_flags_20190112.txt ./SS_20190716/c_04.png ./SS_20190716/c_77_edge.png

• Navigation • Extraction • Exploration • Validation •

ftp://www13.warehouse.org/path/file-listing http://www.lexcorp.com/ http://www.umbrella.com/hive/plan_for_reopening

my $end_of_prefix = -1; for my $scheme ( qw<https http
ftp> ) { my $prefix = $scheme . '://'; my $p_len = length $prefix; if ( substr( $url, 0, $p_len ) eq $prefix ) { $end_of_prefix = $p_len; last; } } if ( $end_of_prefix == -1 ) { warn "Unexpected URL format: $url"; next; } my $slash_pos = index $url, '/', $end_of_prefix; if ( $slash_pos == -1 ) { warn "Unexpected URL format: $url"; next; } my $dot_pos = rindex $url, '.', $slash_pos; if ( $dot_pos == -1 ) { warn "Unexpected URL format: $url"; next; } my $start = $end_of_prefix; if ( my $i = index substr( $url, $start, $dot_pos - $start ), '.' ) { $start += $i + 1 if $i != -1; } say substr $url, $start, $dot_pos - $start;

my $url_re = qr{ ^ (https?|ftp) :// ([^/]+ \.)? ([^/]+)
\. ([^/\.]+) / }msx; $url =~ /$url_re/ or warn "Unexpected URL format: $url" and next; say $3;

• Navigation • Extraction • Exploration • Validation •

• Navigation • Extraction • Exploration • Validation • Understanding

ﬁnd . -ls

$ find . -ls 48707625 0 drwxr-xr-x 38 bruce_pro staff
1216 Jun 11 09:32 ./SS_20200503 37742599 0 drwxr-xr-x 73 bruce_pro staff 2336 Jul 31 2019 ./SS_20190828 - HPMoR

48707625 0 drwxr-xr-x 38 bruce_pro staff 1216 Jun 11 09:32
./SS_20200503 37742599 0 drwxr-xr-x 73 bruce_pro staff 2336 Jul 31 2019 ./SS_20190828 - HPMoR my ( $inode, $blocks, $perms, $links, $owner, $group, $size, $mod_time, $path ) = split /\s+/, $line;

./SS_20200503 37742599 0 drwxr-xr-x 73 bruce_pro staff 2336 Jul 31 2019 ./SS_20190828 - HPMoR ^^ ^^ my ( $inode, $blocks, $perms, $links, $owner, $group, $size, $mod_time, $path ) = split /\s+/, $line;

./SS_20200503 37742599 0 drwxr-xr-x 73 bruce_pro staff 2336 Jul 31 2019 ./SS_20190828 - HPMoR my ( $inode, $blocks, $perms, $links, $owner, $group, $size, $mod_time, $path ) = split /\s+/, $line, 9;

./SS_20200503 37742599 0 drwxr-xr-x 73 bruce_pro staff 2336 Jul 31 2019 ./SS_20190828 - HPMoR my ( $inode, $blocks, $perms, $links, $owner, $group, $size, $mod_time, $path ) = split /\s+/, $line;

my @F = split /\s+/, $line, 9; shift @F if
$F[0] eq ''; my ( $inode, $blocks, $perms, $links, $owner, $group, $size, $mod_time, $path ) = @F;

my @F = split /\s+/, $line, 9; if ( $F[0]
eq '' ) { ( my $empty, @F ) = split /\s+/, $line, 10; } my ( $inode, $blocks, $perms, $links, $owner, $group, $size, $mod_time, $path ) = @F;

$line =~ s{^\s+}{}; my ( $inode, $blocks, $perms, $links, $owner,
$group, $size, $mod_time, $path ) = split /\s+/, $line, 9;

./SS_20200503 37742599 0 drwxr-xr-x 73 bruce_pro staff 2336 Jul 31 2019 ./SS_20190828 - HPMoR $line =~ s{^\s+}{}; my ( $inode, $blocks, $perms, $links, $owner, $group, $size, $mod_time, $path ) = split /\s+/, $line, 9;

$line =~ s{^\s+}{}; my ( $inode, $blocks, $perms, $links, $owner,
$group, $size, $mod_time, $path ) = split /\s+/, $line, 9; my ( $year, $hour, $minute ); if ( substr($mod_time, 2, 1) eq ':' ) { ($hour, $minute) = split ':', $mod_time; } else { $year = $mod_time }

./SS_20200503 37742599 0 drwxr-xr-x 73 bruce_pro staff also 2336 Jul 31 2019 ./SS_20190828 - HPMoR $line =~ s{^\s+}{}; my ( $inode, $blocks, $perms, $links, $owner, $group, $size, $mod_time, $path ) = split /\s+/, $line, 9; my ( $year, $hour, $minute ); if ( substr($mod_time, 2, 1) eq ':' ) { ($hour, $minute) = split ':', $mod_time; } else { $year = $mod_time }

./SS_20200503 37742599 0 drwxr-xr-x 73 bruce_pro staff 2336 Jul 31 2019 ./SS_20190828 - HPMoR my $mod_re = qr{ (?<MOD_TIME_MONTH> Jan|Feb|Mar|Apr|May|Jun| Jul|Aug|Sep|Oct|Nov|Dec) [ ]{1,2} (?<MOD_TIME_DAY> [1-3]?\d) [ ] (?: (?<MOD_TIME_HHMM> [0-2]\d:[0-5]\d) | [ ](?<MOD_TIME_YEAR> \d\d\d\d) ) }msx;

my $re = qr{ \A \s* (?<INODE> \d+ ) \s+
(?<BLOCKS> \d+ ) \s+ (?<PERMS> [-dlcbsp] [-r] [-w] [-xs] [-r] [-w] [-xsS] [-r] [-w] [-xtT] ) \s+ (?<LINKS> \d+ ) \s+ (?<OWNER> \w+ ) \s+ (?<GROUP> \w+ ) \s+ (?<SIZE> \d+ ) \s+ (?<MOD_TIME> $mod_re ) \s+ (?<PATH> .+? ) \s* \Z }msx;

$line =~ /$find_ls_re/ or die "Failed to match '$line'"; print
$line if $+{SIZE} > 1024;

grammar find_dash_ls { rule TOP { \s* <inode> <blocks> <perms>
<links> <owner> <group> <size> <modified> <path> } token inode { \d+ } token blocks { \d+ } token perms { <[\-dlcbsp]> <[\-r]> <[\-w]> <[\-xs]> <[\-r]> <[\-w]> <[\-xsS]> <[\-r]> <[\-w]> <[\-xtT]> } token links { \d+ } token owner { \w+ } token group { \w+ } token size { \d+ } token path { \S .* } constant @days = 1 .. 31; constant @months = <Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec>; token month { @months } token day { @days } token hour { <[0..2]> \d } token minute { <[0..5]> \d } token year { \d\d\d\d } rule modified { <month> <day> [ <year> || <hour>':'<minute> ] } }

./SS_20200503 37742599 0 drwxr-xr-x 73 bruce_pro staff also 2336 Jul 31 2019 ./SS_20190828 - HPMoR

./SS_20200503 37742599 0 drwxr-xr-x 73 bruce_pro staff also 2336 Jul 31 2019 ./SS_20190828 - HPMoR $line =~ /$find_ls_re/ or die "Failed to match '$line'";

./SS_20200503 37742599 0 drwxr-xr-x 73 bruce_pro staff also 2336 Jul 31 2019 ./SS_20190828 - HPMoR #use re 'debug'; # Uncomment to get a trace. $line =~ /$find_ls_re/ or die "Failed to match '$line'"; 18 < 0 > <drwxr-xr-x> | 5| 20:OPEN3 'PERMS'(22) 18 < 0 > <drwxr-xr-x> | 5| 22:ANYOF[\-bcdlps](33) 19 < 0 d> <rwxr-xr-x > | 5| 33:ANYOF[\-r](44)

• Navigation • Extraction • Exploration • Validation • Understanding

Regex Resources • http://regex.info/book.html  The O'Reilly book "Mastering Regular Expressions" 
- Perl, Python, and more…even covers .NET ! • https://www.rubyguides.com/2015/06/ruby-regex/  Ruby-oriented • https://towardsdatascience.com/regular-expressions- explained-c9bce508e672  Python-oriented • https://learning.oreilly.com/videos/understanding-regular- expressions/9781491996300  Damian Conway's 5-hour video tutorial (subscriber-only)

Multi-Dimensional Data Structures

Single-Dimensional Data Structures • Array: List, Vector, Stack, Queue…   
  • Hash: Map, Record, Dictionary…

Single-Dimensional Data Structures • Array; List, Vector, Stack, Queue…   
  • Hash; Map, Record, Dictionary…

AoWhat? • AoA Array of Arrays (or List of Lists)
• AoH Array of Hashes • HoA Hash of Arrays • HoH Hash of Hashes • AoHoHoA Array of Hashes of Hashes of Arrays

Atlanta.pm says: • Who could conceptualize more than 3 levels?

Util 1.1.1

AoHoHoA • A Hospital •

AoHoHoA • A Hospital • has multiple numbered ﬂoors •

with multiple wards/units per ﬂoor •

with multiple wards/units per ﬂoor • with multiple nurses per ward •

with multiple wards/units per ﬂoor • with multiple nurses per ward • with a list of patients, in room-number order •

with multiple wards/units per ﬂoor • with multiple nurses per ward • with a list of patients, in room-number order • $hospital->[3]{'MedSurg'}{'Sarah'}[4];

Herman Munster 1313 Mockingbird Lane Mockingbird Heights Sherlock Holmes 221b
Baker Street London …

@name = ( 'Herman Munster', 'Sherlock Holmes', # … );
@addr = … @city = …

for my $i ( keys @name ) { say join
':', $name[$i], $addr[$i], $city[$i]; }

for my $i ( reverse keys @name ) { next
if $name[$i] !~ /herman/i; splice @name, $i, 1; splice @addr, $i, 1; splice @city, $i, 1; }

my @to_keep = grep { $name[$i] !~ /herman/i } keys
@name; @name = @name[@to_keep]; @addr = @addr[@to_keep]; @city = @city[@to_keep];

https://perldoc.perl.org/functions/keys.html keys HASH keys ARRAY Called in list context, returns
a list consisting of all the keys of the named hash, or in Perl 5.12 or later only, the indices of an array.

@name = ( 'Herman Munster', 'Sherlock Holmes', # … );
@addr = … @city = …

my @people_AoH = ( { NAME => 'Herman Munster', ADDR
=> '1313 Mockingbird Lane', CITY => 'Mockingbird Heights', }, { NAME => 'Sherlock Holmes', ADDR => '221b Baker Street', CITY => 'London', }, # … many more people loaded from the file … );

@people_AoH = grep { $->{NAME} !~ /herman/i } @people_AoH;

@people_AoH .= grep: *.<NAME> !~~ m:i/herman/;

Harry Potter

HPMoR: Harry Potter and the Methods of Rationality

Seventh Horcrux

Potter Who and the Wossname's Thingummy

half-blood prince Harry Potter "He accused me of being Dumbledore's
man through and through. ... I told him I was." HPMoR Narrator And from that day onward, no matter what Hermione tried to tell anyone, it would be an accepted legend of Hogwarts that Harry Potter could make absolutely anything happen by snapping his fingers. HPMoR Narrator Either Harry Potter had thought of a lot of very good ideas very fast, or for some unimaginable reason he'd already spent a lot of time working out how to fight underwater. HPMoR Harry Potter Harry's Internal Critic promptly awarded him the All-Time Award for the Worst Acting in the History of Ever. wossnames thingummy The Potter M:"...they called me...do you know what they called me?" Flame and air? "The Sorting Hat called you brilliant," he said. "That's a Ravenclaw tie you're wearing. Where's your wand, Myrtle Smith?" wossnames thingummy The Potter "90% of a cup of coffee is the smell. And this is 200% coffee, Jamaican Blue Mountain." And it was working: Myrtle Smith was now available in colour. seventh horcrux Potter In retrospect, I have absolutely no idea how Horcruxes work. seventh horcrux Potter "Gryffindor." S.Hat: "But you would do so well in Slytherin" "I've already done well in Slytherin. Now I want to do well, in *Gryffindor*." seventh horcrux Potter "Hermione," I said sweetly, "Do you want to be friends?" Merlin bless the simple interactions of children. HPMoR Harry Potter (to Umbridge) "I make you this one offer," said the Boy-Who-Lived. "I never learn that you've been interfering with me or any of mine. And you never find out why the unkillable soul-eating monster is scared of me. Now sit down and shut up." HPMoR Greengrass The enemy is attacking Hogwarts students... And Hogwarts, is going to *fight* *back*.”

HPMoR Harry Potter Harry's Internal Critic promptly awarded him the
All-Time Award for the Worst Acting in the History of Ever.

half-blood prince Harry Potter "He accused me of being Dumbledore's
man through and through. ... I told him I was." HPMoR Narrator And from that day onward, no matter what Hermione tried to tell anyone, it would be an accepted legend of Hogwarts that Harry Potter could make absolutely anything happen by snapping his fingers. HPMoR Narrator Either Harry Potter had thought of a lot of very good ideas very fast, or for some unimaginable reason he'd already spent a lot of time working out how to fight underwater. HPMoR Harry Potter Harry's Internal Critic promptly awarded him the All-Time Award for the Worst Acting in the History of Ever. wossnames thingummy The Potter M:"...they called me...do you know what they called me?" Flame and air? "The Sorting Hat called you brilliant," he said. "That's a Ravenclaw tie you're wearing. Where's your wand, Myrtle Smith?" wossnames thingummy The Potter "90% of a cup of coffee is the smell. And this is 200% coffee, Jamaican Blue Mountain." And it was working: Myrtle Smith was now available in colour. seventh horcrux Potter In retrospect, I have absolutely no idea how Horcruxes work. seventh horcrux Potter "Gryffindor." S.Hat: "But you would do so well in Slytherin" "I've already done well in Slytherin. Now I want to do well, in *Gryffindor*." seventh horcrux Potter "Hermione," I said sweetly, "Do you want to be friends?" Merlin bless the simple interactions of children. HPMoR Harry Potter (to Umbridge) "I make you this one offer," said the Boy-Who-Lived. "I never learn that you've been interfering with me or any of mine. And you never find out why the unkillable soul-eating monster is scared of me. Now sit down and shut up." HPMoR Greengrass The enemy is attacking Hogwarts students... And Hogwarts, is going to *fight* *back*.”

HPMoR Greengrass The enemy is attacking Hogwarts students... HPMoR Harry
Potter Harry's Internal Critic promptly awarded him the All-Time Award HPMoR Harry Potter (to Umbridge) "I make you this one offer," said the Boy-Who-Lived. HPMoR Narrator And from that day onward, no matter what Hermione tried to tell anyone, HPMoR Narrator Either Harry Potter had thought of a lot of very good ideas very fast, half-blood prince Harry Potter "He accused me of being Dumbledore's man through and through. seventh horcrux Potter In retrospect, I have absolutely no idea how Horcruxes work. seventh horcrux Potter "Gryffindor." seventh horcrux Potter "Hermione," I said sweetly, "Do you want to be friends?" wossnames thingummyThe Potter M:"...they called me...do you know what they called me?" wossnames thingummyThe Potter "90% of a cup of coffee is the smell. And this is 200% coffee, Jamaican Blue Mountain."

All-Time Award for the Worst Acting in the History of Ever. Book Name Character Name One or more lines describing the Moment

Book Character First line that described the Moment Tabs HPMoR
Harry Potter Harry's Internal Critic promptly awarded h…

All-Time Award for the Worst Acting in the History of Ever. Book Name Character Name One or more lines describing the Moment Book Character First line that described the Moment Tabs HPMoR Harry Potter Harry's Internal Critic promptly awarded h…

idea from    TVTropes.org

SQLite FTW?

GROUP BY book, character ORDER BY book, character

# After writing @lines out to tempfile: @lines = `sort
-nr -k 5,7 tempfile`; # …versus keeping it in Perl: @lines = { $b->[4] <=> $a->[4] or $b->[6] <=> $a->[6] } @lines;

Summary should be… • clustered by Book • clustered by
Character • •

Remaining data looks like… • • • multiple (kept-in-order) Moments
• multiple (kept-in-order) lines of text per Moment.

HoHoAoA • clustered by Book • clustered by Character •
multiple (kept-in-order) Moments • multiple (kept-in-order) lines of text per Moment.

%main{book}{char}[moment_num][line_num]

Took a Level in Badass

use 5.010; $/ = ''; # Paragraph mode my %book_char_lines_HoHoAoA;
while (<>) { chomp; my ( $book, $character, @lines ) = split "\n"; push @{ $book_char_lines_HoHoAoA{$book}{$character} }, [@lines]; } for my $book ( sort keys %book_char_lines_HoHoAoA ) { for my $char ( sort keys %{ $book_char_lines_HoHoAoA{$book} } ) { for my $aref ( @{ $book_char_lines_HoHoAoA{$book}{$char} } ) { say join "\t", $book, $char, $aref->[0]; } } say ''; }

use 5.024; $/ = ''; # Paragraph mode my %book_char_lines_HoHoAoA;
while (<>) { chomp; my ( $book, $character, @lines ) = split "\n"; push $book_char_lines_HoHoAoA{$book}{$character}->@*, [@lines]; } for my $book ( sort keys %book_char_lines_HoHoAoA ) { for my $char ( sort keys $book_char_lines_HoHoAoA{$book}->%* ) { for my $aref ( $book_char_lines_HoHoAoA{$book}{$char}->@* ) { say join "\t", $book, $char, $aref->[0]; } } say ''; }

my %book_char_lines_HoHoAoA; for 'cmoa.txt'.IO.slurp.split("\n\n") { my ( $book, $character, @lines
) = .split: "\n"; push %book_char_lines_HoHoAoA{$book}{$character}, [@lines]; } for %book_char_lines_HoHoAoA .keys.sort -> $book { for %book_char_lines_HoHoAoA{$book} .keys.sort -> $char { for %book_char_lines_HoHoAoA{$book}{$char}.list -> @lines { say join "\t", $book, $char, @lines[0]; } } say ''; }

my %book_char_lines_HoHoAoA; for 'cmoa.txt'.IO.slurp.split("\n\n") { my ( $book, $character, @lines
) = .split: "\n"; push %book_char_lines_HoHoAoA{$book}{$character}, [@lines]; } # More efficient. Less readable, or more readable? for %book_char_lines_HoHoAoA.sort -> (:key($book), :value(%book_hash )) { for %book_hash\ .sort -> (:key($char), :value(@char_array)) { for @char_array -> @lines { say join "\t", $book, $char, @lines[0]; } } say ''; }

If that sounds good…

Deep DS Resources • https://perldoc.perl.org/perldsc.html  Perl: see also: perllol, perlref,
and perlreftut.html • https://www.perlmonks.org/?node_id=172833  Auto-Viviﬁcation Explanation • https://web.stanford.edu/class/archive/cs/cs106ap/ cs106ap.1198/lectures/15-NestedCollections/15- Nested_Data_Structures.pdf  Python: Nested DS starts at page 70

%main{book}{char}[moment_num][line_num]

Trees in the wild • Objects (Nodes) • Direction (Edges)
• Hierarchy (Parent/Child) • Order (Siblings) • Navigation (Methods)

<html><head><title>Doc 1</title></head> <body> Stuff <hr> 2000-08-17 </body></html>

<html><head><title>Doc 1</title></head> <body> Stuff <hr> 2000-08-17 </body></html> <html> <head> <title>
Doc 1 </title> </head> <body> Stuff <hr> 2000-08-17 </body> </html>

<html><head><title>Doc 1</title></head> <body> Stuff <hr> 2000-08-17 </body></html> • html •
head • title • "Doc 1" • body • "Stuﬀ" • hr • "2000-08-17" <html> <head> <title> Doc 1 </title> </head> <body> Stuff <hr> 2000-08-17 </body> </html>

<html><head><title>Doc 1</title></head> <body> Stuff <hr> 2000-08-17 </body></html> • html •
head • title • "Doc 1" • body • "Stuﬀ" • hr • "2000-08-17" html / \ head body / / | \ title "Stuff" hr "2000-08-17" | "Doc 1" <html> <head> <title> Doc 1 </title> </head> <body> Stuff <hr> 2000-08-17 </body> </html>

html / \ head body / / | \ title
"Stuff" hr "2000-08-17" | "Doc 1"

<html><head><title>Doc 1</title></head> <body> Stuff <hr> 2000-08-17 </body></html>

curl -o weather.html 'https://www.wunderground.com/weather/us/al/auburn'

<div class="condition-wind small-6 medium-12 columns"> <div class="wind-compass-wrap"> <div class="wind-compass" style="transform:rotate(313deg);">
<div class="dial"> <div class="arrow-direction"> </div> </div> </div> <div class="wind-north"> N </div> <strong> 0 </strong> </div> <p class="ng-star-inserted"> Gusts <strong> <span class="test-false wu-unit wu-unit-speed"> <span class="wu-value wu-value-to" style="color:;"> 2 </span> <span class="wu-label"> <span class="ng-star-inserted"> mph </span> </span> </span> </strong> </div> </div>

<div class="dial"> <div class="arrow-direction"> </div> </div> </div> <div class="wind-north"> N </div> <strong> 0 </strong> </div> <p class="ng-star-inserted"> Gusts <strong> <span class="test-false wu-unit wu-unit-speed"> <span class="wu-value wu-value-to" style="color:;"> 2 </span> <span class="wu-label"> <span class="ng-star-inserted"> mph </span> </span> </span> </strong> </div>  </div>

<div class="dial"> <div class="arrow-direction"> </div> </div> </div> <div class="wind-north"> N </div> <strong> 0 </strong> </div> <p class="ng-star-inserted"> Gusts <strong> <span class="test-false wu-unit wu-unit-speed"> <span class="wu-value wu-value-to" style="color:;"> 2 </span> <span class="wu-label"> <span class="ng-star-inserted"> mph </span> </span> </span> </strong> </div> </div>

<div class="dial"> <div class="arrow-direction"> </div> </div> </div> <div class="wind-north"> N </div> <strong> 0 </strong> </div> <p class="ng-star-inserted"> Gusts <strong> <span class="test-false wu-unit wu-unit-speed"> <span class="wu-value wu-value-to" style="color:;"> 2 </span> <span class="wu-label"> <span class="ng-star-inserted"> mph </span> </span> </span> </strong> </div>

use Modern::Perl; use File::Slurp; $_ = read_file('weather.html'); s{\A.+<div [^>]*class="condition-wind[^>]*>((?:.+?</div>){6}).+\z}{$1}ms; s{</?[^>]+>}{
}msg; s{ } { }msg; tr{ }{}s; s{^\s+}{}; s{\s+$}{}; say;

N 0 Gusts 2 mph

my $from_div_to_strong = / '<div ' <-[>]>* 'class="condition-wind' <-[>]>* '>'
[ .+? '</div>' ] ** 6 /; my $tag_or_nbsp = / '<' '/'? <-[>]>+ '>' | ' ' /; say 'weather.html'.IO.slurp\ .match( $from_div_to_strong )\ .subst( $tag_or_nbsp, ' ', :global )\ .trans( ' ' => ' ', :squash )\ .trim;

use TokeParser;

<strong class="Humongulus" > primary clank in this location </strong>

<strong class="Humongulus"> primary clank in this location </strong>

use Modern::Perl; use HTML::TokeParser; my $p = HTML::TokeParser->new('weather.html') or die
"Can't open: $!"; $p->empty_element_tags(1); my $in_wind = 0; my $level = 0; my @texts; # this_loop_is_on_the_next_slide(); s/ / /g for @texts; say @texts;

while ( my $t = $p->get_token ) { my $type
= $t->[0]; if ( $type eq 'S' ) { my ( $tag, $attr ) = @{$t}[1,2]; $in_wind = 1 if $tag eq 'div' and ($attr->{class} // '') =~ /^condition-wind /; $level++ if $in_wind; } elsif ( $type eq 'E' and $in_wind ) { my $tag = $t->[1]; $level--; $in_wind = 0 if $level == 0; } elsif ( $type eq 'T' and $in_wind ) { push @texts, $t->[1]; } }

while ( my $t = $p->get_token ) { last if
$t->[0] eq 'S' and $t->[1] eq 'div' and($t->[2]{class} // '') =~ /^condition-wind /; } my ( $level, @texts ) = (1); while ( my $t = $p->get_token ) { if ( $t->[0] eq 'S' ) { $level++ } elsif ( $t->[0] eq 'E' ) { $level--; last if !$level; } elsif ( $t->[0] eq 'T' ) { push @texts, $t->[1]; } }

use HTML::TreeBuilder;

use Modern::Perl; use HTML::TreeBuilder; my $tree = HTML::TreeBuilder->new_from_file('weather.html') or die;

my $wind = $tree->look_down(

my $wind = $tree->look_down( _tag => 'div',

my $wind = $tree->look_down( _tag => 'div', class => qr/^condition-wind /, );

my $wind = $tree->look_down( _tag => 'div', class => qr/^condition-wind /, ); $wind

my $wind = $tree->look_down( _tag => 'div', class => qr/^condition-wind /, ); $wind->as_trimmed_text

my $wind = $tree->look_down( _tag => 'div', class => qr/^condition-wind /, ); $wind->as_trimmed_text( extra_chars => '\xA0' );

my $wind = $tree->look_down( _tag => 'div', class => qr/^condition-wind /, ); say $wind->as_trimmed_text( extra_chars => '\xA0' );

N 0 Gusts 2 mph

my $wind = $tree->look_down( _tag => 'div', class => qr/^condition-wind /, ); say $wind->as_trimmed_text( extra_chars => '\xA0' );

Micro-languages • Strings • Regex • Trees • XPath •
JSON • jq • JSONPath

Tree Resources • https://metacpan.org/pod/HTML::Tree::Scanning • https://www.perlmonks.org/?node_id=153259  Introduction to the more
generic Tree::DAG_Node • https://perlhacks.com/2014/04/data-munging-perl/  Book: Data Munging with Perl  Now freely available for download! • https://docs.python-guide.org/scenarios/scrape/ • https://stackoverﬂow.com/questions/14172028/html- parse-tree-using-python-2-7  Answers also for Javascript

>>> >>> <<<

Bonus Round

Static Analysis • Lint (and other linters) • Scanners for
style issues and common bugs: • Perl::Critic • RubyCritic • Rubocop • Prospector  • Scanners for security issues: • Brakeman • Bandit • Assisted Refactoring • PyCharm • ReSharper

style issues and common bugs: • Perl::Critic • RubyCritic • Rubocop • Prospector  • Scanners for security issues: • Brakeman • Bandit • Assisted Refactoring • PyCharm • ReSharper • Automated Refactoring • Blue Tiger

>>> >>> <<<

Thanks!

Copyrights

Copyright Information: Images • Camelia • (c) 2009 by Larry
Wall  http://github.com/perl6/mu/raw/master/misc/ camelia.txt • Regular Expressions • © Randall Munroe  https://xkcd.com/208/ • Brantley Grayson Wolters • (c) 2011 by his mother, Amy Wolters  Util 1.1.1 is my eldest grandchild  (in RCS numbering)

Copyright Information: This Talk This work is licensed under a
Creative Commons Attribution 4.0 International License. CC BY https://creativecommons.org/licenses/by/4.0/ (email me for the original Apple Keynote .key ﬁle)

History • v 1.01 2020-06-12  Presented at Southeast LinuxFest  60
minutes with Q&A • v 1.02 2020-06-24  Presented at Perl & Raku Conference in the Cloud  50 minutes with Q&A

Removed  (Not presented,   but maybe worth reading)

Key knowledge (Perl) • List::Util first max min sum all
any none uniq shuffle • List::UtilsBy max_by min_by count_by • perlfunc Perl Functions by Category • Test::Tutorial Intro to Automated Testing • Devel::Cover Shows code that lacks testing • Benchmark Performance comparisons • Devel::NYTProf Perfomance profiler

<div class="dial"> <div class="arrow-direction"> <div class="wind-north"> "N" <strong> "0" <p class="ng-star-inserted"> " Gusts " <strong> <span class="test-false wu-unit wu-unit-speed ng-star-inserted"> <span class="wu-value wu-value-to" style="color:;"> "2" <span class="wu-label"> <span class="ng-star-inserted"> "mph"

XXX If that sounds good…

style issues and common bugs: • Perl::Critic • RubyCritic • Rubocop • Prospector  • Scanners for security issues: • Brakeman • Bandit • Assisted Refactoring • PyCharm • ReSharper • Automated Refactoring • Blue Tiger

• TODO: • Add Regex history • Credit Damian on
P6 Regex • Review all long slide text, and split

Raku ==   Perl 5 minus Warts plus Awesome

Refactoring_and_Readability_-_CRHS_-_Perl_Confe...

Refactoring_and_Readability_-_CRHS_-_Perl_Conference_20200624.pdf

More Decks by Bruce Gray

Featured

Transcript