ETL in Raku

Convincing users to adopt Raku requires finding a niche. Turns out that Raku is an ideal language for designing frameworks: Pugins in a variety of languages are easily added and the basic language constructs make firewalling simple. This talk looks at an example of designing an ETL framework and a sample transform plugin. It provides an example that would be easily adaptable to Bioinformatics, an area with complicated ETL that currently is heavily invested in Perl.

Transcript

1. Sipping Objects from a Firehose: Raku for ETL Steven Lembark Workhorse Computing lembark@wrkhors.com
2. Raku has a lot of features. More than most people know. More than most people use. They fit togeher so nicely.
3. Raku has a lot of features. More than most people know. More than most people use. They fit togeher so nicely. Trick now is getting someone to use it.
4. Who wants them? Q: Who would care about the features we have? Handle complex, variable requrements. Generate both boilerplate and plugins. Needs parallel execution.
5. Complex Processing: ETL “Extract, Transform, and Load”
6. Complex Processing: ETL Feeding data warehouses.
7. Complex Processing: ETL Feeding data warehouses. Sounds simple: Extract some data. Transform it to local format. Load it into a database/-lake/-cesspool.
8. Complex Processing: ETL Feeding data warehouses. Sounds simple: Extract some data. Transform it to local format. Load it into a database/-lake/-cesspool. Catch: It ain’t.
9. Complex Processing: ETL Extraction fails. Transforms require complex, variable scrubbing logic. Loading has extensive bookkeeping.
10. Complex Processing: ETL The whole is messier than the sum of its failures. Failures propagate: Extraction create orhpan data. Transforms create unloadable data. Loading creates bookeeping errors.
11. Why this fits Raku “Modern programming” isn’t about programming. It’s about fitting pices into a ‘framework’. Smaller pieces are better. Raku provides useful tools for frameworks. Nice ways to integrate the pieces.
12. Raku features for ETL Functors gather+take lazy+hyper+map try+catch junctions smartmatch (~~)
13. Raku features for ETL Functors avoid exposing framework hooks. gather/take simplify data acquisition. lazy simplifies handling large datasets. Junctions encapsulate complex, multi-level logic. Smartmatch simplifies comparing messy data.
14. Raku features for ETL Orthogonal, additive constructs. More opportunities for declarative approaches. Fine-grained control over execution. Effective for ETL framework and plugins.
15. Raku features for ETL Well-integrated interface to external libraries. Allows leveraging c, Rust, etc, code. Speed Compatility. Portability.
16. Lifecycle Extraction is mosly boilerplate: Remote system access: curl, wget, ssh. Exception handling. Minimal configuration: remote location, authn.
17. Lifecycle Extraction is mosly boilerplate: try/catch logic. Shell commands: wget, curl, ssh, scp. HTTP, FTP endpoints.
18. Lifecycle Transforms are mostly logic: Grammars. Junctions. Handle plugins: gather/take Parallel, large data: lazy/hyper/map
19. Lifecycle Loading is mostly bookkeeping: Loading data. Loading metadata. Loading audit data. Parallel execution.
20. Lifecycle All wrapped in shell executables. Configuration. Command line. Process control. Error handling.
21. Basic framework Top level try block. Catches aborts. Framework methods daisy-chain via: sub X(ETL $etl ->ETL); try { ETL .new .prior .extract .transform .load .after } catch { # audit log for morgue. }
22. Basic framework Transform stage: Takes the post-extract object. Generates the transformed output. try { ETL .new .prior .extract .transform .load .after } catch { ... }
23. Basic framework Transform stage: read-record() & transform-row() are closures. else-blocks handle failure to open (undef). method transform( -> ETL ) { with Plugin.open-input( $.in-path ) -> &read-record { my \input_rows = lazy gather loop { read-record() or last }; with Plugin.open-output( $.out-path ) -> &tranform-row { input_rows .hyper( degree => $.threads ) .map ( { sink transform-row $_ } ); } else { … } } else { … } self }
24. Basic framework Transform stage: input_rows is a bare promise returned from the lazy gather. This calls the loop once each time a new record is required. method transform( -> ETL ) { with Plugin.open-input( $.in-path ) -> &read-record { my \input_rows = lazy gather loop { read-record() or last }; with Plugin.open-output( $.out-path ) -> &tranform-row { input_rows .hyper( degree => $.threads ) .map ( { sink transform-row $_ } ); } else { … } } else { … } self }
25. Basic framework Transform stage: input_rows has a threaded map. Passed the gathered rows from input_rows. method transform( -> ETL ) { with Plugin.open-input( $.in-path ) -> &read-record { my \input_rows = lazy gather loop { read-record() or last }; with Plugin.open-output( $.out-path ) -> &tranform-row { input_rows .hyper( degree => $.threads ) .map ( { sink transform-row $_ } ); } else { … } } else { … } self }
26. Basic framework Transform stage: the object is passed out for the next stage. method transform( -> ETL ) { with Plugin.open-input( $.in-path ) -> &read-record { my \input_rows = lazy gather loop { read-record() or last }; with Plugin.open-output( $.out-path ) -> &tranform-row { input_rows .hyper( degree => $.threads ) .map ( { sink transform-row $_ } ); } else { … } } else { … } self }
27. Basic plugin Plugin reader: open-input returns a closure. $in-path is a string: open-input has no access to the ETL object. method open-input ( Str $in-path -> Code ) { with $in-path.IO.open -> my $fh will end {.close} { sub( -> Str ) { $fh.lines } } else { die "Failed open: $in-path" } }
28. Basic plugin Plugin writer: Writes to gzip via “run(...).in”, gzip’s stdin. method open-output ( Str $path -> Code ) { with $path.IO.child( "$path.gz" ).open( :w ) -> $out will end {.close} { my $gzip = run ( < /bin/gzip ‑9 >, :out( $out ), :in ).in ; sub { $gzip.print( $^line ) } } else { die "Failed open: $path, $!" } }
29. Basic plugin Plugin writer: Returns a closure that writes to gzip. method open-output ( Str $path -> Code ) { with $path.IO.child( "$path.gz" ).open( :w ) -> $out will end {.close} { my $gzip = run ( < /bin/gzip ‑9 >, :out( $out ), :in ).in ; sub { $gzip.print( $^line ) } } else { die "Failed open: $path, $!" } }
30. Framework and Plugin Are firewalled: gather doesn’t specify the take. take doesn’t know about the gather. Closures accept strings, not ETL objects. Closures encapsulate their filehandles. The isolation simplifies both layers. One more reason Raku is nice for frameworks.
31. But wait, there’s MORE!!! Junctions: Encapsulate messy, layered logic. Smartmatch: Simple, flexible code. Works nicely with Junctions.
32. But wait, there’s MORE!!! Effective, fast external library interface. Make use of fast Rust, etc, lib’s. Re-cycle published interfaces. Use Raku framework to exec multi-language stages.
33. Who would use it? Bioinformatics: Grammars handle Truly Messy ™ formats. Smart match simplifies validation & testing: if( $protein ~~ $dna ) … if( $protein ~~ $junction ) … Common requirement for external lib’s. Common requirement for speed.
34. Who would use it Finance: High-volume with metadata & audit requirements. External libraries helpful. Simple plugins.
35. Summary ETL is a nice niche space for Raku: Reliable frameworks. Straightforward plugins. Flexible grammars & comparison tools. It can handle the level of messyness required.
36. Bedside Reading More reading on guts of ETL operations: https://speakerdeck.com/lembark/gathers-and-takes More reading on Raku in general: https://docs.raku.org/