Write ETL or ELT data processing jobs with bricolage. @inohiro at Cookpad Inc.
 RubyKaigi 2019 LT 

self.introduce •@inohiro on Twitter and GitHub •Struggling over data business in Cookpad Inc. 

ETL and ELT https://www.quora.com/Why-are-most-companies-moving-from-ETL-to-ELT SFG 

bricolage •A batch job framework ‣ Designed to work with AWS services but works with PostgreSQL also ‣ For both ETL and ELT, especially ELT ‣ Written in Ruby https://github.com/bricolages/bricolage 

Why bricolage •Simple and flexible ‣ Especially if you write SQL mainly •But we also can use Ruby 

Job •Is written in SQL, Ruby, or even execs other scripts •Declined a job class ‣Help to write frequent patterns jobs ‣load, unload, insert, insert-delta, createview, exec, rebuild-drop, rebuild-rename, adhoc, … •You can define dependencies of jobs as jobnet(s) 

Let’s see a rebuild-rename job 

Rebuild a summary table with backup 

What bricolage does Drop old and temporary tables
 that created previously Create new table and 
 insert summaries ✨ Swap new and old tables ♻ 

Good points for rubyists •Write transformation scripts in Ruby ‣ Use Bricolage::CommandLineApplication class •Define your own useful methods and use them with ERB 

Thank you •Try to use bricolage •Cookpad is looking for data engineers. Let’s talk $