Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Write ETL or ELT data processing jobs with bricolage.

Write ETL or ELT data processing jobs with bricolage.

Introduce bricolage that's a batch job framework used in Cookpad Inc. Also explain ETL and ETL data processing briefly.

Hiroyuki Inoue

April 19, 2019
Tweet

More Decks by Hiroyuki Inoue

Other Decks in Programming

Transcript

  1. bricolage •A batch job framework ‣ Designed to work with

    AWS services but works with PostgreSQL also ‣ For both ETL and ELT, especially ELT ‣ Written in Ruby https://github.com/bricolages/bricolage
  2. Job •Is written in SQL, Ruby, or even execs other

    scripts •Declined a job class ‣Help to write frequent patterns jobs ‣load, unload, insert, insert-delta, createview, exec, rebuild-drop, rebuild-rename, adhoc, … •You can define dependencies of jobs as jobnet(s)
  3. What bricolage does Drop old and temporary tables
 that created

    previously Create new table and 
 insert summaries ✨ Swap new and old tables ♻
  4. Good points for rubyists •Write transformation scripts in Ruby ‣

    Use Bricolage::CommandLineApplication class •Define your own useful methods and use them with ERB