Write ETL or ELT data processing jobs with bricolage.

Write ETL or ELT data processing jobs with bricolage.

Introduce bricolage that's a batch job framework used in Cookpad Inc. Also explain ETL and ETL data processing briefly.

464f7715290af1c0cd41e6bf9d1ee8cc?s=128

Hiroyuki Inoue

April 19, 2019
Tweet

Transcript

  1. 1.
  2. 4.

    bricolage •A batch job framework ‣ Designed to work with

    AWS services but works with PostgreSQL also ‣ For both ETL and ELT, especially ELT ‣ Written in Ruby https://github.com/bricolages/bricolage
  3. 5.
  4. 6.

    Job •Is written in SQL, Ruby, or even execs other

    scripts •Declined a job class ‣Help to write frequent patterns jobs ‣load, unload, insert, insert-delta, createview, exec, rebuild-drop, rebuild-rename, adhoc, … •You can define dependencies of jobs as jobnet(s)
  5. 9.

    What bricolage does Drop old and temporary tables
 that created

    previously Create new table and 
 insert summaries ✨ Swap new and old tables ♻
  6. 10.

    Good points for rubyists •Write transformation scripts in Ruby ‣

    Use Bricolage::CommandLineApplication class •Define your own useful methods and use them with ERB