Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Write ETL or ELT data processing jobs with bricolage.

Write ETL or ELT data processing jobs with bricolage.

Introduce bricolage that's a batch job framework used in Cookpad Inc. Also explain ETL and ETL data processing briefly.

464f7715290af1c0cd41e6bf9d1ee8cc?s=128

Hiroyuki Inoue

April 19, 2019
Tweet

Transcript

  1. Write ETL or ELT data processing jobs with bricolage. @inohiro

    at Cookpad Inc.
 RubyKaigi 2019 LT
  2. self.introduce •@inohiro on Twitter and GitHub •Struggling over data business

    in Cookpad Inc.
  3. ETL and ELT https://www.quora.com/Why-are-most-companies-moving-from-ETL-to-ELT SFG

  4. bricolage •A batch job framework ‣ Designed to work with

    AWS services but works with PostgreSQL also ‣ For both ETL and ELT, especially ELT ‣ Written in Ruby https://github.com/bricolages/bricolage
  5. Why bricolage •Simple and flexible ‣ Especially if you write

    SQL mainly •But we also can use Ruby
  6. Job •Is written in SQL, Ruby, or even execs other

    scripts •Declined a job class ‣Help to write frequent patterns jobs ‣load, unload, insert, insert-delta, createview, exec, rebuild-drop, rebuild-rename, adhoc, … •You can define dependencies of jobs as jobnet(s)
  7. Let’s see a rebuild-rename job

  8. Rebuild a summary table with backup

  9. What bricolage does Drop old and temporary tables
 that created

    previously Create new table and 
 insert summaries ✨ Swap new and old tables ♻
  10. Good points for rubyists •Write transformation scripts in Ruby ‣

    Use Bricolage::CommandLineApplication class •Define your own useful methods and use them with ERB
  11. Thank you •Try to use bricolage •Cookpad is looking for

    data engineers. Let’s talk $