Upgrade to Pro — share decks privately, control downloads, hide ads and more …

ETLツールを1から作り直した話

Sponsored · Ship Features Fearlessly Turn features on and off without deploys. Used by thousands of Ruby developers.
Avatar for katsuyan katsuyan
September 02, 2017

 ETLツールを1から作り直した話

ETLツールを1から作り直した話(embulkを利用したETLツール)

Avatar for katsuyan

katsuyan

September 02, 2017
Tweet

More Decks by katsuyan

Other Decks in Technology

Transcript

  1. © 2017 VASILY,Inc. ࣗݾ঺հ {:name “Katsuya Tajima” :twitter“@katsuyan121” :school {:name“芝浦工大”

    :year 4} :intern “VASILY,Inc.” :langs [:Clojure :Go :Ruby] :likes [:Aikido :Shibainu]}
  2. © 2017 VASILY,Inc. ໊ࣾגࣜձࣾ7"4*-: ϰΝγϦʔ 7"4*-: *OD ઃཱ೥݄ ॴࡏ஍౦ژ౎඼઒۠੢ޒ൓ాδχΞεϏϧ' ैۀһ໊

    ࢿຊۚԯ
 ୅දऔక໾ۚࢁ༟थ
 औక໾ࠓଜխ޾ઍ༿େี
 ओཁגओ άϩʔϏεϕϯνϟʔΩϟϐλϧ ҏ౻஧ςΫϊϩδʔϕϯνϟʔζ (.0ϕϯνϟʔύʔτφʔζ ,%%*גࣜձࣾ גࣜձࣾߨஊࣾ
  3. © 2017 VASILY,Inc. Google Play ϕετΞϓϦ J04ΞϓϦ ɾ"QQ4UPSF#&45 ɾ&TTFOUJBMೝఆ "OESPJEΞϓϦ

    ɾ೥(PPHMF1MBZϕετΞϓϦ ɾ೥(PPHMF1MBZϕετΞϓϦ ɾ೥(PPHMF1MBZϕετΞϓϦ
 ɹɹϕετΠϊϕʔςΟϒΞϓϦେ৆ ɾ(PPHMF͔Βτοϓσϕϩούʔೝఆ ೥࿈ଓϕετΞϓϦड৆͸ ੈքͰ7"4*-:͚ͩ
  4. © 2017 VASILY,Inc. ৽&5-πʔϧ σʔλಉظͷྲྀΕ .Z42-͔ΒεΩʔϚऔಘ ઃఆϑΝΠϧΛ࡞੒ :".-  &NCVMLΛ࣮ߦͯ͠σʔλಉظ

    &NCVMLΛར༻ʂʂ 'JMFJDPO*DPOTNBEFCZ'SFFQJLGSPNXXXqBUJDPODPNJTMJDFOTFECZ'MBUJDPO#BTJD-JDFOTF :.-JDPO*DPOTNBEFCZ:BOOJDLGSPNXXXqBUJDPODPNJTMJDFOTFECZ$$#:
  5. © 2017 VASILY,Inc. &NCVMLͷ࢖͍ํ ‣ :".-ܗࣜͰઃఆϑΝΠϧهड़ ‣ &NCMVLΛ࣮ߦ ‣ FNCVMLSVOʙZNM

    :.-JDPO*DPOTNBEFCZ:BOOJDLGSPNXXXqBUJDPODPNJTMJDFOTFECZ$$#:
  6. © 2017 VASILY,Inc. &NCVMLઃఆϑΝΠϧͷྫ in: type: mysql host: ϗετ໊ user:

    Ϣʔβʔ໊ password: ύεϫʔυ database: ର৅ͷσʔλϕʔε table: ର৅ͷςʔϒϧ select: "*" out: type: bigquery mode: replace auth_method: json_key json_keyfile: json_keyfile΁ͷύε project: ϓϩδΣΫτ໊ dataset: σʔληοτ໊ table: ςʔϒϧ໊ gcs_bucket: GCSͷόέοτ໊ auto_create_gcs_bucket: false compression: GZIP source_format: NEWLINE_DELIMITED_JSON .Z42-ˠ#JH2VFSZͷ৔߹ FNCVMLJOQVUNZTRM FNCVMLPVUQVUCJHRVFSZ
  7. © 2017 VASILY,Inc. (Pϥούʔͷ໾ׂ ‣ ઃఆϑΝΠϧͷੜ੒ ‣ &NCVMLͷ࣮ߦ ‣ ϩάऩू

    ‣ Τϥʔൃੜ࣌ͷॲཧ 'JMFJDPO*DPOTNBEFCZ'SFFQJLGSPNXXXqBUJDPODPNJTMJDFOTFECZ'MBUJDPO#BTJD-JDFOTF :.-JDPO*DPOTNBEFCZ:BOOJDLGSPNXXXqBUJDPODPNJTMJDFOTFECZ$$#: 4BMBEJDPO*DPOTNBEFCZ.BEFCZPMJWFSGSPNXXXqBUJDPODPNJTMJDFOTFECZ'MBUJDPO#BTJD-JDFOTF