ETLツールを1から作り直した話

3e2285cb3972dc9a48000153aa3a0805?s=47 katsuyan
September 02, 2017

 ETLツールを1から作り直した話

ETLツールを1から作り直した話(embulkを利用したETLツール)

3e2285cb3972dc9a48000153aa3a0805?s=128

katsuyan

September 02, 2017
Tweet

Transcript

  1. © 2017 VASILY,Inc. &5-πʔϧΛ͔Β࡞Γ௚ͨ͠࿩ ୈճֶੜΤϯδχΞݶఆ-5େձʂʂʂ ాౡࠀ࠸

  2. © 2017 VASILY,Inc. ࣗݾ঺հ {:name “Katsuya Tajima” :twitter“@katsuyan121” :school {:name“芝浦工大”

    :year 4} :intern “VASILY,Inc.” :langs [:Clojure :Go :Ruby] :likes [:Aikido :Shibainu]}
  3. © 2017 VASILY,Inc. ໊ࣾגࣜձࣾ7"4*-: ϰΝγϦʔ 7"4*-: *OD ઃཱ೥݄ ॴࡏ஍౦ژ౎඼઒۠੢ޒ൓ాδχΞεϏϧ' ैۀһ໊

    ࢿຊۚԯ
 ୅දऔక໾ۚࢁ༟थ
 औక໾ࠓଜխ޾ઍ༿େี
 ओཁגओ άϩʔϏεϕϯνϟʔΩϟϐλϧ ҏ౻஧ςΫϊϩδʔϕϯνϟʔζ (.0ϕϯνϟʔύʔτφʔζ ,%%*גࣜձࣾ גࣜձࣾߨஊࣾ
  4. © 2017 VASILY,Inc. Ҏ্ͷϑΝογϣϯ&$αΠτ͔Βͷ΂ສ఺Λ௒͑Δ঎඼Λܝࡌ ݄ؒສਓҎ্͕ར༻͢Δ೔ຊ࠷େڃͷϑΝογϣϯαΠτ

  5. © 2017 VASILY,Inc. Google Play ϕετΞϓϦ J04ΞϓϦ ɾ"QQ4UPSF#&45 ɾ&TTFOUJBMೝఆ "OESPJEΞϓϦ

    ɾ೥(PPHMF1MBZϕετΞϓϦ ɾ೥(PPHMF1MBZϕετΞϓϦ ɾ೥(PPHMF1MBZϕετΞϓϦ
 ɹɹϕετΠϊϕʔςΟϒΞϓϦେ৆ ɾ(PPHMF͔Βτοϓσϕϩούʔೝఆ ೥࿈ଓϕετΞϓϦड৆͸ ੈքͰ7"4*-:͚ͩ
  6. © 2017 VASILY,Inc. 7"4*-:ΠϯλʔϯͰ΍͖ͬͯͨ͜ͱ ‣ &5-πʔϧͷ࡞Γ௚͠ ‣ %#Ҡߦ .POHP%#ͷ๾໓ 

    ‣ $JSDMF$*΁ͷΞοϓσʔτ ‣ ΫϩʔϥʔͷҰ෦ίϯςφԽ
  7. © 2017 VASILY,Inc. 7"4*-:ΠϯλʔϯͰ΍͖ͬͯͨ͜ͱ ‣ &5-πʔϧͷ࡞Γ௚͠ ‣ %#Ҡߦ .POHP%#ͷ๾໓ 

    ‣ $JSDMF$*΁ͷΞοϓσʔτ ‣ ΫϩʔϥʔͷҰ෦ίϯςφԽ
  8. © 2017 VASILY,Inc. &5-ͱ͸ ‣ &YUSBDU5SBOTGPSN-PBE ‣ &YUSBDU৘ใݯ͔ΒσʔλΛநग़ ‣ 5SBOTGPSNσʔλΛม׵ɾՃ޻

    ‣ -PBE֨ೲઌʹσʔλΛϩʔυ
  9. © 2017 VASILY,Inc. ࠓճ΍Γ͔ͨͬͨ͜ͱ 3%4 .Z42- ͔Β#JH2VFSZ΁ͷσʔλಉظ

  10. © 2017 VASILY,Inc. چ&5-πʔϧ σʔλಉظͷྲྀΕ .Z42-͔ΒεΩʔϚऔಘ #JH2VFSZʹσʔλಉظ 'JMFJDPO*DPOTNBEFCZ'SFFQJLGSPNXXXqBUJDPODPNJTMJDFOTFECZ'MBUJDPO#BTJD-JDFOTF

  11. © 2017 VASILY,Inc. چ&5-πʔϧ σʔλಉظͷྲྀΕ .Z42-͔ΒεΩʔϚऔಘ #JH2VFSZʹσʔλಉظ σʔλͷ૿Ճʹ଱͑ΒΕ ͳ͘ͳͬͨ 'JMFJDPO*DPOTNBEFCZ'SFFQJLGSPNXXXqBUJDPODPNJTMJDFOTFECZ'MBUJDPO#BTJD-JDFOTF

  12. © 2017 VASILY,Inc. ৽&5-πʔϧ σʔλಉظͷྲྀΕ .Z42-͔ΒεΩʔϚऔಘ ઃఆϑΝΠϧΛ࡞੒ :".-  &NCVMLΛ࣮ߦͯ͠σʔλಉظ

    &NCVMLΛར༻ʂʂ 'JMFJDPO*DPOTNBEFCZ'SFFQJLGSPNXXXqBUJDPODPNJTMJDFOTFECZ'MBUJDPO#BTJD-JDFOTF :.-JDPO*DPOTNBEFCZ:BOOJDLGSPNXXXqBUJDPODPNJTMJDFOTFECZ$$#:
  13. © 2017 VASILY,Inc. ᶃ&NCVMLʹΑΔσʔλಉظ ͜ͷ෦෼⾪ :.-JDPO*DPOTNBEFCZ:BOOJDLGSPNXXXqBUJDPODPNJTMJDFOTFECZ$$#:

  14. © 2017 VASILY,Inc. &NCVMLͱ͸ ‣ σʔλసૹϑϨʔϜϫʔΫ ‣ 'MVFOUEͷόον൛ͷΑ͏ͳπʔϧ ‣ ϓϥάΠϯʹΑΓෳ਺ͷೖྗɾग़ྗʹରԠ

    ‣ ೖྗɾग़ྗ࣌ͷΤϥʔॲཧͳͲΛߦͬͯ͘ΕΔ
  15. © 2017 VASILY,Inc. &NCVMLͷ࢖͍ํ ‣ :".-ܗࣜͰઃఆϑΝΠϧهड़ ‣ &NCMVLΛ࣮ߦ ‣ FNCVMLSVOʙZNM

    :.-JDPO*DPOTNBEFCZ:BOOJDLGSPNXXXqBUJDPODPNJTMJDFOTFECZ$$#:
  16. © 2017 VASILY,Inc. &NCVMLઃఆϑΝΠϧͷྫ in: type: mysql host: ϗετ໊ user:

    Ϣʔβʔ໊ password: ύεϫʔυ database: ର৅ͷσʔλϕʔε table: ର৅ͷςʔϒϧ select: "*" out: type: bigquery mode: replace auth_method: json_key json_keyfile: json_keyfile΁ͷύε project: ϓϩδΣΫτ໊ dataset: σʔληοτ໊ table: ςʔϒϧ໊ gcs_bucket: GCSͷόέοτ໊ auto_create_gcs_bucket: false compression: GZIP source_format: NEWLINE_DELIMITED_JSON .Z42-ˠ#JH2VFSZͷ৔߹ FNCVMLJOQVUNZTRM FNCVMLPVUQVUCJHRVFSZ
  17. © 2017 VASILY,Inc. ᶄ(Pϥούʔ ͜ͷ෦෼⾪ 'JMFJDPO*DPOTNBEFCZ'SFFQJLGSPNXXXqBUJDPODPNJTMJDFOTFECZ'MBUJDPO#BTJD-JDFOTF :.-JDPO*DPOTNBEFCZ:BOOJDLGSPNXXXqBUJDPODPNJTMJDFOTFECZ$$#:

  18. © 2017 VASILY,Inc. &NCVML͚ͩͰ࣮ݱͰ͖ͳ͍͜ͱ ‣ ෳ਺ςʔϒϧͷಉظ ‣ ςʔϒϧʹରͯͭ͠ઃఆϑΝΠϧ͕ඞཁ ‣ ॊೈͳϩάऩू

    ‣ Τϥʔൃੜ࣌ͷ௨஌ॲཧ
  19. © 2017 VASILY,Inc. (Pϥούʔͷ໾ׂ ‣ ઃఆϑΝΠϧͷੜ੒ ‣ &NCVMLͷ࣮ߦ ‣ ϩάऩू

    ‣ Τϥʔൃੜ࣌ͷॲཧ 'JMFJDPO*DPOTNBEFCZ'SFFQJLGSPNXXXqBUJDPODPNJTMJDFOTFECZ'MBUJDPO#BTJD-JDFOTF :.-JDPO*DPOTNBEFCZ:BOOJDLGSPNXXXqBUJDPODPNJTMJDFOTFECZ$$#: 4BMBEJDPO*DPOTNBEFCZ.BEFCZPMJWFSGSPNXXXqBUJDPODPNJTMJDFOTFECZ'MBUJDPO#BTJD-JDFOTF
  20. © 2017 VASILY,Inc. ࠷ऴߏ੒ 'JMFJDPO*DPOTNBEFCZ'SFFQJLGSPNXXXqBUJDPODPNJTMJDFOTFECZ'MBUJDPO#BTJD-JDFOTF :.-JDPO*DPOTNBEFCZ:BOOJDLGSPNXXXqBUJDPODPNJTMJDFOTFECZ$$#: 4BMBEJDPO*DPOTNBEFCZ.BEFCZPMJWFSGSPNXXXqBUJDPODPNJTMJDFOTFECZ'MBUJDPO#BTJD-JDFOTF

  21. © 2017 VASILY,Inc. ΋ͬͱศརʹ IUUQUFDIWBTJMZKQFOUSZEJHEBH@BOE@FNCVML

  22. © 2017 VASILY,Inc. Πϯλʔϯืूத IUUQTXXXXBOUFEMZDPNQSPKFDUT