ETLツールを1から作り直した話

3e2285cb3972dc9a48000153aa3a0805?s=47 katsuyan
September 02, 2017
Technology
1
260

 ETLツールを1から作り直した話

ETLツールを1から作り直した話(embulkを利用したETLツール)

3e2285cb3972dc9a48000153aa3a0805?s=128

katsuyan

September 02, 2017
Tweet

More Decks by katsuyan

See All by katsuyan
3e2285cb3972dc9a48000153aa3a0805?s=48 katsuyan
0
570
3e2285cb3972dc9a48000153aa3a0805?s=48 katsuyan
1
710
3e2285cb3972dc9a48000153aa3a0805?s=48 katsuyan
1
3.7k
3e2285cb3972dc9a48000153aa3a0805?s=48 katsuyan
1
330

Other Decks in Technology

See All in Technology
Cb88f765dd38cd942c39aacb5abbea06?s=48 ufoo68
0
110
Ac85568dfa460d48c36095d142632b1a?s=48 wps
0
130
88f4e84b94fe07cddbd9e6479d689192?s=48 soudai
1
310
57e8d22493c141fd3c7e7dbe62196c1b?s=48 uenitty
1
250
0ced4f52ccc9e9b2644b4dd7f475e545?s=48 kkojima
0
140
55c9804c185f493816433678393cbe92?s=48 sumihiro3
0
140
F01c8fdfa64bdfc00a94896e0b1359e0?s=48 tmdoi
0
240
Cd931bc05e7cee46b9a950a63e47ba4c?s=48 you
0
110
0b28f5164b540e5a31bc63cca39b2917?s=48 curry9999
0
140
703c34db2db0e609fddec6a5585ceccc?s=48 kenta_sato3
1
120
C174e1ef0c746f53d989b1902b4e674e?s=48 govargo
11
1.8k
8a84268593355816432ceaf78777d585?s=48 dena_tech
0
240

Featured

See All Featured
B3d6434763caa0ef5dc4b792662c49f7?s=48 smashingmag
228
17k
96270e4c3e5e9806cf7245475c00b275?s=48 addyosmani
308
21k
C4de88ea47538e4594750e3861a341c8?s=48 brettharned
91
2.8k
5b6a2bd86ef643786a381a212e0ef2f1?s=48 geoffreycrofte
10
220
6804f1775cb4babfcc3851298566fbce?s=48 tanoku
85
8.3k
3ab1249be442027903e1180025340b3f?s=48 reverentgeek
165
6.8k
D67fff74178013024d833b3eec6cf27f?s=48 lynnandtonic
265
16k
32de0bd2ba869609d26fd052a4622778?s=48 cromwellryan
100
5.7k
11c84dff30266b907714802088852677?s=48 jasonvnalue
80
7.9k
A16eb159db895e3b01d3dc95767ad595?s=48 jonyablonski
6
600
245cee81a9c424266e5e401d844ea881?s=48 lara
13
2.3k
6804f1775cb4babfcc3851298566fbce?s=48 tanoku
257
24k

Transcript

  1. © 2017 VASILY,Inc. &5-πʔϧΛ͔Β࡞Γ௚ͨ͠࿩ ୈճֶੜΤϯδχΞݶఆ-5େձʂʂʂ ాౡࠀ࠸

  2. © 2017 VASILY,Inc. ࣗݾ঺հ {:name “Katsuya Tajima” :twitter“@katsuyan121” :school {:name“芝浦工大”

    :year 4} :intern “VASILY,Inc.” :langs [:Clojure :Go :Ruby] :likes [:Aikido :Shibainu]}

  3. © 2017 VASILY,Inc. ໊ࣾגࣜձࣾ7"4*-: ϰΝγϦʔ 7"4*-: *OD ઃཱ೥݄ ॴࡏ஍౦ژ౎඼઒۠੢ޒ൓ాδχΞεϏϧ' ैۀһ໊

    ࢿຊۚԯ
 ୅දऔక໾ۚࢁ༟थ
 औక໾ࠓଜխ޾ઍ༿େี
 ओཁגओ άϩʔϏεϕϯνϟʔΩϟϐλϧ ҏ౻஧ςΫϊϩδʔϕϯνϟʔζ (.0ϕϯνϟʔύʔτφʔζ ,%%*גࣜձࣾ גࣜձࣾߨஊࣾ

  4. © 2017 VASILY,Inc. Ҏ্ͷϑΝογϣϯ&$αΠτ͔Βͷ΂ສ఺Λ௒͑Δ঎඼Λܝࡌ ݄ؒສਓҎ্͕ར༻͢Δ೔ຊ࠷େڃͷϑΝογϣϯαΠτ

  5. © 2017 VASILY,Inc. Google Play ϕετΞϓϦ J04ΞϓϦ ɾ"QQ4UPSF#&45 ɾ&TTFOUJBMೝఆ "OESPJEΞϓϦ

    ɾ೥(PPHMF1MBZϕετΞϓϦ ɾ೥(PPHMF1MBZϕετΞϓϦ ɾ೥(PPHMF1MBZϕετΞϓϦ
 ɹɹϕετΠϊϕʔςΟϒΞϓϦେ৆ ɾ(PPHMF͔Βτοϓσϕϩούʔೝఆ ೥࿈ଓϕετΞϓϦड৆͸ ੈքͰ7"4*-:͚ͩ

  6. © 2017 VASILY,Inc. 7"4*-:ΠϯλʔϯͰ΍͖ͬͯͨ͜ͱ ‣ &5-πʔϧͷ࡞Γ௚͠ ‣ %#Ҡߦ .POHP%#ͷ๾໓ 

    ‣ $JSDMF$*΁ͷΞοϓσʔτ ‣ ΫϩʔϥʔͷҰ෦ίϯςφԽ

  7. © 2017 VASILY,Inc. 7"4*-:ΠϯλʔϯͰ΍͖ͬͯͨ͜ͱ ‣ &5-πʔϧͷ࡞Γ௚͠ ‣ %#Ҡߦ .POHP%#ͷ๾໓ 

    ‣ $JSDMF$*΁ͷΞοϓσʔτ ‣ ΫϩʔϥʔͷҰ෦ίϯςφԽ

  8. © 2017 VASILY,Inc. &5-ͱ͸ ‣ &YUSBDU5SBOTGPSN-PBE ‣ &YUSBDU৘ใݯ͔ΒσʔλΛநग़ ‣ 5SBOTGPSNσʔλΛม׵ɾՃ޻

    ‣ -PBE֨ೲઌʹσʔλΛϩʔυ

  9. © 2017 VASILY,Inc. ࠓճ΍Γ͔ͨͬͨ͜ͱ 3%4 .Z42- ͔Β#JH2VFSZ΁ͷσʔλಉظ

  10. © 2017 VASILY,Inc. چ&5-πʔϧ σʔλಉظͷྲྀΕ .Z42-͔ΒεΩʔϚऔಘ #JH2VFSZʹσʔλಉظ 'JMFJDPO*DPOTNBEFCZ'SFFQJLGSPNXXXqBUJDPODPNJTMJDFOTFECZ'MBUJDPO#BTJD-JDFOTF

  11. © 2017 VASILY,Inc. چ&5-πʔϧ σʔλಉظͷྲྀΕ .Z42-͔ΒεΩʔϚऔಘ #JH2VFSZʹσʔλಉظ σʔλͷ૿Ճʹ଱͑ΒΕ ͳ͘ͳͬͨ 'JMFJDPO*DPOTNBEFCZ'SFFQJLGSPNXXXqBUJDPODPNJTMJDFOTFECZ'MBUJDPO#BTJD-JDFOTF

  12. © 2017 VASILY,Inc. ৽&5-πʔϧ σʔλಉظͷྲྀΕ .Z42-͔ΒεΩʔϚऔಘ ઃఆϑΝΠϧΛ࡞੒ :".-  &NCVMLΛ࣮ߦͯ͠σʔλಉظ

    &NCVMLΛར༻ʂʂ 'JMFJDPO*DPOTNBEFCZ'SFFQJLGSPNXXXqBUJDPODPNJTMJDFOTFECZ'MBUJDPO#BTJD-JDFOTF :.-JDPO*DPOTNBEFCZ:BOOJDLGSPNXXXqBUJDPODPNJTMJDFOTFECZ$$#:

  13. © 2017 VASILY,Inc. ᶃ&NCVMLʹΑΔσʔλಉظ ͜ͷ෦෼⾪ :.-JDPO*DPOTNBEFCZ:BOOJDLGSPNXXXqBUJDPODPNJTMJDFOTFECZ$$#:

  14. © 2017 VASILY,Inc. &NCVMLͱ͸ ‣ σʔλసૹϑϨʔϜϫʔΫ ‣ 'MVFOUEͷόον൛ͷΑ͏ͳπʔϧ ‣ ϓϥάΠϯʹΑΓෳ਺ͷೖྗɾग़ྗʹରԠ

    ‣ ೖྗɾग़ྗ࣌ͷΤϥʔॲཧͳͲΛߦͬͯ͘ΕΔ

  15. © 2017 VASILY,Inc. &NCVMLͷ࢖͍ํ ‣ :".-ܗࣜͰઃఆϑΝΠϧهड़ ‣ &NCMVLΛ࣮ߦ ‣ FNCVMLSVOʙZNM

    :.-JDPO*DPOTNBEFCZ:BOOJDLGSPNXXXqBUJDPODPNJTMJDFOTFECZ$$#:

  16. © 2017 VASILY,Inc. &NCVMLઃఆϑΝΠϧͷྫ in: type: mysql host: ϗετ໊ user:

    Ϣʔβʔ໊ password: ύεϫʔυ database: ର৅ͷσʔλϕʔε table: ର৅ͷςʔϒϧ select: "*" out: type: bigquery mode: replace auth_method: json_key json_keyfile: json_keyfile΁ͷύε project: ϓϩδΣΫτ໊ dataset: σʔληοτ໊ table: ςʔϒϧ໊ gcs_bucket: GCSͷόέοτ໊ auto_create_gcs_bucket: false compression: GZIP source_format: NEWLINE_DELIMITED_JSON .Z42-ˠ#JH2VFSZͷ৔߹ FNCVMLJOQVUNZTRM FNCVMLPVUQVUCJHRVFSZ

  17. © 2017 VASILY,Inc. ᶄ(Pϥούʔ ͜ͷ෦෼⾪ 'JMFJDPO*DPOTNBEFCZ'SFFQJLGSPNXXXqBUJDPODPNJTMJDFOTFECZ'MBUJDPO#BTJD-JDFOTF :.-JDPO*DPOTNBEFCZ:BOOJDLGSPNXXXqBUJDPODPNJTMJDFOTFECZ$$#:

  18. © 2017 VASILY,Inc. &NCVML͚ͩͰ࣮ݱͰ͖ͳ͍͜ͱ ‣ ෳ਺ςʔϒϧͷಉظ ‣ ςʔϒϧʹରͯͭ͠ઃఆϑΝΠϧ͕ඞཁ ‣ ॊೈͳϩάऩू

    ‣ Τϥʔൃੜ࣌ͷ௨஌ॲཧ

  19. © 2017 VASILY,Inc. (Pϥούʔͷ໾ׂ ‣ ઃఆϑΝΠϧͷੜ੒ ‣ &NCVMLͷ࣮ߦ ‣ ϩάऩू

    ‣ Τϥʔൃੜ࣌ͷॲཧ 'JMFJDPO*DPOTNBEFCZ'SFFQJLGSPNXXXqBUJDPODPNJTMJDFOTFECZ'MBUJDPO#BTJD-JDFOTF :.-JDPO*DPOTNBEFCZ:BOOJDLGSPNXXXqBUJDPODPNJTMJDFOTFECZ$$#: 4BMBEJDPO*DPOTNBEFCZ.BEFCZPMJWFSGSPNXXXqBUJDPODPNJTMJDFOTFECZ'MBUJDPO#BTJD-JDFOTF

  20. © 2017 VASILY,Inc. ࠷ऴߏ੒ 'JMFJDPO*DPOTNBEFCZ'SFFQJLGSPNXXXqBUJDPODPNJTMJDFOTFECZ'MBUJDPO#BTJD-JDFOTF :.-JDPO*DPOTNBEFCZ:BOOJDLGSPNXXXqBUJDPODPNJTMJDFOTFECZ$$#: 4BMBEJDPO*DPOTNBEFCZ.BEFCZPMJWFSGSPNXXXqBUJDPODPNJTMJDFOTFECZ'MBUJDPO#BTJD-JDFOTF

  21. © 2017 VASILY,Inc. ΋ͬͱศརʹ IUUQUFDIWBTJMZKQFOUSZEJHEBH@BOE@FNCVML

  22. © 2017 VASILY,Inc. Πϯλʔϯืूத IUUQTXXXXBOUFEMZDPNQSPKFDUT