ETLツールを1から作り直した話

ETLツールを1から作り直した話(embulkを利用したETLツール)

Transcript

1. © 2017 VASILY,Inc. &5-πʔϧΛ͔Β࡞Γ௚ͨ͠࿩ ୈճ
   ֶੜΤϯδχΞݶఆ-5େձʂʂʂ
   ాౡ
   ࠀ࠸
2. © 2017 VASILY,Inc. ࣗݾ঺հ {:name “Katsuya Tajima” :twitter“@katsuyan121” :school {:name“芝浦工大” :year 4} :intern “VASILY,Inc.” :langs [:Clojure :Go :Ruby] :likes [:Aikido :Shibainu]}
3. © 2017 VASILY,Inc. ໊ࣾ
   גࣜձࣾ7"4*-: ϰΝγϦʔ
   
   7"4*-: *OD
   ઃཱ
   ೥݄
   ॴࡏ஍
   ౦ژ౎඼઒۠੢ޒ൓ాδχΞεϏϧ
   '
   ैۀһ
   ໊
   ࢿຊۚ
   ԯ
 ୅දऔక໾
   ۚࢁ
   ༟थ
 औక໾ࠓଜ
   խ޾
   
   ઍ༿
   େี
 ओཁגओ
   
   άϩʔϏεϕϯνϟʔΩϟϐλϧ
   ҏ౻஧ςΫϊϩδʔϕϯνϟʔζ
   (.0ϕϯνϟʔύʔτφʔζ
   ,%%*גࣜձࣾ
   גࣜձࣾߨஊࣾ
4. © 2017 VASILY,Inc. Ҏ্ͷϑΝογϣϯ&$αΠτ͔Βͷ΂ສ఺Λ௒͑Δ঎඼Λܝࡌ ݄ؒສਓҎ্͕ར༻͢Δ೔ຊ࠷େڃͷϑΝογϣϯαΠτ
5. © 2017 VASILY,Inc. Google Play ϕετΞϓϦ J04ΞϓϦ
   ɾ"QQ4UPSF
   #&45
   
   ɾ&TTFOUJBMೝఆ "OESPJEΞϓϦ
   ɾ೥
   (PPHMF
   1MBZ
   ϕετΞϓϦ
   ɾ೥
   (PPHMF
   1MBZ
   ϕετΞϓϦ
   ɾ೥
   (PPHMF
   1MBZ
   ϕετΞϓϦ
 ɹɹ
   ϕετΠϊϕʔςΟϒΞϓϦେ৆
   ɾ(PPHMF͔Β
   τοϓσϕϩούʔೝఆ ೥࿈ଓϕετΞϓϦड৆͸
   ੈքͰ7"4*-:͚ͩ
6. © 2017 VASILY,Inc. 7"4*-:ΠϯλʔϯͰ΍͖ͬͯͨ͜ͱ ‣ &5-πʔϧͷ࡞Γ௚͠
   ‣ %#Ҡߦ .POHP%#ͷ๾໓
   ‣ $JSDMF$*΁ͷΞοϓσʔτ
   ‣ ΫϩʔϥʔͷҰ෦ίϯςφԽ
7. © 2017 VASILY,Inc. 7"4*-:ΠϯλʔϯͰ΍͖ͬͯͨ͜ͱ ‣ &5-πʔϧͷ࡞Γ௚͠
   ‣ %#Ҡߦ .POHP%#ͷ๾໓
   ‣ $JSDMF$*΁ͷΞοϓσʔτ
   ‣ ΫϩʔϥʔͷҰ෦ίϯςφԽ
8. © 2017 VASILY,Inc. &5-ͱ͸ ‣ &YUSBDU5SBOTGPSN-PBE
   ‣ &YUSBDU
   ৘ใݯ͔ΒσʔλΛநग़
   ‣ 5SBOTGPSN
   σʔλΛม׵ɾՃ޻
   ‣ -PBE
   ֨ೲઌʹσʔλΛϩʔυ
9. © 2017 VASILY,Inc. ࠓճ΍Γ͔ͨͬͨ͜ͱ 3%4 .Z42- ͔Β#JH2VFSZ΁ͷσʔλಉظ
10. © 2017 VASILY,Inc. چ&5-πʔϧ σʔλಉظͷྲྀΕ
    
    .Z42-͔ΒεΩʔϚऔಘ
    
    #JH2VFSZʹσʔλಉظ 'JMF
    JDPO
    *DPOT
    NBEF
    CZ
    'SFFQJL
    GSPN
    XXXqBUJDPODPN
    JT
    MJDFOTFE
    CZ
    'MBUJDPO
    #BTJD
    -JDFOTF
11. © 2017 VASILY,Inc. چ&5-πʔϧ σʔλಉظͷྲྀΕ
    
    .Z42-͔ΒεΩʔϚऔಘ
    
    #JH2VFSZʹσʔλಉظ σʔλͷ૿Ճʹ଱͑ΒΕ ͳ͘ͳͬͨ 'JMF
    JDPO
    *DPOT
    NBEF
    CZ
    'SFFQJL
    GSPN
    XXXqBUJDPODPN
    JT
    MJDFOTFE
    CZ
    'MBUJDPO
    #BTJD
    -JDFOTF
12. © 2017 VASILY,Inc. ৽&5-πʔϧ σʔλಉظͷྲྀΕ
    
    .Z42-͔ΒεΩʔϚऔಘ
    
    ઃఆϑΝΠϧΛ࡞੒ :".-
    
    &NCVMLΛ࣮ߦͯ͠σʔλಉظ &NCVMLΛར༻ʂʂ 'JMF
    JDPO
    *DPOT
    NBEF
    CZ
    'SFFQJL
    GSPN
    XXXqBUJDPODPN
    JT
    MJDFOTFE
    CZ
    'MBUJDPO
    #BTJD
    -JDFOTF
    :.-
    JDPO
    *DPOT
    NBEF
    CZ
    :BOOJDL
    GSPN
    XXXqBUJDPODPN
    JT
    MJDFOTFE
    CZ
    $$
    
    #:
13. © 2017 VASILY,Inc. ᶃ&NCVMLʹΑΔσʔλಉظ ͜ͷ෦෼⾪ :.-
    JDPO
    *DPOT
    NBEF
    CZ
    :BOOJDL
    GSPN
    XXXqBUJDPODPN
    JT
    MJDFOTFE
    CZ
    $$
    
    #:
14. © 2017 VASILY,Inc. &NCVMLͱ͸ ‣ σʔλసૹϑϨʔϜϫʔΫ
    ‣ 'MVFOUEͷόον൛ͷΑ͏ͳπʔϧ
    ‣ ϓϥάΠϯʹΑΓෳ਺ͷೖྗɾग़ྗʹରԠ
    ‣ ೖྗɾग़ྗ࣌ͷΤϥʔॲཧͳͲΛߦͬͯ͘ΕΔ
15. © 2017 VASILY,Inc. &NCVMLͷ࢖͍ํ ‣ :".-ܗࣜͰઃఆϑΝΠϧهड़
    ‣ &NCMVLΛ࣮ߦ
    ‣
    FNCVML
    SVO
    ʙZNM
    :.-
    JDPO
    *DPOT
    NBEF
    CZ
    :BOOJDL
    GSPN
    XXXqBUJDPODPN
    JT
    MJDFOTFE
    CZ
    $$
    
    #:
16. © 2017 VASILY,Inc. &NCVMLઃఆϑΝΠϧͷྫ in: type: mysql host: ϗετ໊ user: Ϣʔβʔ໊ password: ύεϫʔυ database: ର৅ͷσʔλϕʔε table: ର৅ͷςʔϒϧ select: "*" out: type: bigquery mode: replace auth_method: json_key json_keyfile: json_keyfile΁ͷύε project: ϓϩδΣΫτ໊ dataset: σʔληοτ໊ table: ςʔϒϧ໊ gcs_bucket: GCSͷόέοτ໊ auto_create_gcs_bucket: false compression: GZIP source_format: NEWLINE_DELIMITED_JSON .Z42-
    ˠ
    #JH2VFSZͷ৔߹ FNCVMLJOQVUNZTRM FNCVMLPVUQVUCJHRVFSZ
17. © 2017 VASILY,Inc. ᶄ(Pϥούʔ ͜ͷ෦෼⾪ 'JMF
    JDPO
    *DPOT
    NBEF
    CZ
    'SFFQJL
    GSPN
    XXXqBUJDPODPN
    JT
    MJDFOTFE
    CZ
    'MBUJDPO
    #BTJD
    -JDFOTF
    :.-
    JDPO
    *DPOT
    NBEF
    CZ
    :BOOJDL
    GSPN
    XXXqBUJDPODPN
    JT
    MJDFOTFE
    CZ
    $$
    
    #:
18. © 2017 VASILY,Inc. &NCVML͚ͩͰ࣮ݱͰ͖ͳ͍͜ͱ ‣ ෳ਺ςʔϒϧͷಉظ
    ‣ ςʔϒϧʹରͯͭ͠ઃఆϑΝΠϧ͕ඞཁ
    ‣ ॊೈͳϩάऩू
    ‣ Τϥʔൃੜ࣌ͷ௨஌ॲཧ
19. © 2017 VASILY,Inc. (Pϥούʔͷ໾ׂ ‣ ઃఆϑΝΠϧͷੜ੒
    ‣ &NCVMLͷ࣮ߦ
    ‣ ϩάऩू
    ‣ Τϥʔൃੜ࣌ͷॲཧ 'JMF
    JDPO
    *DPOT
    NBEF
    CZ
    'SFFQJL
    GSPN
    XXXqBUJDPODPN
    JT
    MJDFOTFE
    CZ
    'MBUJDPO
    #BTJD
    -JDFOTF
    :.-
    JDPO
    *DPOT
    NBEF
    CZ
    :BOOJDL
    GSPN
    XXXqBUJDPODPN
    JT
    MJDFOTFE
    CZ
    $$
    
    #:
    4BMBE
    JDPO
    *DPOT
    NBEF
    CZ
    .BEFCZPMJWFS
    GSPN
    XXXqBUJDPODPN
    JT
    MJDFOTFE
    CZ
    'MBUJDPO
    #BTJD
    -JDFOTF
20. © 2017 VASILY,Inc. ࠷ऴߏ੒ 'JMF
    JDPO
    *DPOT
    NBEF
    CZ
    'SFFQJL
    GSPN
    XXXqBUJDPODPN
    JT
    MJDFOTFE
    CZ
    'MBUJDPO
    #BTJD
    -JDFOTF
    :.-
    JDPO
    *DPOT
    NBEF
    CZ
    :BOOJDL
    GSPN
    XXXqBUJDPODPN
    JT
    MJDFOTFE
    CZ
    $$
    
    #:
    4BMBE
    JDPO
    *DPOT
    NBEF
    CZ
    .BEFCZPMJWFS
    GSPN
    XXXqBUJDPODPN
    JT
    MJDFOTFE
    CZ
    'MBUJDPO
    #BTJD
    -JDFOTF
21. © 2017 VASILY,Inc. ΋ͬͱศརʹ IUUQUFDIWBTJMZKQFOUSZEJHEBH@BOE@FNCVML
22. © 2017 VASILY,Inc. Πϯλʔϯืूத IUUQTXXXXBOUFEMZDPNQSPKFDUT