Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
ETLツールを1から作り直した話
Search
katsuyan
September 02, 2017
Technology
1
310
ETLツールを1から作り直した話
ETLツールを1から作り直した話(embulkを利用したETLツール)
katsuyan
September 02, 2017
Tweet
Share
More Decks by katsuyan
See All by katsuyan
realtime_marketing_system
katsuyan
0
2.5k
digdag-3tips
katsuyan
2
4.3k
ファッションチェックランキングRubyKaigiの裏側 / Fashion check ranking app for RubyKaigi2019
katsuyan
1
5.7k
embulkを利用した開発でちょっと困ったこと
katsuyan
1
470
Other Decks in Technology
See All in Technology
アジャイルな開発チームでテスト戦略の話は誰がする? / Who Talks About Test Strategy?
ak1210
1
850
20250309 無冠のわたし これからどう先生きのこれる?
akiko_pusu
1
150
開発者体験を定量的に把握する手法と活用事例
ham0215
0
140
プロダクト開発者目線での Entra ID 活用
sansantech
PRO
0
140
Охота на косуль у древних
ashapiro
0
130
【内製開発Summit 2025】イオンスマートテクノロジーの内製化組織の作り方/In-house-development-summit-AST
aeonpeople
2
1.1k
【Snowflake九州ユーザー会#2】BigQueryとSnowflakeを比較してそれぞれの良し悪しを掴む / BigQuery vs Snowflake: Pros & Cons
civitaspo
3
1.1k
JAWS FESTA 2024「バスロケ」GPS×サーバーレスの開発と運用の舞台裏/jawsfesta2024-bus-gps-serverless
ma2shita
3
370
フォーイット_エンジニア向け会社紹介資料_Forit_Company_Profile.pdf
forit_tech
1
1.7k
Amazon Athenaから利用時のGlueのIcebergテーブルのメンテナンスについて
nayuts
0
120
x86-64 Assembly Essentials
latte72
4
470
【Forkwell】「正しく」失敗できるチームを作る──現場のリーダーのための恐怖と不安を乗り越える技術 - FL#83 / A team that can fail correctly by forkwell
i35_267
2
120
Featured
See All Featured
The Language of Interfaces
destraynor
156
24k
BBQ
matthewcrist
87
9.5k
The Cult of Friendly URLs
andyhume
78
6.2k
Chrome DevTools: State of the Union 2024 - Debugging React & Beyond
addyosmani
4
380
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
134
33k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
47
5.2k
YesSQL, Process and Tooling at Scale
rocio
172
14k
The World Runs on Bad Software
bkeepers
PRO
67
11k
Being A Developer After 40
akosma
89
590k
Designing on Purpose - Digital PM Summit 2013
jponch
117
7.1k
How STYLIGHT went responsive
nonsquared
99
5.4k
Mobile First: as difficult as doing things right
swwweet
223
9.5k
Transcript
© 2017 VASILY,Inc. &5-πʔϧΛ͔Β࡞Γͨ͠ ୈճֶੜΤϯδχΞݶఆ-5େձʂʂʂ ాౡࠀ࠸
© 2017 VASILY,Inc. ࣗݾհ {:name “Katsuya Tajima” :twitter“@katsuyan121” :school {:name“芝浦工大”
:year 4} :intern “VASILY,Inc.” :langs [:Clojure :Go :Ruby] :likes [:Aikido :Shibainu]}
© 2017 VASILY,Inc. ໊ࣾגࣜձࣾ7"4*-: ϰΝγϦʔ 7"4*-: *OD ઃཱ݄ ॴࡏ౦ژ۠ޒాδχΞεϏϧ' ैۀһ໊
ࢿຊۚԯ දऔకۚࢁ༟थ औకࠓଜխઍ༿େี ओཁגओ άϩʔϏεϕϯνϟʔΩϟϐλϧ ҏ౻ςΫϊϩδʔϕϯνϟʔζ (.0ϕϯνϟʔύʔτφʔζ ,%%*גࣜձࣾ גࣜձࣾߨஊࣾ
© 2017 VASILY,Inc. Ҏ্ͷϑΝογϣϯ&$αΠτ͔ΒͷສΛ͑ΔΛܝࡌ ݄ؒສਓҎ্͕ར༻͢Δຊ࠷େڃͷϑΝογϣϯαΠτ
© 2017 VASILY,Inc. Google Play ϕετΞϓϦ J04ΞϓϦ ɾ"QQ4UPSF#&45 ɾ&TTFOUJBMೝఆ "OESPJEΞϓϦ
ɾ(PPHMF1MBZϕετΞϓϦ ɾ(PPHMF1MBZϕετΞϓϦ ɾ(PPHMF1MBZϕετΞϓϦ ɹɹϕετΠϊϕʔςΟϒΞϓϦେ ɾ(PPHMF͔Βτοϓσϕϩούʔೝఆ ࿈ଓϕετΞϓϦड ੈքͰ7"4*-:͚ͩ
© 2017 VASILY,Inc. 7"4*-:ΠϯλʔϯͰ͖ͬͯͨ͜ͱ ‣ &5-πʔϧͷ࡞Γ͠ ‣ %#Ҡߦ .POHP%#ͷ໓
‣ $JSDMF$*ͷΞοϓσʔτ ‣ ΫϩʔϥʔͷҰ෦ίϯςφԽ
© 2017 VASILY,Inc. 7"4*-:ΠϯλʔϯͰ͖ͬͯͨ͜ͱ ‣ &5-πʔϧͷ࡞Γ͠ ‣ %#Ҡߦ .POHP%#ͷ໓
‣ $JSDMF$*ͷΞοϓσʔτ ‣ ΫϩʔϥʔͷҰ෦ίϯςφԽ
© 2017 VASILY,Inc. &5-ͱ ‣ &YUSBDU5SBOTGPSN-PBE ‣ &YUSBDUใݯ͔ΒσʔλΛநग़ ‣ 5SBOTGPSNσʔλΛมɾՃ
‣ -PBE֨ೲઌʹσʔλΛϩʔυ
© 2017 VASILY,Inc. ࠓճΓ͔ͨͬͨ͜ͱ 3%4 .Z42- ͔Β#JH2VFSZͷσʔλಉظ
© 2017 VASILY,Inc. چ&5-πʔϧ σʔλಉظͷྲྀΕ .Z42-͔ΒεΩʔϚऔಘ #JH2VFSZʹσʔλಉظ 'JMFJDPO*DPOTNBEFCZ'SFFQJLGSPNXXXqBUJDPODPNJTMJDFOTFECZ'MBUJDPO#BTJD-JDFOTF
© 2017 VASILY,Inc. چ&5-πʔϧ σʔλಉظͷྲྀΕ .Z42-͔ΒεΩʔϚऔಘ #JH2VFSZʹσʔλಉظ σʔλͷ૿Ճʹ͑ΒΕ ͳ͘ͳͬͨ 'JMFJDPO*DPOTNBEFCZ'SFFQJLGSPNXXXqBUJDPODPNJTMJDFOTFECZ'MBUJDPO#BTJD-JDFOTF
© 2017 VASILY,Inc. ৽&5-πʔϧ σʔλಉظͷྲྀΕ .Z42-͔ΒεΩʔϚऔಘ ઃఆϑΝΠϧΛ࡞ :".- &NCVMLΛ࣮ߦͯ͠σʔλಉظ
&NCVMLΛར༻ʂʂ 'JMFJDPO*DPOTNBEFCZ'SFFQJLGSPNXXXqBUJDPODPNJTMJDFOTFECZ'MBUJDPO#BTJD-JDFOTF :.-JDPO*DPOTNBEFCZ:BOOJDLGSPNXXXqBUJDPODPNJTMJDFOTFECZ$$#:
© 2017 VASILY,Inc. ᶃ&NCVMLʹΑΔσʔλಉظ ͜ͷ෦⾪ :.-JDPO*DPOTNBEFCZ:BOOJDLGSPNXXXqBUJDPODPNJTMJDFOTFECZ$$#:
© 2017 VASILY,Inc. &NCVMLͱ ‣ σʔλసૹϑϨʔϜϫʔΫ ‣ 'MVFOUEͷόον൛ͷΑ͏ͳπʔϧ ‣ ϓϥάΠϯʹΑΓෳͷೖྗɾग़ྗʹରԠ
‣ ೖྗɾग़ྗ࣌ͷΤϥʔॲཧͳͲΛߦͬͯ͘ΕΔ
© 2017 VASILY,Inc. &NCVMLͷ͍ํ ‣ :".-ܗࣜͰઃఆϑΝΠϧهड़ ‣ &NCMVLΛ࣮ߦ ‣ FNCVMLSVOʙZNM
:.-JDPO*DPOTNBEFCZ:BOOJDLGSPNXXXqBUJDPODPNJTMJDFOTFECZ$$#:
© 2017 VASILY,Inc. &NCVMLઃఆϑΝΠϧͷྫ in: type: mysql host: ϗετ໊ user:
Ϣʔβʔ໊ password: ύεϫʔυ database: ରͷσʔλϕʔε table: ରͷςʔϒϧ select: "*" out: type: bigquery mode: replace auth_method: json_key json_keyfile: json_keyfileͷύε project: ϓϩδΣΫτ໊ dataset: σʔληοτ໊ table: ςʔϒϧ໊ gcs_bucket: GCSͷόέοτ໊ auto_create_gcs_bucket: false compression: GZIP source_format: NEWLINE_DELIMITED_JSON .Z42-ˠ#JH2VFSZͷ߹ FNCVMLJOQVUNZTRM FNCVMLPVUQVUCJHRVFSZ
© 2017 VASILY,Inc. ᶄ(Pϥούʔ ͜ͷ෦⾪ 'JMFJDPO*DPOTNBEFCZ'SFFQJLGSPNXXXqBUJDPODPNJTMJDFOTFECZ'MBUJDPO#BTJD-JDFOTF :.-JDPO*DPOTNBEFCZ:BOOJDLGSPNXXXqBUJDPODPNJTMJDFOTFECZ$$#:
© 2017 VASILY,Inc. &NCVML͚ͩͰ࣮ݱͰ͖ͳ͍͜ͱ ‣ ෳςʔϒϧͷಉظ ‣ ςʔϒϧʹରͯͭ͠ઃఆϑΝΠϧ͕ඞཁ ‣ ॊೈͳϩάऩू
‣ Τϥʔൃੜ࣌ͷ௨ॲཧ
© 2017 VASILY,Inc. (Pϥούʔͷׂ ‣ ઃఆϑΝΠϧͷੜ ‣ &NCVMLͷ࣮ߦ ‣ ϩάऩू
‣ Τϥʔൃੜ࣌ͷॲཧ 'JMFJDPO*DPOTNBEFCZ'SFFQJLGSPNXXXqBUJDPODPNJTMJDFOTFECZ'MBUJDPO#BTJD-JDFOTF :.-JDPO*DPOTNBEFCZ:BOOJDLGSPNXXXqBUJDPODPNJTMJDFOTFECZ$$#: 4BMBEJDPO*DPOTNBEFCZ.BEFCZPMJWFSGSPNXXXqBUJDPODPNJTMJDFOTFECZ'MBUJDPO#BTJD-JDFOTF
© 2017 VASILY,Inc. ࠷ऴߏ 'JMFJDPO*DPOTNBEFCZ'SFFQJLGSPNXXXqBUJDPODPNJTMJDFOTFECZ'MBUJDPO#BTJD-JDFOTF :.-JDPO*DPOTNBEFCZ:BOOJDLGSPNXXXqBUJDPODPNJTMJDFOTFECZ$$#: 4BMBEJDPO*DPOTNBEFCZ.BEFCZPMJWFSGSPNXXXqBUJDPODPNJTMJDFOTFECZ'MBUJDPO#BTJD-JDFOTF
© 2017 VASILY,Inc. ͬͱศརʹ IUUQUFDIWBTJMZKQFOUSZEJHEBH@BOE@FNCVML
© 2017 VASILY,Inc. Πϯλʔϯืूத IUUQTXXXXBOUFEMZDPNQSPKFDUT