Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
embulkを利用した開発でちょっと困ったこと
Search
katsuyan
May 17, 2017
Technology
1
490
embulkを利用した開発でちょっと困ったこと
embulk meetup #3で発表した
embulkを利用した開発でちょっと困ったこと
katsuyan
May 17, 2017
Tweet
Share
More Decks by katsuyan
See All by katsuyan
realtime_marketing_system
katsuyan
0
2.6k
digdag-3tips
katsuyan
2
4.6k
ファッションチェックランキングRubyKaigiの裏側 / Fashion check ranking app for RubyKaigi2019
katsuyan
1
5.9k
ETLツールを1から作り直した話
katsuyan
1
320
Other Decks in Technology
See All in Technology
Kubernetes における cgroup v2 でのOut-Of-Memory 問題の解決
pfn
PRO
0
440
見てわかるテスト駆動開発
recruitengineers
PRO
6
2.4k
Kiroと学ぶコンテキストエンジニアリング
oikon48
5
5.6k
DeNA での思い出 / Memories at DeNA
orgachem
PRO
6
1.9k
トヨタ生産方式(TPS)入門
recruitengineers
PRO
6
1.4k
Grafana MCPサーバーによるAIエージェント経由でのGrafanaダッシュボード動的生成
hamadakoji
1
1k
カミナシ社の『ID管理基盤』製品内製 - その意思決定背景と2年間の進化 #AWSUnicornDay / Kaminashi ID - The Big Whys
kaminashi
3
720
kubellが考える戦略と実行を繋ぐ活用ファーストのデータ分析基盤
kubell_hr
0
120
Webブラウザ向け動画配信プレイヤーの 大規模リプレイスから得た知見と学び
yud0uhu
0
130
退屈なことはDevinにやらせよう〜〜Devin APIを使ったVisual Regression Testの自動追加〜
kawamataryo
4
1.1k
シークレット管理だけじゃない!HashiCorp Vault でデータ暗号化をしよう / Beyond Secret Management! Let's Encrypt Data with HashiCorp Vault
nnstt1
2
130
DuckDB-Wasmを使って ブラウザ上でRDBMSを動かす
hacusk
1
140
Featured
See All Featured
Writing Fast Ruby
sferik
628
62k
It's Worth the Effort
3n
187
28k
The World Runs on Bad Software
bkeepers
PRO
70
11k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
139
34k
The Pragmatic Product Professional
lauravandoore
36
6.8k
Being A Developer After 40
akosma
90
590k
Six Lessons from altMBA
skipperchong
28
4k
Practical Tips for Bootstrapping Information Extraction Pipelines
honnibal
PRO
23
1.4k
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
irinanazarova
8
910
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
31
2.2k
Intergalactic Javascript Robots from Outer Space
tanoku
272
27k
Chrome DevTools: State of the Union 2024 - Debugging React & Beyond
addyosmani
7
830
Transcript
&NCVMLΛར༻ͨ͠։ൃ Ͱͪΐͬͱࠔͬͨ͜ͱ !LBUTVZBO
{:name “Katsuya Tajima” :twitter“@katsuyan121” :school {:name“芝浦工大” :year 4} :langs [:Clojure
:Go :Ruby] :likes [:Aikido :Shibainu]} XIPBNJ
࡞ͬͨͷ
Γ͍ͨ͜ͱ 3%4 .Z42- #JH2VFSZ .Z42-͔Β#JH2VFSZͷσʔλಉظ
Γ͍ͨ͜ͱ 3%4 .Z42- #JH2VFSZ .Z42-͔Β#JH2VFSZͷσʔλಉظ ͜ΕΛ&NCVML ͰΓ͍ͨ
FNCVMLͰͰ͖ͳ͍͜ͱ FNCVMLςʔϒϧ୯ҐͰసૹ ෳͷςʔϒϧಉظʹରԠͰ͖ͳ͍
#JH2VFSZ 3%4 .Z42- (Pϥούʔ࡞ FYFD
ຊ
ࠔͬͨ͜ͱᶃ
&NCVMLͷΤϥʔ͕TUEPVUʹ $ embulk run sample.yml 2> error.log 2017-05-15 18:16:38.400
+0900: Embulk v0.8.18 2017-05-15 18:16:50.038 +0900 [INFO] (0001:transaction): Loaded plugin embulk-input-mysql (0.8.2) 2017-05-15 18:16:52.789 +0900 [INFO] (0001:transaction): Loaded plugin embulk-output-bigquery (0.4.5) 2017-05-15 18:16:52.820 +0900 [INFO] (0001:transaction): Fetch size is 10000. Using server-side prepared statement. org.embulk.exec.PartialExecutionException: java.lang.RuntimeException: . . . $ cat error.log $ TUEPVUʹΤϥʔ
ղܾࡦ ऴྃεςʔλεΛར༻ ϩάͷৼΓ͚Λ(PͰཧ &NCVMLͷϩάͦͷ··ར༻ Τϥʔൃੜ࣌TUEFSSʹग़ྗ
ࠔͬͨ͜ͱᶄ
ಉظ࣌࣌ؒͣΕΔ FNCVMLJOQVUKECD .Z42-ϓϥάΠϯΛར༻ ϓϥάΠϯ͕ࣗಈͰ.Z42-ͷ UJNF[POFΛऔಘ͠ͳ͍͜ͱ͕ݪҼ
ಉظ࣌࣌ؒͣΕΔ in: type: mysql host: localhost user: user password:
password database: sample table: sample select: "*" default_timezone: "Asia/Tokyo" column_options: create_time: {type: string, timestamp_format: "%Y-%m-%d %H:%M:%S%:z"} update_time: {type: string, timestamp_format: "%Y-%m-%d %H:%M:%S%:z"} out: . . . TBNQMFZNM Θ͔Γ͍͢Α͏ʹ "TJB5PLZP࣌ؒ ʹͦΖ͑ͯ·͢
ಉظ࣌࣌ؒͣΕΔ mysql> select * from sample; +------+----------+---------------------+ | id
| name | create_time | +------+----------+---------------------+ | 1 | sample1 | 2013-05-27 10:46:08 | | 2 | sample2 | 2013-05-27 10:46:08 | | 3 | sample3 | 2013-05-27 10:46:08 | | 4 | sample4 | 2013-05-27 10:46:08 | | 5 | sample5 | 2013-05-27 10:46:08 | | 6 | sample6 | 2013-05-27 10:46:08 | | 7 | sample7 | 2013-05-27 10:46:08 | | 8 | sample8 | 2013-05-27 10:46:08 | | 9 | sample9 | 2013-05-27 10:46:08 | | 10 | sample10 | 2013-05-27 10:46:08 | +------+----------+---------------------+ $ embulk preview sample.yml +---------+-------------+---------------------------+ | id:long | name:string | create_time:timestamp | +---------+-------------+---------------------------+ | 1 | sample1 | 2013-05-27 19:46:08+09:00 | | 2 | sample2 | 2013-05-27 19:46:08+09:00 | | 3 | sample3 | 2013-05-27 19:46:08+09:00 | | 4 | sample4 | 2013-05-27 19:46:08+09:00 | | 5 | sample5 | 2013-05-27 19:46:08+09:00 | | 6 | sample6 | 2013-05-27 19:46:08+09:00 | | 7 | sample7 | 2013-05-27 19:46:08+09:00 | | 8 | sample8 | 2013-05-27 19:46:08+09:00 | | 9 | sample9 | 2013-05-27 19:46:08+09:00 | | 10 | sample10 | 2013-05-27 19:46:08+09:00 | +---------+-------------+---------------------------+ NZTRM FNCVMLQSFWJFX ͪΐ͏Ͳ࣌ؒ ͣΕͯΔ
Կ͔Φϓγϣϯͳ͍͔ͳ ͳΜ͔ͬͯͳ ͍ͭ͋ͬͨʂ
ղܾࡦ in: type: mysql host: localhost user: user password:
password database: sample table: sample select: "*" options: {useTimezone: true, serverTimezone: Asia/Tokyo} default_timezone: "Asia/Tokyo" column_options: create_time: {type: string, timestamp_format: "%Y-%m-%d %H:%M:%S%:z"} update_time: {type: string, timestamp_format: "%Y-%m-%d %H:%M:%S%:z"} out: . . . ͍ͭ͜ΛՃ TBNQMFZNM
ղܾ mysql> select * from sample; +------+----------+---------------------+ | id
| name | create_time | +------+----------+---------------------+ | 1 | sample1 | 2013-05-27 10:46:08 | | 2 | sample2 | 2013-05-27 10:46:08 | | 3 | sample3 | 2013-05-27 10:46:08 | | 4 | sample4 | 2013-05-27 10:46:08 | | 5 | sample5 | 2013-05-27 10:46:08 | | 6 | sample6 | 2013-05-27 10:46:08 | | 7 | sample7 | 2013-05-27 10:46:08 | | 8 | sample8 | 2013-05-27 10:46:08 | | 9 | sample9 | 2013-05-27 10:46:08 | | 10 | sample10 | 2013-05-27 10:46:08 | +------+----------+---------------------+ $ embulk preview sample.yml +---------+-------------+---------------------------+ | id:long | name:string | create_time:timestamp | +---------+-------------+---------------------------+ | 1 | sample1 | 2013-05-27 10:46:08+09:00 | | 2 | sample2 | 2013-05-27 10:46:08+09:00 | | 3 | sample3 | 2013-05-27 10:46:08+09:00 | | 4 | sample4 | 2013-05-27 10:46:08+09:00 | | 5 | sample5 | 2013-05-27 10:46:08+09:00 | | 6 | sample6 | 2013-05-27 10:46:08+09:00 | | 7 | sample7 | 2013-05-27 10:46:08+09:00 | | 8 | sample8 | 2013-05-27 10:46:08+09:00 | | 9 | sample9 | 2013-05-27 10:46:08+09:00 | | 10 | sample10 | 2013-05-27 10:46:08+09:00 | +---------+-------------+---------------------------+ NZTRM FNCVMLQSFWJFX ͪΌΜͱಉظ͞Εͨ
+%#$ͷΦϓγϣϯ ݹ͍Φϓγϣϯ VTF5JNF[POFUSVF TFSWFS5JNF[POF"TJB5PLZP ৽͍͠Φϓγϣϯ VTF-FHBDZ%BUFUJNF$PEFGBMTF ͜Ε͚ͩͰ ղܾͰ͖ͳ͔ͬͨ
ࢀߟ FNCVMLJOQVUKECDͷ.Z42-ϓϥά ΠϯͰ͕࣌ؒ࣌ؒͣΕΔ IUUQRJJUBDPNLBUTVZBOJUFNT GEDBCBD
·ͱΊ ϩάͷग़ྗઌ֬ೝ͠·͠ΐ͏ ࣌ؒͷҋਂ͍ Φϓγϣϯશ෦ோΊ·͠ΐ͏ &NCVMLศར