embulkを利用した開発でちょっと困ったこと

 embulkを利用した開発でちょっと困ったこと

embulk meetup #3で発表した
embulkを利用した開発でちょっと困ったこと

3e2285cb3972dc9a48000153aa3a0805?s=128

katsuyan

May 17, 2017
Tweet

Transcript

  1. &NCVMLΛར༻ͨ͠։ൃ Ͱͪΐͬͱࠔͬͨ͜ͱ !LBUTVZBO 

  2. {:name “Katsuya Tajima” :twitter“@katsuyan121” :school {:name“芝浦工大” :year 4} :langs [:Clojure

    :Go :Ruby] :likes [:Aikido :Shibainu]} XIPBNJ
  3. ࡞ͬͨ΋ͷ 

  4. ΍Γ͍ͨ͜ͱ 3%4 .Z42- #JH2VFSZ .Z42-͔Β#JH2VFSZ΁ͷσʔλಉظ

  5. ΍Γ͍ͨ͜ͱ 3%4 .Z42- #JH2VFSZ .Z42-͔Β#JH2VFSZ΁ͷσʔλಉظ ͜ΕΛ&NCVML Ͱ΍Γ͍ͨ

  6. FNCVMLͰͰ͖ͳ͍͜ͱ FNCVML͸ςʔϒϧ୯ҐͰసૹ ෳ਺ͷςʔϒϧಉظʹରԠͰ͖ͳ͍

  7. #JH2VFSZ 3%4 .Z42- (Pϥούʔ࡞੒ FYFD

  8. ຊ୊ 

  9. ࠔͬͨ͜ͱᶃ 

  10. &NCVMLͷΤϥʔ͕TUEPVUʹ  $ embulk run sample.yml 2> error.log 2017-05-15 18:16:38.400

    +0900: Embulk v0.8.18 2017-05-15 18:16:50.038 +0900 [INFO] (0001:transaction): Loaded plugin embulk-input-mysql (0.8.2) 2017-05-15 18:16:52.789 +0900 [INFO] (0001:transaction): Loaded plugin embulk-output-bigquery (0.4.5) 2017-05-15 18:16:52.820 +0900 [INFO] (0001:transaction): Fetch size is 10000. Using server-side prepared statement. org.embulk.exec.PartialExecutionException: java.lang.RuntimeException: . . . $ cat error.log $ TUEPVUʹΤϥʔ
  11. ղܾࡦ ऴྃεςʔλεΛར༻ ϩάͷৼΓ෼͚Λ(PͰ؅ཧ &NCVMLͷϩά͸ͦͷ··ར༻ Τϥʔൃੜ࣌͸TUEFSSʹ΋ग़ྗ 

  12. ࠔͬͨ͜ͱᶄ 

  13. ಉظ࣌࣌ؒͣΕΔ FNCVMLJOQVUKECD .Z42-ϓϥάΠϯΛར༻ ϓϥάΠϯ͕ࣗಈͰ.Z42-ͷ UJNF[POFΛऔಘ͠ͳ͍͜ͱ͕ݪҼ 

  14. ಉظ࣌࣌ؒͣΕΔ  in: type: mysql host: localhost user: user password:

    password database: sample table: sample select: "*" default_timezone: "Asia/Tokyo" column_options: create_time: {type: string, timestamp_format: "%Y-%m-%d %H:%M:%S%:z"} update_time: {type: string, timestamp_format: "%Y-%m-%d %H:%M:%S%:z"} out: . . . TBNQMFZNM Θ͔Γ΍͍͢Α͏ʹ "TJB5PLZP࣌ؒ ʹͦΖ͑ͯ·͢
  15. ಉظ࣌࣌ؒͣΕΔ  mysql> select * from sample; +------+----------+---------------------+ | id

    | name | create_time | +------+----------+---------------------+ | 1 | sample1 | 2013-05-27 10:46:08 | | 2 | sample2 | 2013-05-27 10:46:08 | | 3 | sample3 | 2013-05-27 10:46:08 | | 4 | sample4 | 2013-05-27 10:46:08 | | 5 | sample5 | 2013-05-27 10:46:08 | | 6 | sample6 | 2013-05-27 10:46:08 | | 7 | sample7 | 2013-05-27 10:46:08 | | 8 | sample8 | 2013-05-27 10:46:08 | | 9 | sample9 | 2013-05-27 10:46:08 | | 10 | sample10 | 2013-05-27 10:46:08 | +------+----------+---------------------+ $ embulk preview sample.yml +---------+-------------+---------------------------+ | id:long | name:string | create_time:timestamp | +---------+-------------+---------------------------+ | 1 | sample1 | 2013-05-27 19:46:08+09:00 | | 2 | sample2 | 2013-05-27 19:46:08+09:00 | | 3 | sample3 | 2013-05-27 19:46:08+09:00 | | 4 | sample4 | 2013-05-27 19:46:08+09:00 | | 5 | sample5 | 2013-05-27 19:46:08+09:00 | | 6 | sample6 | 2013-05-27 19:46:08+09:00 | | 7 | sample7 | 2013-05-27 19:46:08+09:00 | | 8 | sample8 | 2013-05-27 19:46:08+09:00 | | 9 | sample9 | 2013-05-27 19:46:08+09:00 | | 10 | sample10 | 2013-05-27 19:46:08+09:00 | +---------+-------------+---------------------------+ NZTRM FNCVMLQSFWJFX ͪΐ͏Ͳ࣌ؒ ͣΕͯΔ
  16. Կ͔Φϓγϣϯͳ͍͔ͳ  ͳΜ͔࢖ͬͯͳ ͍΍ͭ͋ͬͨʂ

  17. ղܾࡦ  in: type: mysql host: localhost user: user password:

    password database: sample table: sample select: "*" options: {useTimezone: true, serverTimezone: Asia/Tokyo} default_timezone: "Asia/Tokyo" column_options: create_time: {type: string, timestamp_format: "%Y-%m-%d %H:%M:%S%:z"} update_time: {type: string, timestamp_format: "%Y-%m-%d %H:%M:%S%:z"} out: . . . ͍ͭ͜Λ௥Ճ TBNQMFZNM
  18. ղܾ  mysql> select * from sample; +------+----------+---------------------+ | id

    | name | create_time | +------+----------+---------------------+ | 1 | sample1 | 2013-05-27 10:46:08 | | 2 | sample2 | 2013-05-27 10:46:08 | | 3 | sample3 | 2013-05-27 10:46:08 | | 4 | sample4 | 2013-05-27 10:46:08 | | 5 | sample5 | 2013-05-27 10:46:08 | | 6 | sample6 | 2013-05-27 10:46:08 | | 7 | sample7 | 2013-05-27 10:46:08 | | 8 | sample8 | 2013-05-27 10:46:08 | | 9 | sample9 | 2013-05-27 10:46:08 | | 10 | sample10 | 2013-05-27 10:46:08 | +------+----------+---------------------+ $ embulk preview sample.yml +---------+-------------+---------------------------+ | id:long | name:string | create_time:timestamp | +---------+-------------+---------------------------+ | 1 | sample1 | 2013-05-27 10:46:08+09:00 | | 2 | sample2 | 2013-05-27 10:46:08+09:00 | | 3 | sample3 | 2013-05-27 10:46:08+09:00 | | 4 | sample4 | 2013-05-27 10:46:08+09:00 | | 5 | sample5 | 2013-05-27 10:46:08+09:00 | | 6 | sample6 | 2013-05-27 10:46:08+09:00 | | 7 | sample7 | 2013-05-27 10:46:08+09:00 | | 8 | sample8 | 2013-05-27 10:46:08+09:00 | | 9 | sample9 | 2013-05-27 10:46:08+09:00 | | 10 | sample10 | 2013-05-27 10:46:08+09:00 | +---------+-------------+---------------------------+ NZTRM FNCVMLQSFWJFX ͪΌΜͱಉظ͞Εͨ
  19. +%#$ͷΦϓγϣϯ ݹ͍Φϓγϣϯ VTF5JNF[POFUSVF TFSWFS5JNF[POF"TJB5PLZP ৽͍͠Φϓγϣϯ VTF-FHBDZ%BUFUJNF$PEFGBMTF  ͜Ε͚ͩͰ͸ ղܾͰ͖ͳ͔ͬͨ

  20. ࢀߟ FNCVMLJOQVUKECDͷ.Z42-ϓϥά ΠϯͰ͕࣌ؒ࣌ؒͣΕΔ IUUQRJJUBDPNLBUTVZBOJUFNT GEDBCBD 

  21. ·ͱΊ ϩάͷग़ྗઌ͸֬ೝ͠·͠ΐ͏ ࣌ؒͷҋ͸ਂ͍ Φϓγϣϯ͸શ෦ோΊ·͠ΐ͏ &NCVML͸ศར