Upgrade to Pro — share decks privately, control downloads, hide ads and more …

digdagで支えるデータパイプライン / Building a data pipeline ...

Yukiyan
April 16, 2019

digdagで支えるデータパイプライン / Building a data pipeline with digdag

2019/04/16 Data Pipeline Casual Talk #2 #dpct
https://dpct.connpass.com/event/121371/

@yukiyan_w
https://github.com/yukiyan/Curriculum-Vitae

Yukiyan

April 16, 2019
Tweet

More Decks by Yukiyan

Other Decks in Technology

Transcript

  1. ѻͬͯΔσʔλͷछྨ   ɾΞΫηεϩά 3BJMT΍OHJOY  ɾΞϓϦέʔγϣϯσʔλ %#ʹ֨ೲ͢ΔλΠϓͷσʔλ  ɾΞΫγϣϯϩά

    ΫϦοΫϩά΍ΠϯϓϨογϣϯϩά౳ͷ޿ࠂϩά  ɾSEQBSUZπʔϧ΁ૹͬͯΔϩά ޿ࠂܭଌπʔϧ΁ૹͬͯΔϩά  ʜFUD
  2. EPDLFS࣮ߦʹ͍ͭͯ   ɾগ͠บ͕͋ΔͷͰӈهࢀর EJHEBHλεΫͷEPDLFS࣮ߦʹ͍ͭͯ  ɾWBSSVOEPDLFSTPDLΛϚ΢ϯτ͢Δඞཁ͕͋ΔͷͰ஫ҙ ɾEJHEBHࣗମΛEPDLFS &$4 Ͱߏஙͯ͠Δ৔߹ɺEJOEʹͳͬͯ͠·͍

    ߟ͑Δ͜ͱ͕૿͑Δ ɾͦ͜·Ͱੵۃతʹ࢖ͬͯ͸͍ͳ͍ ɾͰ΋ɺվળ͸ΊͬͪΌਐΜͰΔͷͰظ଴ @FYQPSUEPDLFSTVQQPSUTEPDLFSPQUJPOBSHVNFOUT OFYUBEE UFTUT 
  3. ࣦഊஊ σʔλύΠϓϥΠϯߏங࣌ʹ͍ͭͯ   ɾ#JH2VFSZͷσʔληοτͷ෼͚ํ ɾσʔληοτʹແ਺ͷςʔϒϧ͕͋Δ ɾEFWTUHQSPEΛ($1ϓϩδΣΫτͰ෼͚ͨΓɺσʔληοτ໊Λ ޻෉͢΂͖ͩͬͨ ɾࠩ෼ಉظ͔Ͳ͏͔஌Βͳ͍ͱूܭϛεΔ໰୊ ɾར༻ऀʹύΠϓϥΠϯࣄ৘Λҙࣝͤ͞ͳ͍ઃܭʹ͢Δ΂͖ͩͬͨ

    ɾ͜Ε͔Β࡞Δਓ͸ඞಡˠࢲͷߟ͑ͨ࠷ڧͷϩάˍϞχλϦϯάઃ ܭԼொ༘ࢠԫࠚهCZ!ZV[VUBT ɾ#JH2VFSZͷεΩʔϚࣗಈਪ࿦Λ࢖͏৔߹͸஫ҙ ɾ࿦ཧ࡟আܥͷϑϥάͳͲɺ΄΅/6--ͳΧϥϜ͕͋ΔͱɺεΩʔϚ ͷBVUPEFUFDUΛ͢Γൈ͚Δ શεΩϟϯ͢ΔΘ͚Ͱ͸ͳ͍ͷͰ ˞EJHEBHؔ܎ͳ͍෦෼