Pro Yearly is on sale from $80 to $50! »

digdagで支えるデータパイプライン / Building a data pipeline with digdag

E74600a3064ae510f783a9bb8214e8ff?s=47 Yukiyan
April 16, 2019

digdagで支えるデータパイプライン / Building a data pipeline with digdag

2019/04/16 Data Pipeline Casual Talk #2 #dpct
https://dpct.connpass.com/event/121371/

@yukiyan_w
https://github.com/yukiyan/Curriculum-Vitae

E74600a3064ae510f783a9bb8214e8ff?s=128

Yukiyan

April 16, 2019
Tweet

Transcript

  1. EJHEBHͰࢧ͑ΔσʔλύΠϓϥΠϯ  %BUB1JQFMJOF$BTVBM5BML एࢁ༟ૅ  

  2. ࣗݾ঺հ   एࢁ༟ૅ :VLJ8BLBZBNB  !ZVLJZBO@X IUUQTHJUIVCDPNZVLJZBO$VSSJDVMVN7JUBF ɾ+6#*-&&803,4Ͱ5JNF5SFF ΧϨϯμʔγΣΞΞϓϦ

    ͷ։ൃ ɾओʹσʔλ෼ੳج൫ͷ࢓ࣄΛ΍͍ͬͯ·͢
  3.   ΧϨϯμʔʹ͓ޓ͍ͷ༧ఆΛॻ͖ࠐΉ ΧϨϯμʔγΣΞΞϓϦ 5JNF5SFFͷ঺հ ܰ͘

  4.   ձࣾ৘ใͱن໛ײ ɾઃཱ೥݄ ɾݱࡏͷैۀһ਺໊͘Β͍ ɾ5JNF5SFFͷϦϦʔε೥݄ ɾϢʔβʔ਺ສ ɾάϩʔόϧʹϢʔβʔ͕͍ͯɺ ೔ຊϢʔβʔׂ͕ఔ౓Λ઎ΊΔ ɾେ͖͍ςʔϒϧͰ਺ඦ(#

    ਺ेԯϨίʔυఔ౓
  5. લఏ ૊৫ʹ͓͚Δࢲͷ໾ׂ   ɾۀ຿ʹ͓͍ͯ໌֬ͳ໾ׂ෼୲͸͍ͯ͠ͳ͍ ɾϥϕϧͱͯ͠͸όοΫΤϯυΤϯδχΞ͕ͩɺʮࣗ෼͕΍Γ͍ͨ͜ͱʯ ͔ͭʮձࣾʹඞཁͦ͏ͳ͜ͱʯΛࣗ෼Ͱ൑அͯ͠΍͍ͬͯΔ ɾ3BJMTͰͷ৽ػೳ։ൃͯ͠Δͱ͖΋͋Ε͹ɺσʔλύΠϓϥΠϯ੔උ ʹऴ࢝͢Δ͜ͱ΋͋Δ ɾ৽ن1+اըϚωʔδΛ͢Δ͜ͱ΋͋Δ

    ɾػցֶशج൫ΛҰਓͰߏங͢Δͱ͍͏͜ͱ
  6. ࠓ೔࿩͢͜ͱ   ɾσʔλύΠϓϥΠϯΛͲ͏૊ΜͰ͍Δͷ͔ ɾEJHEBHͷ঺հ ɾ࣮ࡍʹ૊ΜͩύΠϓϥΠϯࣄྫ ɾΠϯϑϥߏ੒΍σϓϩΠϑϩʔ ɾࣦഊஊ

  7. ࠓ೔࿩͞ͳ͍͜ͱ   ɾෳ਺ϓϩμΫτԣஅͷσʔλج൫ ɾฐࣾͷϓϩμΫτ͸5JNF5SFFͷΈͳͷͰ࿩ͤͳ͍ ɾσʔλύΠϓϥΠϯͷ඼࣭อূ ɾϞχλϦϯά΍ςετʹ͍ͭͯ͸·ͩϕετͳํ๏͕Θ͔ͬͯͳ͍ ɾ૊৫ͬΆ͍࿩ ɾ࢖ͬͯ΋Β͑Δσʔλج൫ͱ͸ ܥͷ͋Ε͜Ε

    ɾσʔλαΠΤϯςΟετͱͷίϛϡχέʔγϣϯ͋Ε͜Ε ɾͦ΋ͦ΋ฐࣾʹ%4͸͍ͳ͍ ŝũŞƀ
  8. ѻͬͯΔσʔλͷछྨ   ɾΞΫηεϩά 3BJMT΍OHJOY  ɾΞϓϦέʔγϣϯσʔλ %#ʹ֨ೲ͢ΔλΠϓͷσʔλ  ɾΞΫγϣϯϩά

    ΫϦοΫϩά΍ΠϯϓϨογϣϯϩά౳ͷ޿ࠂϩά  ɾSEQBSUZπʔϧ΁ૹͬͯΔϩά ޿ࠂܭଌπʔϧ΁ૹͬͯΔϩά  ʜFUD
  9. σʔλͷྲྀΕ   ɾҰ୴ɺ4ʹू·ΔΑ͏ͳύΠϓϥΠϯ ɾ4ͱ($4ʹಉ͡σʔλ͕͋Δ ɾ4΍($4ʹݩσʔλ͕͋Ε͹ɺকདྷతʹ Ԡ༻ͷ෯͕޿͕Γ΍͍͢ $MPVE%BUBqPX౳

  10. EJHEBHͷબఆཧ༝   ɾҎલ͔ΒqVFOUE FNCVMLΛศརʹ࢖͍ͬͯͨͷͰɺ5SFBTVSF%BUB ͷ੡඼ʹ৴པײ͕͋ͬͨ ɾ೥ࠒʹൃද͞Ε͙ͯ͢ʹ࢖ͬͯΈͯɺߏங͕؆୯ͩ͠ʮ͜Ε͸ ΠέΔʯͱࢥͬͨ lσʔλॲཧϫʔΫϑϩʔͱ͍͏΋ͷ͸ԟʑʹͯ͠ෳࡶʹͳΓɺδϣϒؒͷґଘؔ܎͕Ͳ͏ͯ͠΋ ग़ͯ͠·͏΋ͷ͕ͩɺEJHEBHͷઃఆϑΝΠϧ͸಄ʹු͔ΜͩϫʔΫϑϩʔΛઃఆϑΝΠϧʹམͱ

    ͠ࠐΈ΍͍͢͠௚ײతͩ͠ػೳ΋๛෋ͳͷͰͱͯ΋ظ଴ͷͰ͖Δ΋ͷͩͬͨɻz %JHEBH FNCVML #JH2VFSZ 3FEBTIͰσʔλ෼ੳج൫ߏஙͷເΛݟΔࡶͳϝϞ
  11. EJHEBHͷબఆཧ༝   ɾ"JSqPX΍-VJHJʹ͍ͭͯ͸ɺݫີͳൺֱ͸͍ͯ͠ͳ͍͕ͲΕ΋ඞཁͳ ػೳ͸Χόʔͯ͠Δͱࢥ͏ͷͰɺνʔϜͷಛੑʹ߹ΘͤͯͲΕ࢖ͬͯ΋ ͍͍ͱࢥ͏ ͨͩEJHEBH͸ΠΠͧ

  12. EJHEBHͷ঺հ   ɾ؅ཧը໘ ɾϦτϥΠ ɾΤϥʔ௨஌ ɾEJHϑΝΠϧͷه๏

  13. δϣϒͷ༷ࢠ͕ͻͱ໨ͰΘ͔ΔͷͰɺΤϥʔ͕ى͖ͨͱ͖ʹศར ؅ཧը໘  

  14. ϦτϥΠ  

  15. Τϥʔ௨஌   δϣϒ͕ࣦഊͨ͠ͱ͖ʹɺTMBDL௨஌Ͱ͖Δ T[ZOEJHEBHTMBDLͷϓϥάΠϯΛ࢖͍ͬͯΔ

  16. EJHϑΝΠϧͷه๏   εέδϡʔϧઃఆ TIFMMTDSJQU࣮ߦ 3VCZͷ࣮ߦ 1ZUIPO΋Մ

  17. EJHϑΝΠϧͷه๏   εΫϦϓτؒͰͷม਺౉͠ .Z42-ͷςʔϒϧҰཡͱ͔εΫϦϓτͰऔಘͯ֨͠ೲͨ͠Γ͢Δ

  18. EJHϑΝΠϧͷه๏   ฒྻ࣮ߦ

  19. EJHϑΝΠϧͷه๏   CRΦϖϨʔλͰதؒςʔϒϧΛ࡞੒ SFETIJGU FNSΦϖϨʔλ౳ศརͳ΋ͷ͕͍͔ͭ͋͘Δ

  20. EJHϑΝΠϧͷه๏   λεΫΛ೚ҙͷEPDLFSΠϝʔδ಺Ͱ࣮ߦ

  21. EPDLFS࣮ߦʹ͍ͭͯ   ɾগ͠บ͕͋ΔͷͰӈهࢀর EJHEBHλεΫͷEPDLFS࣮ߦʹ͍ͭͯ  ɾWBSSVOEPDLFSTPDLΛϚ΢ϯτ͢Δඞཁ͕͋ΔͷͰ஫ҙ ɾEJHEBHࣗମΛEPDLFS &$4 Ͱߏஙͯ͠Δ৔߹ɺEJOEʹͳͬͯ͠·͍

    ߟ͑Δ͜ͱ͕૿͑Δ ɾͦ͜·Ͱੵۃతʹ࢖ͬͯ͸͍ͳ͍ ɾͰ΋ɺվળ͸ΊͬͪΌਐΜͰΔͷͰظ଴ @FYQPSUEPDLFSTVQQPSUTEPDLFSPQUJPOBSHVNFOUT OFYUBEE UFTUT 
  22. EJHEBHͰ࣮ࡍʹ૊ΜͩύΠϓϥΠϯࣄྫ   ʰ.Z42-ͷσʔλΛ#JH2VFSZ΁ྲྀ͢ʱ 3VCZͰςʔϒϧҰཡΛऔಘ FNCVMLJOQVUNZTRMͰ4΁ΤΫεϙʔτ 4ͱ($4ΛTZOD #JH2VFSZ΁ϩʔυ͢Δ 3FEBTIͷSFGSFTI"1*Λୟ͍ͯάϥϑߋ৽

  23. ʰಛఆσʔλͷநग़݁ՌΛTMBDL΁Ξοϓϩʔυʱ EJHEBHͰ࣮ࡍʹ૊ΜͩύΠϓϥΠϯࣄྫ  

  24. ʰ/14ͷ಺༰ΛTMBDL΁௨஌ʱ (PPHMF$MPVE5SBOTMBUJPO"1*΋࢖͍ɺଟݴޠʹ΋ରԠ EJHEBHͰ࣮ࡍʹ૊ΜͩύΠϓϥΠϯࣄྫ  

  25. ʰ͋Δςʔϒϧͷ೚ҙͷΧϥϜΛLVSPNPKJͰܗଶૉղੳޙɺ໊ࢺΛ (PPHMF/BUVSBM-BOHVBHF"1*ʹט·ͤͯΤϯςΟςΟ෼ੳ͢Δʱ #JH2VFSZͰର৅ͷநग़΍લॲཧ ($4΁ΤΫεϙʔτ FNCVMLpMUFSLVSPNPKJͰ໊ࢺΛநग़ #JH2VFSZ΁ϩʔυ͢Δ ($4΁ΤΫεϙʔτ FNCVMLpMUFSHPPHMF@OBUVSBM@MBOHVBHF@BQJͰΤϯςΟςΟ෼ੳ #JH2VFSZ΁ϩʔυ͢Δ ΤϯςΟςΟ෼ੳ݁ՌͷUZQF͕-0$"5*0/Ͱ͋ΔςʔϒϧΛ࡞੒

    ఆظ࣮ߦܥͷॲཧ͚ͩͰ͸ͳ͘ɺ ͜ͷΑ͏ͳ୯ൃͷΞυϗοΫͳύΠϓϥΠϯΛ࣮ߦ͢Δͷʹ΋ศར EJHEBHͰ࣮ࡍʹ૊ΜͩύΠϓϥΠϯࣄྫ  
  26. ʰ͋Δςʔϒϧͷ೚ҙͷΧϥϜΛࡐྉʹɺ GBTU5FYUͰ/-1ʹ͓͚Δ୯ޠͷ෼ࢄදݱΛऔಘʱ #JH2VFSZͰର৅ͷநग़΍લॲཧ ($4΁ΤΫεϙʔτ FNCVMLpMUFSLVSPNPKJͰ໊ࢺΛநग़ GBTU5FYUͰ෼ࢄදݱΛऔಘ ϞσϧΛ4ʹΞοϓϩʔυ EJHEBHͰ࣮ࡍʹ૊ΜͩύΠϓϥΠϯࣄྫ  

    .-0QTʹ΋Ԡ༻Ͱ͖Δ
  27. ʰ4QBSL EBUBCSJDLT ͷͨΊͷσʔλ४උͱֶशͷ࣮ߦʱ #JH2VFSZͰର৅ͷநग़΍લॲཧ ($4ͱ4΁ΤΫεϙʔτ EBUBCSJDLTͷ"1*Λୟ͘ SEQBSUZπʔϧͱͷ࿈ܞʹ΋Ԡ༻Ͱ͖Δ EJHEBHͰ࣮ࡍʹ૊ΜͩύΠϓϥΠϯࣄྫ  

  28.   ඇج൫ΤϯδχΞʮ͜ͷॲཧͷϩάΛ#JH2VFSZʹೖΕ͍ͨͳ͊ʯ

  29. σʔλͷྲྀΕ ࠶ܝ   ඇج൫ΤϯδχΞʮ͜ͷॲཧͷϩάΛ#JH2VFSZʹೖΕ͍ͨͳ͊ʯ ৽ͨͳ޿ࠂϩάΛ௥Ճ͍ͨ͠ͱ͢Δ

  30.   ɾpSFIPTFʹೖΕΔ͜ͱ͚ͩΛߟ͑Ε͹͍͍Α͏ʹ͍ͯ͠Δ UFSSBGPSN qVFOUMPHHFSSVCZ qVFOUEDPOG EJHϑΝΠϧΛগ͍͠͡Δ͚ͩ ඇج൫ΤϯδχΞʮ͜ͷॲཧͷϩάΛ#JH2VFSZʹೖΕ͍ͨͳ͊ʯ

  31. Πϯϑϥߏ੒   &$4্ʹߏஙɻ 4QPUqFFUͰ҆͘ӡ༻͍ͯ͠Δɻ

  32. σϓϩΠϑϩʔ   EJHϑΝΠϧॻ͘ 1VMM3FRVFTUग़͢ Ϛʔδ͞ΕΔͱ$JSDMF$*͕σϓϩΠ͢Δ ɾ$JSDMF$*಺ͰEJHEBHQVTI ઃఆϑΝΠϧͷ൓ө Λ͢Δ ɾEJHϑΝΠϧͷมߋͷΈͷ৔߹͸ɺ࠶ىಈෆཁ

    ࠶ىಈ͍ͨ͠৔߹͸ɺTMBDLCPUʹ໋ྩ
  33. ຊ൪δϣϒ࣮ߦதʹ࠶ىಈͯ͠΋໰୊ͳ͍ɻ EJHEBHͷ%#͕ঢ়ଶΛอ͍࣋ͯͯ͠ɺ ͏·੍͘ޚ͍ͯ͠Δɻ σϓϩΠϑϩʔ  

  34. ࣦഊஊ   ɾEJHϑΝΠϧ͸%3:ʹ͗͢͠ͳ͍ ɾJODMVEFͱ͔࢖ͬͯ%3:ʹͰ͖Δ͚Ͳɺ۪௚ʹॻ͍ͨ΄͏͕ॲཧΛ௥ ͍΍͍͢ ɾ00.ରࡦ͕৬ਓ͡ΈͯΔ ɾฒྻ౓Λ্͛͗͢ͳ͍ ɾ৔߹ʹΑͬͯ͸ɺ@QBSBMMFMUSVFΛආ͚Δ ɾEJHEBHͷىಈΦϓγϣϯNBYUBTLUISFBETΛௐ੔

    ɾFNCVML+9NYHSVOMBSHF@KPCZNM ɾFNCVMLͷZNMͷFYFD۟ͷNBY@UISFBET΍NJO@PVUQVU@UBTLTΛௐ੔ ɾͦ΋ͦ΋ͷύΠϓϥΠϯͷઃܭΛݟ௚͢
  35. ࣦഊஊ σʔλύΠϓϥΠϯߏங࣌ʹ͍ͭͯ   ɾ#JH2VFSZͷσʔληοτͷ෼͚ํ ɾσʔληοτʹແ਺ͷςʔϒϧ͕͋Δ ɾEFWTUHQSPEΛ($1ϓϩδΣΫτͰ෼͚ͨΓɺσʔληοτ໊Λ ޻෉͢΂͖ͩͬͨ ɾࠩ෼ಉظ͔Ͳ͏͔஌Βͳ͍ͱूܭϛεΔ໰୊ ɾར༻ऀʹύΠϓϥΠϯࣄ৘Λҙࣝͤ͞ͳ͍ઃܭʹ͢Δ΂͖ͩͬͨ

    ɾ͜Ε͔Β࡞Δਓ͸ඞಡˠࢲͷߟ͑ͨ࠷ڧͷϩάˍϞχλϦϯάઃ ܭԼொ༘ࢠԫࠚهCZ!ZV[VUBT ɾ#JH2VFSZͷεΩʔϚࣗಈਪ࿦Λ࢖͏৔߹͸஫ҙ ɾ࿦ཧ࡟আܥͷϑϥάͳͲɺ΄΅/6--ͳΧϥϜ͕͋ΔͱɺεΩʔϚ ͷBVUPEFUFDUΛ͢Γൈ͚Δ શεΩϟϯ͢ΔΘ͚Ͱ͸ͳ͍ͷͰ ˞EJHEBHؔ܎ͳ͍෦෼
  36. ·ͱΊ   ɾσʔλύΠϓϥΠϯΛͲ͏૊ΜͰ͍Δͷ͔ ɾҰ୴ɺ4ʹू·ΔΑ͏ͳύΠϓϥΠϯʹͯ͠Δ ɾ'JSFIPTFڬΜͰ͓͘ͱศར ɾEJHEBH͸ศར ɾఆظ࣮ߦܥͷॲཧҎ֎ʹ΋༗ޮ ɾ.-0QTʹ΋Ԡ༻Ͱ͖Δ ɾΠϯϑϥߏ੒΍σϓϩΠϑϩʔ

    ɾ&$4XJUI4QPUqFFUͰ҆͘ӡ༻ ɾ$JSDMF$*΍$IBUCPUͷ׆༻ ɾ00.ʹؾΛ͚ͭΑ͏ ɾ#JH2VFSZΛ࢖͏৔߹͸σʔληοτͷઃܭʹ஫ҙ͠Α͏