digdagで支えるデータパイプライン / Building a data pipeline with digdag

2019/04/16 Data Pipeline Casual Talk #2 #dpct
https://dpct.connpass.com/event/121371/

@yukiyan_w
https://github.com/yukiyan/Curriculum-Vitae

Transcript

1. EJHEBHͰࢧ͑ΔσʔλύΠϓϥΠϯ 
   %BUB
   1JQFMJOF
   $BTVBM
   5BML
   
   एࢁ
   ༟ૅ
   
2. ࣗݾ঺հ
    एࢁ
   ༟ૅ
   :VLJ
   8BLBZBNB
   !ZVLJZBO@X
   IUUQTHJUIVCDPNZVLJZBO$VSSJDVMVN7JUBF ɾ+6#*-&&
   803,4Ͱ5JNF5SFF ΧϨϯμʔγΣΞΞϓϦ ͷ։ൃ
   ɾओʹσʔλ෼ੳج൫ͷ࢓ࣄΛ΍͍ͬͯ·͢
3. 
    ΧϨϯμʔʹ͓ޓ͍ͷ༧ఆΛॻ͖ࠐΉ
   ΧϨϯμʔγΣΞΞϓϦ 5JNF5SFFͷ঺հ ܰ͘
4. 
    ձࣾ৘ใͱن໛ײ ɾઃཱ
   ೥݄
   ɾݱࡏͷैۀһ਺
   ໊͘Β͍
   ɾ5JNF5SFFͷϦϦʔε
   ೥݄
   ɾϢʔβʔ਺
   ສ
   ɾάϩʔόϧʹϢʔβʔ͕͍ͯɺ
   
   
   
   ೔ຊϢʔβʔׂ͕ఔ౓Λ઎ΊΔ
   ɾେ͖͍ςʔϒϧͰ਺ඦ(#
   ਺ेԯϨίʔυఔ౓
5. લఏ
   ૊৫ʹ͓͚Δࢲͷ໾ׂ
    ɾۀ຿ʹ͓͍ͯ໌֬ͳ໾ׂ෼୲͸͍ͯ͠ͳ͍
   ɾϥϕϧͱͯ͠͸όοΫΤϯυΤϯδχΞ͕ͩɺʮࣗ෼͕΍Γ͍ͨ͜ͱʯ ͔ͭʮձࣾʹඞཁͦ͏ͳ͜ͱʯΛࣗ෼Ͱ൑அͯ͠΍͍ͬͯΔ
   ɾ3BJMTͰͷ৽ػೳ։ൃͯ͠Δͱ͖΋͋Ε͹ɺσʔλύΠϓϥΠϯ੔උ ʹऴ࢝͢Δ͜ͱ΋͋Δ
   ɾ৽ن1+اըϚωʔδΛ͢Δ͜ͱ΋͋Δ
   
   
   
   
   ɾػցֶशج൫ΛҰਓͰߏங͢Δͱ͍͏͜ͱ
6. ࠓ೔࿩͢͜ͱ
    ɾσʔλύΠϓϥΠϯΛͲ͏૊ΜͰ͍Δͷ͔
   ɾEJHEBHͷ঺հ
   ɾ࣮ࡍʹ૊ΜͩύΠϓϥΠϯࣄྫ
   ɾΠϯϑϥߏ੒΍σϓϩΠϑϩʔ
   ɾࣦഊஊ
7. ࠓ೔࿩͞ͳ͍͜ͱ
    ɾෳ਺ϓϩμΫτԣஅͷσʔλج൫
   
   
   
   ɾฐࣾͷϓϩμΫτ͸5JNF5SFFͷΈͳͷͰ࿩ͤͳ͍
   ɾσʔλύΠϓϥΠϯͷ඼࣭อূ
   
   
   
   
   ɾϞχλϦϯά΍ςετʹ͍ͭͯ͸·ͩϕετͳํ๏͕Θ͔ͬͯͳ͍
   ɾ૊৫ͬΆ͍࿩
   
   
   
   
   ɾ࢖ͬͯ΋Β͑Δσʔλج൫ͱ͸
   ܥͷ͋Ε͜Ε
   
   
   
   
   ɾσʔλαΠΤϯςΟετͱͷίϛϡχέʔγϣϯ͋Ε͜Ε
   
   
   
   
   ɾͦ΋ͦ΋ฐࣾʹ%4͸͍ͳ͍ ŝũŞƀ
8. ѻͬͯΔσʔλͷछྨ
    ɾΞΫηεϩά 3BJMT΍OHJOY
   ɾΞϓϦέʔγϣϯσʔλ %#ʹ֨ೲ͢ΔλΠϓͷσʔλ
   ɾΞΫγϣϯϩά ΫϦοΫϩά΍ΠϯϓϨογϣϯϩά౳ͷ޿ࠂϩά
   ɾSE
   QBSUZπʔϧ΁ૹͬͯΔϩά ޿ࠂܭଌπʔϧ΁ૹͬͯΔϩά
   ʜFUD
9. σʔλͷྲྀΕ
    ɾҰ୴ɺ4ʹू·ΔΑ͏ͳύΠϓϥΠϯ
   ɾ4ͱ($4ʹಉ͡σʔλ͕͋Δ
   ɾ4΍($4ʹݩσʔλ͕͋Ε͹ɺকདྷతʹ
   
   
   
   Ԡ༻ͷ෯͕޿͕Γ΍͍͢ $MPVE
   %BUBqPX౳
10. EJHEBHͷબఆཧ༝
     ɾҎલ͔ΒqVFOUE
    FNCVMLΛศརʹ࢖͍ͬͯͨͷͰɺ5SFBTVSF
    %BUB ͷ੡඼ʹ৴པײ͕͋ͬͨ
    ɾ೥ࠒʹൃද͞Ε͙ͯ͢ʹ࢖ͬͯΈͯɺߏங͕؆୯ͩ͠ʮ͜Ε͸ ΠέΔʯͱࢥͬͨ lσʔλॲཧϫʔΫϑϩʔͱ͍͏΋ͷ͸ԟʑʹͯ͠ෳࡶʹͳΓɺδϣϒؒͷґଘؔ܎͕Ͳ͏ͯ͠΋ ग़ͯ͠·͏΋ͷ͕ͩɺEJHEBHͷઃఆϑΝΠϧ͸಄ʹු͔ΜͩϫʔΫϑϩʔΛઃఆϑΝΠϧʹམͱ ͠ࠐΈ΍͍͢͠௚ײతͩ͠ػೳ΋๛෋ͳͷͰͱͯ΋ظ଴ͷͰ͖Δ΋ͷͩͬͨɻz
    %JHEBH
    
    FNCVML
    
    #JH2VFSZ
    
    3FEBTI
    Ͱσʔλ෼ੳج൫ߏஙͷເΛݟΔ
    
    ࡶͳϝϞ
11. EJHEBHͷબఆཧ༝
     ɾ"JSqPX΍-VJHJʹ͍ͭͯ͸ɺݫີͳൺֱ͸͍ͯ͠ͳ͍͕ͲΕ΋ඞཁͳ ػೳ͸Χόʔͯ͠Δͱࢥ͏ͷͰɺνʔϜͷಛੑʹ߹ΘͤͯͲΕ࢖ͬͯ΋ ͍͍ͱࢥ͏ ͨͩEJHEBH͸ΠΠͧ
12. EJHEBHͷ঺հ
     ɾ؅ཧը໘
    ɾϦτϥΠ
    ɾΤϥʔ௨஌
    ɾEJHϑΝΠϧͷه๏
13. δϣϒͷ༷ࢠ͕ͻͱ໨ͰΘ͔ΔͷͰɺΤϥʔ͕ى͖ͨͱ͖ʹศར ؅ཧը໘
    
14. ϦτϥΠ
    
15. Τϥʔ௨஌
     δϣϒ͕ࣦഊͨ͠ͱ͖ʹɺTMBDL௨஌Ͱ͖Δ
    T[ZOEJHEBHTMBDL
    ͷϓϥάΠϯΛ࢖͍ͬͯΔ
16. EJHϑΝΠϧͷه๏
     εέδϡʔϧઃఆ TIFMM
    TDSJQU࣮ߦ 3VCZͷ࣮ߦ 1ZUIPO΋Մ
17. EJHϑΝΠϧͷه๏
     εΫϦϓτؒͰͷม਺౉͠
    .Z42-ͷςʔϒϧҰཡͱ͔εΫϦϓτͰऔಘͯ֨͠ೲͨ͠Γ͢Δ
18. EJHϑΝΠϧͷه๏
     ฒྻ࣮ߦ
19. EJHϑΝΠϧͷه๏
     CRΦϖϨʔλͰதؒςʔϒϧΛ࡞੒
    SFETIJGU
    FNSΦϖϨʔλ౳ศརͳ΋ͷ͕͍͔ͭ͋͘Δ
20. EJHϑΝΠϧͷه๏
     λεΫΛ೚ҙͷEPDLFSΠϝʔδ಺Ͱ࣮ߦ
21. EPDLFS࣮ߦʹ͍ͭͯ
     ɾগ͠บ͕͋ΔͷͰӈهࢀর EJHEBH
    λεΫͷ
    EPDLFS
    ࣮ߦʹ͍ͭͯ
    ɾWBSSVOEPDLFSTPDL
    ΛϚ΢ϯτ͢Δඞཁ͕͋ΔͷͰ஫ҙ
    ɾEJHEBHࣗମΛEPDLFS &$4 Ͱߏஙͯ͠Δ৔߹ɺEJOEʹͳͬͯ͠·͍ ߟ͑Δ͜ͱ͕૿͑Δ
    ɾͦ͜·Ͱੵۃతʹ࢖ͬͯ͸͍ͳ͍
    ɾͰ΋ɺվળ͸ΊͬͪΌਐΜͰΔͷͰظ଴
    
    
    
    
    @FYQPSU
    EPDLFS
    TVQQPSUT
    EPDLFS
    PQUJPO
    BSHVNFOUT
    OFYU
    BEE
    UFTUT
    
22. EJHEBHͰ࣮ࡍʹ૊ΜͩύΠϓϥΠϯࣄྫ
     ʰ.Z42-ͷσʔλΛ#JH2VFSZ΁ྲྀ͢ʱ 
    3VCZͰςʔϒϧҰཡΛऔಘ
    
    FNCVMLJOQVUNZTRMͰ4΁ΤΫεϙʔτ
    
    4ͱ($4ΛTZOD
    
    #JH2VFSZ΁ϩʔυ͢Δ
    
    3FEBTIͷSFGSFTI
    "1*Λୟ͍ͯάϥϑߋ৽
23. ʰಛఆσʔλͷநग़݁ՌΛTMBDL΁Ξοϓϩʔυʱ EJHEBHͰ࣮ࡍʹ૊ΜͩύΠϓϥΠϯࣄྫ
    
24. ʰ/14ͷ಺༰ΛTMBDL΁௨஌ʱ
    (PPHMF
    $MPVE
    5SBOTMBUJPO
    "1*΋࢖͍ɺଟݴޠʹ΋ରԠ EJHEBHͰ࣮ࡍʹ૊ΜͩύΠϓϥΠϯࣄྫ
    
25. ʰ͋Δςʔϒϧͷ೚ҙͷΧϥϜΛ
    LVSPNPKJ
    Ͱܗଶૉղੳޙɺ໊ࢺΛ
    (PPHMF
    /BUVSBM
    -BOHVBHF
    "1*
    ʹט·ͤͯΤϯςΟςΟ෼ੳ͢Δʱ 
    #JH2VFSZͰର৅ͷநग़΍લॲཧ
    
    ($4΁ΤΫεϙʔτ
    
    FNCVMLpMUFSLVSPNPKJ
    Ͱ໊ࢺΛநग़
    
    #JH2VFSZ΁ϩʔυ͢Δ
    
    ($4΁ΤΫεϙʔτ
    
    FNCVMLpMUFSHPPHMF@OBUVSBM@MBOHVBHF@BQJͰΤϯςΟςΟ෼ੳ
    
    #JH2VFSZ΁ϩʔυ͢Δ
    
    ΤϯςΟςΟ෼ੳ݁ՌͷUZQF͕-0$"5*0/Ͱ͋ΔςʔϒϧΛ࡞੒ ఆظ࣮ߦܥͷॲཧ͚ͩͰ͸ͳ͘ɺ
    ͜ͷΑ͏ͳ୯ൃͷΞυϗοΫͳύΠϓϥΠϯΛ࣮ߦ͢Δͷʹ΋ศར EJHEBHͰ࣮ࡍʹ૊ΜͩύΠϓϥΠϯࣄྫ
    
26. ʰ͋Δςʔϒϧͷ೚ҙͷΧϥϜΛࡐྉʹɺ
    GBTU5FYUͰ/-1ʹ͓͚Δ୯ޠͷ෼ࢄදݱΛऔಘʱ 
    #JH2VFSZͰର৅ͷநग़΍લॲཧ
    
    ($4΁ΤΫεϙʔτ
    
    FNCVMLpMUFSLVSPNPKJ
    Ͱ໊ࢺΛநग़
    
    GBTU5FYUͰ෼ࢄදݱΛऔಘ
    
    ϞσϧΛ4ʹΞοϓϩʔυ EJHEBHͰ࣮ࡍʹ૊ΜͩύΠϓϥΠϯࣄྫ
     .-
    0QTʹ΋Ԡ༻Ͱ͖Δ
27. ʰ4QBSL EBUBCSJDLT ͷͨΊͷσʔλ४උͱֶशͷ࣮ߦʱ 
    #JH2VFSZͰର৅ͷநग़΍લॲཧ
    
    ($4ͱ4΁ΤΫεϙʔτ
    
    EBUBCSJDLTͷ"1*Λୟ͘ SE
    QBSUZπʔϧͱͷ࿈ܞʹ΋Ԡ༻Ͱ͖Δ EJHEBHͰ࣮ࡍʹ૊ΜͩύΠϓϥΠϯࣄྫ
    
28. 
     ඇج൫ΤϯδχΞʮ͜ͷॲཧͷϩάΛ#JH2VFSZʹೖΕ͍ͨͳ͊ʯ
29. σʔλͷྲྀΕ ࠶ܝ
     ඇج൫ΤϯδχΞʮ͜ͷॲཧͷϩάΛ#JH2VFSZʹೖΕ͍ͨͳ͊ʯ ৽ͨͳ޿ࠂϩάΛ௥Ճ͍ͨ͠ͱ͢Δ
30. 
    
    ɾpSFIPTFʹೖΕΔ͜ͱ͚ͩΛߟ͑Ε͹͍͍Α͏ʹ͍ͯ͠Δ
    
    
    
    UFSSBGPSN
    qVFOUMPHHFSSVCZ
    qVFOUEDPOG
    EJHϑΝΠϧΛগ͍͠͡Δ͚ͩ ඇج൫ΤϯδχΞʮ͜ͷॲཧͷϩάΛ#JH2VFSZʹೖΕ͍ͨͳ͊ʯ
31. Πϯϑϥߏ੒
     &$4্ʹߏஙɻ
    4QPU
    qFFUͰ҆͘ӡ༻͍ͯ͠Δɻ
32. σϓϩΠϑϩʔ
     
    EJHϑΝΠϧॻ͘
    
    1VMM3FRVFTUग़͢
    
    Ϛʔδ͞ΕΔͱ$JSDMF$*͕σϓϩΠ͢Δ
    
    
    
    
    ɾ$JSDMF$*಺Ͱ
    EJHEBH
    QVTI ઃఆϑΝΠϧͷ൓ө
    Λ͢Δ
    
    
    
    
    ɾEJHϑΝΠϧͷมߋͷΈͷ৔߹͸ɺ࠶ىಈෆཁ
    
    ࠶ىಈ͍ͨ͠৔߹͸ɺTMBDL
    CPUʹ໋ྩ
33. ຊ൪δϣϒ࣮ߦதʹ࠶ىಈͯ͠΋໰୊ͳ͍ɻ
    EJHEBHͷ%#͕ঢ়ଶΛอ͍࣋ͯͯ͠ɺ
    ͏·੍͘ޚ͍ͯ͠Δɻ σϓϩΠϑϩʔ
    
34. ࣦഊஊ
     ɾEJHϑΝΠϧ͸%3:ʹ͗͢͠ͳ͍
    
    
    
    
    ɾJODMVEFͱ͔࢖ͬͯ%3:ʹͰ͖Δ͚Ͳɺ۪௚ʹॻ͍ͨ΄͏͕ॲཧΛ௥ ͍΍͍͢
    ɾ00.ରࡦ͕৬ਓ͡ΈͯΔ
    
    
    
    
    ɾฒྻ౓Λ্͛͗͢ͳ͍
    
    
    
    
    ɾ৔߹ʹΑͬͯ͸ɺ
    @QBSBMMFM
    USVF
    Λආ͚Δ
    
    
    
    
    ɾEJHEBHͷىಈΦϓγϣϯ
    NBYUBTLUISFBET
    Λௐ੔
    
    
    
    
    ɾFNCVML
    +9NYH
    SVO
    MBSHF@KPCZNM
    
    
    
    
    ɾFNCVMLͷZNMͷFYFD۟ͷNBY@UISFBET΍NJO@PVUQVU@UBTLTΛௐ੔
    
    
    
    
    ɾͦ΋ͦ΋ͷύΠϓϥΠϯͷઃܭΛݟ௚͢
35. ࣦഊஊ σʔλύΠϓϥΠϯߏங࣌ʹ͍ͭͯ
     ɾ#JH2VFSZͷσʔληοτͷ෼͚ํ
    
    
    
    
    ɾσʔληοτʹແ਺ͷςʔϒϧ͕͋Δ
    
    
    
    
    ɾEFWTUHQSPEΛ($1ϓϩδΣΫτͰ෼͚ͨΓɺσʔληοτ໊Λ ޻෉͢΂͖ͩͬͨ
    
    
    
    
    ɾࠩ෼ಉظ͔Ͳ͏͔஌Βͳ͍ͱूܭϛεΔ໰୊
    
    
    
    
    ɾར༻ऀʹύΠϓϥΠϯࣄ৘Λҙࣝͤ͞ͳ͍ઃܭʹ͢Δ΂͖ͩͬͨ
    
    
    
    
    ɾ͜Ε͔Β࡞Δਓ͸ඞಡˠ
    ࢲͷߟ͑ͨ࠷ڧͷϩάˍϞχλϦϯάઃ ܭ
    
    Լொ༘ࢠԫࠚه
    CZ
    !ZV[VUBT
    ɾ#JH2VFSZͷεΩʔϚࣗಈਪ࿦Λ࢖͏৔߹͸஫ҙ
    
    
    
    
    ɾ࿦ཧ࡟আܥͷϑϥάͳͲɺ΄΅/6--ͳΧϥϜ͕͋ΔͱɺεΩʔϚ ͷBVUPEFUFDUΛ͢Γൈ͚Δ શεΩϟϯ͢ΔΘ͚Ͱ͸ͳ͍ͷͰ ˞
    EJHEBHؔ܎ͳ͍෦෼
36. ·ͱΊ
     ɾσʔλύΠϓϥΠϯΛͲ͏૊ΜͰ͍Δͷ͔
    
    
    
    
    ɾҰ୴ɺ4ʹू·ΔΑ͏ͳύΠϓϥΠϯʹͯ͠Δ
    
    
    
    
    ɾ'JSFIPTFڬΜͰ͓͘ͱศར
    ɾEJHEBH͸ศར
    
    
    
    
    ɾఆظ࣮ߦܥͷॲཧҎ֎ʹ΋༗ޮ
    
    
    
    
    ɾ.-
    0QTʹ΋Ԡ༻Ͱ͖Δ
    ɾΠϯϑϥߏ੒΍σϓϩΠϑϩʔ
    
    
    
    
    ɾ&$4
    XJUI
    4QPUqFFUͰ҆͘ӡ༻
    
    
    
    
    ɾ$JSDMF$*΍$IBUCPUͷ׆༻
    ɾ00.ʹؾΛ͚ͭΑ͏
    ɾ#JH2VFSZΛ࢖͏৔߹͸σʔληοτͷઃܭʹ஫ҙ͠Α͏