digdagで支えるデータパイプライン / Building a data pipeline with digdag

by Yukiyan

Slide 1

Slide 1 text

EJHEBHͰࢧ͑ΔσʔλύΠϓϥΠϯ %BUB1JQFMJOF$BTVBM5BML एࢁ༟ૅ

Slide 2

Slide 2 text

ࣗݾ঺հ एࢁ༟ૅ :VLJ8BLBZBNB !ZVLJZBO@X IUUQTHJUIVCDPNZVLJZBO$VSSJDVMVN7JUBF ɾ+6#*-&&803,4Ͱ5JNF5SFF ΧϨϯμʔγΣΞΞϓϦ ͷ։ൃ ɾओʹσʔλ෼ੳج൫ͷ࢓ࣄΛ΍͍ͬͯ·͢

Slide 3

Slide 3 text

ΧϨϯμʔʹ͓ޓ͍ͷ༧ఆΛॻ͖ࠐΉ ΧϨϯμʔγΣΞΞϓϦ 5JNF5SFFͷ঺հ ܰ͘

Slide 4

Slide 4 text

ձࣾ৘ใͱن໛ײ ɾઃཱ೥݄ ɾݱࡏͷैۀһ਺໊͘Β͍ ɾ5JNF5SFFͷϦϦʔε೥݄ ɾϢʔβʔ਺ສ ɾάϩʔόϧʹϢʔβʔ͕͍ͯɺ ೔ຊϢʔβʔׂ͕ఔ౓Λ઎ΊΔ ɾେ͖͍ςʔϒϧͰ਺ඦ(# ਺ेԯϨίʔυఔ౓

Slide 5

Slide 5 text

લఏ ૊৫ʹ͓͚Δࢲͷ໾ׂ ɾۀ຿ʹ͓͍ͯ໌֬ͳ໾ׂ෼୲͸͍ͯ͠ͳ͍ ɾϥϕϧͱͯ͠͸όοΫΤϯυΤϯδχΞ͕ͩɺʮࣗ෼͕΍Γ͍ͨ͜ͱʯ ͔ͭʮձࣾʹඞཁͦ͏ͳ͜ͱʯΛࣗ෼Ͱ൑அͯ͠΍͍ͬͯΔ ɾ3BJMTͰͷ৽ػೳ։ൃͯ͠Δͱ͖΋͋Ε͹ɺσʔλύΠϓϥΠϯ੔උ ʹऴ࢝͢Δ͜ͱ΋͋Δ ɾ৽ن1+اըϚωʔδΛ͢Δ͜ͱ΋͋Δ ɾػցֶशج൫ΛҰਓͰߏங͢Δͱ͍͏͜ͱ

Slide 6

Slide 6 text

ࠓ೔࿩͢͜ͱ ɾσʔλύΠϓϥΠϯΛͲ͏૊ΜͰ͍Δͷ͔ ɾEJHEBHͷ঺հ ɾ࣮ࡍʹ૊ΜͩύΠϓϥΠϯࣄྫ ɾΠϯϑϥߏ੒΍σϓϩΠϑϩʔ ɾࣦഊஊ

Slide 7

Slide 7 text

ࠓ೔࿩͞ͳ͍͜ͱ ɾෳ਺ϓϩμΫτԣஅͷσʔλج൫ ɾฐࣾͷϓϩμΫτ͸5JNF5SFFͷΈͳͷͰ࿩ͤͳ͍ ɾσʔλύΠϓϥΠϯͷ඼࣭อূ ɾϞχλϦϯά΍ςετʹ͍ͭͯ͸·ͩϕετͳํ๏͕Θ͔ͬͯͳ͍ ɾ૊৫ͬΆ͍࿩ ɾ࢖ͬͯ΋Β͑Δσʔλج൫ͱ͸ ܥͷ͋Ε͜Ε ɾσʔλαΠΤϯςΟετͱͷίϛϡχέʔγϣϯ͋Ε͜Ε ɾͦ΋ͦ΋ฐࣾʹ%4͸͍ͳ͍ ŝũŞƀ

Slide 8

Slide 8 text

ѻͬͯΔσʔλͷछྨ ɾΞΫηεϩά 3BJMT΍OHJOY ɾΞϓϦέʔγϣϯσʔλ %#ʹ֨ೲ͢ΔλΠϓͷσʔλ ɾΞΫγϣϯϩά ΫϦοΫϩά΍ΠϯϓϨογϣϯϩά౳ͷ޿ࠂϩά ɾSEQBSUZπʔϧ΁ૹͬͯΔϩά ޿ࠂܭଌπʔϧ΁ૹͬͯΔϩά ʜFUD

Slide 9

Slide 9 text

σʔλͷྲྀΕ ɾҰ୴ɺ4ʹू·ΔΑ͏ͳύΠϓϥΠϯ ɾ4ͱ($4ʹಉ͡σʔλ͕͋Δ ɾ4΍($4ʹݩσʔλ͕͋Ε͹ɺকདྷతʹ Ԡ༻ͷ෯͕޿͕Γ΍͍͢ $MPVE%BUBqPX౳

Slide 10

Slide 10 text

EJHEBHͷબఆཧ༝ ɾҎલ͔ΒqVFOUE FNCVMLΛศརʹ࢖͍ͬͯͨͷͰɺ5SFBTVSF%BUB ͷ੡඼ʹ৴པײ͕͋ͬͨ ɾ೥ࠒʹൃද͞Ε͙ͯ͢ʹ࢖ͬͯΈͯɺߏங͕؆୯ͩ͠ʮ͜Ε͸ ΠέΔʯͱࢥͬͨ lσʔλॲཧϫʔΫϑϩʔͱ͍͏΋ͷ͸ԟʑʹͯ͠ෳࡶʹͳΓɺδϣϒؒͷґଘؔ܎͕Ͳ͏ͯ͠΋ ग़ͯ͠·͏΋ͷ͕ͩɺEJHEBHͷઃఆϑΝΠϧ͸಄ʹු͔ΜͩϫʔΫϑϩʔΛઃఆϑΝΠϧʹམͱ ͠ࠐΈ΍͍͢͠௚ײతͩ͠ػೳ΋๛෋ͳͷͰͱͯ΋ظ଴ͷͰ͖Δ΋ͷͩͬͨɻz %JHEBHFNCVML#JH2VFSZ3FEBTIͰσʔλ෼ੳج൫ߏஙͷເΛݟΔࡶͳϝϞ

Slide 11

Slide 11 text

EJHEBHͷબఆཧ༝ ɾ"JSqPX΍-VJHJʹ͍ͭͯ͸ɺݫີͳൺֱ͸͍ͯ͠ͳ͍͕ͲΕ΋ඞཁͳ ػೳ͸Χόʔͯ͠Δͱࢥ͏ͷͰɺνʔϜͷಛੑʹ߹ΘͤͯͲΕ࢖ͬͯ΋ ͍͍ͱࢥ͏ ͨͩEJHEBH͸ΠΠͧ

Slide 12

Slide 12 text

EJHEBHͷ঺հ ɾ؅ཧը໘ ɾϦτϥΠ ɾΤϥʔ௨஌ ɾEJHϑΝΠϧͷه๏

Slide 13

Slide 13 text

δϣϒͷ༷ࢠ͕ͻͱ໨ͰΘ͔ΔͷͰɺΤϥʔ͕ى͖ͨͱ͖ʹศར ؅ཧը໘

Slide 14

Slide 14 text

ϦτϥΠ

Slide 15

Slide 15 text

Τϥʔ௨஌ δϣϒ͕ࣦഊͨ͠ͱ͖ʹɺTMBDL௨஌Ͱ͖Δ T[ZOEJHEBHTMBDLͷϓϥάΠϯΛ࢖͍ͬͯΔ

Slide 16

Slide 16 text

EJHϑΝΠϧͷه๏ εέδϡʔϧઃఆ TIFMMTDSJQU࣮ߦ 3VCZͷ࣮ߦ 1ZUIPO΋Մ

Slide 17

Slide 17 text

EJHϑΝΠϧͷه๏ εΫϦϓτؒͰͷม਺౉͠ .Z42-ͷςʔϒϧҰཡͱ͔εΫϦϓτͰऔಘͯ֨͠ೲͨ͠Γ͢Δ

Slide 18

Slide 18 text

EJHϑΝΠϧͷه๏ ฒྻ࣮ߦ

Slide 19

Slide 19 text

EJHϑΝΠϧͷه๏ CRΦϖϨʔλͰதؒςʔϒϧΛ࡞੒ SFETIJGU FNSΦϖϨʔλ౳ศརͳ΋ͷ͕͍͔ͭ͋͘Δ

Slide 20

Slide 20 text

EJHϑΝΠϧͷه๏ λεΫΛ೚ҙͷEPDLFSΠϝʔδ಺Ͱ࣮ߦ

Slide 21

Slide 21 text

EPDLFS࣮ߦʹ͍ͭͯ ɾগ͠บ͕͋ΔͷͰӈهࢀর EJHEBHλεΫͷEPDLFS࣮ߦʹ͍ͭͯ ɾWBSSVOEPDLFSTPDLΛϚ΢ϯτ͢Δඞཁ͕͋ΔͷͰ஫ҙ ɾEJHEBHࣗମΛEPDLFS &$4 Ͱߏஙͯ͠Δ৔߹ɺEJOEʹͳͬͯ͠·͍ ߟ͑Δ͜ͱ͕૿͑Δ ɾͦ͜·Ͱੵۃతʹ࢖ͬͯ͸͍ͳ͍ ɾͰ΋ɺվળ͸ΊͬͪΌਐΜͰΔͷͰظ଴ @FYQPSUEPDLFSTVQQPSUTEPDLFSPQUJPOBSHVNFOUT OFYUBEE UFTUT

Slide 22

Slide 22 text

EJHEBHͰ࣮ࡍʹ૊ΜͩύΠϓϥΠϯࣄྫ ʰ.Z42-ͷσʔλΛ#JH2VFSZ΁ྲྀ͢ʱ 3VCZͰςʔϒϧҰཡΛऔಘ FNCVMLJOQVUNZTRMͰ4΁ΤΫεϙʔτ 4ͱ($4ΛTZOD #JH2VFSZ΁ϩʔυ͢Δ 3FEBTIͷSFGSFTI"1*Λୟ͍ͯάϥϑߋ৽

Slide 23

Slide 23 text

ʰಛఆσʔλͷநग़݁ՌΛTMBDL΁Ξοϓϩʔυʱ EJHEBHͰ࣮ࡍʹ૊ΜͩύΠϓϥΠϯࣄྫ

Slide 24

Slide 24 text

ʰ/14ͷ಺༰ΛTMBDL΁௨஌ʱ (PPHMF$MPVE5SBOTMBUJPO"1*΋࢖͍ɺଟݴޠʹ΋ରԠ EJHEBHͰ࣮ࡍʹ૊ΜͩύΠϓϥΠϯࣄྫ

Slide 25

Slide 25 text

ʰ͋Δςʔϒϧͷ೚ҙͷΧϥϜΛLVSPNPKJͰܗଶૉղੳޙɺ໊ࢺΛ (PPHMF/BUVSBM-BOHVBHF"1*ʹט·ͤͯΤϯςΟςΟ෼ੳ͢Δʱ #JH2VFSZͰର৅ͷநग़΍લॲཧ ($4΁ΤΫεϙʔτ FNCVMLpMUFSLVSPNPKJͰ໊ࢺΛநग़ #JH2VFSZ΁ϩʔυ͢Δ ($4΁ΤΫεϙʔτ FNCVMLpMUFSHPPHMF@OBUVSBM@MBOHVBHF@BQJͰΤϯςΟςΟ෼ੳ #JH2VFSZ΁ϩʔυ͢Δ ΤϯςΟςΟ෼ੳ݁ՌͷUZQF͕-0$"5*0/Ͱ͋ΔςʔϒϧΛ࡞੒ ఆظ࣮ߦܥͷॲཧ͚ͩͰ͸ͳ͘ɺ ͜ͷΑ͏ͳ୯ൃͷΞυϗοΫͳύΠϓϥΠϯΛ࣮ߦ͢Δͷʹ΋ศར EJHEBHͰ࣮ࡍʹ૊ΜͩύΠϓϥΠϯࣄྫ

Slide 26

Slide 26 text

ʰ͋Δςʔϒϧͷ೚ҙͷΧϥϜΛࡐྉʹɺ GBTU5FYUͰ/-1ʹ͓͚Δ୯ޠͷ෼ࢄදݱΛऔಘʱ #JH2VFSZͰର৅ͷநग़΍લॲཧ ($4΁ΤΫεϙʔτ FNCVMLpMUFSLVSPNPKJͰ໊ࢺΛநग़ GBTU5FYUͰ෼ࢄදݱΛऔಘ ϞσϧΛ4ʹΞοϓϩʔυ EJHEBHͰ࣮ࡍʹ૊ΜͩύΠϓϥΠϯࣄྫ .-0QTʹ΋Ԡ༻Ͱ͖Δ

Slide 27

Slide 27 text

ʰ4QBSL EBUBCSJDLT ͷͨΊͷσʔλ४උͱֶशͷ࣮ߦʱ #JH2VFSZͰର৅ͷநग़΍લॲཧ ($4ͱ4΁ΤΫεϙʔτ EBUBCSJDLTͷ"1*Λୟ͘ SEQBSUZπʔϧͱͷ࿈ܞʹ΋Ԡ༻Ͱ͖Δ EJHEBHͰ࣮ࡍʹ૊ΜͩύΠϓϥΠϯࣄྫ

Slide 28

Slide 28 text

ඇج൫ΤϯδχΞʮ͜ͷॲཧͷϩάΛ#JH2VFSZʹೖΕ͍ͨͳ͊ʯ

Slide 29

Slide 29 text

σʔλͷྲྀΕ ࠶ܝ ඇج൫ΤϯδχΞʮ͜ͷॲཧͷϩάΛ#JH2VFSZʹೖΕ͍ͨͳ͊ʯ ৽ͨͳ޿ࠂϩάΛ௥Ճ͍ͨ͠ͱ͢Δ

Slide 30

Slide 30 text

ɾpSFIPTFʹೖΕΔ͜ͱ͚ͩΛߟ͑Ε͹͍͍Α͏ʹ͍ͯ͠Δ UFSSBGPSN qVFOUMPHHFSSVCZ qVFOUEDPOG EJHϑΝΠϧΛগ͍͠͡Δ͚ͩ ඇج൫ΤϯδχΞʮ͜ͷॲཧͷϩάΛ#JH2VFSZʹೖΕ͍ͨͳ͊ʯ

Slide 31

Slide 31 text

Πϯϑϥߏ੒ &$4্ʹߏஙɻ 4QPUqFFUͰ҆͘ӡ༻͍ͯ͠Δɻ

Slide 32

Slide 32 text

σϓϩΠϑϩʔ EJHϑΝΠϧॻ͘ 1VMM3FRVFTUग़͢ Ϛʔδ͞ΕΔͱ$JSDMF$*͕σϓϩΠ͢Δ ɾ$JSDMF$*಺ͰEJHEBHQVTI ઃఆϑΝΠϧͷ൓ө Λ͢Δ ɾEJHϑΝΠϧͷมߋͷΈͷ৔߹͸ɺ࠶ىಈෆཁ ࠶ىಈ͍ͨ͠৔߹͸ɺTMBDLCPUʹ໋ྩ

Slide 33

Slide 33 text

ຊ൪δϣϒ࣮ߦதʹ࠶ىಈͯ͠΋໰୊ͳ͍ɻ EJHEBHͷ%#͕ঢ়ଶΛอ͍࣋ͯͯ͠ɺ ͏·੍͘ޚ͍ͯ͠Δɻ σϓϩΠϑϩʔ

Slide 34

Slide 34 text

ࣦഊஊ ɾEJHϑΝΠϧ͸%3:ʹ͗͢͠ͳ͍ ɾJODMVEFͱ͔࢖ͬͯ%3:ʹͰ͖Δ͚Ͳɺ۪௚ʹॻ͍ͨ΄͏͕ॲཧΛ௥ ͍΍͍͢ ɾ00.ରࡦ͕৬ਓ͡ΈͯΔ ɾฒྻ౓Λ্͛͗͢ͳ͍ ɾ৔߹ʹΑͬͯ͸ɺ@QBSBMMFMUSVFΛආ͚Δ ɾEJHEBHͷىಈΦϓγϣϯNBYUBTLUISFBETΛௐ੔ ɾFNCVML+9NYHSVOMBSHF@KPCZNM ɾFNCVMLͷZNMͷFYFD۟ͷNBY@UISFBET΍NJO@PVUQVU@UBTLTΛௐ੔ ɾͦ΋ͦ΋ͷύΠϓϥΠϯͷઃܭΛݟ௚͢

Slide 35

Slide 35 text

ࣦഊஊ σʔλύΠϓϥΠϯߏங࣌ʹ͍ͭͯ ɾ#JH2VFSZͷσʔληοτͷ෼͚ํ ɾσʔληοτʹແ਺ͷςʔϒϧ͕͋Δ ɾEFWTUHQSPEΛ($1ϓϩδΣΫτͰ෼͚ͨΓɺσʔληοτ໊Λ ޻෉͢΂͖ͩͬͨ ɾࠩ෼ಉظ͔Ͳ͏͔஌Βͳ͍ͱूܭϛεΔ໰୊ ɾར༻ऀʹύΠϓϥΠϯࣄ৘Λҙࣝͤ͞ͳ͍ઃܭʹ͢Δ΂͖ͩͬͨ ɾ͜Ε͔Β࡞Δਓ͸ඞಡˠࢲͷߟ͑ͨ࠷ڧͷϩάˍϞχλϦϯάઃ ܭԼொ༘ࢠԫࠚهCZ!ZV[VUBT ɾ#JH2VFSZͷεΩʔϚࣗಈਪ࿦Λ࢖͏৔߹͸஫ҙ ɾ࿦ཧ࡟আܥͷϑϥάͳͲɺ΄΅/6--ͳΧϥϜ͕͋ΔͱɺεΩʔϚ ͷBVUPEFUFDUΛ͢Γൈ͚Δ શεΩϟϯ͢ΔΘ͚Ͱ͸ͳ͍ͷͰ ˞EJHEBHؔ܎ͳ͍෦෼

Slide 36

Slide 36 text

·ͱΊ ɾσʔλύΠϓϥΠϯΛͲ͏૊ΜͰ͍Δͷ͔ ɾҰ୴ɺ4ʹू·ΔΑ͏ͳύΠϓϥΠϯʹͯ͠Δ ɾ'JSFIPTFڬΜͰ͓͘ͱศར ɾEJHEBH͸ศར ɾఆظ࣮ߦܥͷॲཧҎ֎ʹ΋༗ޮ ɾ.-0QTʹ΋Ԡ༻Ͱ͖Δ ɾΠϯϑϥߏ੒΍σϓϩΠϑϩʔ ɾ&$4XJUI4QPUqFFUͰ҆͘ӡ༻ ɾ$JSDMF$*΍$IBUCPUͷ׆༻ ɾ00.ʹؾΛ͚ͭΑ͏ ɾ#JH2VFSZΛ࢖͏৔߹͸σʔληοτͷઃܭʹ஫ҙ͠Α͏