Cloud Composer & Dataflow によるバッチETLの再構築 #data_ml_engineering / 20190719

Cloud Composer & Dataﬂow ʹΑΔ  όονETLͷ࠶ߏங 2019-07-19  #data_ml_engineering  presented by
@yuzutas0   https://www.pexels.com/photo/architecture-blur-building-colourful-392031/ https://www.pexels.com/photo/architecture-blur-building-colourful-392031/

WEBʹެ։ࡁΈͰ͢ #data_ml_engineering  ɹࡱӨ΍ϝϞ͸ෆཁͰ͢ɻϦϥοΫεͯ͠ฉ͍͍͚ͯͨͩΕ͹ͱࢥ͍·͢ɻ    εϥΠυ 70+ ຕ  ɹΞδΣϯμʲ4ʳΛॏ఺తʹɺଞ͸ϥΠτχϯάͰτʔΫ͠·͢ɻ  ɹ࠙਌λΠϜɾSNSͰͷQ&AαϙʔτΛલఏͱͨ͠಺༰ʹͳΓ·͢ɻ ςΫϊϩδʔ͸෢ثͩͱࢥ͍ͬͯ·͢ 
ɹ໨తɾ੍໿ʹԠͯ͡࢖͍෼͚·͠ΐ͏ɻಛఆͷٕज़ཁૉΛਪ঑͢ΔൃදͰ͸͋Γ·ͤΜɻ ɹ஫ҙɾ໔੹

1. ͸͡Ίʹ 2. ίϯςΩετ 3. ܭଌɺݕ౼ɺ߹ҙܗ੒ 4. ϦϏϧυˍϦϦʔε 5. ͓ΘΓʹ
ɹΞδΣϯμ

ɹ@yuzutas0 ɹɹ    

ɹաڈͷొஃࢿྉ σʔλج൫ͷϊ΢ϋ΢ɾ஌ݟΛఏڙ͍ͯ͠·͢    PyCon JP ϕεττʔΫΞϫʔυ༏ल৆ σϒαϛՆ Ξϯέʔτຬ଍౓No.1

ʮ࠶ߏஙʯͷࣄྫΛఏڙ͢Δ ͋͘·Ͱ1ͭͷࣄྫͳͷͰ  ࣗ͝਎ͷٕज़ཁૉ΍૊৫ঢ়گͱൺ΂ͳ͕Βߟ͑ͯ  ࣗ෼ͳΓͷֶͼΛಘ͍ͯͩ͘͞ ɹຊ೔ͷझࢫ

ϩάऩू΍ETLʹ͍ͭͯ  γεςϜߏஙɾӡ༻ͷ࣮຿Λ୲͏  ιϑτ΢ΣΞΤϯδχΞ ͱɺͦͷΫϥΠΞϯτɾϚωʔδϟʔʢʹͳΔ༧ఆͷਓʣ ɹຊ೔ͷ૝ఆλʔήοτ

ɹΞδΣϯμ

ɹϝϧΧϦʢCtoCϑϦϚʣ

FY2019.6 3Q ܾࢉઆ໌ձࢿྉ https://pdf.irpocket.com/C4385/eHSm/vwwn/oECA.pdf ɹࣄۀ੒௕ʢʹσʔλ૿ྔʣ

ɹάϩʔόϧɾ৽نࣄۀ

https://speakerdeck.com/hik0107/mercari-bi-team-data-analytics-summit-2018 ɹੵۃతͳσʔλ׆༻

ɾϓϩμΫτ͕৳ͼ͍ͯΔ ɾσʔλྔ͕ٸܹʹ૿͍͑ͯΔ  ɾάϩʔόϧ΍৽نࣄۀΛ৳͹͢ମ੍Λ࡞͍ͬͯΔ ɾ෼ੳ΍MLͳͲσʔλΛੵۃతʹ׆༻͍ͯ͠Δ ɹ·ͱΊ of ಛ௃

ʮBQͷσʔλ͕ߋ৽͞Ε͍ͯͳ͍ΜͰ͚͢Ͳʂʯ   ʢҰ෦ͷςʔϒϧ͸൒݄΋ࢭ·͍ͬͯͨʣ ɹݱ৔Ͱੜ͍ͯͨ͡՝୊

ɹ੒௕௧ ϓϩμΫτˢ σʔλˢ ෛՙˢ ར༻ऀˢ Good Good Bad
Bad ❌ γεςϜɺବ໨Ͱ͢ʂ ߋ৽͞Ε͍ͯͳ͍΍Μʂ

ɹྺ࢙తܦҢ ETL System ETL  for US ETL  for
JP ࡞ͬͨʂ ϝϯςʂ US Team ຊۀͷ๣Β ળҙͰαϙʔτ  ʢਖ਼௚ݶք͕͋Δʣ JP SRE JP BI JP΋ཉ͍͠ʂ  ૬৐Γͤͯ͞ʂ ґཔ USΞϓϦΛ ྑ͘͢Δͧʂ JPΞϓϦຊ൪؀ڥ  ͕࠷༏ઌͩʂ ෼ੳۀ຿ʹ ઐ೦͢Δͧʂ ETL  for UK

ɹ͜ͷҊ݅ͷείʔϓᶃ ϓϩμΫτ Ϣʔβʔ DBɾϩά ࢪࡦɾۀ຿ BigQuery ऩू ૄ௨
׆༻ Ձ஋ %BUB0QTʹ͓͍ͯ ࠷େԽ͢΂͖໨తม਺

ɹ͜ͷҊ݅ͷείʔϓᶄ Other Product  DB .POPMJUI "11#& Other Other
BigQuery ॱ࣍Ҡ؅༧ఆ Read Only Replica ػີ৘ใ ϚεΩϯά DB .JDSP  TFSWJDFT DB .JDSP  TFSWJDFT DB .JDSP  TFSWJDFT ੴङDC GCP

ɾ൒݄΋ߋ৽͞Ε͍ͯͳ͍σʔλ ɾ͋ͳͨͩͬͨΒͲ͏͠·͔͢ʁ ɹToday’s Issue

ɹΞδΣϯμ

ɹؔ܎ऀώΞϦϯά ՝୊ ղܾ ΞφϦετ ʮࢭ·ͬͯΔʂʯ ʮࠓ΄͍͠ʂ࢑ఆରԠΛʂʯ σϕϩού ʮݴ͏΄Ͳ͔ʁʯ
ʮ࠶ߏஙͨ͠΄͏͕͍͍ʂʯ

ɹܭଌ͢Δ ՝୊ ղܾ ΞφϦετ ʮࢭ·ͬͯΔʂʯ ʮࠓ΄͍͠ʂ࢑ఆରԠΛʂʯ σϕϩού ʮݴ͏΄Ͳ͔ʁʯ

ؔ܎ऀҰಉʮ༧૝ΑΓ൵ࢂͳ͜ͱʹͳ͍ͬͯΔʯ ɹBQߋ৽஗ԆbotΛ࡞ͬͨ

ຖ࣮࣌ߦ dataset.__TABLES__ ΛSELECT  ϝλ৘ใΛεφοϓγϣοτอଘ pandas.read_csv() Ͱऔಘ  νΣοΫ࣌ؒɺର৅ςʔϒϧ  ௨஌ઌνϟϯωϧ pandas.read_gbq() Ͱ
ςʔϒϧ໊ͱ  ࠷ऴߋ৽೔࣌Λऔಘ ߋ৽༗ແΛ൑ఆ slackweb.Slack(). notify() Ͱ ࢦఆνϟϯωϧʹ௨஌ ɹBQ update checker / implementation IUUQTXXXqBUJDPODPNGSFFJDPODTW@ ύωϧσʔλΛ෼ੳͰ͖ΔΑ͏ʹ஝ੵ

ɹBQ update checker / design http://yuzutas0.hatenablog.com/entry/2017/05/23/073000 BigQuery

ɹBQ update checker / docs for user (1)

ɹBQ update checker / docs for user (2)

ɹՄࢹԽ → ߹ҙܗ੒ ՝୊ ղܾ ΞφϦετ ʮࢭ·ͬͯΔʂʯ ʮࠓ΄͍͠ʂ࢑ఆରԠΛʂʯ
σϕϩού ʮݴ͏΄Ͳ͔ʁʯ ʮ࠶ߏஙͨ͠΄͏͕͍͍ʂʯ ༏ઌॱΛ্͛ͯରԠʂ

ɹԆ໋͢Δ ՝୊ ղܾ ΞφϦετ ʮࢭ·ͬͯΔʂʯ ʮࠓ΄͍͠ʂ࢑ఆରԠΛʂʯ σϕϩού ʮݴ͏΄Ͳ͔ʁʯ

  ΞφϦετͱҰॹʹʮͱΓ͋͑ͣϦτϥΠʯ    ஗Ԇ͍ͯ͠ͳ͍ςʔϒϧͷ࿈ܞ·Ͱಓ࿈ΕͰશ໓  ʢೋ࣍ࡂ֐ʣ ʮར༻ऀ͕૝ఆ͍ͯ͠Δ΄Ͳ؆୯ͳঢ়گͰ͸ͳ͍ʯ͕ՄࢹԽ͞Εͨ  ɹ࢑ఆରԠ IUUQTXXXQFYFMTDPNQIPUPCSPXOBOEXIJUFUBCCZLJUUFO

USݖݶΛ࢑ఆൃߦͯ͠΋Βͬͯௐࠪ։࢝  ॏ͗ͯ͢؅ཧը໘͕։͚ͳ͍ ίπΛڭ͑ͯ΋Β͏ͱ͜Ζ͔Β…… http://{ip_or_domain}/admin/airﬂow/tree?dag_id={id}&num_runs=1 ɹ҉த໛ࡧ IUUQTXXXQFYFMTDPNQIPUPHSFZDPODSFUFSPBE

ɾσʔλ૿Ճʹ൐͏λΠϜΞ΢τ͕ଟൃ  ɾશδϣϒ͕௚ྻ࣮ߦͰޙଓॲཧΛר͖ࠐΉ  ʢJDBC→DBͷΞΫηεෛՙΛ཈͑ΔҙਤͰͷઃܭʣ  ɾUSνʔϜ΋ಉ͡࢓૊Έ͕ͩδϣϒͷ෼͚ํΛ޻෉  ɾJP͸ͦ͜·Ͱग़དྷ͍ͯͳ͔ͬͨ  ʢ૬৐Γʴยखؒͷળҙαϙʔτͩͱݶք͕͋Δʣ ɹௐࠪ

Ԧಓͷखஈͱͯ͠͸USνʔϜͱಉ༷ͷνϡʔχϯά ʢ҆қͳ࠶ߏஙʹಀ͛ͳ͍ʂʣ   ͨͩ͠  ɾ࢓૊ΈΛΩϟονΞοϓ͢Δͱ͜Ζ͔Βελʔτ ɾෛՙͰΤϥʔ͕ى͖͍ͯΔطଘγεςϜӨڹΛߟྀ͠ͳ͕Β࡞ۀ ɹνϡʔχϯά͔ʁ

ϝϧϖΠDataplatformTeam͔ΒఏҊ  ʮ͜ΜͳΜ࡞ͬͨΜ͚ͩͲྑ͔ͬͨΒԣల։͠·ͤΜʁʯ ɹϦϏϧυ͔ʁ ϝϧϖΠʹ͓͚Δେن໛όονॲཧ - Mercari Engineering Blog 
https://tech.mercari.com/entry/2019/06/05/120000

̋ ̋ ˕ ˕ ɹൺֱݕ౼ γεςϜ αϙʔτ 64 
&5-4ZTUFN "JSqPXPO(,& 4QBSLFBSMZ    νϡʔχϯά͢Ε͹ػೳཁ݅ΛຬͨͤΔ ͸ͣ ஍ཧɾ͕࣌ࠩ͋Δ    ඇಉظͰ૬ஊ͸Մೳ .FSQBZ  #BUDI1JQFMJOF $MPVE$PNQPTFS %BUBqPXMBUFMZ    ػೳཁ݅ΛຬͨͤΔ  GVMMNBOBHFEͰ૬ରతʹ࢖͍΍͍͢ ͸ͣ ෺ཧతʹΦϑΟε͕͍ۙ    ૬ஊ͠΍͍͢

໌Β͔ʹ “ETLγεςϜઃܭ” ͷ໰୊Ͱ͸ͳ͘  ”JPઐ೚ϝϯςφͷ௕ظෆࡏ” ͱ “ͦ͏ͳΔʹࢸͬͨ૊৫తྗֶ” ͕  ਅʹղ͘΂͖Πγϡʔ    
“σʔλૄ௨͕ࢭ·͍ͬͯΔ” ͸ණࢁͷҰ֯  ͳΔ΂͘ϚΠϯυγΣΞΛׂ͔ͣʹࡁΉΑ͏ʹ  “͍͔ʹٕज़໘ͰϥΫͯ͠ରԠ͢Δ͔” ͕ҙࢥܾఆͷ࣠ͱͳΔ ɹҙࢥܾఆͷϙΠϯτ IUUQTXXXJSBTVUPZBDPNCMPHQPTU@IUNM

  https://www.pexels.com/photo/architecture-blur-building-colourful-392031/ ࠶ߏஙʴར༻੾ସͷ΄͏͕ૣ͘׬ྃͰ͖Δͱ൑அ ʢ҆қͳ࠶ߏஙʹಀ͛·ͨ͠ʂʣ ɹϦϏϧυʂ ͪͳΈʹΦν͸ ɹᶃϝϧϖΠͷύΠϓϥΠϯ͸ϑϧGCPલఏͷߏ੒ͳͷͰɺͦͷ··ͷԣల։͸ग़དྷͳ͔ͬͨ  ɹᶄUSνʔϜ͸USνʔϜͰ஗ԆՄࢹԽΛड͚ͯJPͷδϣϒΛվमͯͩͬͨ͘͠͞

ɹՄࢹԽ → ߹ҙܗ੒ ՝୊ ղܾ ΞφϦετ ʮࢭ·ͬͯΔʂʯ ʮࠓ΄͍͠ʂ࢑ఆରԠΛʂʯ
σϕϩού ʮݴ͏΄Ͳ͔ʁʯ ʮ࠶ߏஙͨ͠΄͏͕͍͍ʂʯ ϑΥʔΧε͢Δ ࢑ఆରԠʹ࣌ؒɾ࿑ྗΛׂ͔ͳ͍

ɾ୅ସςʔϒϧͷ֓ࢉ஋Ͱࡁ·ͤΔ ɾBQʹͳ͍σʔλΛεΫϦϓτͰࢀর͢Δ ɾ஌ݟ΍πʔϧΛੵۃతʹڞ༗͠߹͏ ෆ҆ఆͳγεςϜʹա৒ґଘͤͣʹۀ຿Λ਱ߦ͢ΔੌΈ͕͋Δοʂ ʢ໨త஍ʹḷΓணͨ͘Ίͷखஈɾܦ࿏͸1ͭͰ͸ͳ͍ʣ ɹΞφϦετͷ޻෉͕͋ͬͯͦ͜ https://www.pexels.com/photo/group-hand-ﬁst-bump-1068523/

ɹ߹ҙܗ੒·ͱΊ ՝୊ ղܾ ΞφϦετ ʮࢭ·ͬͯΔʂʯ ʮࠓ΄͍͠ʂ࢑ఆରԠΛʂʯ σϕϩού ʮݴ͏΄Ͳ͔ʁʯ

ɹΞδΣϯμ

ɹγεςϜߏ੒ Replica DB

ɹγεςϜߏ੒ Replica DB ͜͜͸  !TJSPLFO͞Μ͕ ྑ͍ײ͡ʹ  ΍ͬͯ͘Ε·ͨ͠

ɹγεςϜߏ੒ Replica DB ͜͜Λ࿩͠·͢

ɹCloud Composer: DAG Runs ᶃόϦσʔγϣϯ ᶄDataﬂow࣮ߦ ᶅGCSϑΝΠϧऔಘ ᶆBQ
Load (ࠩ෼ or શ݅)

ɹComposer → Dataﬂow ʢਖ਼֬ʹ͸GCS্ʹ഑උ͞Ε͍ͯΔʣTemplate Λࢦఆͯ͠ Cloud Dataﬂow ʹ࣮ߦ໋ྩΛૹΔ

ɹCloud Dataﬂow: ETL ᶃGCS͔ΒdumpϑΝΠϧΛread ᶄѱຐվ଄ͷม׵ॲཧͰσʔλΛmodify ᶅGCSʹBQ LoadableͳϑΝΠϧΛwrite ಈ࡞֬ೝͰΤϥʔΛ௵͠ͳ͕Β
ม׵ॲཧΛ࡞ΓࠐΉ ※ΤϯϋϯεͷͨΊ࠷৽ঢ়گͱဃ཭͕͋Γ·͢ɻ

ɹWhy Dataflow? ɾmysqldumpͷTSVϑΥʔϚοτͰ͸BigQueryʹLoadͰ͖ͳ͍ → ཁ੔ܗ  ɹɹɾdouble-quotation-marks escaped by
double-quotation-marks in double-quotation-marks  ɹɹɾnew-line escaped by double backslashes   ɾσʔλྔ͕ଟ͍ͷͰDBෛՙˍύϑΥʔϚϯε؍఺͔Β  ɹεέʔϥϏϦςΟͷߴ͍DataflowʹॲཧΛدͤͨ    ɾDataflow͸ม׵૷ஔͱͯ͠੹຿ΛׂΓ੾͍ͬͯΔͷͰ ɹDataflow → BigQuery ʹ௚LoadͤͣɺGCSʹม׵ޙϑΝΠϧΛஔ͍͍ͯΔ    ɾ࣮ߦ؀ڥ͸Python3.5 (supported at Apache Beam 2.11.0 / Mar 5, 2019)

ɹDataﬂow Onboard by @rilmayer_jp

ɹTest Code for Transform σόοάͰΤϥʔ͕ग़ͨ  σʔλύλʔϯΛςετʹ࢖͏ σόοάͰΤϥʔ͕ग़ͨ  ςʔϒϧͷσʔλΛςετʹ࢖͏
beamϞδϡʔϧ͸  MagicMockʹͯ͠ ϩδοΫ෦෼͚ͩ ίʔυͰςετ

ɹComposer → BQ: શ݅ߋ৽ GCS → BQ Load

ɹComposer → BQ: ࠩ෼ߋ৽ ݩςʔϒϧ + tmpςʔϒϧ  ˠ
Union ALL → ॏෳআڈ → ্ॻ͖ tmpςʔϒϧΛ࡟আ ࠩ෼σʔλΛtmpςʔϒϧʹload ৄ͘͠͸ҎԼͷهࣄΛࢀর͍ͩ͘͞ʂ ਺ඦGBͷσʔλΛMySQL͔ΒBigQuery΁ಉظ͢Δ  https://tech.mercari.com/entry/2018/06/28/100000

ɹRebuilt BQ / docs for user (1)

ɹRebuilt BQ / docs for user (2)

ɹRebuilt BQ / docs for user (3) ʢ൒݄์ஔ͞Ε͍ͯΔʣݱঢ়ΑΓ͸ 
ʮϚγʹͳΔʯͰσʔλར༻ऀͱѲΔ ɹɾա৒඼࣭ʹ͠ͳ͍ ɹɾܭଌʢ஗Ԇ؂ࢹʣͱαϙʔτ͸໌ه ɹɾᐆດͳ΋ͷ͸ᐆດͰ͋Δ͜ͱΛ໌ه

Ұ෦νʔϜʹఏڙ → ؀ڥґଘͷো֐ → ݕ஌ɾՐফ͠ɾରԠϑϩʔͷ੔උ ɹCanary Release

Sprint + Increment: ܧଓతվળͷϦζϜΛ࡞Δ ɹִिස౓Ͱஈ֊ϦϦʔε W W W
0QT Ұ෦ͷνʔϜ͔Βఏڙ ࣍ͷνʔϜʹ΋ఏڙ ʜʜ ར༻Ҋ಺W  2"ɾϑΟʔυόοΫ ར༻Ҋ಺W 2"ɾϑΟʔυόοΫ ʜʜ %BUB શ݅ߋ৽ͰࡁΉςʔϒϧ ࠩ෼ߋ৽͠ͳ͍ͱਏ͍ςʔϒϧ ʜʜ NZTRMEVNQͰ$47ϑΝΠϧ͕  (#ҎԼʹ෼ׂ͞ΕΔςʔϒϧ %BUBqPXͰ$47Λ෼ׂ͠ͳ͍ͱ  #2-PBE͕ࣦഊ͢Δςʔϒϧ ʜʜ վળ վળ վળ վળ վળ վળ վળ վળ

7hͰλΠϜΞ΢τ͍ͯͨ͠ߪങσʔλ࿈ܞ͕ɺ2.5hͰແࣄʹSuccessʂ 01:00 02:00 03:00 04:00 05:00 06:00 07:00 08:00 09:00
Before After ɹ݁Ռ ❌ ✅   લͷॲཧ

ɹΞδΣϯμ

ݸੑ๛͔ͳλϨϯτϓϨΠϠʔ͕ଟ͍૊৫ͳͷͰ ࣗ෼ͷྲّྀ΍ઃܭࢥ૝Λԡ͠௨͢ͷͰ͸ͳ͘ ӢͷΑ͏ʹॊೈʹܗΛม͑ͯʢCloudʣ ࢦشऀͷΑ͏ʹશମΛݟ౉͠ʢComposerʣ ৘ใͷྲྀΕΛ੔ཧ͠ͳ͕ΒਐΊͨʢDataﬂowʣ ·͞ʹ "Cloud Composer & Dataﬂow
ʹΑΔόονETLͷ࠶ߏங” ɹҙࣝͨ͜͠ͱ https://www.pexels.com/photo/hd-457881/

[BI / PM] @mattsun, @shoei, @hase-ryo, @hikaru, @nakatomo,  ɹɹɹɹ @natsume,
@igachan-san, @tsudar, @anboo, @hiza  [JP Dev] @siroken3, @shoe116, @ichirin2501, @bokko, @catatsuy, @shinpei   [Merpay Dev] @laughingman7743, @syucream, @cocoiti, @kazegusuri, @sfujjiwara   [US Dev/ML] @hatone, @yu   [JP ML / Search] @furusawa, @tairosan ɹSpecial Thanks account-name in team Slack

ɹࠓޙͷ՝୊ of Batch ETL in Mercari JP ୹ظ
l࢖ΘΕΔzج൫ͷຏ͖ࠐΈ ϓϩμΫτϚωδϝϯτ γεςϜ։ൃ XJUI#*43&%BUB1MBUGPSN தظ lഁյͱ૑଄z͔Βlܭଌͱվળz΁ͷγϑτ αʔϏεϚωδϝϯτʢ*5*-ʣ σʔλϚωδϝϯτʢ%.#0,ʣ XJUIIBTFSZPTBO ௕ظ lہॴ࠷దz͔Βͷ୤٫ શࣾσʔλઓུࡦఆʢ%BUB0QTʣ XJUIUBJSPTBO

݈શͳ෼ੳ͸  ݈શͳσʔλͷ্ʹ੒Γཱͪ·͢   ݈શͳσʔλ͸  ݈શͳϓϩηεͱγεςϜͷ্ʹ੒Γཱͪ·͢ ·ͣ͸໨ͷલͷখ͞ͳ1า͔Β  σʔλΛ੔උ͍͖ͯ͠·͠ΐ͏ʂ ɹ·ͱΊ

๛෋ͳσʔλ׆༻ࣄྫͱ߹Θͤͯ Ҋ݅ɾϓϩηεɾγεςϜɾνʔϜɾΧϧνϟʔΛ  ͍͔ʹ݈શͳঢ়ଶ΁ͱϋοΫ͢Δ͔͝঺հ ɹએ఻

ݽ܉ฃಆͰؤு͍ͬͯΔݱ৔୲౰ͷօ༷  ݱঢ়Λෆ҆ࢹ͍ͯ͠ΔϚωʔδϟʔͷօ༷  ͥͻ @yuzutas0 ʹ͓੠ֻ͚͍ͩ͘͞  AsIs → ToBe ొΓํͷ੔ཧΛ͓ख఻͍͠·͢
ɹަྲྀλΠϜʹ޲͚ͯ

ྫ͑͹Cloud Dataﬂow͸खܰʹεέʔϧͰ͖ΔҰํͰίετ΋ֻ͔Γ·͢ ࣄۀن໛΍׆༻ํ๏ʹΑͬͯ͸ROI؍఺ͰϖΠ͠ͳ͍͔΋͠Ε·ͤΜ ɾεέʔϥϒϧͳγεςϜΛ࡞Δલʹ΍Δ͜ͱ͸ࢁఔ͋ΔͷͰ͸ʁ  ɾද໘తͳٕज़ཁૉΛऔΓೖΕΔ͜ͱ͕໨తԽ͍ͯ͠ͳ͍ʁ ɾͦͷσʔλૄ௨Ͱຊ౰ʹܦӦ՝୊ΛղܾͰ͖Δʁ ҆қͳγεςϜ։ൃʹඈͼͭ͘લʹɺͥͻҰ౓ߟ͑ͯΈ͍ͯͩ͘͞ ɹ஫ҙɿਖ਼͍͠΋ͷΛɺਖ਼͘͠࡞Γ·͠ΐ͏

ʮ࠶ߏஙʯͷࣄྫΛఏڙ͢Δ ͋͘·Ͱ1ͭͷࣄྫͳͷͰ  ࣗ͝਎ͷٕज़ཁૉ΍૊৫ঢ়گͱൺ΂ͳ͕Βߟ͑ͯ  ࣗ෼ͳΓͷֶͼΛಘ͍ͯͩ͘͞ ɹຊ೔ͷझࢫʢ࠶ܝʣ

ࢲ͸͜͏͠·ͨ͠ɻ ͋ͳͨͩͬͨΒͲ͏͠·͔͢ʁ

͋ͳ͕ͨ͝୲౰͍ͯ͠Δ  ϏδωεɺϓϩηεɺγεςϜɺνʔϜɺΧϧνϟʔͱ Ͳ͕͜ಉ͡Ͱ͔ͨ͠ʁͲ͕͜ҧ͍·͔ͨ͠ʁ ͦͷڞ௨ɾࠩҟ͸ɺͳͥੜ͍ͯ͡·͔͢ʁ

͋ͳͨͷ୲౰ݱ৔͸ࠓͷঢ়ଶ͕ϕετͰ͔͢ʁ  ͦΕͱ΋վળ༨஍͸͋Γͦ͏Ͱ͔͢ʁ খͯ͘͞΋͍͍ͷͰม͑ΒΕΔ͜ͱ͸͋Γ·͔͢ʁ

ࠓ͙͢1ͭΞΫγϣϯΛى͜͢ͱͨ͠Β Կ͕Ͱ͖ͦ͏Ͱ͔͢ʁ

  https://www.pexels.com/photo/architecture-blur-building-colourful-392031/ ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠

Cloud Composer & Dataflow によるバッチETLの再構築 #data_ml_engineering / 20190719

Cloud Composer & Dataflow によるバッチETLの再構築 #data_ml_engineering / 20190719

More Decks by yuzutas0

Other Decks in Technology

Featured

Transcript