DeNA流データエンジニアリングの極意

Slide 1

Slide 1 text

Slide 2

Slide 2 text

˞%F/"ͷػցֶशج൫ͱ෼ੳج൫ ͷࢿྉ͔Β෼ੳج൫ͷύʔτ ʮσʔλΤϯδχΞϦϯάͷۃҙʯ Λൈ͖ग़ͨ͠΋ͷͰ͢ 2 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. ˞ػցֶशج൫ͷύʔτ͸ͪ͜Β IUUQTTQFBLFSEFDLDPNTPOPUTEFOB BJTJTVUFNVCVOJPLFSVLVSBVEPXPIVPZPOHTJUBKJKJF YVFYJKJQBOGBMTFHPV[IV

Slide 3

Slide 3 text

Slide 4

Slide 4 text

Slide 5

Slide 5 text

σʔλϨΠΫͱσʔλ΢ΣΞϋ΢ε 5 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. σʔλϨΠΫͱDWHʹ෼͚Δ΂͠ DB Log Data Lake ౷߹DWH Data Source શͯ͸Ұ౓σʔλ ϨΠΫʹೖΔ ҙࢥܾఆͷͨΊʹ ໨తผʹฤ੒͞Εɺ ౷߹͞Εͨ࣌ܥྻ σʔλϕʔε ※ ͞ΒʹΞϓϦέʔγϣϯ༻ʹ༻ҙͨ͠ Data MartΛ࣋ͭ૊৫΋͋Δׂ͕Ѫ

Slide 6

Slide 6 text

Slide 7

Slide 7 text

7 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. σʔλϨΠΫͱDWHʹ෼͚Δ΂͠ • ऩू͚ͩͰ΋ࠔ೉͕͋Δ • σʔλ͕͚͍ܽͯͳ͍͔ • ٯʹɺσʔλ͕ॏෳ͍ͯ͠ͳ͍͔ • ଞͷ͜ͱ·Ͱಉ࣌ʹߟ͑Δͷ͸೉͍͠ • ϑΥʔϚοτͷ౷Ұ • ΧϥϜ໊ͷ౷Ұ • σʔλόϦσʔγϣϯ • ΦϯϥΠϯεΩʔϚมߋ • ूΊΔ͜ͱ͚ͩʹઐ೦͢Δ • શͯͷσʔλ͸ͦ͜ʹ͋Δ ͱʹ͔͘σʔλϨΠΫʹूΊΔ

Slide 8

Slide 8 text

Slide 9

Slide 9 text

Slide 10

Slide 10 text

10 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. ετϦʔϛϯάऩूͱόονऩू • ϦΞϧλΠϜʹ͍ۙ ετϦʔϛϯάऩू • ӡ༻͕೉͍͠ • ϓϩηεΛσʔϞϯԽ͠ৗறԽͤ͞ΔͨΊϝϞ ϦϦʔΫͷ໰୊ΛҾ͖ى͜͠΍͍͢ • Ұ෦ܽଛͨ͠৔߹ͷ࠶సૹ͕ࠔ೉ • ߴτϥϑΟοΫͳ࣌ؒͰͷෛՙ্ঢʹΑΔసૹ ஗Ԇͷൃੜ ϦΞϧλΠϜੑͷ୲อ ❌ܽ఺ ⭕ར఺

Slide 11

Slide 11 text

11 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. ετϦʔϛϯάऩूͱόονऩू όονऩू • ϓϩηε͕ৗற͠ͳ͍ͨΊɺϝϞϦϦʔΫ΋ॏେͳ ໰୊ͱ͸ͳΒͣπʔϧͷ࣮૷͕༰қ • σʔλͷऔΓ௚͠΋Ұൠతʹ࠶࣮ߦ͢Δ͚ͩͰ༰қ • ϦΞϧλΠϜੑ͕ͳ͍ • ϚγϯϦιʔεΛ࠷େʹར༻͠Α͏ͱ͢ΔͨΊɺଞ ͷγεςϜʹӨڹΛ༩͑ͯ͠·͏͜ͱ͕͋Δ • ॲཧ͕ऴΘͬͨΒશ͘ϦιʔεΛ࢖Θͳ͘ͳΔ %BJMZ΋͘͠͸)PVSMZͰόονπʔϧΛىಈ ⭕ར఺ ❌ܽ఺

Slide 12

Slide 12 text

Slide 13

Slide 13 text

13 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Web Web Web Cassandra Storm HBase ログコレクタ Hadoop HDFS ログAPI イベントログ %F/"ͷϩάऩूج൫ • qVFOUE ·ͨ͸IBEPPQMPHEBFNPOͱ͍ ͏಺੡πʔϧ ͕ϩά"1*ʹσʔλΛૹΔ • )%'4ʹ֨ೲ͞ΕΔ ετϦʔϛϯάऩूͱόονऩू

Slide 14

Slide 14 text

14 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. DB (MySQL) Sqoop Importdb MapReduce ジョブ Map HDFS Map HDFS Map HDFS 起動 *NQPSUECϚελσʔλͷόονऩू • Ϣʔβ৘ใͳͲͷϚελσʔλΛ࣋ͭαʔϏεͷσʔ λϕʔε .Z42- ͔Β೔࣌Ͱऔಘ • 4RPPQΛϕʔεʹͨ͠಺੡*NQPSUECπʔϧɺ .BQ3FEVDFͰ)%'4ʹσʔλΛ֨ೲ ετϦʔϛϯάऩूͱόονऩू

Slide 15

Slide 15 text

Rails (Web UI) Embulk Sidekiq Worker Hadoop HDFS Vertica アナリスト設定情報 (MySQL) ジョブキュー (Redis) Embulk Sidekiq Worker 15 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. .FEKFEόονͰσʔλసૹ • )%'4ͷσʔλΛ7FSUJDBPS#JH2VFSZʹҠ͢ • &NCVMLΛόοΫΤϯυʹͨ͠಺੡3BJMTΞϓϦ έʔγϣϯ.FEKFEͰσʔλΛసૹ ετϦʔϛϯάऩूͱόονऩू

Slide 16

Slide 16 text

Slide 17

Slide 17 text

17 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Web Web Web Cassandra Storm HBase ログコレクタ Hadoop HDFS ログAPI イベントログ %F/"ͷϩάऩूج൫ • $BTTBOESB͕όοϑΝϨΠϠʔ • ࡉ͔͘ॻ͖ࠐΉͱɺ*0ෛՙ͕ߴ·Γɺ·ͨஅยԽ ͕ى͜Γ΍͍͢ • ૹ৴ଆ qVFOUE ʹ΋όοϑΝϨΠϠʔ͕͋Γɺ σʔλΛ·ͱΊͯૹΔ͜ͱͰޮ཰Խ͍ͯ͠Δ

Slide 18

Slide 18 text

Slide 19

Slide 19 text

Slide 20

Slide 20 text

Slide 21

Slide 21 text

21 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. ᶃ΂͖౳ʹ͢΂͠ • ϦτϥΠͰಉ͡σʔλΛೋॏʹೖΕͯ͠·͏ Retryableʹ͢΂͠ ❌ϦτϥΠػೳͰΑ͘ൃੜ͢Δ໰୊ ⭕ରॲ๏ ͋Δૢ࡞Λ1ճߦͬͯ΋ෳ਺ճߦͬͯ΋݁Ռ͕ಉ͡Ͱ͋Δ͜ͱ ႈ౳ੑ • ௥هͰ͸ͳ্͘ॻ͖͕ݪଇ • ྫ #2ͷ৔߹83*5&@"11&/%Ͱ͸ͳ͘83*5&@536/$"5& • ͦΕͰ΋௥ه͍ͨ͠৔߹͸ϢχʔΫͳKPC@JEΛৼΓɺϦτϥ Π͕૸ͬͯ΋ॏʹ࣮ߦ͞Εͳ͍Α͏ʹ͢Δ

Slide 22

Slide 22 text

22 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. FNCVMLPVUQVUCJHRVFSZ ੿࡞ • BQQFOEϞʔυͱSFQMBDFϞʔυ • 83*5&@"11&/%Λ࢖ͬͨBQQFOEϞʔυ • 83*5&@536/$"5&Λ࢖ͬͨSFQMBDFϞʔυ • SFQMBDFϞʔυΛ࢖͏‷ ŋŷŋ و ̑̑ • ಺෦తʹVOJRVFͳKPC@JEΛੜ੒ • #JH2VFSZଆͰෆҙʹϦτϥΠ͕૸ͬͯ΋σʔλ ͕ॏෳ͠ͳ͍Α͏ʹ͍ͯ͠Δ ΂͖౳ʹ͢΂͠

Slide 23

Slide 23 text

23 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. ᶄBUPNJDʹ͢΂͠ • ్த·Ͱ͔͠σʔλ͕֨ೲ͞Ε͍ͯͳ͍ঢ়ଶͰޙଓ ͷδϣϒΛ࣮ߦͯ͠͠·͏ • ϦτϥΠ͢Δͱɺ֨ೲઌσʔλ͕Ұ࣌తʹফ͞Εͯ ͠·͍ɺσʔλ͕ଘࡏ͍ͯ͠ͳ͍ॠ͕ؒൃੜ͢Δ τϥϯβΫγϣϯʹؚ·ΕΔλεΫ͕શ࣮ͯߦ͞ΕΔ ͔ɺ͋Δ͍͸શ࣮͘ߦ͞Εͳ͍͜ͱΛอূ͢Δੑ࣭ Retryableʹ͢΂͠ atomicity ❌Α͘ൃੜ͢Δ໰୊

Slide 24

Slide 24 text

24 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. %F/"ͷ৔߹ • BUPNJDͳૢ࡞ΛετϨʔδ͝ͱʹௐࠪ • )%'4 • UNQσΟϨΫτϦʹσʔλΛॻ͍ͯશͯ׬ྃ ͨ͠ΒSFOBNF • 7FSUJDB • UNQςʔϒϧʹσʔλΛॻ͍ͯɺ"-5&3 5"#-&3&/".&50ͰTXBQ • #JH2VFSZ • UNQςʔϒϧʹσʔλΛॻ͍ͯɺDPQZKPC atomicʹ͢΂͠

Slide 25

Slide 25 text

Slide 26

Slide 26 text

Slide 27

Slide 27 text

Slide 28

Slide 28 text

28 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 7BMJEBUJPO͢΂͠ • ͳʹ΋Τϥʔ͕ى͜ΒͣͲ͔͜Ͱσʔλ͕ /6--஋ʹมΘ͍ͬͯͨ • ྫ %8)ͷΧϥϜܕΛJOUܕʹ͍͕ͯͨ͠ɺ σʔλϨΠΫʹ ͷΑ͏ͳσʔλ͕ೖΔ͜ ͱ͕͋ͬͨ ❌Α͘ൃੜ͢Δ໰୊ • 7BMJEBUJPOͯ͠ૣΊʹؾͮ͘΂͠ ⭕ରॲ๏ Validation͢΂͠ 4

Slide 29

Slide 29 text

Slide 30

Slide 30 text

Slide 31

Slide 31 text

Slide 32

Slide 32 text

Slide 33

Slide 33 text

Slide 34

Slide 34 text

Slide 35

Slide 35 text

Slide 36

Slide 36 text

Slide 37

Slide 37 text

37 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. ᶄ࣌ؒύϥϝʔλͷࢦఆΛՄೳʹ͢΂͠ • ࣌ؒύϥϝʔλΛEBUFίϚϯυ΍/08 ͷΑ ͏ͳ42-ؔ਺Ͱࢦఆ • ࣮ߦλΠϛϯάͰύϥϝʔλ͕ܾ·ͬͯ͠·͍ɺ࠶ ࣮ߦͰ͖ͳ͍ ❌Α͋͘Δ໰୊ • σϑΥϧτ஋ͱͯ͠ͷ/08 Ҏ֎ʹ΋ɺ࠶࣮ߦ ࣌ʹɺ࣌ؒύϥϝʔλΛࢦఆͰ͖ΔΑ͏ʹ͓ͯ͘͠ ⭕ରॲ๏ ࠶औΓࠐΈΛ༰қʹ͢΂͠

Slide 38

Slide 38 text

Slide 39

Slide 39 text

39 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. ᶅσʔλൃੜ࣌ࠁͰσʔλΛ෼͚Δ΂͠ • σʔλൃੜ࣌ࠁͰσΟϨΫτϦΛ෼ׂ • ϩάൃੜ࣌ࠁɺߦߋ৽࣌ࠁ ·ͨ͸औΓࠐΈ࣌ࠁ • ಛఆσΟϨΫτϦͷΈͷ࠶औΓࠐΈ‷ ŋŷŋ و ̑̑ • )%'4ͷσΟϨΫτϦΛ೔࣌Ͱ෼͚͍ͯͳ͍ • ࠶औΓࠐΈ͕શσʔλऔΓࠐΈʹͳΔ • σʔλճऩ࣌ࠁͰσΟϨΫτϦΛ෼ׂ • ෼ੳର৅೔ͷσʔλΛ࠶औΓࠐΈ͢Δͷʹ ϑΟϧλϦϯάͱϚʔδ͕ඞཁʹͳΔ ❌Α͋͘Δ໰୊ ⭕ରॲ๏ ࠶औΓࠐΈΛ༰қʹ͢΂͠

Slide 40

Slide 40 text

Slide 41

Slide 41 text

Slide 42

Slide 42 text

42 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. • σʔλϨΠΫͱ%8)ʹ෼͚Δ΂͠ • ετϦʔϛϯάऩूʹ͸όοϑΝΛઃ͚Δ΂͠ • 3FUSZBCMFʹ͢΂͠ • ΂͖౳ʹ͢΂͠ • BUPNJDʹ͢΂͠ • VVJEͰॏෳഉআ͢΂͠ • 7BMJEBUJPO͢΂͠ • 42-Ͱ෼ੳ͢΂͠ • ࠶औΓࠐΈΛ༰қʹ͢΂͠ • 3FUSZBCMFʹ͢΂͠ • ࣌ؒύϥϝʔλͷࢦఆΛՄೳʹ͢΂͠ • σʔλൃੜ࣌ࠁͰσʔλΛ෼͚Δ΂͠ • σʔλͷͭͳ͕ΓΛ؅ཧ͢Δ΂͠ ·ͱΊ 1 2 3 4 5 6