データ分析とプロダクトエンジニアリング / 20180417

Slide 1

Slide 1 text

σʔλ෼ੳͱ  ϓϩμΫτΤϯδχΞϦϯά !ZV[VUBT ೥౓ΤϯδχΞίʔε৽ೖࣾһݚमσʔλղੳϋοΧιϯ IUUQTVOTQMBTIDPNQIPUPTIQK4L66:46

Slide 2

Slide 2 text

ຊࢿྉͰ͸ϋοΧιϯͰ࢖͏σʔλج൫ʹ͍ͭͯઆ໌͠·͢ ϋοΧιϯࣗମͷਐΊํ͸!OBPZBOBPͷࢿྉʢࣾ֎ඇެ։ʣΛࢀর

Slide 3

Slide 3 text

!ZV[VUBT ͓ؾܰʹ͓੠ֻ͚͍ͩ͘͞

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

࿩͞ͳ͍͜ͱ σʔλ෼ੳͷ۩ମతͳख๏࿦ ɹɹɹɹɾڭΘΔͷͰ͸ͳࣗ͘෼ͰֶͿͨΊͷߨ࠲ʹͳΓ·͢  ɹɹɹɹɾྑ͍ࢿྉ͸ੈʹࢁ΄Ͳ͋ΔͷͰάά͍ͬͯͩ͘͞  ɹɹɹɹɾϋοΧιϯͰखΛಈ͔ͯ͠ࢼߦࡨޡͯ͠Έ͍ͯͩ͘͞

Slide 6

Slide 6 text

࿩͢͜ͱ ɾσʔλج൫ͷͬ͘͟Γઆ໌ ɾσʔλ෼ੳͷͬ͘͟Γઆ໌ ɾͳͥΤϯδχΞ͕σʔλ෼ੳʁ ɾ#JH2VFSZͷͬ͘͟Γઆ໌

Slide 7

Slide 7 text

σʔλج൫ͷͬ͘͟Γઆ໌

Slide 8

Slide 8 text

σʔλج൫ #JH2VFSZΛத৺ͱͨ͠σʔλͷऩूˠ஝ੵˠར༻

Slide 9

Slide 9 text

σʔλ׆༻

Slide 10

Slide 10 text

ৄࡉ͸ͪ͜Β IUUQTTQFBLFSEFDLDPNZV[VUBT

Slide 11

Slide 11 text

σʔλج൫ͷͬ͘͟Γઆ໌ ׬

Slide 12

Slide 12 text

σʔλ෼ੳͷͬ͘͟Γઆ໌

Slide 13

Slide 13 text

ϓϩμΫτ։ൃͱԾઆݕূ IUUQFDOJLLFJCQDPKQJUFNCPPLT1IUNM

Slide 14

Slide 14 text

࣮ࢪͨ͠ࢪࡦͷ݁ՌΛ෼ੳ

Slide 15

Slide 15 text

͜Μͳײ͡Ͱ΍Γ·ͨ͠

Slide 16

Slide 16 text

Slide 17

Slide 17 text

։ൃνʔϜʹΑΔԾઆݕূαΠΫϧ Ұؾ௨؏ʹΑΔଟ༷ͳ؍఺Ͱͷֶश  ऩӹ΍ӡ༻Λ౿·͑ͯɺސ٬Ձ஋Λ࠷େԽ͢ΔϓϩμΫτ։ൃ ઃܭ ࣮૷ ςετ ϦϦʔε ޮՌଌఆ Ծઆ

Slide 18

Slide 18 text

σʔλ෼ੳͷͬ͘͟Γઆ໌ ׬

Slide 19

Slide 19 text

࿩͢͜ͱ ɾσʔλج൫ͷͬ͘͟Γઆ໌ ɾσʔλ෼ੳͷͬ͘͟Γઆ໌ ɾͳͥΤϯδχΞ͕σʔλ෼ੳʁ ɾ#JH2VFSZͷͬ͘͟Γઆ໌

Slide 20

Slide 20 text

ͳͥΤϯδχΞ͕σʔλ෼ੳʁ

Slide 21

Slide 21 text

ྫɿ8ࠗ׆ͷओཁϓϨΠϠʔ͸উͪےΛ໛ࡧத ࠃ಺ ւ֎ 0NJBJ͸৽نΞϓϦ։ൃ λοϓϧ஀ੜ͸σʔλݚڀϥϘ 1BJST͸ϢʔβʔώΞϦϯά ຽ଒΍फڭ΍ݴޠ͕伴͔ʁ ը૾͕Ճ޻͞ΕΔͳΒಈը͔ʁ ग़ձ͍ͷ޻ఔ࡟ݮPSޙ޻ఔ͔ʁ IUUQTFVSFKQXQDPOUFOUVQMPBETCECDFEFFECGDEQEG IUUQTXXXDZCFSBHFOUDPKQOFXTEFUBJMJE IUUQDPOUFOUTYKTUPSBHFKQYDPOUFOUT"4FDCEFDFCDDECQEG CZIUUQTXXXMJOLFEJODPNJONBSDPTWDFMJP CZIUUQTUXJUUFSDPNDODPPL IUUQTHJ[NPEPDPNUIFGVUVSFPGPOMJOFEBUJOHJTVOTFYZBOECSVUBMMZF⒎F

Slide 22

Slide 22 text

ຊ࣭తʹϓϩμΫτ։ൃ͸ ෆ࣮֬ੑͱͷઓ͍ʹͳΔ IUUQTVOTQMBTIDPNQIPUPTW#QEK-9T

Slide 23

Slide 23 text

ϦΫϧʔτͷྺ࢙ IUUQTXXXOJLLFJCPPLDPNCPPL

Slide 24

Slide 24 text

σʔλ͕ࢧ͑Δ஌ࣝ૑଄ উͪے ɹσʔλͷຽओԽ BigQuery ɹSECIϞσϧ σβΠφʔ ϓϩσϡʔαʔ ސ٬Ձ஋ Ϛʔέλʔ ΤϯδχΞ ࢧԉऀɾ։୓ऀͱͯ͠ͷ  σʔλαΠΤϯςΟετ

Slide 25

Slide 25 text

Slide 26

Slide 26 text

Slide 27

Slide 27 text

Slide 28

Slide 28 text

ͳͥΤϯδχΞ͕σʔλ෼ੳʁ ׬

Slide 29

Slide 29 text

࿩͢͜ͱ ɾσʔλج൫ͷͬ͘͟Γઆ໌ ɾσʔλ෼ੳͷͬ͘͟Γઆ໌ ɾͳͥΤϯδχΞ͕σʔλ෼ੳʁ ɾ#JH2VFSZͷͬ͘͟Γઆ໌

Slide 30

Slide 30 text

#JH2VFSZͷͬ͘͟Γઆ໌

Slide 31

Slide 31 text

IUUQTXXXTMJEFTIBSFOFUSFDSVJUDPKQIBEPPQCR IUUQUFDIUBSHFUJUNFEJBDPKQUUOFXTOFXTIUNM

Slide 32

Slide 32 text

IUUQTDMPVEHPPHMFDPNCJHRVFSZXIBUJTCJHRVFSZ IMKB

Slide 33

Slide 33 text

6 ετϨʔδͱΤϯδϯͷ෼཭: -  ͲͪΒ΋࢖ͬͨ෼͚ͩ -  ෼཭ͨ͠՝ۚ -  ύϑΥʔϚϯεӨڹ͸ִ཭ ෼ࢄετϨʔδ -  ෳ਺σʔληϯλ΁ෳ੡ -  1 ͭͷετϨʔδʹݟ͑Δ -  DWHͷαΠϩԽ͕ͳ͍ Federated Storage -  ϩʔυ͠ͳ͘ͱ΋෼ੳՄೳ -  ૣ͍ϩʔυΤϯδϯͱͯ͠΋ ར༻Մೳ σʔλγΣΞɺࡉ͔ͳACL -  1ͭͷετϨʔδͰACLઃఆ ͢Δ͚ͩͰڞ༗ -  σʔλΛڞ༗͢ΔจԽʹ Ingestion: [streaming, batch] -  ͲͪΒ΋ωΠςΟϒରԠ (PPHMF$MPVEൃදࢿྉΑΓҾ༻

Slide 34

Slide 34 text

ϝϦοτᶃ Πϯϑϥอकӡ༻ͷ޻਺Λ࡟ݮͰ͖Δ

Slide 35

Slide 35 text

Confidential & Proprietary Now: యܕతσʔλ෼ੳ ϓϩάϥϛϯά Ϧιʔεͷ ϓϩϏδϣϯ ύϑΥʔϚϯε νϡʔχϯά ؂ࢹ ৴པੑ ֬อ σϓϩΠͱઃఆ ૿ઃରԠ ࢖༻཰ͷ ޮ཰Խ Next: Google ͱ࣮ݱ͢Δ Ϗοάσʔλ෼ੳ ͢΂͕ͯࣗಈԽ ϓϩάϥϛϯά ෼ੳͱಎ࡯ (PPHMF$MPVEൃදࢿྉΑΓҾ༻

Slide 36

Slide 36 text

Confidential & Proprietary ઐ༻ઢκʔϯ جװܥγεςϜ A σʔλϕʔε جװܥγεςϜ B σʔλϕʔε άϧʔϓձࣾ ސ٬σʔλϕʔε ސ٬ σʔλϕʔε ϑΝΠϧ ετϨʔδ ϑΝΠϧ ετϨʔδ ϑΝΠϧ ετϨʔδ ϑΝΠϧ ετϨʔδ FTP ଐੑͷΈऔΓ ग़͠FTP FTP VPN άϧʔϓձࣾΠϯτϥ άϧʔϓձࣾ઀ଓκʔϯ όονॲཧ αʔόʔ ՄࢹԽπ ʔϧ ෼ੳ πʔϧ Ϩϙʔτ ؂ࠪ πʔϧ σʔλϚʔτ ॲཧ ຊ౰ʹඞཁͳ࡞ۀ͸෼ੳɺಎ࡯ ෼ੳ׭ Ϗδωεاը खಈ࡞ۀ DWH DWH DWH Hadoop cluster (PPHMF$MPVEൃදࢿྉΑΓҾ༻

Slide 37

Slide 37 text

ϝϦοτᶄ ଟ༷ͳϢʔεέʔεʹରԠ͍ͯ͠Δ

Slide 38

Slide 38 text

42-Λୟ͚ͩ͘

Slide 39

Slide 39 text

ओཁ#*πʔϧͱͷ࿈ܞ IUUQTDMPVEHPPHMFDPNCJHRVFSZUIJSEQBSUZUPPMT IMKB

Slide 40

Slide 40 text

ϓϩάϥϜͱͷ࿈ܞ

Slide 41