分析基盤と組織のあり方 - DeNAの事例

Data Engineering Study #3 #%()! DeNA 2020/9/16 -*& ",# +,
$'

(& *Ryoji Hasegawa+ • -> IT
• %)@"SaaS • 20169 DeNA ◦ ! ' $#

54B'/ 3 • %+!#"(6,: <=9C 38 ;B DeNA.?)&7(6,: 9C
$ (6,: 2<=((6,:<=) 1 - B • <=A@*>0

1 2 DeNA DeNA 3

DeNA 5 1

• 9e<i • (EgW5`J0KP \a 7^1W5VY:? 6
$ % " • (Eh+ZgQF&/A =6S,D • UC>;H83[ ! $ % " A(Ec B(Ec C(Ec … (Ec.VY * N f O b M G[-VY … A(Ec B(Ec C(Ec * N f O b M #"%VY … • (EcgW5`J2 4d]L • (EcgR_,D • ()(Ec.G [- • (EQF&/A=6, D • @BTX)dI' L

* A Q K ; 6 G
A Q DeNA JM=? 4BQ: + CN9 "&. 7 2*1020+04BAQ (24BAQ '/4BAQ !2$4BAQ 4BQ:JM CN9JM 4BQ: K;6G 54CN9JM 2 3OHDeNA *0%- 07I8P CN9JM EF> # , | ) 0 . | A Q D < L @ A Q 0 A Q

$ < M G 3 . B
< M & | # ) ' | < M @ 7 H 9 < M ) < M 0=5DFI ->M2FI ?J1FI 8 6 8 +$*)+)%)-><M "+-><M !(-><M + -><M ->M2FI ?J1FI A4 $<MN, /E0=5D ; +$->M0=FI CK 0 = : L M D P G

DeNA 9 2

• 6F@RNO=QG6F<LM2 37(1$<!1+#A : B8>; • RDBExcel6F •
!1+#6F@4#$&,5J#$&,0/"*?D • -1 '/%1.I6F@I(1$-1)>; !1+#HCK6F P 6F<LE97 (2010) 10 Log Server … DB Server … App Server App Server App Server Batch Server DB Server DB Server … Marketing Data Mart (MySQL)

HR34/1;W.'17,C(!$ → :MFI^ <?LB 7)JA]XGa 7,:M\>?LE_UVQZ (2010) 11 7,65
7"5-5$:MK\ :M\@UV ?L\ … • 7,:M\>%0$?L BS5&8= UV NICYO#7 ?LB SE_UVT 9 • 5*2+ Hadoop [P`D

2)1?TD^]IR (2011) 12 # + S l 4+32
4#2,2%5USl 5UlBbd Ve@bd • 4+5U'/%4+5U OJ • 68?Tf 5U9 G *-(4.2 W&,2_hLn ? T D ^ G ?TD^Q oKC ;`Im bd>c ?Tl … ? P D ^ k \ ? T [ E a Z < i $ | a Z ? T " | 0 m ] • ?TD^2(Ve@bd OJ • Q 04)=Ve@ !4+cN • Ve XC D^oKCFY ◦ oj ?TFY: Vertica H: ◦ oA7MFY: BI"40ETL"40=g

• DeNA ,!4 ( • *-"% &'). $
5 + 15 + BI Monthly Active User 1,000 - ,) #+

• • 15

• • 16

vM7\qt ]7\4>em6 dIDL#%)/0'5%<Y >exyN7\/RpdIJA9=ZHh7\qta =ZHh?UnX: 3-1=ZHhs| 17 PW=ZHhc`
On-Premises 7\ A 7\ B 7\ C … … 7\ A 7\ B 7\ C … • Q{jGNrf:^6o@%,5*5T Fz • bK7\C dI!%(.&z • MLAIiE }OBlSw • 7\4-2$"+_q8g • URpkVB; • 7\_dI[u!%(.&

O • Google Cloud Data Platform Day #2
Google Cloud 3 . '*!%.);QCA GJHF https://cloudonair.withgoogle.c om/events/jp-data-platform- day?talk=transformation • DeNA TechCon 2020 DeNA .'*!%.) NKM6#*, https://engineer.dena.com/post s/2020.03/techcon_2020_live/ GCP(.4P7AE@ $&+*"J 5D:L4P7A 18 BI Tools Service A Service Env App Server DB Server Data Platform Service A BigQuery Cloud Storage GKE digdag batch web app Argus app logs snapshot Service B Service C Service B Service C --- --- I9:L7A78B<0 2>?/<01=

• • 19

uW~@'80{} • NsR\ {tYl#Uo • ,AE9*K.FI,07Mm • Vr^z)I/9%D3>K.FI e
• NsRP→ahQ • "$Pn#| Mah !%8G07-K;0R[ #S] ah ! Z Xxc O^z ~#i ! ~i: Nsi 20 +KBJ)I2K6'ICI7Nsq 0?K5Nsq =H0*%Nsq :47-K;0Nsq V r ^ z G ^z}M JybpkwfJvguW~ ^zL`_j x<(K1TduV[<&47

IcsL\^ 0F,#D*3!no 21 IcsLVdrx ] • IcsM0F,#D*3!no\^ %F+N •
g&F;IcsOaWknoVqenoy0F,#D*3!AD$$BF 7z!2A+1cTShU'+1QfY ^b H Z • G_noC"=X ':>3%F)?D'+1XcTtpmv ShiLw &F;E#D,F/"D<D1Ic`s +9F.Ic`s 8B+%!Ic`s 4-1(F6+Ic`s O a W k G 0F,#D*3!AD$G • ':>3%F)?D'+1XR → !2A+10F,#D*3! OaWk#D*3!3u[': >3%F)?D • cTtp → Jlj0F,5"7 @"D PK m

.A)nt ,A<@Vwu • BI • .A)$?'jgX { .A)kCP\ & 20?'ZP
• (A%7<;'1?"VOCKf • looker YF\ • .A)MtplSP@cP { mODh.A)nt^] • .A)34:?`E@t • 0;'/^] { • #A5Dh.A)?&1GKfVwG~TGyH} J • jL}[ *A5QKjLxvLJ %'-5e qUB}Nb 22 KfVwG K_Vwt #A5@?)A-?6?/Dhe KfsW|r H.A)|r Kf,A<u %'-5e Kf\ .A)?&1;?"G .A)?&1;?"G Dh^] ia4>&!/ (;8A%9??&1;?"G KfVw.A)?&1;?"0=+&I Hd oDhF ^]iavR}CYz

$3G!)()AF LF?1 6>B *? )AF " 6>4 $AF6>M/-@8
23 *?;:/)AF '9K 3GLF /=.E3G ,5(LF OS()&# IE … DC *?;:/)AF '9K $HJ7 %M/W ,5(LF 3GLF %M/W ,5(LF Google + 6>0B 3G<2 3G<2 *? )AF " 6> 4

! 24 • HmC9XGE>WFaegLQ(h&68Xs@eg Zi ?eg o^& • $
[n7uLQ"%N&jbjb>W Fa_\OJVregLQ(Ydf # • DeNAGE • PRt7+528Xs=>WFa(c6 M • ;b<pZi?egC>WFa$&wKC4IlYC(q! • VU*3)0Sk(D%:'>WFa(AU&"#%8XTq4. 5-]`v]B(,1/#&

DeNA Tech Twitter ! DeNA !! ) &$(% '#
* Twitter " *

分析基盤と組織のあり方 - DeNAの事例

分析基盤と組織のあり方 - DeNAの事例

DeNA_Tech

More Decks by DeNA_Tech

Other Decks in Technology

Featured

Transcript

Data Engineering Study #3 #%()! DeNA 2020/9/16 -*& ",# +,

(& *Ryoji Hasegawa+ • -> IT

54B'/ 3 • %+!#"(6,: <=9C 38 ;B DeNA.?)&7(6,: 9C

1 2 DeNA DeNA 3

DeNA 5 1

• 9e<i • (EgW5`J0KP \a 7^1W5VY:? 6

* A Q K ; 6 G

$ < M G 3 . B

DeNA 9 2

• 6F@RNO=QG6F<LM2 37(1$<!1+#A : B8>; • RDBExcel6F •

HR34/1;W.'17,C(!$ → :MFI^ <?LB 7)JA]XGa 7,:M\>?LE_UVQZ (2010) 11 7,65

2)1?TD^]IR (2011) 12 # + S l 4+32

• DeNA ,!4 ( • *-"% &'). $

14 3

• • 15

• • 16

vM7\qt ]7\4>em6 dIDL#%)/0'5%<Y >exyN7\/RpdIJA9=ZHh7\qta =ZHh?UnX: 3-1=ZHhs| 17 PW=ZHhc`

O • Google Cloud Data Platform Day #2

• • 19

uW~@'80{} • NsR\ {tYl#Uo • ,AE9*K.FI,07Mm • Vr^z)I/9%D3>K.FI e

IcsL\^ 0F,#D3!no 21 IcsLVdrx ] • IcsM0F,#D3!no\^ %F+N •

.A)nt ,A<@Vwu • BI • .A)$?'jgX { .A)kCP\ & 20?'ZP

$3G!)()AF LF?1 6>B *? )AF " 6>4 $AF6>M/-@8

! 24 • HmC9XGE>WFaegLQ(h&68Xs@eg Zi ?eg o^& • $

DeNA Tech Twitter ! DeNA !! ) &$(% '#