Applibotデータ分析基盤の刷新 / Applibot

サイバーエージェントの技術者（エンジニア・クリエイター）向けカンファレンス『CA BASE CAMP 2019』

Applibotデータ分析基盤の刷新
小川 詩織 伊藤 皓程

Transcript

1. Applibot 
      /
2. 1.')! 2.    (" 3.(&*+ 4.
   & 5.& % 6.$#
3. 
   
4. 48 MF / Shiori Ogawa •  - $.".# •  - (6H: .+.( - ED
   )+
   
   J? etc. … •  - 2015.04  gloops - Ameba/GREE/FP Web(&,31>5 (K<!-) - /L0 - 2017.08
       - 2017.08 2@!-,!-*.%.31&,, - 2018.03; =I(A.R.T. G7
   /L=IB9 - 2018.10; &,'!AC'"
5. # &" / Kotei Ito •  -    •  -  •  - 2015.04  QualiArts - () -   
    - 2017
     AJA  - AJA SSP - 2019
        - A.R.T.%$!
6. 6fNc") D.I.V.E. • 
    (Data/Infrastructure/Value/Elevation) 6fNc
   6f<]IBb8 ! & •  GV2M •  Q.35
   g`Nc EIJ7 Nci;/4KY
   @C1M- RPEI8X
   ;NcDOh2?T ;W',#SA ^9
   ;Nc[^= 1 [d?T Q.115
   g`Nc HA  F: #+(-% a_Ue  Zj*-$L> g`',#\38X0
7. 
    
   
    
8.     
      
9. !" #B • ## &;$ 96#B08% /; • !$ 7.(5  /;
   2<*A:4   • $ @,)>=+'3-1?    
   
    " #
10. &R*"@) 6;AJ N:5Q /8 -(=F 1S E< 0CN: OD>B
    M@
    $3H5
    #! 
    OD*1S
    6;?+
    OD%
    7, 
    ' )
    KPI>B
    L;G9 
    2P.I>B
    4K OD>B
11. 
     … • K%"W2`_ • ;ZPO)74J6V
    9X51 • < C
    AF<  51 • S)/4LBT"( • ' Y2I
    Q="( • #&0:,+ 
    :U\3L> • AF*]4LN @
    DG  • .R/?E • M^!$8-[H  etc. ………
12. 
       …
13. *3( /+ • :HF7 !!#'-2 $, ".  %  "0)4 &1  &  68<2 5 ;&#0#-.3… *% 1'!($G $"/)2 … @A,+E?92 … =C2  D4B> 
14. 4I  @5_1 •  
     01-BE 9> ,.BE &#BE !,; -4FG= !,7A < ,./2:6=  •   !,,.2:H /8 -?  '0+ -C3J $1D  (1% *)"!
15. 3F  =7_2 =OJ8 !-/2:UU9;>
    DF5:  D:@ IRB ,'0*)."4 ?@#!0<  $-C8+#+1&(%B56 
    ."ADE  
    &.Q>G ,3(9# 2Q>G  .0*1K @ G 2:<AMP? @ HC 4E/";E/")$6LN&.G %!'(7 +3-3(7C
16.    
      
17. 9b!L\TN • +  &- ", 
    $ (!#.  *% .'CD9b
    P<7C+)1Q8.'Y; WVOJ#>! &(*#[TN=@:MB ! .'C #d` 46FE^>"
    aX%,cS#Z  • )'&- $-/ ($0# G?5
    .'Y3" Q8 !_H
    2AI45!…U]KLR !!
18.  
    
19. 
     
20.  
    
       
21. Applibot8316 • 
     
    #%)2, !
    $DWH(Redshift)/' (7# 5(/') •  Redshift !20 5(Redshift !/20 •   RedshiftBI
    "*4& - TableauGeckoBoard .+
22. *7 #<)Va • 1  
     '(# % $& UZA1/%*7::.HM "GB#XI # >R •  )* ! !#8,DB \D$/'6=)1;+3b ?!  *7C'5=2O
    &'07%1SF#&(-9= &'07YT 8,.0/.\D API<) ^J;+3# •  % +$
    % "! &'07YT<)
    8,'5=2[]_K
    .0/.
    '=4;E WN!P! #QL O
    ^J<) `@#
23. /,?D'28- • DB  /(6/,&UDBdump
    Af$ P<C5*/,Update %
    @:=aI !idJ% /, 1:M>c%$>
    RNbE Z OKApplibot • DB History DB/,\
    History.17&?D$"[ /, 1:M>c%$ •   History?D/,&
    8)h ?DBB%
    VHAf $ ;F _#DWH SeT $
    ETLYG!^]X`gT $ +0639)%6'7,(4RNJQ LWApplibot
24. 
    
          DB
     + History DB
     
        ()
25. .+L?&04, • DWH(
     ) ;X\.+.+%:#"5JVL?@ZY.+N?()-2 PrestoHive, DremelUE%1) RedshiftBigQuery 7P CIApplibot Applibot<G[ • DB DB3'.+History.+%L? 65JV.+O "L?RK T • 
      (+DWH) .+%" QJV=$!>W  
    FS/*
    FS3'
    /*8$9I.+%H"  "MB DAApplibot
26. =32 B7 logloglogloglog logloglogloglog loglogloglogolo gloglog logloglogloglog logloglogloglog loglogloglogolo gloglog >@1
    '
    # ( *
    , *
    )$
    +" 40!-
    %*
    F.O.X. etc…  :E +96
    &) DA
    GJ<H. F5/I;?1:E(BI) HadoopC:E=8/I:E
27. '  E0B4 logloglogloglog logloglogloglog loglogloglogolo gloglog logloglogloglog logloglogloglog loglogloglogolo gloglog >A.
    #
     $ &
    ( &
    %!
    ' 2,)
    "&
    F.O.X. etc… /C5H=:
    *L 1F
    -8?9IK  M  '  7 G  7G H3+J;@.7G(BI) HadoopD7G <6+J7G
28. 20180424 AWS Black Belt Online Seminar AWS     https://www.slideshare.net/AmazonWebServicesJapan/20180424-aws-black-belt-online-seminar-aws?ref=https://aws.amazon.com/jp/blogs/news/webinar- bb-datalake-2018/
29. 
     
30. $'.(-6RW • Redshift]g y:H (^Jc85%…) onMf \! aZ(&8 `l>K@ /,on  • aZ(&8 Y
    pEaZ _" Redshift#ftOX2%3%  SortKeyNqH! (7+,Ue! 
    ACG;v
    ik= • 1+0/,#wrm SF+0859)*4+Flydata#^ … <DETLdIQLz! • P[hV
    bsj?T/,#B!
    aZuxH
31. OEIK • -)S^?PEIK RedshiftFlydata 7@1600HMg -)%C_ &(.g#… • a@/+*UDY dfWD $
    XD8J a@/+*-)%>$# UD0)4\L!$dfB • Redshift-)%V>$#bFFlydata`Ti [RUD%9"
    h_,236 =]'1(%N"
    c;A%G# eZ 15:-)QKPI<$ !…
32. !$.%,9T\ (}t) • RedshiftajT\ • .6=k[)$7esyY@F dS`P')0 |  • Redshift RpO 
    /+AIJ |  • n/+HS_eOHg i • w]ZDBB]ZTDQXz • %;)+Yh
    GINB%"< ^NBlM urKv  • Ex$*0$
    Vacuumpb35#7>),:1>& AW • ?C/+fm€~7-,V  • 2)0/+ {qoUL)0<8>&(4)Flydata c  
33. jX\` (‘ˆ) • mX\`RedshiftFlydata1600[d  • L9/:/ EJ)4>!•$+*tcƒ v
    L!€k.b F Š 
    4>Op"ga)++  • NU L#G^ i,F'cƒ' "]H '  • B0 "Sw ;@;@lz uV-  • 83.f‰!se%QT Œ_+
    (( • …R;54"rW { ŒŽxW-
    yWG^  • nEJh}KBI6A"?C7G^,F *”!‹, • Redshift!83.sP-,&!†YFlydata"„q “ }o rW.I+
    ’‚"7=A D  O~ 1<2.h+
    ‡"M T.Z, |
34. Redshifti & • D1=8*F23F2kS&<E$
    Gdj63fRW -,D027'Xq#027G-,D QJ $ Z TZ 63#fR$ • Lrp\ZDB"#o[ • Ua]
    DWHcn b $
    63'N%^ %lHYo[ • ?27>C-5)2"
    B1/2OeImP:+;+ ] g$ !(4A9F.
    _M$@F5hVK %Ua`b
35.  
     • E6O? • )* &T4.G> • -V8AHSWX0I • 2,KPIG>C81F • % F.O.X.
    (dump# (QDBOK)N# /30I • %HSM<!&P@  • G>,'+"$+B;  O? • DGG>9= 7U JL
    YR:K5
36.  
       
38. 
    
     
39. 
    
40. ETL
    
41. 
    
42. 
     Lambda ""(#event ./16
    : *9 =>-  ! 3;!70 $,)!</'% →  8& 5+42 API
43.  
     ^LpX 7$8,%5tH@ `W ?wPZbsxyCd E:KPI`W\8D_ ! 4+,F.O.X.n1,BGCd +(4bskU#-5qY !  `W:69039+[T# !pX RV O u!eizrSfK → M200QF/> → 10Q+α/> M1/20'.2 →  → `W)&7#aPdkU! → ADI; lo<" j → S3]APIqc →=NJ/8hm →=NJ/8hm →/8hmgv+*8
44. 
    
45. 
     • %#')F@ • )!)"0A/L< • >9%4J2 • RedshiftD! I, • Athena ()&$6K ( :*%%) • =-%6ECG2 • >91.Glue B3? NM2  (5;H) • 8%0 4J2+7Redshift Spectrum …
46. >M(6.6-O;G  •   
     JV QD 
    LHT5'IY8  LH 5' QD&%2." NC :X)12ETB 9 • KPI   SAJ7KPI?!R5' 8=P  KPI05*#&/<,4KPI&$2KF(+/ W • BI   BI-3 U@QD…
47.    
    
48.  …
      mail: ogawa_shiori@applibot.co.jp Facebook: https://www.facebook.com/100008453594381 Chatwork
      mail: ito_kotei@applibot.co.jp Facebook: https://www.facebook.com/mk.mdrm.cmtr Chatwork   …m(_ _)m
49.