Applibotデータ分析基盤の刷新 / Applibot

2016ba6b977a2e6691811fa66d5f4336?s=47 CyberAgent
February 22, 2019

Applibotデータ分析基盤の刷新 / Applibot

サイバーエージェントの技術者(エンジニア・クリエイター)向けカンファレンス『CA BASE CAMP 2019』

Applibotデータ分析基盤の刷新
小川 詩織 伊藤 皓程

2016ba6b977a2e6691811fa66d5f4336?s=128

CyberAgent

February 22, 2019
Tweet

Transcript

  1. Applibot    /

  2. 1.')! 2.    (" 3.(&*+ 4.& 5.& %

    6.$#
  3. 

  4. 48 MF / Shiori Ogawa •  - $.".# •

     - (6H: .+.( - ED)+J? etc. … •  - 2015.04  gloops - Ameba/GREE/FP Web(&,31>5 (K<!-) - /L0 - 2017.08    - 2017.08 2@!-,!-*.%.31&,, - 2018.03; =I(A.R.T. G7 /L=IB9 - 2018.10; &,'!AC'"
  5. # &" / Kotei Ito •  -  

     •  -  •  - 2015.04  QualiArts - () -    - 2017  AJA  - AJA SSP - 2019     - A.R.T.%$!
  6. 6fNc") D.I.V.E. •  (Data/Infrastructure/Value/Elevation) 6fNc6f<]IBb8 ! & • 

    GV2M •  Q.35g`Nc EIJ7 Nci;/4KY@C1M- RPEI8X;NcDOh2?T ;W',#SA ^9 ;Nc[^= 1 [d?T Q.115g`Nc HA  F: #+(-% a_Ue  Zj*-$L> g`',#\38X0
  7.     

  8.        

  9. !" #B • ## &;$ 96#B08% /; • !$ 7.(5

     /;2<*A:4   • $ @,)>=+'3-1?       " #
  10. &R*"@) 6;AJ N:5Q /8 -(=F 1S E< 0CN: OD>B M@

    $3H5 #!  OD*1S 6;?+ OD% 7,  ' ) KPI>B L;G9  2P.I>B 4K OD>B
  11.  … • K%"W2`_ • ;ZPO)74J6V9X51 • < CAF< 

    51 • S)/4LBT"( • ' Y2IQ="( • #&0:,+ :U\3L> • AF*]4LN @ DG  • .R/?E • M^!$8-[H  etc. ………
  12.    …

  13. *3( /+ • :HF7 !!#'-2 $, ".  % 

    "0)4 &1  &  68<2 5 ;&#0#-.3… *% 1'!($G $"/)2 … @A,+E?92 … =C2  D4B> 
  14. 4I  @5_1 •    01-BE 9> ,.BE

    &#BE !,; -4FG= !,7A < ,./2:6=  •   !,,.2:H /8 -?  '0+ -C3J $1D  (1% *)"!
  15. 3F  =7_2 =OJ8 !-/2:UU9;>DF5:  D:@ IRB ,'0*)."4 ?@#!0<

     $-C8+#+1&(%B56  ."ADE  &.Q>G ,3(9# 2Q>G  .0*1K @ G 2:<AMP? @ HC 4E/";E/")$6LN&.G %!'(7 +3-3(7C
  16.      

  17. 9b!L\TN • +  &- ",  $ (!#. 

    *% .'CD9b  P<7C+)1Q8.'Y; WVOJ#>! &(*#[TN=@:MB ! .'C #d` 46FE^>"aX%,cS#Z  • )'&- $-/ ($0# G?5 .'Y3" Q8 !_H 2AI45!…U]KLR !!
  18.   

  19.  

  20.       

  21. Applibot8316 •   #%)2, !$DWH(Redshift)/' (7# 5(/') • 

    Redshift !20 5(Redshift !/20 •   RedshiftBI"*4& - TableauGeckoBoard .+
  22. *7 #<)Va • 1   '(# % $& UZA1/%*7::.HM

    "GB#XI # >R •  )* ! !#8,DB \D$/'6=)1;+3b ?!  *7C'5=2O&'07%1SF#&(-9= &'07YT 8,.0/.\D API<) ^J;+3# •  % +$% "! &'07YT<)8,'5=2[]_K .0/.'=4;E WN!P! #QL O^J<) `@#
  23. /,?D'28- • DB  /(6/,&UDBdumpAf$ P<C5*/,Update %@:=aI !idJ% /, 1:M>c%$>RNbE

    Z OKApplibot • DB History DB/,\History.17&?D$"[ /, 1:M>c%$ •   History?D/,&8)h ?DBB%VHAf $ ;F _#DWH SeT $ ETLYG!^]X`gT $ +0639)%6'7,(4RNJQ LWApplibot
  24.         DB 

    + History DB      ()
  25. .+L?&04, • DWH(  ) ;X\.+.+%:#"5JVL?@ZY.+N?()-2 PrestoHive, DremelUE%1) RedshiftBigQuery 7P

    CIApplibot Applibot<G[ • DB DB3'.+History.+%L? 65JV.+O "L?RK T •    (+DWH) .+%" QJV=$!>W   FS/*FS3' /*8$9I.+%H"  "MB DAApplibot
  26. =32 B7 logloglogloglog logloglogloglog loglogloglogolo gloglog logloglogloglog logloglogloglog loglogloglogolo gloglog

    >@1 ' # ( * , * )$ +" 40!- %* F.O.X. etc…  :E +96&) DAGJ<H. F5/I;?1:E(BI) HadoopC:E=8/I:E
  27. '  E0B4 logloglogloglog logloglogloglog loglogloglogolo gloglog logloglogloglog logloglogloglog loglogloglogolo

    gloglog >A. #   $ & ( & %! ' 2,) "& F.O.X. etc… /C5H=:*L 1F-8?9IK  M  '  7 G  7G H3+J;@.7G(BI) HadoopD7G <6+J7G
  28. 20180424 AWS Black Belt Online Seminar AWS   

     https://www.slideshare.net/AmazonWebServicesJapan/20180424-aws-black-belt-online-seminar-aws?ref=https://aws.amazon.com/jp/blogs/news/webinar- bb-datalake-2018/
  29.  

  30. $'.(-6RW • Redshift]g y:H (^Jc85%…) onMf \! aZ(&8 `l>K@ /,on

     • aZ(&8 Y pEaZ _" Redshift#ftOX2%3%  SortKeyNqH! (7+,Ue! ACG;v ik= • 1+0/,#wrm SF+0859)*4+Flydata#^ … <DETLdIQLz! • P[hVbsj?T/,#B!aZuxH
  31. OEIK • -)S^?PEIK RedshiftFlydata 7@1600HMg -)%C_ &(.g#… • a@/+*UDY dfWD

    $ XD8J a@/+*-)%>$# UD0)4\L!$dfB • Redshift-)%V>$#bFFlydata`Ti [RUD%9"h_,236 =]'1(%N" c;A%G# eZ 15:-)QKPI<$ !…
  32. !$.%,9T\ (}t) • RedshiftajT\ • .6=k[)$7esyY@F dS`P')0 |  •

    Redshift RpO /+AIJ |  • n/+HS_eOHg i • w]ZDBB]ZTDQXz • %;)+YhGINB%"< ^NBlM urKv  • Ex$*0$Vacuumpb35#7>),:1>& AW • ?C/+fm€~7-,V  • 2)0/+ {qoUL)0<8>&(4)Flydata c  
  33. jX\` (‘ˆ) • mX\`RedshiftFlydata1600[d  • L9/:/ EJ)4>!•$+*tcƒ vL!€k.b F

    Š 4>Op"ga)++  • NU L#G^ i,F'cƒ' "]H '  • B0 "Sw ;@;@lz uV-  • 83.f‰!se%QT Œ_+(( • …R;54"rW { ŒŽxW- yWG^  • nEJh}KBI6A"?C7G^,F *”!‹, • Redshift!83.sP-,&!†YFlydata"„q “ }o rW.I+’‚"7=A D  O~ 1<2.h+‡"M T.Z, |
  34. Redshifti & • D1=8*F23F2kS&<E$Gdj63fRW -,D027'Xq#027G-,D QJ $ Z TZ 63#fR$

    • Lrp\ZDB"#o[ • Ua] DWHcn b $ 63'N%^ %lHYo[ • ?27>C-5)2"B1/2OeImP:+;+ ] g$ !(4A9F._M$@F5hVK %Ua`b
  35.   • E6O? • )* &T4.G> • -V8AHSWX0I •

    2,KPIG>C81F • % F.O.X.(dump# (QDBOK)N# /30I • %HSM<!&P@  • G>,'+"$+B;  O? • DGG>9= 7U JLYR:K5
  36.     

  37. None
  38.      

  39. 

  40. ETL

  41. 

  42.  Lambda ""(#event ./16  : *9 =>-  !

    3;!70 $,)!</'% →  8& 5+42 API
  43.   ^LpX 7$8,%5tH@ `W ?wPZbsxyCd E:KPI`W\8D_ ! 4+,F.O.X.n1,BGCd +(4bskU#-5qY

    !  `W:69039+[T# !pX RV O u!eizrSfK → M200QF/> → 10Q+α/> M1/20'.2 →  → `W)&7#aPdkU! → ADI; lo<" j → S3]APIqc →=NJ/8hm →=NJ/8hm →/8hmgv+*8
  44. 

  45.  • %#')F@ • )!)"0A/L< • >9%4J2 • RedshiftD! I,

    • Athena ()&$6K ( :*%%) • =-%6ECG2 • >91.Glue B3? NM2  (5;H) • 8%0 4J2+7Redshift Spectrum …
  46. >M(6.6-O;G  •    JV QD LHT5'IY8 

    LH 5' QD&%2." NC :X)12ETB 9 • KPI   SAJ7KPI?!R5' 8=P  KPI05*#&/<,4KPI&$2KF(+/ W • BI   BI-3 U@QD…
  47.     

  48.  …   mail: ogawa_shiori@applibot.co.jp Facebook: https://www.facebook.com/100008453594381 Chatwork 

     mail: ito_kotei@applibot.co.jp Facebook: https://www.facebook.com/mk.mdrm.cmtr Chatwork   …m(_ _)m
  49.