Powering a Startup with Apache SPark

Apache Summit 2017 발표 자료

Transcript

1. Kevin, Between (VCNC) kevin@between.us Powering a Startup with Apache Spark #EUent8
2. 4FPVM
   4PVUI
   ,PSFB
3. (BOHOBN )POHEBF *UBFXPO .ZVOHEPOH
5. 
   
   CFUB
   VTFST
   
   
   SFMFBTF
   .
   EPXOMPBET
   
   .
   EPXOMPBET
   HMPCBM
   MBVODIFT
   
   #FUXFFO
   
   .
   EPXOMPBET
   
   #FUXFFO
   
   
   4UBSUT
   NPOFUJ[BUJPO
   .
   EPXOMPBET
   
   (MPCBM
   FYQBOTJPO
   OFX
   CVTJOFTT
   UFBN
   PG
   
   
6. put your #assignedhashtag here by setting the footer in view-header/footer ,FWJO
   ,JN • Came from Seoul, South Korea • Co-founder, used to be a product developer • Now a data analyst, engineer, team leader • Founder of Korea Spark User Group • Committer and PMC member of Apache Zeppelin 6
7. #FUXFFO
   %BUB
   5FBN 7
8. *OUSP
   UP
   #FUXFFO
   %BUB
   5FBN • Data engineer * 4 – Manager, engineer with various stack of knowledge and experience – Junior engineer, used to be a server engineer – Senior engineer, has lots of exps and skills – Data engineer, used to be a top level Android developer • Hiring data analyst and machine learning expert 8
9. #FUXFFO
   %BUB
   5FBN
   JT
   EPJOH • Analysis – Service monitoring – Analysis usage of new features and build product strategies • Data Infrastructure – Build and manage infrastructure – Spark, Zeppelin, AWS, BI Tools, etc • Third Party Management – Mobile Attribution Tools for marketing (Kochava, Tune, Appsflyer, etc) – Google Analytics, Firebase, etc – Ad Networks 9
10. #FUXFFO
    %BUB
    5FBN
    JT
    EPJOH • Machine Learning Study & Research – For the next business model • Support team – To build business, product, monetization strategies • Performance Marketing Analysis – Monitoring effectiveness of marketing budgets • Product Development – Improves client performance, server architecture, etc 10
12. 
    1.
    _ 12 Sunset @ Between Office
13. 5FDIOPMPHJFT 13
14. 3FRVJSFNFOUT • Big Data – 2TB/day of log data from millions of DAU – 20M of users • Small Team – Team of 4, need to support 50 • Tiny Budget – Company is just over BEP (Break Even Point) • Need very efficient tech stack! 14
15. 8BZ
    8F
    8PSL • Use Apache Spark as a general processing engine • Scriptify everything with Apache Zeppelin • Heavy utilization of AWS and Spot instances to cut cost • Proper selection of BI Dashboard Tools 15
16. "QBDIF
    4QBSL
    (FOFSBM
    &OHJOF • Definitely the best way to deal with big data (as you all know!) • It’s performance, agility exactly meets startup requirements – Used Spark from 2014 • Great match with Cloud Service, especially with Spot instance – Utilizing burst nature of Cloud Service 16
17. 4DSJQUJGZ
    &WFSZUIJOH
    XJUI
    ;FQQFMJO • Doing everything on Zeppelin! • Daily batch tasks in form of Spark scripts (using Zeppelin scheduler) • Ad hoc analysis • Cluster control scripts • The world first user of Zeppelin! • More than 200 Zeppelin notebooks 17
18. "84
    $MPVE • Spot Instance is my friend! – Mostly use spot instance for analysis – only 10 ~ 20% of cost compare to on-demand instances • Dynamic cluster launch with Auto Scale – Launch clusters automatically for batch analysis – Manually launch more clusters on Zeppelin, with Auto Scale script – Automatically diminish clusters when no usage 18
19. #*
    %BTICPBSE
    5PPMT • Use Zeppelin as a dashboard using Spark SQL with ZEPL • Holistics (holistics.io) or Dash (plot.ly/products/dash/) 19
20. 2VFTUJPOT
    
    $IBMMFOHFT 20
21. 3%%
    "1*
    PS
    %BUB'SBNF
    "1* • Now Spark has very different style of APIs – Programmatic RDD API – SQL-like DataFrame, DataSet API • In case of having many, simple ad-hoc queries – DataFrame works • Having more complex, deep dive analytic questions – RDD works • For a while, mostly use RDD, DataFrame for ML or simple ad hoc tasks 21
22. 4VTIJ
    PS
    $PPLFE
    %BUB • Keeping data in a raw form as possible! – Doing ETL’s usually makes trouble, increasing management cost – The Sushi Principle (Joseph & Robert in Strata) – Drastically reduce operation & management cost – Apache Spark is a great tool for extracting insight from raw data 22 fresh data!
23. 5P
    )JSF
    %BUB
    "OBMZTU
    PS
    /PU • For data analyst, expected skill set are.. – Excel, SQL, R, .. • Those skills are not expected.. – Programatic API like Spark RDD – Cooking raw data • Prefer data engineer with analytic skills • May need to add some ETL tasks to work with data analyst 23
24. #FUUFS
    'BTUFS
    5FBN
    4VQQPSU • Better - Zeppelin is great for analyzing data, but not enough for sharing data for team – We have really few alternatives – Increase of using BI dashboard tools? – Still finding a good way • Faster - Launching a Spark cluster takes few minutes – Not bad, but we want it faster – Google BigQuery or AWS Athena – SQL Database with ETL 24
25. 'VUVSF
    1MBO • Prepare for exploding # of data operations! – Team is growing, business is growing – # of tasks – # of 3rd party data products – Communication cost • Operations with machine learning & deep learning – Better way to manage task & data flow 25
26. -FUsT
    XSBQ
    VQ 26
27. 8IBU
    .BUUFST
    GPS
    6T • Support Team – Each Team should see proper data and make good decision from it – Regular meetings, fast response to adhoc data request – Ultimately, our every activity should be related to company’s business • Technical Lead – Technical investments for competence of both company and individual – Working in Between should be a best experience for each individuals • Social Impact – Our activity on work has valuable impact for society? – Open source, activity on community 27
28. )PX
    "QBDIF
    4QBSL
    JT
    1PXFSJOH
    B
    4UBSUVQ • One great tool for general purpose – Daily batch tasks – Agile, adhoc analysis – Drawing dashboard – Many more.. • Helps saving time, reducing cost of data operations • Great experience for engineer and analyst • Sharing know-how’s to / from community 28
29. 8PSL
    BT
    B
    EBUB
    FOHJOFFS
    BU
    4UBSUVQ • Fascinating, fast evolution of tech • Need hard work and labor • Data work will shine only when it is understood and used by teammates 29 Two Peasants Digging, Vincent van Gogh Two Men Digging, Jean-Francois Millet
30. 5IBOL
    ZPV 30