Spark + S3 + R3 을 이용한 데이터 분석 시스템 만들기

Spark
- 최대한 메모리를 활용하는 MapReduce
- 훨씬 다양한 API (distinct, groupByKey, sort, join, …)
- 재사용하는 데이터는 cache
- Map같은 연산은 lazy 하게 처리
Spark + S3
- 로그를 저장해 주는 HDFS layer를 따로 운영하지 않아도 됨
- s3n / s3 file system을 통해서 Hadoop에 잘 integration 되어 있음
- S3 EC2 간 데이터 전송이 무료!
Spark + R3 instance
• R3 instance
- 최근에 출시된, 메모리 optimized instance
- 메모리 244GB ( r3.8xlarge기준 )
- 320GB SSD x 2 (Shuffle 결과 임시 저장할 때 유리함)
- Spark에 최적화된 instance type
결론
Spark + S3 + R3 instance 조합으로 처리 용량을 늘리거나 줄일 수 있는 자유도가 있고 직접 운영하지 않아도 되는 10배 이상 빠른 분석 시스템을 만들었습니다

Transcript

1. 4QBSL43ਸ੉ਊೠ
   ؘ੉ఠ࠙ࢳदझమٜ݅ӝ
   7$/$ѐߊ౱ӣݺࠁ
   "84ೠҴਬ੷Ӓܛ BXTLSVH
   
   
   

   View Slide

2. ߊ಴੗ࣗѐ
   ↟ ӣݺࠁ
   ↟ 7$/$ীࢲ࠺౟ਦਸѐߊೞҊ੓חѐߊ੗
   ↟ ࢲߡ౱җؘ੉ఠ౱ীࢲ੟׮ೠѪٜਸҊ஖ח઺
   ↟ ഥࢎীࢲ"84ীتਸॳחѪਸ׸׼
   

   View Slide

3. ࠺౟ਦ
   ↟ ழ೒ٜਸਤೠݽ߄ੌࢲ࠺झ
   ↟ ই੉ಪ উ٘۽੉٘য೒ܻா੉࣌ઁҕ
   ↟ ଻౴ ӝ֛ੌ ࢎ૓ ݫݽ ஫ܽ؊ӝמઁҕ
   ↟ ੹ࣁ҅ীࢲ݅׮਍۽٘ BTPG
   
   ↟ IUUQCFUXFFOVT
   ↟ IUUQFOHJOFFSJOHWDODDPLS
   

   View Slide

4. ֙ୡ
   7$/$ؘ੉ఠ࠙ࢳ౱j
   

   View Slide

5. Ӓٜ਷טযաחؘ੉ఠী
   Ҋా߉Ҋ੓঻חؘj
   

   View Slide

6. ӝઓؘ੉ఠ࠙ࢳदझమ
   4
   4FSWFS
   )%'4)#BTF
   .BQ3FEVDF
   .Z42-
   

   View Slide

7. ӝઓؘ੉ఠ࠙ࢳदझమ
   4
   4FSWFS
   )%'4)#BTF
   .BQ3FEVDF
   .Z42-
   6QMPBEMPH
   

   View Slide

8. ӝઓؘ੉ఠ࠙ࢳदझమ
   4
   4FSWFS
   )%'4)#BTF
   .BQ3FEVDF
   .Z42-
   %PXOMPBEMPH
   #VMLJOTFSU
   

   View Slide

9. ӝઓؘ੉ఠ࠙ࢳदझమ
   4
   4FSWFS
   )%'4)#BTF
   .BQ3FEVDF
   .Z42-
   "OBMZTJT
   

   View Slide

10. ӝઓؘ੉ఠ࠙ࢳदझమ
    4
    4FSWFS
    )%'4)#BTF
    .BQ3FEVDF
    .Z42-
    3FQPSU
    

    View Slide

11. ӝઓؘ੉ఠ࠙ࢳदझమ
    ↟ ݒੌجܻח઱ӝ੸ੋ࠙ࢳ੘সҗ0OEFNBOE࠙ࢳ੉ਃҳؽ
    ↟ ࢲߡо+TPOਵ۽۽Ӓܳթӣ
    ↟ \XIP\BEESFTTYYYYYY VTFS*E VTFS"HFOU#FUXFFO
    4)7&,"OESPJELP@,3
    ^ XIBU
    \PCKFDU*E@YYYY QBSFOU*E PCKFDU5ZQF64&3@5)3&"%@.&44"(
    & NFTTBHF\MFOHUI^^ IPX\BDUJPO"%%^ XIFO\UJNF.JMMJT
    ^ WFSTJPO^
    ↟ ࢲߡীࢲೞܖীೠߣঀ4۽স۽٘
    ↟ ഥࢎղীؘ੉ఠ࠙ࢳਊ)%'4)#BTF.BQ3FEVDF௿۞झఠઓ੤
    ↟ ইஜী4۽ࠗఠ۽Ӓܳ߉ই৬ࢲ)#BTFী॔ইࠗ਺
    ↟ ࠙ࢳਸج۰Ѿҗܳ.Z42-ਸ੷੢ೣ
    

    View Slide

12. ӝઓؘ੉ఠ࠙ࢳदझమ
    ↟ ୊਺ূदр߆ীউѦܻҊજওભj
    ↟ ҳࢿറ֙੉૑աפदрبֈѱѦܿ
    ↟ ૑Әࠗఠਬ੷ࣻоߓоغݶ
    ↟ ৈ۞؀੄ࢲߡ৬য়೑ࣗझझఖਸ૒੽ҙܻ೧ঠೣ
    ↟ ыо૑դҙҗ੢গ੄োࣘ
    

    View Slide

13. ыо૑੢গೞ٘ਝয
    ↟ ೞ٘٣झ௼
    ↟ ݫݽܻ
    ↟ ೞنତীDPSSVQUJPO੉ಌ૑חߡӒઓ੤
    ↟ ݫݽܻח࢖ࢿ
    ↟ ֎౟ਕ௼
    ↟ (ח߽ݾ (ח࠺श
    

    View Slide

14. ݔզೞ٘ਝযҊ஖۞
    ׮֝णפ׮
    

    View Slide

15. ղоѐߊ੗ੋ૑ਊ౹ੋ૑
    

    View Slide

16. ыо૑੢গࣗ೐౟ਝয
    ↟ ($UJNFPVU
    ↟ )%'4)#BTF.BQ3FEVDF
    ↟ ബҗ੸ੋݫݽܻೡ׼੄য۰਑
    ↟ .Z42-੉ցޖוܿ
    

    View Slide

17. ыо૑դҙ߄׮
    ↟ ইஜী4۽ࠗఠ۽Ӓࠂࢎоցޖয়ېѦܿ
    ↟ ࢎ੢שয়ט࠙ࢳѾҗ঱ઁա৬ਃ
    ↟ ؘ੉ఠ౱۽Ӓ౵ੌٜই૒߄׮ѤցҊ੓חؘਃj
    ↟ ࢎ੢שj
    

    View Slide

18. ؀উݽ࢝
    ↟ &MBTUJD4FBSDI
    ↟ ؘ੉ఠо݆ਵݶוܻ׮חফӝоj
    ↟ )%'4.BQ3FEVDF )#BTFઁѢ
    
    ↟ ৈ੹൤JOEFYJOH੉೙ਃೠؘj
    ↟ 4QBSL
    ↟ ߄׮ѤցীࢲIPUೞ׮חؘj
    

    View Slide

19. ↟ ୭؀ೠݫݽܻܳഝਊೞח.BQ3FEVDF
    ↟ ഻ঁ׮নೠ"1*
    ↟ EJTUJODU HSPVQ#Z,FZ TPSU KPJO j
    ↟ ੤ࢎਊೞחؘ੉ఠחDBDIF
    ↟ .BQэ਷ো࢑਷MB[Zೞѱ୊ܻ
    

    View Slide

20. 74
    %BUB 3FTVMU
    .BQ3FEVDF੄ৈ۞ױ҅࠙ࢳ
    %BUB 3FTVMU
    4QBSLীࢲ੄ৈ۞ױ҅࠙ࢳ
    

    View Slide

21. Ӓېࢲઁо
    4QBSLܳೠߣॄࠌणפ׮
    

    View Slide

22. View Slide

23. 4QBSL੢੼
    ↟ ڪযդࣘب
    ↟ রMJOFDPVOUীୡ੿ب
    ↟ উ੿ࢿ
    ↟ 4QBSL݅ਵ۽୽࠙೧ࢲࣗ೐౟ਝযр੄ݫݽܻೡ׼੉गоࢎ
    ۄ૗
    ↟ 4DBMFPVU
    ↟ "EIPDRVFSZоמ 3&1-
    
    

    View Slide

24. ࢎप੹.BQ3FEVDFܳ
    ૞ࠄ੸੉হणפ׮
    

    View Slide

25. ೞ૑݅4QBSLחجܾࣻ
    ੓णפ׮
    

    View Slide

26. ೞ૑݅4QBSLחجܾࣻ
    ੓णפ׮
    ߓ਎ѱহ؍ؘਃ
    

    View Slide

27. 4QBSL৘ઁݫࣁ૑ѐࣻ
    
    ↟ WBMMPHTTDUFYU'JMF tTONZMPHTMPHT
    
    ↟ WBMNTH-PHTMPHTGJMUFS @DPOUBJOT t"%%@.4(u
    
    
    ↟ WBMNTH$PVOUNTH-PHTDPVOU
    

    View Slide

28. 4QBSL৘ઁݫࣁ૑ѐࣻ
    
    ↟ TDUFYU'JMF tTONZMPHTMPHTu
    GJMUFS @DPOUBJOT t"%%@.4(u
    
    DPVOU
    

    View Slide

29. 4QBSL৘ઁݫࣁ૑ࠁմਬ੷ࣻ
    ↟ WBMMPHTTDUFYU'JMF tTONZMPHTMPHTu
    
    ↟ WBMNTH-PHTMPHTGJMUFS @DPOUBJOT t"%%@.4(u
    
    
    ↟ WBMNTH6TFSTNTH-PHTNBQ @UP"DUJPO-PHVTFS*E
    EJTUJODU
    ↟ WBMNTH6TFS$PVOUTNTH6TFSTDPVOU
    

    View Slide

30. 1$ࢎਊ۝$PIPSUҳೞӝ
    ↟ WBMMPHTTDUFYU'JMF tTOMPHTCFUXFFOMPH
    
    ↟ WBMQD""TMPHTGJMUFS @DPOUBJOT t#FUXFFO1$u
    
    NBQ @UP""
    
    ↟ WBMQD6TFS%BUFTQD""TNBQ MPH MPHVTFS*E GPSNBUGPSNBU MPHXIFOUJNF.JMMJT
    
    EJTUJODU
    ↟ WBMQD6TFS'JSTU%BUFQD6TFS%BUFTHSPVQ#Z,FZ
    NBQ LW LW@ LW@UP-JTUTPSUFEIFBE
    
    
    ↟ WBMQD6TFS'JSTU%BUF"DUJWF%BUFQD6TFS'JSTU%BUFKPJO QD6TFS%BUFT
    
    ↟ WBMSFTVMUQD6TFS'JSTU%BUF"DUJWF%BUFNBQ @@
    
    SFEVDF#Z,FZ @@
    TPSU#Z,FZ USVF
    
    

    View Slide

31. 4QBSL"84
    

    View Slide

32. ੗੉ઁӒۢ੉ઁ૓૞۽
    "84ীࢲ਍৔غח
    4QBSL࠙ࢳदझమਸٜ݅যࠇद׮
    

    View Slide

33. 4UPSBHFMBZFSח
    যڌѱೞ૑
    

    View Slide

34. ۽Ӓחযରೖ4ী੓ਗ਼ই
    4ܳӒրॳ੗
    

    View Slide

35. 4QBSL4
    ↟ ۽Ӓܳ੷੢೧઱ח)%'4MBZFSܳٮ۽਍৔ೞ૑ঋইبؽ
    ↟ TOTGJMFTZTUFNਸా೧ࢲ)BEPPQীੜJOUFHSBUJPOغয੓਺
    ↟ 4&$рؘ੉ఠ੹࣠੉ޖܐ
    

    View Slide

36. 4QBSL4
    ↟ &$ীࢲ4੄੽Ӕࣘببજ਺
    ↟ SYMBSHF (#/*$
    ীࢲ_.#ZUFTTFDա১
    

    View Slide

37. 4QBSL4
    ↟ 4UPSBHFMBZFS੄GBOPVUਸѣ੿ೡ೙ਃоহ਺
    ↟ ૒੽4UPSBHFMBZFSܳ਍৔ೠ׮ݶ زदী׮ࣻ੄4QBSLDMVTUFSо۽Ӓ
    ী੽ӔೡٸחUISPVHIQVUী߽ݾ੉ࢤӡࣻ߆ীহ਺
    ↟ 4ীࢲח੉۞ೠޙઁܳযו੿ب೧Ѿ೧ષ
    4QBSLDMVTUFS 4QBSLDMVTUFS
    -PH4
    

    View Slide

38. 4QBSL&$
    ↟ ೙ਃೡٸDMVTUFS௼ӝܳః਋Ѣա୶о੸ੋDMVTUFSܳڸ਎ࣻ੓׮
    ↟ ৈ۞ѐ੄࠙ࢳਸ೙ਃೡ҃਋زदীجܾࣻ੓׮
    ↟ زदীجܾࣻ੓ਵݶ оѺ਷ਬ૑ೞݶࢲݽٚ࠙ࢳ੉՘աחदрਸ
    খ׼ӡࣻ੓׮
    ↟ э਷੘সী݆਷&$JOTUBODFܳࢎਊೞ؊ۄبѾҴэ਷࠺ਊ
    

    View Slide

39. 4QBSL3JOTUBODF
    ↟ 3JOTUBODF
    ↟ ୭Ӕী୹दػ ݫݽܻPQUJNJ[FEJOTUBODF
    ↟ ݫݽܻ(# SYMBSHFӝળ
    
    ↟ (#44%Y
    ↟ 4IVGGMFѾҗ੐द੷੢ೡٸਬܻೣ
    ↟ 4QBSLী୭੸ചػJOTUBODFUZQF
    

    View Slide

40. 4QBSL1MBDFNFOU(SPVQ
    ↟ 1MBDFNFOU(SPVQ
    ↟ JOTUBODFр੄OFUXPSLMBUFODZоծই૗
    ↟ XJEFEFQFOEFODZTIVGGMFPQFSBUJPOীࢲਬܻೣ
    

    View Slide

41. 4QBSL"84ؘ੉ఠ࠙ࢳदझమ
    -PHCVDLFU
    0OEFNBOE
    4QBSL$MVTUFS
    0OEFNBOE
    4QBSL$MVTUFS
    %BJMZ"OBMZTJT
    4QBSL$MVTUFS
    1MBDFNFOU(SPVQ
    ;FQQFMJO%BTICPBSE 6TFST
    

    View Slide

42. 4QBSL۽৤ӟറj
    ↟ ೞܖ஖࠙ࢳदрदрղ৻۽хࣗ
    ↟ ೠߣ$BDIF೧ࢲJUFSBUJWFೞѱجࣻ੓਺
    ↟ ೞ٘ਝযҙܻ৬ৈ۞झఖ੄ࣗ೐౟ਝযҙܻ੄Ҧ۽਑ীࢲߩযթ
    ↟ )%'4)#BTF.BQ3FEVDF4QBSL
    ↟ *OIPVTFDMVTUFSী࠺೧ࢲࣽࣻ࠺ਊ਷࠺तೞѢա׮ࣗ࠺व٠
    ↟ ഻ঁ׮নೠदبٜ੉оמ೧૗
    

    View Slide

43. 4QBSL۽৤ӟറj
    ↟ ֙஖ؘ੉ఠী؀೧ࢲ࠙ࢳਸ೧ࠁҊरয
    ↟ җѢীח׼ੌ࠙ࢳਸ઺ױदఃݶࢲয়ېѦܻח࠙ࢳਸجܻӝয۰਑
    ↟ അ੤ח׼ੌ࠙ࢳҗदр੉য়ېѦܻח࠙ࢳਸزदীप೯оמ
    ↟ ࠺౟ਦ੄ߡӒ੉ग୶੸ী௾ب਑੉ؽ
    ↟ যڃਬ੷оজ੉۽Ӓইਓغ঻׮חؘ੉ਬܳݽܰѷ׮
    ↟ ࢲߡ੄ߡӒਬ੷੄׳੹੄೯زী੄೧ࢲࢤӟޙઁ
    ↟ ׳஖੄೧׼ਬ੷੄೯ز۽Ӓ৬&YDFQUJPO۽Ӓܳ࠺Ү࠙ࢳೞৈਗੋߊѼ
    ↟ ૓ੑ੢߷੉ծইઉ؊݆਷ѐߊ౱ਗٜ੉ؘ੉ఠ࠙ࢳীҙबਸо૑ӝद੘ೣ
    

    View Slide

44. 4QBSLӒ৻੄5JQ
    

    View Slide

45. 4QBSL5JQT
    ↟ 4IVGGMFೡٸࢤӝח੐द੷੢౵ੌٜ੉ѐࣻоষ୒݆੉ࢤӡࣻ੓਺
    ↟ ٣झ௼חթחؘJOPEFоݽ੗ۆࢎకߊࢤ
    ↟ NLGTFYUJэ਷২࣌ਵ۽ழߡ
    ↟ NEBENਵ۽44%فѐޘযࢲ4QBSLUFNQಫ؊ীNPVOU
    

    View Slide

46. 4QBSL5JQT
    ↟ 4ীࢲ੍য৬ࢲ୊ܻೡ҃਋౵ੌ੉࠺तೠ௼ӝ۽ଂѐઉ੓חѱਬܻ
    ↟ ௼ӝо ੋ۽Ӓٜ۽੷੢غয੓חѪࠁ׮
    ૞ܻѐо੓חѱਬܻೣ
    ↟ DPSFܳݽفࢎਊೞ੗
    ↟ ਗېࢲߡীࢲೞܖীೠߣ۽Ӓܳস۽٘ೞ؍Ѫਸೠदрীೠߣਵ۽
    ߸҃ೣ
    

    View Slide

47. 4QBSL5JQT;FQQFMJO
    ↟ ؘ੉ఠ࠙ࢳਸਤೠਢӝ߈֢౟ؘ࠘੉ఠWJTVBMJ[BUJPOUPPM
    ↟ 4QBSL߂4QBSL42-җোز೧ࢲಞܻೞѱؘ੉ఠ୊ܻ Ӓې೐Ӓܻח੘
    স١ਸ૓೯ೡࣻ੓਺
    ↟ ٮ۽؀एࠁٜ٘ܳ݅೙ਃоহয૗
    ↟ 5BKP١׮ؘܲ੉ఠ࠙ࢳోҗ੄ా೤ب૓೯઺
    

    View Slide

48. 4QBSL5JQT;FQQFMJO
    

    View Slide

49. Ѿۿ
    

    View Slide

50. ↟ 4QBSL43JOTUBODFઑ೤ਵ۽
    ↟ ୊ܻਊ۝ਸטܻѢա઴ੌࣻ੓ח੗ਬبо੓Ҋ
    ↟ ૒੽਍৔ೞ૑ঋইبغח
    ↟ ߓ੉࢚ࡅܲ࠙ࢳदझమਸٜ݅঻णפ׮
    Ѿۿ
    

    View Slide

51. View Slide

52. 2"
    

    View Slide