Pro Yearly is on sale from $80 to $50! »

Spark + S3 + R3 을 이용한 데이터 분석 시스템 만들기

6a11050c8147e4f5fbf2637907c27964?s=47 VCNC
December 20, 2014

Spark + S3 + R3 을 이용한 데이터 분석 시스템 만들기

Spark
- 최대한 메모리를 활용하는 MapReduce
- 훨씬 다양한 API (distinct, groupByKey, sort, join, …)
- 재사용하는 데이터는 cache
- Map같은 연산은 lazy 하게 처리
Spark + S3
- 로그를 저장해 주는 HDFS layer를 따로 운영하지 않아도 됨
- s3n / s3 file system을 통해서 Hadoop에 잘 integration 되어 있음
- S3 EC2 간 데이터 전송이 무료!
Spark + R3 instance
• R3 instance
- 최근에 출시된, 메모리 optimized instance
- 메모리 244GB ( r3.8xlarge기준 )
- 320GB SSD x 2 (Shuffle 결과 임시 저장할 때 유리함)
- Spark에 최적화된 instance type
결론
Spark + S3 + R3 instance 조합으로 처리 용량을 늘리거나 줄일 수 있는 자유도가 있고 직접 운영하지 않아도 되는 10배 이상 빠른 분석 시스템을 만들었습니다

6a11050c8147e4f5fbf2637907c27964?s=128

VCNC

December 20, 2014
Tweet

Transcript

  1. 4QBSL 4 3ਸ੉ਊೠ ؘ੉ఠ࠙ࢳदझమٜ݅ӝ 7$/$ѐߊ౱ӣݺࠁ "84ೠҴਬ੷Ӓܛ BXTLSVH  

  2. ߊ಴੗ࣗѐ ↟ ӣݺࠁ ↟ 7$/$ীࢲ࠺౟ਦਸѐߊೞҊ੓חѐߊ੗ ↟ ࢲߡ౱җؘ੉ఠ౱ীࢲ੟׮ೠѪٜਸҊ஖ח઺ ↟ ഥࢎীࢲ"84ীتਸॳחѪਸ׸׼

  3. ࠺౟ਦ ↟ ழ೒ٜਸਤೠݽ߄ੌࢲ࠺झ ↟ ই੉ಪ উ٘۽੉٘য೒ܻா੉࣌ઁҕ ↟ ଻౴ ӝ֛ੌ ࢎ૓

    ݫݽ ஫ܽ؊ӝמઁҕ ↟ ੹ࣁ҅ীࢲ݅ ׮਍۽٘ BTPG  ↟ IUUQCFUXFFOVT ↟ IUUQFOHJOFFSJOHWDODDPLS
  4. ֙ୡ 7$/$ؘ੉ఠ࠙ࢳ౱j

  5. Ӓٜ਷טযաחؘ੉ఠী Ҋా߉Ҋ੓঻חؘj

  6. ӝઓؘ੉ఠ࠙ࢳदझమ 4 4FSWFS )%'4 )#BTF .BQ3FEVDF .Z42-

  7. ӝઓؘ੉ఠ࠙ࢳदझమ 4 4FSWFS )%'4 )#BTF .BQ3FEVDF .Z42- 6QMPBEMPH

  8. ӝઓؘ੉ఠ࠙ࢳदझమ 4 4FSWFS )%'4 )#BTF .BQ3FEVDF .Z42- %PXOMPBEMPH #VMLJOTFSU

  9. ӝઓؘ੉ఠ࠙ࢳदझమ 4 4FSWFS )%'4 )#BTF .BQ3FEVDF .Z42- "OBMZTJT

  10. ӝઓؘ੉ఠ࠙ࢳदझమ 4 4FSWFS )%'4 )#BTF .BQ3FEVDF .Z42- 3FQPSU

  11. ӝઓؘ੉ఠ࠙ࢳदझమ ↟ ݒੌجܻח઱ӝ੸ੋ࠙ࢳ੘সҗ0OEFNBOE࠙ࢳ੉ਃҳؽ ↟ ࢲߡо+TPOਵ۽۽Ӓܳթӣ ↟ \XIP\BEESFTTYYYYYY VTFS*E VTFS"HFOU#FUXFFO 4)7&,"OESPJELP@,3

    ^ XIBU \PCKFDU*E@YYYY QBSFOU*E PCKFDU5ZQF64&3@5)3&"%@.&44"( & NFTTBHF\MFOHUI^^ IPX\BDUJPO"%%^ XIFO\UJNF.JMMJT ^ WFSTJPO^ ↟ ࢲߡীࢲೞܖীೠߣঀ4۽স۽٘ ↟ ഥࢎղীؘ੉ఠ࠙ࢳਊ)%'4 )#BTF .BQ3FEVDF௿۞झఠઓ੤ ↟ ইஜী4۽ࠗఠ۽Ӓܳ߉ই৬ࢲ)#BTFী॔ইࠗ਺ ↟ ࠙ࢳਸج۰Ѿҗܳ.Z42-ਸ੷੢ೣ
  12. ӝઓؘ੉ఠ࠙ࢳदझమ ↟ ୊਺ূदр߆ীউѦܻҊજওભj ↟ ҳࢿറ֙੉૑աפदрبֈѱѦܿ ↟ ૑Әࠗఠਬ੷ࣻоߓоغݶ  ↟ ৈ۞؀੄ࢲߡ৬য়೑ࣗझझఖਸ૒੽ҙܻ೧ঠೣ

    ↟ ыо૑դҙҗ੢গ੄োࣘ
  13. ыо૑੢গೞ٘ਝয ↟ ೞ٘٣झ௼ ↟ ݫݽܻ ↟ ೞنତীDPSSVQUJPO੉ಌ૑חߡӒઓ੤ ↟ ݫݽܻח࢖ࢿ ↟

    ֎౟ਕ௼ ↟ (ח߽ݾ (ח࠺श
  14. ݔզೞ٘ਝযҊ஖۞ ׮֝णפ׮

  15. ղоѐߊ੗ੋ૑ਊ౹ੋ૑

  16. ыо૑੢গࣗ೐౟ਝয ↟ ($UJNFPVU ↟ )%'4)#BTF.BQ3FEVDF ↟ ബҗ੸ੋݫݽܻೡ׼੄য۰਑ ↟ .Z42-੉ցޖוܿ

  17. ыо૑դҙ߄׮ ↟ ইஜী4۽ࠗఠ۽Ӓࠂࢎоցޖয়ېѦܿ ↟ ࢎ੢שয়ט࠙ࢳѾҗ঱ઁա৬ਃ  ↟ ؘ੉ఠ౱۽Ӓ౵ੌٜই૒߄׮ѤցҊ੓חؘਃj ↟ ࢎ੢שj

  18. ؀উݽ࢝ ↟ &MBTUJD4FBSDI ↟ ؘ੉ఠо݆ਵݶוܻ׮חফӝоj ↟ )%'4 .BQ3FEVDF )#BTFઁѢ 

    ↟ ৈ੹൤JOEFYJOH੉೙ਃೠؘj ↟ 4QBSL ↟ ߄׮ѤցীࢲIPUೞ׮חؘj
  19. ↟ ୭؀ೠݫݽܻܳഝਊೞח.BQ3FEVDF ↟ ഻ঁ׮নೠ"1* ↟ EJTUJODU HSPVQ#Z,FZ TPSU KPJO j

    ↟ ੤ࢎਊೞחؘ੉ఠחDBDIF ↟ .BQэ਷ো࢑਷MB[Zೞѱ୊ܻ
  20. 74 %BUB 3FTVMU .BQ3FEVDF੄ৈ۞ױ҅࠙ࢳ %BUB 3FTVMU 4QBSLীࢲ੄ৈ۞ױ҅࠙ࢳ

  21. Ӓېࢲઁо 4QBSLܳೠߣॄࠌणפ׮

  22. None
  23. 4QBSL੢੼ ↟ ڪযդࣘب ↟ রMJOFDPVOUীୡ੿ب ↟ উ੿ࢿ ↟ 4QBSL݅ਵ۽୽࠙೧ࢲࣗ೐౟ਝযр੄ݫݽܻೡ׼੉गоࢎ ۄ૗

    ↟ 4DBMFPVU ↟ "EIPDRVFSZоמ 3&1-
  24. ࢎप੹.BQ3FEVDFܳ ૞ࠄ੸੉হणפ׮

  25. ೞ૑݅4QBSLחجܾࣻ ੓णפ׮

  26. ೞ૑݅4QBSLחجܾࣻ ੓णפ׮ ߓ਎ѱহ؍ؘਃ

  27. 4QBSL৘ઁݫࣁ૑ѐࣻ  ↟ WBMMPHTTDUFYU'JMF tTONZMPHTMPHT   ↟ WBMNTH-PHTMPHTGJMUFS @DPOUBJOT

    t"%%@.4(u  ↟ WBMNTH$PVOUNTH-PHTDPVOU
  28. 4QBSL৘ઁݫࣁ૑ѐࣻ  ↟ TDUFYU'JMF tTONZMPHTMPHT u GJMUFS @DPOUBJOT t"%%@.4(u DPVOU

  29. 4QBSL৘ઁݫࣁ૑ࠁմਬ੷ࣻ ↟ WBMMPHTTDUFYU'JMF tTONZMPHTMPHT u  ↟ WBMNTH-PHTMPHTGJMUFS @DPOUBJOT t"%%@.4(u

     ↟ WBMNTH6TFSTNTH-PHTNBQ @UP"DUJPO-PHVTFS*E EJTUJODU ↟ WBMNTH6TFS$PVOUTNTH6TFSTDPVOU
  30. 1$ࢎਊ۝$PIPSUҳೞӝ ↟ WBMMPHTTDUFYU'JMF tTOMPHTCFUXFFOMPH   ↟ WBMQD""TMPHTGJMUFS @DPOUBJOT t#FUXFFO1$u

    NBQ @UP""  ↟ WBMQD6TFS%BUFTQD""TNBQ MPH MPHVTFS*E GPSNBUGPSNBU MPHXIFOUJNF.JMMJT EJTUJODU ↟ WBMQD6TFS'JSTU%BUFQD6TFS%BUFTHSPVQ#Z,FZ NBQ LW LW@ LW@UP-JTUTPSUFEIFBE  ↟ WBMQD6TFS'JSTU%BUF"DUJWF%BUFQD6TFS'JSTU%BUFKPJO QD6TFS%BUFT  ↟ WBMSFTVMUQD6TFS'JSTU%BUF"DUJWF%BUFNBQ @@  SFEVDF#Z,FZ @ @ TPSU#Z,FZ USVF
  31. 4QBSL "84

  32. ੗੉ઁӒۢ੉ઁ૓૞۽ "84ীࢲ਍৔غח 4QBSL࠙ࢳदझమਸٜ݅যࠇद׮

  33. 4UPSBHFMBZFSח যڌѱೞ૑

  34. ۽Ӓחযରೖ4ী੓ਗ਼ই  4ܳӒրॳ੗

  35. 4QBSL 4 ↟ ۽Ӓܳ੷੢೧઱ח)%'4MBZFSܳٮ۽਍৔ೞ૑ঋইبؽ ↟ TOTGJMFTZTUFNਸా೧ࢲ)BEPPQীੜJOUFHSBUJPOغয੓਺ ↟ 4&$рؘ੉ఠ੹࣠੉ޖܐ

  36. 4QBSL 4 ↟ &$ীࢲ4੄੽Ӕࣘببજ਺ ↟ SYMBSHF (#/*$ ীࢲ_.#ZUFTTFDա১

  37. 4QBSL 4 ↟ 4UPSBHFMBZFS੄GBOPVUਸѣ੿ೡ೙ਃоহ਺ ↟ ૒੽4UPSBHFMBZFSܳ਍৔ೠ׮ݶ زदী׮ࣻ੄4QBSLDMVTUFSо۽Ӓ ী੽ӔೡٸחUISPVHIQVUী߽ݾ੉ࢤӡࣻ߆ীহ਺ ↟ 4ীࢲח੉۞ೠޙઁܳযו੿ب೧Ѿ೧ષ

    4QBSLDMVTUFS 4QBSLDMVTUFS -PH4
  38. 4QBSL &$ ↟ ೙ਃೡٸDMVTUFS௼ӝܳః਋Ѣա୶о੸ੋDMVTUFSܳڸ਎ࣻ੓׮ ↟ ৈ۞ѐ੄࠙ࢳਸ೙ਃೡ҃਋زदীجܾࣻ੓׮ ↟ زदীجܾࣻ੓ਵݶ оѺ਷ਬ૑ೞݶࢲݽٚ࠙ࢳ੉՘աחदрਸ খ׼ӡࣻ੓׮

    ↟ э਷੘সী݆਷&$JOTUBODFܳࢎਊೞ؊ۄبѾҴэ਷࠺ਊ
  39. 4QBSL 3JOTUBODF ↟ 3JOTUBODF ↟ ୭Ӕী୹दػ ݫݽܻPQUJNJ[FEJOTUBODF ↟ ݫݽܻ(# SYMBSHFӝળ

     ↟ (#44%Y ↟ 4IVGGMFѾҗ੐द੷੢ೡٸਬܻೣ ↟ 4QBSLী୭੸ചػJOTUBODFUZQF
  40. 4QBSL 1MBDFNFOU(SPVQ ↟ 1MBDFNFOU(SPVQ ↟ JOTUBODFр੄OFUXPSLMBUFODZоծই૗ ↟ XJEFEFQFOEFODZTIVGGMFPQFSBUJPOীࢲਬܻೣ

  41. 4QBSL "84ؘ੉ఠ࠙ࢳदझమ -PHCVDLFU 0OEFNBOE 4QBSL$MVTUFS 0OEFNBOE 4QBSL$MVTUFS %BJMZ"OBMZTJT 4QBSL$MVTUFS 1MBDFNFOU(SPVQ

    ;FQQFMJO%BTICPBSE 6TFST
  42. 4QBSL۽৤ӟറj ↟ ೞܖ஖࠙ࢳदрदрղ৻۽хࣗ ↟ ೠߣ$BDIF೧ࢲJUFSBUJWFೞѱجࣻ੓਺ ↟ ೞ٘ਝযҙܻ৬ৈ۞झఖ੄ࣗ೐౟ਝযҙܻ੄Ҧ۽਑ীࢲߩযթ ↟ )%'4)#BTF.BQ3FEVDF4QBSL ↟

    *OIPVTFDMVTUFSী࠺೧ࢲࣽࣻ࠺ਊ਷࠺तೞѢա׮ࣗ࠺व٠ ↟ ഻ঁ׮নೠदبٜ੉оמ೧૗
  43. 4QBSL۽৤ӟറj ↟ ֙஖ؘ੉ఠী؀೧ࢲ࠙ࢳਸ೧ࠁҊरয ↟ җѢীח׼ੌ࠙ࢳਸ઺ױदఃݶࢲয়ېѦܻח࠙ࢳਸجܻӝয۰਑ ↟ അ੤ח׼ੌ࠙ࢳҗदр੉য়ېѦܻח࠙ࢳਸزदীप೯оמ ↟ ࠺౟ਦ੄ߡӒ੉ग୶੸ী௾ب਑੉ؽ ↟

    যڃਬ੷оজ੉۽Ӓইਓغ঻׮חؘ੉ਬܳݽܰѷ׮ ↟ ࢲߡ੄ߡӒ ਬ੷੄׳੹੄೯زী੄೧ࢲࢤӟޙઁ ↟ ׳஖੄೧׼ਬ੷੄೯ز۽Ӓ৬&YDFQUJPO۽Ӓܳ࠺Ү࠙ࢳೞৈਗੋߊѼ ↟ ૓ੑ੢߷੉ծইઉ؊݆਷ѐߊ౱ਗٜ੉ؘ੉ఠ࠙ࢳীҙबਸо૑ӝद੘ೣ
  44. 4QBSLӒ৻੄5JQ

  45. 4QBSL5JQT ↟ 4IVGGMFೡٸࢤӝח੐द੷੢౵ੌٜ੉ѐࣻоষ୒݆੉ࢤӡࣻ੓਺ ↟ ٣झ௼חթחؘJOPEFоݽ੗ۆࢎకߊࢤ ↟ NLGTFYUJэ਷২࣌ਵ۽ழߡ ↟ NEBENਵ۽44%فѐޘযࢲ4QBSLUFNQಫ؊ীNPVOU

  46. 4QBSL5JQT ↟ 4ীࢲ੍য৬ࢲ୊ܻೡ҃਋౵ੌ੉࠺तೠ௼ӝ۽ଂѐઉ੓חѱਬܻ ↟ ௼ӝо    ੋ۽Ӓٜ۽੷੢غয੓חѪࠁ׮ ૞ܻѐо੓חѱਬܻೣ

    ↟ DPSFܳݽفࢎਊೞ੗ ↟ ਗېࢲߡীࢲೞܖীೠߣ۽Ӓܳস۽٘ೞ؍Ѫਸೠदрীೠߣਵ۽ ߸҃ೣ
  47. 4QBSL5JQT;FQQFMJO ↟ ؘ੉ఠ࠙ࢳਸਤೠਢӝ߈֢౟࠘ ؘ੉ఠWJTVBMJ[BUJPOUPPM ↟ 4QBSL߂4QBSL42-җোز೧ࢲಞܻೞѱؘ੉ఠ୊ܻ Ӓې೐Ӓܻח੘ স١ਸ૓೯ೡࣻ੓਺ ↟ ٮ۽؀एࠁٜ٘ܳ݅೙ਃоহয૗

    ↟ 5BKP١׮ؘܲ੉ఠ࠙ࢳోҗ੄ా೤ب૓೯઺
  48. 4QBSL5JQT;FQQFMJO

  49. Ѿۿ

  50. ↟ 4QBSL 4 3JOTUBODFઑ೤ਵ۽ ↟ ୊ܻਊ۝ਸטܻѢա઴ੌࣻ੓ח੗ਬبо੓Ҋ ↟ ૒੽਍৔ೞ૑ঋইبغח ↟ ߓ੉࢚ࡅܲ࠙ࢳदझమਸٜ݅঻णפ׮

    Ѿۿ
  51. None
  52. 2"