Upgrade to Pro — share decks privately, control downloads, hide ads and more …

NDC 2015 Cookie Run Log System

NDC 2015 Cookie Run Log System

NDC 2015에서 발표한 쿠키런 로그 시스템 슬라이드를 공유합니다.

Jung-gun Lim

May 20, 2015
Tweet

Other Decks in Programming

Transcript

  1. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ ੐઺Ӕ ߊ಴੗ࣗѐ  ߊ಴ࣗѐ  ؘ࠳दझఠૉ

    _അ੤ /$40'5 _ )BEPPQӝ߈..031( ۽Ӓदझమҳ୷߂਍৔ ࠙࢑झ౟ܻ߁42-௪ܻ ূ૓ѐߊ߂਍৔ ѱ੐ࢲߡ ѐߊ߂਍৔
  2. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ ా҅߂૑಴ %"6 ."6 8"6 16 j

    ࢜۽੉ѐߊೠ૑಴ܳҗѢী؀೧ࢲࡳח׮ ૑಴ѐߊ੉औҊࡈۄ૓׮  җѢ۽Ӓܳ৵੷੢ೞભ  
  3. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ ௢ః۠ѱ੐ࢲߡ +BWB 4QSJOH.7$ 5PNDBU $PVDICBTF .Z42-

    3FEJT ই݃ઓਢࢲ࠺झ ޖ੼Ѩޖ઺ױࢲ࠺झܳ૑ೱ  ௢ః۠۽Ӓदझమ੄୍೟ 
  4. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ ࢲߡஹನք౟੄ઑѤ • ݽٚࢲߡ೐۽ࣁझחਗ஖ঋਸٸલח׮ – ৘࢚஖ޅೠद੼ীલӝبೞ૑݅ –

    ӟәࠁউಁ஖о੓ӝبೠ׮ • ঱ઁաݫੋఠքझ৬ಁ஖ܳоמೞѱ – ৵ջೞݶj  ௢ః۠۽Ӓदझమ੄୍೟ 
  5. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ r߄ࢁ׮s • ஶబஎѐߊೞוۄ߄ࢁ׮ • ߊࢤೠޙઁחӒٸӒٸ೧ѾೞҊ੕੗ •

    ઱ӝ੸ݫੋఠքझ୭ࣗച • ѐߊ੗ஹೊఠীࢲऔѱ੽Ӕоמೞѱ  ௢ః۠۽Ӓदझమ੄୍೟ 
  6. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ rоդೞ׮s • ѱ੐ࢲߡ࠺ਊࠁ׮۽Ӓࢲߡ࠺ਊ੉࠺ऱݶҌۆ • ޖܻೠୡӝై੗ܳೡࣻহ׮ –

    ѱ੐੄ࢿಁחইޖبݽܲ׮ • ࢤпࠁ׮഻ঁ੷۴ೠਬ૑࠺۽ҳ୷ೡࣻ੓׮  ௢ః۠۽Ӓदझమ੄୍೟ 
  7. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ ௢ః۠۽Ӓ • ݒੌ_(#੄+40/ఫझ౟ഋध – ֙஖୨۝5#੉࢚ •

    ࠺੿ഋझః݃ • ৵߄੉ցܻನݘউॳաਃ ª߄ࡅਃ – ۽Ӓܳ୶оೡٸ݃׮੹୓दझమਸসؘ੉౟ೞӝफযਃ  ௢ః۠۽Ӓदझమ੄ѐਃ 
  8. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ ௢ః۠۽Ӓदझమҳࢿ Tomcat Kafka ࣻ૘ ѐߊ ୊ܻ

    ࢲ࠺झ ੿ઁ ੸੤ Amazon S3 (Daily Log) Real Time Log Server LevelDB Log Broker CS Server Amazon EMR (Hadoop) CS ѐߊ੗1$  ௢ః۠۽Ӓदझమ੄ѐਃ 
  9. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ ۽Ӓࣻ૘ • "QBDIF,BGLBࢎਊ • ࠙࢑ݫࣁ૚दझమ –

    1VCMJTI4VCTDSJCF • ੌ߹ష೗ աൗ஖੷੢ Kafka  ۽Ӓࣻ૘"QBDIF,BGLB  Tomcat
  10. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ ,BGLB੄੢੼ • ݫࣁ૑ܳࠂઁೞৈ੷੢ – ֢٘੢গীъೣ –

    ࢲߡসؘ੉౟߂੤ࠗ౴੉࢚दоמ • ױࣽೠ੷੢ࣗҳઑ Ҋ੢੉ੜউդ׮ • ֫਷੍ӝॳӝ؀৉಩  ۽Ӓࣻ૘"QBDIF,BGLB 
  11. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ -PHTUBTIª,BGLB • ஹನք౟ܳݥ୶ݶয٥оޙઁоੌযդ׮ – ೐۽ࣁझоલਸٸ۽Ӓਬपߊࢤ •

    3FEJTܳղ۷׮ৢܻݶ௾ੌաਃ – 3FEJTীӟә੼Ѩझாે੉Ѧܻݶਕ௼ࢪٸޅ֥ইਃ • -PHTUBTIח1VTIߑध ,BGLBח1VMMߑध – ۽Ӓ੍ܳח೐۽ࣁझܳղ۷׮ৢܾࣻ੓׮  ۽Ӓࣻ૘"QBDIF,BGLB 
  12. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ ,BGLBࢎਊ੄઱੄੼ • ݫࣁ૑઺ࠂ਷ೖೡࣻহ׮ – 5XP(FOFSBMTs1SPCMFN –

    ӝ۾ࢿҕ਽׹ਸޅ߉ਵݶ઺ࠂߊࢤ • ઺ࠂઁѢח੿ഛೠా҅ܳਤ೧೙ࣻ  ۽Ӓࣻ૘"QBDIF,BGLB 
  13. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ ۽Ӓߔস • ੗ӝపੑߔস  • /"4

     • .Z42- ੌ߹ߔস  • )BEPPQ௿۞झఠ  • )#BTF$BTTBOESB௿۞झఠ  ۽Ӓߔস"NB[PO4 
  14. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ ۽Ӓߔস • ੗ӝపੑߔস ª߄ࡅਃ • /"4

    ªоդ೧ਃ • .Z42- ੌ߹ߔস ª߄ࡅਃ • )BEPPQ௿۞झఠ ªоդ೧ਃ • )#BTF$BTTBOESB௿۞झఠ ª߄ࢁҊоդ೧ਃ  ۽Ӓߔস"NB[PO4 
  15. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ Kafka ই݃ઓ4 • ޖೠਊ۝ • Ҋоਊࢿ

    • ؘ੉ఠܳ޻য֍ਸٸ੄ ਊ۝׼࠺ਊহ਺ • ੷۴ೠਬ૑࠺ – $16࠺ਊਸղ૑ঋ਺ Amazon S3 (Daily Log)  ۽Ӓߔস"NB[PO4 
  16. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ ই݃ઓ4੄ഝਊ • ౵ੌदझమ੉ۄࠗܰӝূ૑ਗغחӝמ੉ݻহ׮ – ౵ੌ੉ೠߣٜ݅য૑ݶࣻ੿੉উػ׮ •

    ౵ੌ੄੍ࠗ࠙ӝܳ૑ਗೣ – ࠶۟঑୷߂ੋؙयоמ • ࡅܲઑഥ৬ߓ஖࠙ࢳ੉оמೠನݘਵ۽ࠁҙ  ۽Ӓߔস"NB[PO4 
  17. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ ੌ߹ߔস • ,BGLB۽ࠗఠযઁ஖۽Ӓష೗ਸؒ೐߉ח׮ • যઁ۽Ӓܳ੿۳ೠ׮ –

    ః<"DUJPO .FNCFS*E 5JNFTUBNQ )BTI> – ਗೞחؘ੉ఠܳݽৈ੓ѱೠ׮ – ࡅܰѱ઺ࠂઁѢೡࣻ੓׮ – ঑୷੉ੜػ׮  ۽Ӓߔস"NB[PO4 
  18. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ ࠶۟঑୷ • 4OBQQZ'SBNFE'PSNBUࢎਊ • .#ਗࠄ࠶۟ױਤ۽঑୷ –

    ঑୷റ࠶۟௼ӝ.#޷݅۽઴যٚ׮ – ౠ੿ੌ੗੄ղ೒ۨ੉ӝ۾ਸৌۈೞחؘী.#੍݅ਵݶػ׮ • ঑୷࠶۟ਸ੉যࠢ੉ݶೞա੄Ѣ؀ೠ঑୷౵ੌ – 4ীࢲ׮਍۽٘߉ইഝਊೞӝಞೞ׮  ۽Ӓߔস"NB[PO4 
  19. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ ੌ߹ߔস౵ੌҳઑ T[ buy,  id:10127,  10:12  am

      buy,  id:10127,  10:34  am   … buy,  id:91729,  10:12  am   login,  id:10013,  01:13  am   … … play,  id:10013,  10:14  am   play,  id:10013,  10:23  am   … JOEFYKTPO CVZ JE BN CVZ JE BN j QMBZ JE BN ୡ ੌ߹ߔস  ۽Ӓߔস"NB[PO4 
  20. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ ࢠ೒݂ • ۽Ӓ੿۳दী.FNCFSJEܳ৉ࣽਵ۽ٍ૘਺ • ª ª

    • rs rsਵ۽՘աחNFNCFSJE੄۽Ӓоݽੋ׮ •  ࢠ೒݂ػ۽Ӓܳࡅܰѱ࠙ࢳ – ѐߊ੗1$ীࢲࡅؘܲ੉ఠয೒ܻா੉࣌ѐߊ߂పझ౟  ۽Ӓߔস"NB[PO4 
  21. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ Ѿҗ • ౠ੿ਬ੷ೞܖ஖೒ۨ੉۽Ӓઑഥୡ – যઁ۽Ӓٚ֙੹۽Ӓٚэ਷ࣘب۽ •

    ౠ੿ਬ੷ੌ֙஖೒ۨ੉۽Ӓઑഥୡ – "NB[PO&$DYMBSHF ௏যӝળ  ۽Ӓߔস"NB[PO4 
  22. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ Ѿҗ • ࢠ೒ؘ݂੉ఠੌ઱ੌ஖ – ۽ஸীࢲୡ݅ী࠙ࢳ •

    ੌ֙஖೒ۨ੉۽Ӓ੹୓ – ؀ই݃ઓ&.3௿۞झఠীࢲ࠙উী࠙ࢳ • ઑഥ ࠙ࢳݽفझாੌইਓоמ – ࢲߡܳטܾࣻ۾ࢿמ੉ࡈۄ૓׮  ۽Ӓߔস"NB[PO4 
  23. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ ࠺ਊ • ୡӝై੗࠺ਊহ਺ • ֙஖੹୓ਊ۝5#ղ৻ •

    ਘਬ૑࠺޷݅ – ࢎޖप੹ӝࣁࠁ׮ऱ׮ • җѢ۽Ӓܳ(MBDJFSী৤ӝחѪਵ۽࠺ਊ੺хоמ  ۽Ӓߔস"NB[PO4 
  24. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ पदр۽Ӓઑഥ • $4ܳਤ೧ࢲחয়ט۽Ӓبઑഥ೧ঠೠ׮ • যڃदझమਸॶѪੋо 

    – ୡ׼੍ӝॳӝ౟ے੥࣌ࣻо௿Ѫ – ߧਤ௪ܻܳ૑ਗೡѪ • ਤઑѤਸ݅઒ೞחदझమ੉Ӓ׮૑݆૑ঋ׮  पदр۽Ӓदझమ
  25. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ Kafka -FWFM%# • ,FZ7BMVFझషܻ૑ • ޺ҊॳחҳӖઁಿ

    – ҳӖ௼܁ীࢎਊ • ߧਤ௪ܻ૑ਗ • ୡ׼݅Ѥॳӝоמ Real Time Log Server LevelDB  पदр۽Ӓदझమ
  26. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ पदр۽Ӓࢲߡ • -FWFM%#ূ֎౟ਕ௼"1*оহ਺ • 3&45"1*ࢲߡܳ(P۽੘ࢿ –

    ֫਷زदࢿ – ݣ౭௏যܳഝਊೠಞೠझாੌ • ੌ߹-FWFM%# – ੌ߹஠೐஠ష೗ী؀਽ – ઺ࠂઁѢ  पदр۽Ӓदझమ Real Time Log Server Real Time Log Server LevelDB
  27. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ ۽Ӓ࠳۽ழ • (P۽੘ࢿ • җѢ۽Ӓח4 •

    ୭Ӕ۽Ӓחपदр۽Ӓࢲߡ • زੌೠੋఠಕ੉झ۽֙੹۽Ӓ৬ ୡ੹۽Ӓܳݽف੽Ӕоמ Real Time Log Server LevelDB  पदр۽Ӓदझమ Log Broker Amazon S3 (Daily Log)
  28. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ ࢎ۹೒ۨ੉܀ߔ • ౠ੿௢ఃীؘ੉ఠয়ܨߊѼ – ࠺੿࢚੸ਵ۽֫਷੼ࣻߊࢤ –

    ࠺੿࢚೒ۨ੉۽ੋधغয੿૑׼ೞחਬ੷ߊࢤ • ௢ః۠਷ӝ۾҃੬ѱ੐ ҕ੿ೣ੉઺ਃೞ׮  ۽Ӓഝਊ৘
  29. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ ೒ۨ੉܀ߔ • য়ט೒ۨ੉Ѩ࢝ਵ۽৔ೱ߉਷ਬ੷ഛੋ • ࣻߔݺ੄৔ೱ߉਷ਬ੷ٜী؀ೞৈ –

    ݽٚ೒ۨ੉ӝ۾ਸоઉ৬ࢲ – ޙઁ੓ח೒ۨ੉ܳ೙ఠ݂ೞҊ – ޙઁহח೒ۨ੉઺୭Ҋ੼ࣻ۽јन • ਬ੷׼ୡݶоמªࣁदр݅ী೧Ѿ  ۽Ӓഝਊ৘
  30. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ $4؀਽݃஠܄ਢࢲ࠺झ • ஘੺ೠੋఠಕ੉झ۽۽Ӓܳࠁৈળ׮ – ௼ܻझఎੑ୹ –

    ই੉మੑ୹ – ೒ۨ੉ӝ۾ • ੿ഛೠ੹റ࢚ടਸ౸߹ೞҊ$4؀਽оמ – ೲਤ$4ߑ૑  ۽Ӓഝਊ৘
  31. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ 4DBMEJOHਸഝਊೠ࠙ࢳ class ActiveUserCount(args : Args) extends

    Job(args) {
 CookierunLog("member/connect", args("sampling"),
 args("timeStart"), args("timeEnd"),
 Tuple1("memberSeq"))
 .groupBy(‘memberSeq){ _.take(1) }
 .groupAll{ _.size }
 .write(Csv(args("output")))
 }  ۽Ӓഝਊ৘
  32. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ ؘੌܻۨನ౟ಃா੊ • "NB[PO&.3ਤীࢲоز – ই݃ઓ੉ਃ୒ীٮۄ੗زਵ۽ڸਕ઱ח)BEPPQ௿۞झఠ •

    4QPU*OTUBODFࢎਊ – ௏যࢲߡ DYMBSHF ؀ೠदр࠽ܻח࠺ਊ޷݅ – ୹ӔҮా࠺ࣻળ  ۽Ӓഝਊ৘
  33. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ ଵҊ੗ܐ • ഘࢿ૓ ௢ః۠֙ࢲߡѐߊ࠙ైӝ /%$ •

    ੉࢚Ҍ 1ZUIPOਵ۽௢ః۠਍৔ೞӝ 1ZDPO,PSFB • ӣ޹਋ оߺҊਬোೞѱؘ੉ఠ࠙ࢳೞӝ ,($ • ੿ହઁ "84ഝਊೞৈ%BJMZ3FQPSUٜ݅ӝ "84(MPCBM 4VNNJU4FPVM