Upgrade to Pro — share decks privately, control downloads, hide ads and more …

NDC 2015 Cookie Run Log System

Sponsored · Ship Features Fearlessly Turn features on and off without deploys. Used by thousands of Ruby developers.

NDC 2015 Cookie Run Log System

NDC 2015에서 발표한 쿠키런 로그 시스템 슬라이드를 공유합니다.

Avatar for Jung-gun Lim

Jung-gun Lim

May 20, 2015
Tweet

Other Decks in Programming

Transcript

  1. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ ੐઺Ӕ ߊ಴੗ࣗѐ  ߊ಴ࣗѐ  ؘ࠳दझఠૉ

    _അ੤ /$40'5 _ )BEPPQӝ߈..031( ۽Ӓदझమҳ୷߂਍৔ ࠙࢑झ౟ܻ߁42-௪ܻ ূ૓ѐߊ߂਍৔ ѱ੐ࢲߡ ѐߊ߂਍৔
  2. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ ా҅߂૑಴ %"6 ."6 8"6 16 j

    ࢜۽੉ѐߊೠ૑಴ܳҗѢী؀೧ࢲࡳח׮ ૑಴ѐߊ੉औҊࡈۄ૓׮  җѢ۽Ӓܳ৵੷੢ೞભ  
  3. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ ௢ః۠ѱ੐ࢲߡ +BWB 4QSJOH.7$ 5PNDBU $PVDICBTF .Z42-

    3FEJT ই݃ઓਢࢲ࠺झ ޖ੼Ѩޖ઺ױࢲ࠺झܳ૑ೱ  ௢ః۠۽Ӓदझమ੄୍೟ 
  4. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ ࢲߡஹನք౟੄ઑѤ • ݽٚࢲߡ೐۽ࣁझחਗ஖ঋਸٸલח׮ – ৘࢚஖ޅೠद੼ীલӝبೞ૑݅ –

    ӟәࠁউಁ஖о੓ӝبೠ׮ • ঱ઁաݫੋఠքझ৬ಁ஖ܳоמೞѱ – ৵ջೞݶj  ௢ః۠۽Ӓदझమ੄୍೟ 
  5. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ r߄ࢁ׮s • ஶబஎѐߊೞוۄ߄ࢁ׮ • ߊࢤೠޙઁחӒٸӒٸ೧ѾೞҊ੕੗ •

    ઱ӝ੸ݫੋఠքझ୭ࣗച • ѐߊ੗ஹೊఠীࢲऔѱ੽Ӕоמೞѱ  ௢ః۠۽Ӓदझమ੄୍೟ 
  6. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ rоդೞ׮s • ѱ੐ࢲߡ࠺ਊࠁ׮۽Ӓࢲߡ࠺ਊ੉࠺ऱݶҌۆ • ޖܻೠୡӝై੗ܳೡࣻহ׮ –

    ѱ੐੄ࢿಁחইޖبݽܲ׮ • ࢤпࠁ׮഻ঁ੷۴ೠਬ૑࠺۽ҳ୷ೡࣻ੓׮  ௢ః۠۽Ӓदझమ੄୍೟ 
  7. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ ௢ః۠۽Ӓ • ݒੌ_(#੄+40/ఫझ౟ഋध – ֙஖୨۝5#੉࢚ •

    ࠺੿ഋझః݃ • ৵߄੉ցܻನݘউॳաਃ ª߄ࡅਃ – ۽Ӓܳ୶оೡٸ݃׮੹୓दझమਸসؘ੉౟ೞӝफযਃ  ௢ః۠۽Ӓदझమ੄ѐਃ 
  8. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ ௢ః۠۽Ӓदझమҳࢿ Tomcat Kafka ࣻ૘ ѐߊ ୊ܻ

    ࢲ࠺झ ੿ઁ ੸੤ Amazon S3 (Daily Log) Real Time Log Server LevelDB Log Broker CS Server Amazon EMR (Hadoop) CS ѐߊ੗1$  ௢ః۠۽Ӓदझమ੄ѐਃ 
  9. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ ۽Ӓࣻ૘ • "QBDIF,BGLBࢎਊ • ࠙࢑ݫࣁ૚दझమ –

    1VCMJTI4VCTDSJCF • ੌ߹ష೗ աൗ஖੷੢ Kafka  ۽Ӓࣻ૘"QBDIF,BGLB  Tomcat
  10. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ ,BGLB੄੢੼ • ݫࣁ૑ܳࠂઁೞৈ੷੢ – ֢٘੢গীъೣ –

    ࢲߡসؘ੉౟߂੤ࠗ౴੉࢚दоמ • ױࣽೠ੷੢ࣗҳઑ Ҋ੢੉ੜউդ׮ • ֫਷੍ӝॳӝ؀৉಩  ۽Ӓࣻ૘"QBDIF,BGLB 
  11. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ -PHTUBTIª,BGLB • ஹನք౟ܳݥ୶ݶয٥оޙઁоੌযդ׮ – ೐۽ࣁझоલਸٸ۽Ӓਬपߊࢤ •

    3FEJTܳղ۷׮ৢܻݶ௾ੌաਃ – 3FEJTীӟә੼Ѩझாે੉Ѧܻݶਕ௼ࢪٸޅ֥ইਃ • -PHTUBTIח1VTIߑध ,BGLBח1VMMߑध – ۽Ӓ੍ܳח೐۽ࣁझܳղ۷׮ৢܾࣻ੓׮  ۽Ӓࣻ૘"QBDIF,BGLB 
  12. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ ,BGLBࢎਊ੄઱੄੼ • ݫࣁ૑઺ࠂ਷ೖೡࣻহ׮ – 5XP(FOFSBMTs1SPCMFN –

    ӝ۾ࢿҕ਽׹ਸޅ߉ਵݶ઺ࠂߊࢤ • ઺ࠂઁѢח੿ഛೠా҅ܳਤ೧೙ࣻ  ۽Ӓࣻ૘"QBDIF,BGLB 
  13. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ ۽Ӓߔস • ੗ӝపੑߔস  • /"4

     • .Z42- ੌ߹ߔস  • )BEPPQ௿۞झఠ  • )#BTF$BTTBOESB௿۞झఠ  ۽Ӓߔস"NB[PO4 
  14. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ ۽Ӓߔস • ੗ӝపੑߔস ª߄ࡅਃ • /"4

    ªоդ೧ਃ • .Z42- ੌ߹ߔস ª߄ࡅਃ • )BEPPQ௿۞झఠ ªоդ೧ਃ • )#BTF$BTTBOESB௿۞झఠ ª߄ࢁҊоդ೧ਃ  ۽Ӓߔস"NB[PO4 
  15. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ Kafka ই݃ઓ4 • ޖೠਊ۝ • Ҋоਊࢿ

    • ؘ੉ఠܳ޻য֍ਸٸ੄ ਊ۝׼࠺ਊহ਺ • ੷۴ೠਬ૑࠺ – $16࠺ਊਸղ૑ঋ਺ Amazon S3 (Daily Log)  ۽Ӓߔস"NB[PO4 
  16. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ ই݃ઓ4੄ഝਊ • ౵ੌदझమ੉ۄࠗܰӝূ૑ਗغחӝמ੉ݻহ׮ – ౵ੌ੉ೠߣٜ݅য૑ݶࣻ੿੉উػ׮ •

    ౵ੌ੄੍ࠗ࠙ӝܳ૑ਗೣ – ࠶۟঑୷߂ੋؙयоמ • ࡅܲઑഥ৬ߓ஖࠙ࢳ੉оמೠನݘਵ۽ࠁҙ  ۽Ӓߔস"NB[PO4 
  17. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ ੌ߹ߔস • ,BGLB۽ࠗఠযઁ஖۽Ӓష೗ਸؒ೐߉ח׮ • যઁ۽Ӓܳ੿۳ೠ׮ –

    ః<"DUJPO .FNCFS*E 5JNFTUBNQ )BTI> – ਗೞחؘ੉ఠܳݽৈ੓ѱೠ׮ – ࡅܰѱ઺ࠂઁѢೡࣻ੓׮ – ঑୷੉ੜػ׮  ۽Ӓߔস"NB[PO4 
  18. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ ࠶۟঑୷ • 4OBQQZ'SBNFE'PSNBUࢎਊ • .#ਗࠄ࠶۟ױਤ۽঑୷ –

    ঑୷റ࠶۟௼ӝ.#޷݅۽઴যٚ׮ – ౠ੿ੌ੗੄ղ೒ۨ੉ӝ۾ਸৌۈೞחؘী.#੍݅ਵݶػ׮ • ঑୷࠶۟ਸ੉যࠢ੉ݶೞա੄Ѣ؀ೠ঑୷౵ੌ – 4ীࢲ׮਍۽٘߉ইഝਊೞӝಞೞ׮  ۽Ӓߔস"NB[PO4 
  19. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ ੌ߹ߔস౵ੌҳઑ T[ buy,  id:10127,  10:12  am

      buy,  id:10127,  10:34  am   … buy,  id:91729,  10:12  am   login,  id:10013,  01:13  am   … … play,  id:10013,  10:14  am   play,  id:10013,  10:23  am   … JOEFYKTPO CVZ JE BN CVZ JE BN j QMBZ JE BN ୡ ੌ߹ߔস  ۽Ӓߔস"NB[PO4 
  20. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ ࢠ೒݂ • ۽Ӓ੿۳दী.FNCFSJEܳ৉ࣽਵ۽ٍ૘਺ • ª ª

    • rs rsਵ۽՘աחNFNCFSJE੄۽Ӓоݽੋ׮ •  ࢠ೒݂ػ۽Ӓܳࡅܰѱ࠙ࢳ – ѐߊ੗1$ীࢲࡅؘܲ੉ఠয೒ܻா੉࣌ѐߊ߂పझ౟  ۽Ӓߔস"NB[PO4 
  21. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ Ѿҗ • ౠ੿ਬ੷ೞܖ஖೒ۨ੉۽Ӓઑഥୡ – যઁ۽Ӓٚ֙੹۽Ӓٚэ਷ࣘب۽ •

    ౠ੿ਬ੷ੌ֙஖೒ۨ੉۽Ӓઑഥୡ – "NB[PO&$DYMBSHF ௏যӝળ  ۽Ӓߔস"NB[PO4 
  22. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ Ѿҗ • ࢠ೒ؘ݂੉ఠੌ઱ੌ஖ – ۽ஸীࢲୡ݅ী࠙ࢳ •

    ੌ֙஖೒ۨ੉۽Ӓ੹୓ – ؀ই݃ઓ&.3௿۞झఠীࢲ࠙উী࠙ࢳ • ઑഥ ࠙ࢳݽفझாੌইਓоמ – ࢲߡܳטܾࣻ۾ࢿמ੉ࡈۄ૓׮  ۽Ӓߔস"NB[PO4 
  23. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ ࠺ਊ • ୡӝై੗࠺ਊহ਺ • ֙஖੹୓ਊ۝5#ղ৻ •

    ਘਬ૑࠺޷݅ – ࢎޖप੹ӝࣁࠁ׮ऱ׮ • җѢ۽Ӓܳ(MBDJFSী৤ӝחѪਵ۽࠺ਊ੺хоמ  ۽Ӓߔস"NB[PO4 
  24. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ पदр۽Ӓઑഥ • $4ܳਤ೧ࢲחয়ט۽Ӓبઑഥ೧ঠೠ׮ • যڃदझమਸॶѪੋо 

    – ୡ׼੍ӝॳӝ౟ے੥࣌ࣻо௿Ѫ – ߧਤ௪ܻܳ૑ਗೡѪ • ਤઑѤਸ݅઒ೞחदझమ੉Ӓ׮૑݆૑ঋ׮  पदр۽Ӓदझమ
  25. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ Kafka -FWFM%# • ,FZ7BMVFझషܻ૑ • ޺ҊॳחҳӖઁಿ

    – ҳӖ௼܁ীࢎਊ • ߧਤ௪ܻ૑ਗ • ୡ׼݅Ѥॳӝоמ Real Time Log Server LevelDB  पदр۽Ӓदझమ
  26. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ पदр۽Ӓࢲߡ • -FWFM%#ূ֎౟ਕ௼"1*оহ਺ • 3&45"1*ࢲߡܳ(P۽੘ࢿ –

    ֫਷زदࢿ – ݣ౭௏যܳഝਊೠಞೠझாੌ • ੌ߹-FWFM%# – ੌ߹஠೐஠ష೗ী؀਽ – ઺ࠂઁѢ  पदр۽Ӓदझమ Real Time Log Server Real Time Log Server LevelDB
  27. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ ۽Ӓ࠳۽ழ • (P۽੘ࢿ • җѢ۽Ӓח4 •

    ୭Ӕ۽Ӓחपदр۽Ӓࢲߡ • زੌೠੋఠಕ੉झ۽֙੹۽Ӓ৬ ୡ੹۽Ӓܳݽف੽Ӕоמ Real Time Log Server LevelDB  पदр۽Ӓदझమ Log Broker Amazon S3 (Daily Log)
  28. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ ࢎ۹೒ۨ੉܀ߔ • ౠ੿௢ఃীؘ੉ఠয়ܨߊѼ – ࠺੿࢚੸ਵ۽֫਷੼ࣻߊࢤ –

    ࠺੿࢚೒ۨ੉۽ੋधغয੿૑׼ೞחਬ੷ߊࢤ • ௢ః۠਷ӝ۾҃੬ѱ੐ ҕ੿ೣ੉઺ਃೞ׮  ۽Ӓഝਊ৘
  29. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ ೒ۨ੉܀ߔ • য়ט೒ۨ੉Ѩ࢝ਵ۽৔ೱ߉਷ਬ੷ഛੋ • ࣻߔݺ੄৔ೱ߉਷ਬ੷ٜী؀ೞৈ –

    ݽٚ೒ۨ੉ӝ۾ਸоઉ৬ࢲ – ޙઁ੓ח೒ۨ੉ܳ೙ఠ݂ೞҊ – ޙઁহח೒ۨ੉઺୭Ҋ੼ࣻ۽јन • ਬ੷׼ୡݶоמªࣁदр݅ী೧Ѿ  ۽Ӓഝਊ৘
  30. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ $4؀਽݃஠܄ਢࢲ࠺झ • ஘੺ೠੋఠಕ੉झ۽۽Ӓܳࠁৈળ׮ – ௼ܻझఎੑ୹ –

    ই੉మੑ୹ – ೒ۨ੉ӝ۾ • ੿ഛೠ੹റ࢚ടਸ౸߹ೞҊ$4؀਽оמ – ೲਤ$4ߑ૑  ۽Ӓഝਊ৘
  31. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ 4DBMEJOHਸഝਊೠ࠙ࢳ class ActiveUserCount(args : Args) extends

    Job(args) {
 CookierunLog("member/connect", args("sampling"),
 args("timeStart"), args("timeEnd"),
 Tuple1("memberSeq"))
 .groupBy(‘memberSeq){ _.take(1) }
 .groupAll{ _.size }
 .write(Csv(args("output")))
 }  ۽Ӓഝਊ৘
  32. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ ؘੌܻۨನ౟ಃா੊ • "NB[PO&.3ਤীࢲоز – ই݃ઓ੉ਃ୒ীٮۄ੗زਵ۽ڸਕ઱ח)BEPPQ௿۞झఠ •

    4QPU*OTUBODFࢎਊ – ௏যࢲߡ DYMBSHF ؀ೠदр࠽ܻח࠺ਊ޷݅ – ୹ӔҮా࠺ࣻળ  ۽Ӓഝਊ৘
  33. ௢ః۠ ۽Ӓ दझమ ߄ࢁҊоդೠѐߊ੗ܳਤ೧ ଵҊ੗ܐ • ഘࢿ૓ ௢ః۠֙ࢲߡѐߊ࠙ైӝ /%$ •

    ੉࢚Ҍ 1ZUIPOਵ۽௢ః۠਍৔ೞӝ 1ZDPO,PSFB • ӣ޹਋ оߺҊਬোೞѱؘ੉ఠ࠙ࢳೞӝ ,($ • ੿ହઁ "84ഝਊೞৈ%BJMZ3FQPSUٜ݅ӝ "84(MPCBM 4VNNJU4FPVM