Upgrade to Pro — share decks privately, control downloads, hide ads and more …

2009-douban-tech-story

 2009-douban-tech-story

2009年 洪教授回顾豆瓣架构经验

6002ee051e03f0b762642ee7fafd111f?s=128

Zoom.Quiet

April 26, 2009
Tweet

Transcript

  1. ׸ϵຩ࠯ඌࡏܒ֥ ؿᅚ৥ӱ 2009.4 ޠ఼ୡ hongqn@douban.com

  2. ׸ϵຩࡥࢺ • 2005୍3ᄅഈཌ • ၛٳཚބؿགྷູ ނྏ֥ഠ౵ • ؀඀a׈႕aၻ ুaཬቆa๝ Ӭaࣴׄ

    • ໡֥׸ϵaႶਣ
  3. ၂ུඔऌ • 2.9MᇿҨႨ޼đჿ1/4ࠃᄁႨ޼ • తຣࠩ٤ᇿҨႨ޼ • 23M׮෿౨౰/฿đڂᆴ500~600/sec • 23෻௴๙PCڛༀఖ(1U*15/2U*8) •

    12෻ิ܂ཌഈڛༀ • 38G memcached
  4. ֆڛༀఖ • ֆ෻1Uڛༀఖ (frodo) • ֆނAMD Athlon 64 1.8GHz •

    1Gଽթđ160G SATA*2 • Gentoo Linux • MySQL 5 • Quixote (a Python web framework) • Lighttpd + SCGI (shire) • Memcached (!)
  5. Gentoo Linux • ಸၞົ޹ • emerge mysql • ebuild ьႿܵ৘

    patch • ᆺνልླေ֥ת༆ • νಆྟ • GLSA(Gentoo Linux Security Advisories)
  6. MySQL • The world’s most popular open source database •

    ཿഒ؀؟/ཿ؟؀ഒ ==> MyISAM • ؀ཿѩؿۚ ==> InnoDB • Replicate for backup
  7. Python • षؿ࿡෎ • Battery Included • ֻ೘ٚ९Ӯඃ • ഠ౵ӮӉᇏ

    • CPUG: http://python.cn/
  8. Quixote • ࡥֆđ౞ਈđၞႿൌགྷRESTڄ֥۬URL • ֒ൈߎીႵDjango, TurboGears, Pylonsᆃུ࿊ᄴđᆺႵ၂ ۱Шᇗ֥ZOPE • http://www.douban.com/subject/1000001

    # luz/subject/__init__.py def _q_lookup(request, name): subject = get_subject(name) return lambda req: subject_ui(req, subject) # luz/subject/subject_ui.ptl def subject_ui [html] (request, subject): site_header(request) “<h1>%s</h1>” % subject.title site_footer(request)
  9. Lighttpd • ޓݺ֥׮෿ބ࣡෿ྟି • ჰളSCGIᆦӻ • SCGI: ၂۱ࡥ߄ϱЧ֥FastCGIđႮ Quixoteषؿᆀषؿ •

    ෮Ⴕ֥౨౰׻๙ݖ80؊१֥lighttpdࣉӱ ٳؿđ׮෿ଽಸሼSCGI֞localhostഈ֥ Quixoteࣉӱb
  10. Memcache • Ֆഈཌఏࣼᄝ൐ႨđႵིࡨ౞MySQLڵք • ؓlibmemcacheቓਔpythonٿልč൐ႨPyrexĎđྟି൞ Ղpythonϱ֥3x+ def get_subject(subject_id): subject =

    mc.get(‘s:’+subject_id) if subject is None: store.farm.execute(“select xxx, xxx from subject where id=%s”, subject_id) subject = Subject(*store.farm.fetchone()) mc.set(‘s:’+subject_id, subject) return subject
  11. Internet MySQL Lighttpd App SCGI Memcache Static Files FS

  12. ໙ีԛགྷ • 1.2M׮෿౨౰/฿ • Ո஍IOӮູ௞࣠ • ླေ࿙ᅳྍࠏٜ

  13. ࢳथٚσ • ܓઙਆ෻1Uڛༀఖ • pippin ބ meriadoc (ުڿ଀merry) • චނ,

    4Gଽթđ250G SATA*3 • ၂෻ቔູႋႨڛༀఖđ၂෻ቔູඔऌ९ڛༀఖ • థ၍֞චཌචIPࠏٜđ൐ႨDNSࢳ༅҂๝ຩ؍ IP -_-b • ष൓؟ದླྀቔषؿđfrodoቓູषؿႨࠏ (subversion, trac, etc...)
  14. Internet DNS Lighttpd (!") Lighttpd (#$) App MySQL HTTP Proxy

    Memcache Static Files FS SCGI
  15. ࠫׄؿགྷ • ඔऌ९֥ଽթٳ஥ؓྟି႕ཙᇗն • innodb_buffer_pool_size • Ո஍ෛࠏ࿙֡෎؇б๽๳ਈ۷ᇗေ • ຩഈᅳট֥IP؍ٳ҃ޓ҂ौ௶

  16. ໙ีԛགྷ • 1.5M׮෿౨౰/฿đഉໃ֞ྟି௞࣠ • ࠏٜ҂ौ௶đ௔َܣᅰ • IP؍ٳ҃ඔऌ҂ौ௶đႨ޼ّ႘٠໙ߏત

  17. ࢳथٚσ • ߐ֞ौ௶֥ࠏٜđ؟ཌֆIP(BGP) • ܓઙਔ၂෻ྍڛༀఖ (arwen) • 74G 1wሇ SATA

    * 3 • ቓູඔऌ९ڛༀఖ • ष൓൐Ⴈህ૊֥ڛༀఖቔު෻࠹ෘ
  18. Internet Lighttpd App Memcache Data Mining MySQL Master MySQL Slave

    Replicate Static Files SCGI read write
  19. ໙ีԛགྷ • 2M׮෿౨౰/฿ • ࣡෿໓ࡱڛༀՈ஍IOӮູ௞࣠ • ഈϤຣ֥ཬ๭ோčႨ޼๨ཞaٿ૫๭ ோ, etc...Ď •

    ඔऌ९ڛༀఖࢤ࣍௞࣠
  20. ࢳथٚσ • ܓઙ೘෻ڛༀఖđචނđ4Gđ250G SATA*3 • ࡼ๭ோՖ၂۱նଢ੣్ٳӮ10000۱໓ࡱ၂۱ଢ੣ • mod_rewriteЌӻURL҂э • ׿৫֥๭ோlighttpdࣉӱđఓႨmod_memcacheଆॶđ

    ߏթཬ๭ோ • ࡨཬՈ஍IO်ؓ૫٠໙֥႕ཙ • ࡼႋႨڛༀՖwebڛༀఖ׿৫ԛಀ • Ϝ۷؟֥ଽթٳ஥۳࣡෿໓ࡱڛༀ • ᄹࡆ၂۱ᆺ؀ඔऌ९
  21. Web Service Internet Lighttpd App Memcache MySQL Master MySQL Slave

    Replicate Static Files Lighttpd WebDAV WebDAV SCGI !"#$% Memcache Spiders Data Mining MySQL Slave Replicate Lighttpd (w/ mod_memcache) HTTP Proxy store.farm store.farmr write read
  22. ᆺ؀ඔऌ९ • storeᄹࡆfarmrඋྟđູ၂۱ॖႨ֥ᆺ؀ඔऌ९Ⴓѓ • ๨ฮ֥replicate delay໙ี • ڣ९گᇅླေൈࡗ • ۷ྍᇶ९ުđ༯၂۱౨౰ສສࣼ൞ေ؀ඔऌč۷ྍඔऌު඗

    ྍ်૫Ď • Ֆڣ९؀߶֝ᇁcache৚թ٢֥൞ࣸඔऌ • ਲၳ൙ࡱĆ • ࢳथٚمğ۷ྍඔऌ९ުđᄝყ௹ॖି߶ઔഈႨ֥֞౦ঃ༯đᇶ ׮඗ྍߏթ • ......҂ປૅđbut it works
  23. х૧replicate delayႄ ఏ֥ਲၳ൙ࡱ def get_subject(sid): sbj = mc.get(‘s:’+sid) if sbj

    is None: sbj = flush_subject(sid, store.farmr) return sbj def flush_subject(sid, cursor=None): cursor = cursor or store.farm cursor.execute(“select ... from subject”) subject = Subject(*cursor.fetchone()) mc.set(‘s:’+sid, subject) return subject def update_subject(subject, props): store.farm.execute(“update subject ...”) store.farm.connection.commit() flush_subject(subject.id, store.farm)
  24. ໙ีԛགྷ • 2.5M׮෿౨౰/฿ • ඔऌ९Ո஍ॢࡗ҂ܔਔ • ໡ഈ/ࣴׄඔऌਈஔն • SATA஍ܣᅰੱۚ •

    ඔऌ९࿢৯ᄹն
  25. ࢳथٚσ • Scale Upđܓઙඹ෻1Uڛༀఖ • 16Gଽթđ147G SCSI *2 + 500G

    SATA • SCSI ቓ RAID-0 • ႨMySQL SlaveটЌᆣ಺Ⴥ • ᄹࡆmemcachedࢫׄඔଢ • ෮Ⴕ֥MyISAMі׻ڿູInnoDBі • ิۚଽթ০Ⴈིੱ • ࡼಆ໓ෆ෬၍ᇀSphinx
  26. Internet Lighttpd App Memcache MySQL Master MySQL Slave Replicate Static

    Files Lighttpd WebDAV WebDAV SCGI Memcache Spiders Lighttpd (w/ mod_memcache) HTTP Proxy store.farm store.farmr Sphinx Web Service Memcache Memcache Web Service
  27. ໙ีԛགྷ • 5.2M׮෿౨౰/฿ • ๭ோੀਈٮႨӮູቋնӮЧ • Webڛༀఖ֥Ո஍IOߎ൞߶႕ཙ׮෿် ૫ྟି • ႋႨڛༀఖࣉӱඔ҂ܔਔ

    • ࠏ݊ॢࡗ҂ܔਔ
  28. ࢳथٚσ • ฿ࣃ֥ࠏٜьၒ၂ུ :) • ӵք๭ோੀਈ • ު෻ඔऌ຋ड࠹ෘ • ಸᄙСٺ

    • ܓઙ3෻1Uڛༀఖğ4ނđ32Gଽթđ1T SATA * 3 • Ⴊ߄భ؊đఓႨ otho.douban.com ބ lotho.douban.com თ଀ • lighttpd 1.5 with aio support • ҆ඇLVS • Scale Up: ႋႨڛༀఖଽթശࠩ 4G -> 8G
  29. Internet Lighttpd Static Files Lighttpd WebDAV Lighttpd HTTP Proxy Lighttpd

    1.5 (w/ mod_cache) Lighttpd 1.5 (w/ mod_cache) LVS LB (Master) LVS LB (backup) Keepalived www.douban.com otho.douban.com
  30. MySQL Master MySQL Slave Replicate !"#$% Data Mining !"#$% Data

    Mining MySQL Slave replicate read read write write
  31. ໙ีԛགྷ • 6.4M׮෿౨౰/฿ (5M PV) • ႋႨڛༀఖӮູ௞࣠ • ଽթğᅝႨሹ൞ᄹӉđරެႵଽթ྄ ੝

    • CPUğmemcacheؓའ྽ਙ߄/ّ྽ਙ߄
  32. ࢳथٚσ • ֻؽ෻ႋႨڛༀఖഈཌ • lighttpd֥mod_scgiᆺିround-robin • lighttpd 1.5҂໗ק • mod_proxy

    • proxy.balance = fair (load based, passive balancing) • ֒ࣉӱᅝႨଽթӑݖᚐᆴđ֒భ౨౰ປӮުሱ೪ • ൐Ⴈspreadऊކರᆽ
  33. Lighttpd Static Files Lighttpd WebDAV Lighttpd HTTP Proxy App Memcache

    Lighttpd App Memcache Lighttpd Internet SCGI SCGI HTTP Proxy HTTP Proxy Log Aggregator spread spread
  34. ໙ีԛགྷ • 11M׮෿౨౰/฿č3෻ႋႨڛༀఖĎ • ॴࠏٜཿೆӮູު෻࠹ෘ֥௞࣠ • Sphinx֥קᇅྟ҂ܔ • ཌྷҨӁ௖षؿᇏđླေࢳथ๭ோթԥ໙ ี

    • ਲၳགྷའğ ຩᅟэતđࠒᄟਔնਈ৵ࢤ ֤҂֞ԩ৘đloadಏ҂ۚ
  35. ࢳथٚσ • ඔऌ९ٳ९ • ࣴׄཌྷܱі׿৫ԛট • ඔऌ຋डཌྷܱі׿৫ԛটđᇶ९٢ᄝ ฿ࣃđКࣘᆺ؀ • Sphinx

    -> Xapian • ൐ႨMogileFS
  36. !" Master #$ Master %&'() Data Mining %&'() Data Mining

    #$ Slave !" Slave *+,- Slave !" Slave #$ Slave *+,- Master replicate write write replicate replicate read read replicate replicate
  37. ؟ඔऌ९৵ࢤ • і଀ಆअື၂đົ޹၂۱і଀ᇀඔऌ९֥႘ഝ • store.farm[r] -> store.get_cursor(table=‘xxx’, ro=True/False) def flush_subject(sid,

    ro=False): cursor = store.get_cursor(table=‘subject’, ro=ro) cursor.execute(“select ... from subject”) subject = Subject(*cursor.fetchone()) mc.set(‘s:’+sid, subject) return subject • ᄝඔऌ९ࡗ୳іэ֤ಸၞđᄝᇶڣ९ࡗ௜ޙڵ ᄛ္э֤ಸၞ
  38. MogileFS Master Uploader MogileFS Node MogileFS Node nginx (w/ proxy_store)

    MogileFS Tracker FileStorage Gateway Internet upload.douban.com otho.douban.com App signed POST form HTTP redirect
  39. ࢳथٚσč࿃Ď • libmemcache -> libmemcachedđ൐Ⴈconsistent hashࢆ֮memcacheטᆜսࡎ • ྩᆞlibmemcached֥consistent hashཌྷܱbug •

    ႋႨڛༀఖശࠩᇀඹނCPU • ྩᆞlibmemcached֥failoverཌྷܱbug • Ⴈnginxูսlighttpdቓload balance • ቋުؿགྷቌঊࠋ൮൞spreadđ⯝ • ڿӮႨnginx࠺੣ರᆽ
  40. Consistent Hash ๭ோϱಃඋႿcharlee http:// tech.idv2.com/2008/07/24/ memcached-004/

  41. Lighttpd Static Files Lighttpd WebDAV Lighttpd HTTP Proxy App Lighttpd

    Internet SCGI HTTP Proxy Nginx HTTP Proxy App Lighttpd SCGI HTTP Proxy App Lighttpd SCGI HTTP Proxy
  42. ໙ีԛགྷ • 13M׮෿౨౰/฿ • ࠹߃ࡼ෮Ⴕ࣡෿๭ோ׻֝ೆMogileFS • ໓ࡱཬđඔਈնđTracker DBॖିӮູ ௞࣠ •

    ཌྷҨӁ௖ޓ൳ߋ႒đթԥॢࡗष൓ࣅᅦ
  43. ࢳथٚσ • ܓઙ8෻ྍڛༀఖ • 32GଽթđඹނCPU • (300G SCSI×2 + 1T

    SATA) × 3 • (1T SATA × 3) × 5 • 6෻Кࣘđ2෻฿ࣃ • षؿDoubanFS
  44. DoubanFS • ીႵᇏྏඔऌ९đᆰࢤοᅶ໓ࡱ଀hashҰᅳ෮ᄝࢫ ׄđॖഥ෪ྟ۷ݺ • οᅶhashᆴթӮଢ੣ඎđૄ۱ࢫׄڵᄳ၂ቆhashᆴ ౵თ • קൈ๝҄đࠎႿ໓ࡱྩڿൈࡗ֥Merkle Tree

    • ০Ⴈconsistent hashࡨഒᄹ೷ࢫׄջট֥ඔऌ၍׮ਈ • WebDAVቔູ؀ཿࢤ१ • ؀ྟିູMogileFS֥3Пđཿྟିູ50П
  45. Merkle Tree

  46. Uploader DoubanFS Node DoubanFS Node nginx (w/ proxy_store) FileStorage Gateway

    Internet upload.douban.com otho.douban.com App signed POST form HTTP redirect
  47. ໙ีԛགྷ • 16M׮෿౨౰/฿ • ඔऌ९ն໓Чሳ؍࿸ᇗ႕ཙਔඔऌ९ྟ ି • DoubanFSᇏཬ๭ோ֝ᇁIOᄹۚ • ඔऌ९ॖႨྟေ౰ิۚ

  48. ࢳथٚσ • षؿDoubanDB • ۷ݺ֥ഥ෪ྟ • ն໓Чሳ؍၍ԛުđMySQL֥ྟି֤֞ ᄹ఼ • MySQLචMasterٚσ

    • failover۷ࡥֆ • ࢳथreplicate delay໙ี
  49. DoubanDB • ٳ҃ൔKey-Valueඔऌ९ • ՖAmazon Dynamoࠆ֤ਲۋđቓਔ၂ུࡥ߄ • ೘۱ࢤ१ğset(key, value), get(key),

    delete(key) • memcacheླྀၰቔູ؀ཿࢤ१ • ᆇᆞ֥Merkle TreeđࠎႿଽթđުڿູࠎႿՈ஍໓ࡱ • Consistent Hash • ႨTokyoCabinetቓָູҪթԥ • ඔऌ९ᇏ֥ն໓Чሳ؍ࣉೆDoubanDB • DoubanFS 2.0 ࠎႿ DoubanDB ൌགྷ
  50. ๭ோϱಃඋႿcharlee http:// tech.idv2.com/2008/07/24/ memcached-004/

  51. !" Master1 #$ Master1 %&'() Data Mining %&'() Data Mining

    #$ Master2 !" Master2 *+,- Slave !" Slave #$ Slave *+,- Master replicate write write replicate replicate read read replicate replicate
  52. ቋྍ֥၂ུڿ׮ • ࡼDoubanFSބඔऌ຋डӱ྽၍߭Кࣘࠏٜ • ࡥ߄໓ࡱഈԮੀӱބս઒ • ۷ݺ֥০Ⴈ႗ࡱሧჷ • ൐Ⴈngnixቔູቋభ؊ •

    www.douban.com္҆ඇLVS • ൐ႨRabbitMQսูspread
  53. ၂ུࣜဒ • Ϝఫ޽ᄝଽթഈ൞ᆴ֤֥ • ࡹ৫ਅݺ֥profile۽ऎđѩ০Ⴈᆭ • memcacheѩ҂৷ࡎđር༥॥ᇅcache֥ؓའնཬ ބ٠໙ٚൔ • х૧ඔऌ९֥joinҠቔ

    • ᄝӁ௖ഈቓԛཋᇅၛх૧ݖત֥Ұ࿘ • ር༥ҷٳభު෻ᄎෘ • Ⴈၘᅧ໤֥࠯ඌࢳथ໙ี
  54. ྆྆