Upgrade to Pro — share decks privately, control downloads, hide ads and more …

2009-douban-tech-story

 2009-douban-tech-story

2009年 洪教授回顾豆瓣架构经验

Zoom.Quiet

April 26, 2009
Tweet

More Decks by Zoom.Quiet

Other Decks in Technology

Transcript

  1. ֆڛༀఖ • ֆ෻1Uڛༀఖ (frodo) • ֆނAMD Athlon 64 1.8GHz •

    1Gଽթđ160G SATA*2 • Gentoo Linux • MySQL 5 • Quixote (a Python web framework) • Lighttpd + SCGI (shire) • Memcached (!)
  2. Gentoo Linux • ಸၞົ޹ • emerge mysql • ebuild ьႿܵ৘

    patch • ᆺνልླေ֥ת༆ • νಆྟ • GLSA(Gentoo Linux Security Advisories)
  3. MySQL • The world’s most popular open source database •

    ཿഒ؀؟/ཿ؟؀ഒ ==> MyISAM • ؀ཿѩؿۚ ==> InnoDB • Replicate for backup
  4. Quixote • ࡥֆđ౞ਈđၞႿൌགྷRESTڄ֥۬URL • ֒ൈߎીႵDjango, TurboGears, Pylonsᆃུ࿊ᄴđᆺႵ၂ ۱Шᇗ֥ZOPE • http://www.douban.com/subject/1000001

    # luz/subject/__init__.py def _q_lookup(request, name): subject = get_subject(name) return lambda req: subject_ui(req, subject) # luz/subject/subject_ui.ptl def subject_ui [html] (request, subject): site_header(request) “<h1>%s</h1>” % subject.title site_footer(request)
  5. Lighttpd • ޓݺ֥׮෿ބ࣡෿ྟି • ჰളSCGIᆦӻ • SCGI: ၂۱ࡥ߄ϱЧ֥FastCGIđႮ Quixoteषؿᆀषؿ •

    ෮Ⴕ֥౨౰׻๙ݖ80؊१֥lighttpdࣉӱ ٳؿđ׮෿ଽಸሼSCGI֞localhostഈ֥ Quixoteࣉӱb
  6. Memcache • Ֆഈཌఏࣼᄝ൐ႨđႵིࡨ౞MySQLڵք • ؓlibmemcacheቓਔpythonٿልč൐ႨPyrexĎđྟି൞ Ղpythonϱ֥3x+ def get_subject(subject_id): subject =

    mc.get(‘s:’+subject_id) if subject is None: store.farm.execute(“select xxx, xxx from subject where id=%s”, subject_id) subject = Subject(*store.farm.fetchone()) mc.set(‘s:’+subject_id, subject) return subject
  7. ࢳथٚσ • ܓઙਆ෻1Uڛༀఖ • pippin ބ meriadoc (ުڿ଀merry) • චނ,

    4Gଽթđ250G SATA*3 • ၂෻ቔູႋႨڛༀఖđ၂෻ቔູඔऌ९ڛༀఖ • థ၍֞චཌචIPࠏٜđ൐ႨDNSࢳ༅҂๝ຩ؍ IP -_-b • ष൓؟ದླྀቔषؿđfrodoቓູषؿႨࠏ (subversion, trac, etc...)
  8. ࢳथٚσ • ߐ֞ौ௶֥ࠏٜđ؟ཌֆIP(BGP) • ܓઙਔ၂෻ྍڛༀఖ (arwen) • 74G 1wሇ SATA

    * 3 • ቓູඔऌ९ڛༀఖ • ष൓൐Ⴈህ૊֥ڛༀఖቔު෻࠹ෘ
  9. ࢳथٚσ • ܓઙ೘෻ڛༀఖđචނđ4Gđ250G SATA*3 • ࡼ๭ோՖ၂۱նଢ੣్ٳӮ10000۱໓ࡱ၂۱ଢ੣ • mod_rewriteЌӻURL҂э • ׿৫֥๭ோlighttpdࣉӱđఓႨmod_memcacheଆॶđ

    ߏթཬ๭ோ • ࡨཬՈ஍IO်ؓ૫٠໙֥႕ཙ • ࡼႋႨڛༀՖwebڛༀఖ׿৫ԛಀ • Ϝ۷؟֥ଽթٳ஥۳࣡෿໓ࡱڛༀ • ᄹࡆ၂۱ᆺ؀ඔऌ९
  10. Web Service Internet Lighttpd App Memcache MySQL Master MySQL Slave

    Replicate Static Files Lighttpd WebDAV WebDAV SCGI !"#$% Memcache Spiders Data Mining MySQL Slave Replicate Lighttpd (w/ mod_memcache) HTTP Proxy store.farm store.farmr write read
  11. ᆺ؀ඔऌ९ • storeᄹࡆfarmrඋྟđູ၂۱ॖႨ֥ᆺ؀ඔऌ९Ⴓѓ • ๨ฮ֥replicate delay໙ี • ڣ९گᇅླေൈࡗ • ۷ྍᇶ९ުđ༯၂۱౨౰ສສࣼ൞ေ؀ඔऌč۷ྍඔऌު඗

    ྍ်૫Ď • Ֆڣ९؀߶֝ᇁcache৚թ٢֥൞ࣸඔऌ • ਲၳ൙ࡱĆ • ࢳथٚمğ۷ྍඔऌ९ުđᄝყ௹ॖି߶ઔഈႨ֥֞౦ঃ༯đᇶ ׮඗ྍߏթ • ......҂ປૅđbut it works
  12. х૧replicate delayႄ ఏ֥ਲၳ൙ࡱ def get_subject(sid): sbj = mc.get(‘s:’+sid) if sbj

    is None: sbj = flush_subject(sid, store.farmr) return sbj def flush_subject(sid, cursor=None): cursor = cursor or store.farm cursor.execute(“select ... from subject”) subject = Subject(*cursor.fetchone()) mc.set(‘s:’+sid, subject) return subject def update_subject(subject, props): store.farm.execute(“update subject ...”) store.farm.connection.commit() flush_subject(subject.id, store.farm)
  13. ࢳथٚσ • Scale Upđܓઙඹ෻1Uڛༀఖ • 16Gଽթđ147G SCSI *2 + 500G

    SATA • SCSI ቓ RAID-0 • ႨMySQL SlaveটЌᆣ಺Ⴥ • ᄹࡆmemcachedࢫׄඔଢ • ෮Ⴕ֥MyISAMі׻ڿູInnoDBі • ิۚଽթ০Ⴈིੱ • ࡼಆ໓ෆ෬၍ᇀSphinx
  14. Internet Lighttpd App Memcache MySQL Master MySQL Slave Replicate Static

    Files Lighttpd WebDAV WebDAV SCGI Memcache Spiders Lighttpd (w/ mod_memcache) HTTP Proxy store.farm store.farmr Sphinx Web Service Memcache Memcache Web Service
  15. ࢳथٚσ • ฿ࣃ֥ࠏٜьၒ၂ུ :) • ӵք๭ோੀਈ • ު෻ඔऌ຋ड࠹ෘ • ಸᄙСٺ

    • ܓઙ3෻1Uڛༀఖğ4ނđ32Gଽթđ1T SATA * 3 • Ⴊ߄భ؊đఓႨ otho.douban.com ބ lotho.douban.com თ଀ • lighttpd 1.5 with aio support • ҆ඇLVS • Scale Up: ႋႨڛༀఖଽթശࠩ 4G -> 8G
  16. Internet Lighttpd Static Files Lighttpd WebDAV Lighttpd HTTP Proxy Lighttpd

    1.5 (w/ mod_cache) Lighttpd 1.5 (w/ mod_cache) LVS LB (Master) LVS LB (backup) Keepalived www.douban.com otho.douban.com
  17. MySQL Master MySQL Slave Replicate !"#$% Data Mining !"#$% Data

    Mining MySQL Slave replicate read read write write
  18. ࢳथٚσ • ֻؽ෻ႋႨڛༀఖഈཌ • lighttpd֥mod_scgiᆺିround-robin • lighttpd 1.5҂໗ק • mod_proxy

    • proxy.balance = fair (load based, passive balancing) • ֒ࣉӱᅝႨଽթӑݖᚐᆴđ֒భ౨౰ປӮުሱ೪ • ൐Ⴈspreadऊކರᆽ
  19. Lighttpd Static Files Lighttpd WebDAV Lighttpd HTTP Proxy App Memcache

    Lighttpd App Memcache Lighttpd Internet SCGI SCGI HTTP Proxy HTTP Proxy Log Aggregator spread spread
  20. !" Master #$ Master %&'() Data Mining %&'() Data Mining

    #$ Slave !" Slave *+,- Slave !" Slave #$ Slave *+,- Master replicate write write replicate replicate read read replicate replicate
  21. ؟ඔऌ९৵ࢤ • і଀ಆअື၂đົ޹၂۱і଀ᇀඔऌ९֥႘ഝ • store.farm[r] -> store.get_cursor(table=‘xxx’, ro=True/False) def flush_subject(sid,

    ro=False): cursor = store.get_cursor(table=‘subject’, ro=ro) cursor.execute(“select ... from subject”) subject = Subject(*cursor.fetchone()) mc.set(‘s:’+sid, subject) return subject • ᄝඔऌ९ࡗ୳іэ֤ಸၞđᄝᇶڣ९ࡗ௜ޙڵ ᄛ္э֤ಸၞ
  22. MogileFS Master Uploader MogileFS Node MogileFS Node nginx (w/ proxy_store)

    MogileFS Tracker FileStorage Gateway Internet upload.douban.com otho.douban.com App signed POST form HTTP redirect
  23. ࢳथٚσč࿃Ď • libmemcache -> libmemcachedđ൐Ⴈconsistent hashࢆ֮memcacheטᆜսࡎ • ྩᆞlibmemcached֥consistent hashཌྷܱbug •

    ႋႨڛༀఖശࠩᇀඹނCPU • ྩᆞlibmemcached֥failoverཌྷܱbug • Ⴈnginxูսlighttpdቓload balance • ቋުؿགྷቌঊࠋ൮൞spreadđ⯝ • ڿӮႨnginx࠺੣ರᆽ
  24. Lighttpd Static Files Lighttpd WebDAV Lighttpd HTTP Proxy App Lighttpd

    Internet SCGI HTTP Proxy Nginx HTTP Proxy App Lighttpd SCGI HTTP Proxy App Lighttpd SCGI HTTP Proxy
  25. ࢳथٚσ • ܓઙ8෻ྍڛༀఖ • 32GଽթđඹނCPU • (300G SCSI×2 + 1T

    SATA) × 3 • (1T SATA × 3) × 5 • 6෻Кࣘđ2෻฿ࣃ • षؿDoubanFS
  26. Uploader DoubanFS Node DoubanFS Node nginx (w/ proxy_store) FileStorage Gateway

    Internet upload.douban.com otho.douban.com App signed POST form HTTP redirect
  27. DoubanDB • ٳ҃ൔKey-Valueඔऌ९ • ՖAmazon Dynamoࠆ֤ਲۋđቓਔ၂ུࡥ߄ • ೘۱ࢤ१ğset(key, value), get(key),

    delete(key) • memcacheླྀၰቔູ؀ཿࢤ१ • ᆇᆞ֥Merkle TreeđࠎႿଽթđުڿູࠎႿՈ஍໓ࡱ • Consistent Hash • ႨTokyoCabinetቓָູҪթԥ • ඔऌ९ᇏ֥ն໓Чሳ؍ࣉೆDoubanDB • DoubanFS 2.0 ࠎႿ DoubanDB ൌགྷ
  28. !" Master1 #$ Master1 %&'() Data Mining %&'() Data Mining

    #$ Master2 !" Master2 *+,- Slave !" Slave #$ Slave *+,- Master replicate write write replicate replicate read read replicate replicate