2009-douban-tech-story

2009年 洪教授回顾豆瓣架构经验

Transcript

1. ׸ϵຩ࠯ඌࡏܒ֥ ؿᅚ৥ӱ 2009.4 ޠ఼ୡ hongqn@douban.com
2. ׸ϵຩࡥࢺ • 2005୍3ᄅഈཌ • ၛٳཚބؿགྷູ ނྏ֥ഠ౵ • ؀඀a׈႕aၻ ুaཬቆa๝ Ӭaࣴׄ • ໡֥׸ϵaႶਣ
3. ၂ུඔऌ • 2.9MᇿҨႨ޼đჿ1/4ࠃᄁႨ޼ • తຣࠩ٤ᇿҨႨ޼ • 23M׮෿౨౰/฿đڂᆴ500~600/sec • 23෻௴๙PCڛༀఖ(1U*15/2U*8) • 12෻ิ܂ཌഈڛༀ • 38G memcached
4. ֆڛༀఖ • ֆ෻1Uڛༀఖ (frodo) • ֆނAMD Athlon 64 1.8GHz • 1Gଽթđ160G SATA*2 • Gentoo Linux • MySQL 5 • Quixote (a Python web framework) • Lighttpd + SCGI (shire) • Memcached (!)
5. Gentoo Linux • ಸၞົ޹ • emerge mysql • ebuild ьႿܵ৘ patch • ᆺνልླေ֥ת༆ • νಆྟ • GLSA(Gentoo Linux Security Advisories)
6. MySQL • The world’s most popular open source database • ཿഒ؀؟/ཿ؟؀ഒ ==> MyISAM • ؀ཿѩؿۚ ==> InnoDB • Replicate for backup
7. Python • षؿ࿡෎ • Battery Included • ֻ೘ٚ९Ӯඃ • ഠ౵ӮӉᇏ • CPUG: http://python.cn/
8. Quixote • ࡥֆđ౞ਈđၞႿൌགྷRESTڄ֥۬URL • ֒ൈߎીႵDjango, TurboGears, Pylonsᆃུ࿊ᄴđᆺႵ၂ ۱Шᇗ֥ZOPE • http://www.douban.com/subject/1000001 # luz/subject/__init__.py def _q_lookup(request, name): subject = get_subject(name) return lambda req: subject_ui(req, subject) # luz/subject/subject_ui.ptl def subject_ui [html] (request, subject): site_header(request) “<h1>%s</h1>” % subject.title site_footer(request)
9. Lighttpd • ޓݺ֥׮෿ބ࣡෿ྟି • ჰളSCGIᆦӻ • SCGI: ၂۱ࡥ߄ϱЧ֥FastCGIđႮ Quixoteषؿᆀषؿ • ෮Ⴕ֥౨౰׻๙ݖ80؊१֥lighttpdࣉӱ ٳؿđ׮෿ଽಸሼSCGI֞localhostഈ֥ Quixoteࣉӱb
10. Memcache • Ֆഈཌఏࣼᄝ൐ႨđႵིࡨ౞MySQLڵք • ؓlibmemcacheቓਔpythonٿልč൐ႨPyrexĎđྟି൞ Ղpythonϱ֥3x+ def get_subject(subject_id): subject = mc.get(‘s:’+subject_id) if subject is None: store.farm.execute(“select xxx, xxx from subject where id=%s”, subject_id) subject = Subject(*store.farm.fetchone()) mc.set(‘s:’+subject_id, subject) return subject
11. Internet MySQL Lighttpd App SCGI Memcache Static Files FS
12. ໙ีԛགྷ • 1.2M׮෿౨౰/฿ • Ո஍IOӮູ௞࣠ • ླေ࿙ᅳྍࠏٜ
13. ࢳथٚσ • ܓઙਆ෻1Uڛༀఖ • pippin ބ meriadoc (ުڿ଀merry) • චނ, 4Gଽթđ250G SATA*3 • ၂෻ቔູႋႨڛༀఖđ၂෻ቔູඔऌ९ڛༀఖ • థ၍֞චཌචIPࠏٜđ൐ႨDNSࢳ༅҂๝ຩ؍ IP -_-b • ष൓؟ದླྀቔषؿđfrodoቓູषؿႨࠏ (subversion, trac, etc...)
14. Internet DNS Lighttpd (!") Lighttpd (#$) App MySQL HTTP Proxy Memcache Static Files FS SCGI
15. ࠫׄؿགྷ • ඔऌ९֥ଽթٳ஥ؓྟି႕ཙᇗն • innodb_buffer_pool_size • Ո஍ෛࠏ࿙֡෎؇б๽๳ਈ۷ᇗေ • ຩഈᅳট֥IP؍ٳ҃ޓ҂ौ௶
16. ໙ีԛགྷ • 1.5M׮෿౨౰/฿đഉໃ֞ྟି௞࣠ • ࠏٜ҂ौ௶đ௔َܣᅰ • IP؍ٳ҃ඔऌ҂ौ௶đႨ޼ّ႘٠໙ߏત
17. ࢳथٚσ • ߐ֞ौ௶֥ࠏٜđ؟ཌֆIP(BGP) • ܓઙਔ၂෻ྍڛༀఖ (arwen) • 74G 1wሇ SATA * 3 • ቓູඔऌ९ڛༀఖ • ष൓൐Ⴈህ૊֥ڛༀఖቔު෻࠹ෘ
18. Internet Lighttpd App Memcache Data Mining MySQL Master MySQL Slave Replicate Static Files SCGI read write
19. ໙ีԛགྷ • 2M׮෿౨౰/฿ • ࣡෿໓ࡱڛༀՈ஍IOӮູ௞࣠ • ഈϤຣ֥ཬ๭ோčႨ޼๨ཞaٿ૫๭ ோ, etc...Ď • ඔऌ९ڛༀఖࢤ࣍௞࣠
20. ࢳथٚσ • ܓઙ೘෻ڛༀఖđචނđ4Gđ250G SATA*3 • ࡼ๭ோՖ၂۱նଢ੣్ٳӮ10000۱໓ࡱ၂۱ଢ੣ • mod_rewriteЌӻURL҂э • ׿৫֥๭ோlighttpdࣉӱđఓႨmod_memcacheଆॶđ ߏթཬ๭ோ • ࡨཬՈ஍IO်ؓ૫٠໙֥႕ཙ • ࡼႋႨڛༀՖwebڛༀఖ׿৫ԛಀ • Ϝ۷؟֥ଽթٳ஥۳࣡෿໓ࡱڛༀ • ᄹࡆ၂۱ᆺ؀ඔऌ९
21. Web Service Internet Lighttpd App Memcache MySQL Master MySQL Slave Replicate Static Files Lighttpd WebDAV WebDAV SCGI !"#$% Memcache Spiders Data Mining MySQL Slave Replicate Lighttpd (w/ mod_memcache) HTTP Proxy store.farm store.farmr write read
22. ᆺ؀ඔऌ९ • storeᄹࡆfarmrඋྟđູ၂۱ॖႨ֥ᆺ؀ඔऌ९Ⴓѓ • ๨ฮ֥replicate delay໙ี • ڣ९گᇅླေൈࡗ • ۷ྍᇶ९ުđ༯၂۱౨౰ສສࣼ൞ေ؀ඔऌč۷ྍඔऌު඗ ྍ်૫Ď • Ֆڣ९؀߶֝ᇁcache৚թ٢֥൞ࣸඔऌ • ਲၳ൙ࡱĆ • ࢳथٚمğ۷ྍඔऌ९ުđᄝყ௹ॖି߶ઔഈႨ֥֞౦ঃ༯đᇶ ׮඗ྍߏթ • ......҂ປૅđbut it works
23. х૧replicate delayႄ ఏ֥ਲၳ൙ࡱ def get_subject(sid): sbj = mc.get(‘s:’+sid) if sbj is None: sbj = flush_subject(sid, store.farmr) return sbj def flush_subject(sid, cursor=None): cursor = cursor or store.farm cursor.execute(“select ... from subject”) subject = Subject(*cursor.fetchone()) mc.set(‘s:’+sid, subject) return subject def update_subject(subject, props): store.farm.execute(“update subject ...”) store.farm.connection.commit() flush_subject(subject.id, store.farm)
24. ໙ีԛགྷ • 2.5M׮෿౨౰/฿ • ඔऌ९Ո஍ॢࡗ҂ܔਔ • ໡ഈ/ࣴׄඔऌਈஔն • SATA஍ܣᅰੱۚ • ඔऌ९࿢৯ᄹն
25. ࢳथٚσ • Scale Upđܓઙඹ෻1Uڛༀఖ • 16Gଽթđ147G SCSI *2 + 500G SATA • SCSI ቓ RAID-0 • ႨMySQL SlaveটЌᆣ಺Ⴥ • ᄹࡆmemcachedࢫׄඔଢ • ෮Ⴕ֥MyISAMі׻ڿູInnoDBі • ิۚଽթ০Ⴈིੱ • ࡼಆ໓ෆ෬၍ᇀSphinx
26. Internet Lighttpd App Memcache MySQL Master MySQL Slave Replicate Static Files Lighttpd WebDAV WebDAV SCGI Memcache Spiders Lighttpd (w/ mod_memcache) HTTP Proxy store.farm store.farmr Sphinx Web Service Memcache Memcache Web Service
27. ໙ีԛགྷ • 5.2M׮෿౨౰/฿ • ๭ோੀਈٮႨӮູቋնӮЧ • Webڛༀఖ֥Ո஍IOߎ൞߶႕ཙ׮෿် ૫ྟି • ႋႨڛༀఖࣉӱඔ҂ܔਔ • ࠏ݊ॢࡗ҂ܔਔ
28. ࢳथٚσ • ฿ࣃ֥ࠏٜьၒ၂ུ :) • ӵք๭ோੀਈ • ު෻ඔऌ຋ड࠹ෘ • ಸᄙСٺ • ܓઙ3෻1Uڛༀఖğ4ނđ32Gଽթđ1T SATA * 3 • Ⴊ߄భ؊đఓႨ otho.douban.com ބ lotho.douban.com თ଀ • lighttpd 1.5 with aio support • ҆ඇLVS • Scale Up: ႋႨڛༀఖଽթശࠩ 4G -> 8G
29. Internet Lighttpd Static Files Lighttpd WebDAV Lighttpd HTTP Proxy Lighttpd 1.5 (w/ mod_cache) Lighttpd 1.5 (w/ mod_cache) LVS LB (Master) LVS LB (backup) Keepalived www.douban.com otho.douban.com
30. MySQL Master MySQL Slave Replicate !"#$% Data Mining !"#$% Data Mining MySQL Slave replicate read read write write
31. ໙ีԛགྷ • 6.4M׮෿౨౰/฿ (5M PV) • ႋႨڛༀఖӮູ௞࣠ • ଽթğᅝႨሹ൞ᄹӉđරެႵଽթ྄ ੝ • CPUğmemcacheؓའ྽ਙ߄/ّ྽ਙ߄
32. ࢳथٚσ • ֻؽ෻ႋႨڛༀఖഈཌ • lighttpd֥mod_scgiᆺିround-robin • lighttpd 1.5҂໗ק • mod_proxy • proxy.balance = fair (load based, passive balancing) • ֒ࣉӱᅝႨଽթӑݖᚐᆴđ֒భ౨౰ປӮުሱ೪ • ൐Ⴈspreadऊކರᆽ
33. Lighttpd Static Files Lighttpd WebDAV Lighttpd HTTP Proxy App Memcache Lighttpd App Memcache Lighttpd Internet SCGI SCGI HTTP Proxy HTTP Proxy Log Aggregator spread spread
34. ໙ีԛགྷ • 11M׮෿౨౰/฿č3෻ႋႨڛༀఖĎ • ॴࠏٜཿೆӮູު෻࠹ෘ֥௞࣠ • Sphinx֥קᇅྟ҂ܔ • ཌྷҨӁ௖षؿᇏđླေࢳथ๭ோթԥ໙ ี • ਲၳགྷའğ ຩᅟэતđࠒᄟਔնਈ৵ࢤ ֤҂֞ԩ৘đloadಏ҂ۚ
35. ࢳथٚσ • ඔऌ९ٳ९ • ࣴׄཌྷܱі׿৫ԛট • ඔऌ຋डཌྷܱі׿৫ԛটđᇶ९٢ᄝ ฿ࣃđКࣘᆺ؀ • Sphinx -> Xapian • ൐ႨMogileFS
36. !" Master #$ Master %&'() Data Mining %&'() Data Mining #$ Slave !" Slave *+,- Slave !" Slave #$ Slave *+,- Master replicate write write replicate replicate read read replicate replicate
37. ؟ඔऌ९৵ࢤ • і଀ಆअື၂đົ޹၂۱і଀ᇀඔऌ९֥႘ഝ • store.farm[r] -> store.get_cursor(table=‘xxx’, ro=True/False) def flush_subject(sid, ro=False): cursor = store.get_cursor(table=‘subject’, ro=ro) cursor.execute(“select ... from subject”) subject = Subject(*cursor.fetchone()) mc.set(‘s:’+sid, subject) return subject • ᄝඔऌ९ࡗ୳іэ֤ಸၞđᄝᇶڣ९ࡗ௜ޙڵ ᄛ္э֤ಸၞ
38. MogileFS Master Uploader MogileFS Node MogileFS Node nginx (w/ proxy_store) MogileFS Tracker FileStorage Gateway Internet upload.douban.com otho.douban.com App signed POST form HTTP redirect
39. ࢳथٚσč࿃Ď • libmemcache -> libmemcachedđ൐Ⴈconsistent hashࢆ֮memcacheטᆜսࡎ • ྩᆞlibmemcached֥consistent hashཌྷܱbug • ႋႨڛༀఖശࠩᇀඹނCPU • ྩᆞlibmemcached֥failoverཌྷܱbug • Ⴈnginxูսlighttpdቓload balance • ቋުؿགྷቌঊࠋ൮൞spreadđ⯝ • ڿӮႨnginx࠺੣ರᆽ
40. Consistent Hash ๭ோϱಃඋႿcharlee http:// tech.idv2.com/2008/07/24/ memcached-004/
41. Lighttpd Static Files Lighttpd WebDAV Lighttpd HTTP Proxy App Lighttpd Internet SCGI HTTP Proxy Nginx HTTP Proxy App Lighttpd SCGI HTTP Proxy App Lighttpd SCGI HTTP Proxy
42. ໙ีԛགྷ • 13M׮෿౨౰/฿ • ࠹߃ࡼ෮Ⴕ࣡෿๭ோ׻֝ೆMogileFS • ໓ࡱཬđඔਈնđTracker DBॖିӮູ ௞࣠ • ཌྷҨӁ௖ޓ൳ߋ႒đթԥॢࡗष൓ࣅᅦ
43. ࢳथٚσ • ܓઙ8෻ྍڛༀఖ • 32GଽթđඹނCPU • (300G SCSI×2 + 1T SATA) × 3 • (1T SATA × 3) × 5 • 6෻Кࣘđ2෻฿ࣃ • षؿDoubanFS
44. DoubanFS • ીႵᇏྏඔऌ९đᆰࢤοᅶ໓ࡱ଀hashҰᅳ෮ᄝࢫ ׄđॖഥ෪ྟ۷ݺ • οᅶhashᆴթӮଢ੣ඎđૄ۱ࢫׄڵᄳ၂ቆhashᆴ ౵თ • קൈ๝҄đࠎႿ໓ࡱྩڿൈࡗ֥Merkle Tree • ০Ⴈconsistent hashࡨഒᄹ೷ࢫׄջট֥ඔऌ၍׮ਈ • WebDAVቔູ؀ཿࢤ१ • ؀ྟିູMogileFS֥3Пđཿྟିູ50П
45. Merkle Tree
46. Uploader DoubanFS Node DoubanFS Node nginx (w/ proxy_store) FileStorage Gateway Internet upload.douban.com otho.douban.com App signed POST form HTTP redirect
47. ໙ีԛགྷ • 16M׮෿౨౰/฿ • ඔऌ९ն໓Чሳ؍࿸ᇗ႕ཙਔඔऌ९ྟ ି • DoubanFSᇏཬ๭ோ֝ᇁIOᄹۚ • ඔऌ९ॖႨྟေ౰ิۚ
48. ࢳथٚσ • षؿDoubanDB • ۷ݺ֥ഥ෪ྟ • ն໓Чሳ؍၍ԛުđMySQL֥ྟି֤֞ ᄹ఼ • MySQLචMasterٚσ • failover۷ࡥֆ • ࢳथreplicate delay໙ี
49. DoubanDB • ٳ҃ൔKey-Valueඔऌ९ • ՖAmazon Dynamoࠆ֤ਲۋđቓਔ၂ུࡥ߄ • ೘۱ࢤ१ğset(key, value), get(key), delete(key) • memcacheླྀၰቔູ؀ཿࢤ१ • ᆇᆞ֥Merkle TreeđࠎႿଽթđުڿູࠎႿՈ஍໓ࡱ • Consistent Hash • ႨTokyoCabinetቓָູҪթԥ • ඔऌ९ᇏ֥ն໓Чሳ؍ࣉೆDoubanDB • DoubanFS 2.0 ࠎႿ DoubanDB ൌགྷ
50. ๭ோϱಃඋႿcharlee http:// tech.idv2.com/2008/07/24/ memcached-004/
51. !" Master1 #$ Master1 %&'() Data Mining %&'() Data Mining #$ Master2 !" Master2 *+,- Slave !" Slave #$ Slave *+,- Master replicate write write replicate replicate read read replicate replicate
52. ቋྍ֥၂ུڿ׮ • ࡼDoubanFSބඔऌ຋डӱ྽၍߭Кࣘࠏٜ • ࡥ߄໓ࡱഈԮੀӱބս઒ • ۷ݺ֥০Ⴈ႗ࡱሧჷ • ൐Ⴈngnixቔູቋభ؊ • www.douban.com္҆ඇLVS • ൐ႨRabbitMQսูspread
53. ၂ུࣜဒ • Ϝఫ޽ᄝଽթഈ൞ᆴ֤֥ • ࡹ৫ਅݺ֥profile۽ऎđѩ০Ⴈᆭ • memcacheѩ҂৷ࡎđር༥॥ᇅcache֥ؓའնཬ ބ٠໙ٚൔ • х૧ඔऌ९֥joinҠቔ • ᄝӁ௖ഈቓԛཋᇅၛх૧ݖત֥Ұ࿘ • ር༥ҷٳభު෻ᄎෘ • Ⴈၘᅧ໤֥࠯ඌࢳथ໙ี
54. ྆྆