Share
Embed
Transcript
54 slides
-
၂ུඔऌ • 2.9MᇿҨႨđჿ1/4ࠃᄁႨ • తຣࠩ٤ᇿҨႨ • 23M౨/฿đڂᆴ500~600/sec • 23௴๙PCڛༀఖ(1U*15/2U*8) •
12ิ܂ཌഈڛༀ • 38G memcached -
ֆڛༀఖ • ֆ1Uڛༀఖ (frodo) • ֆނAMD Athlon 64 1.8GHz •
1Gଽթđ160G SATA*2 • Gentoo Linux • MySQL 5 • Quixote (a Python web framework) • Lighttpd + SCGI (shire) • Memcached (!) -
Gentoo Linux • ಸၞົ • emerge mysql • ebuild ьႿܵ
patch • ᆺνልླေ֥ת༆ • νಆྟ • GLSA(Gentoo Linux Security Advisories) -
MySQL • The world’s most popular open source database •
ཿഒ؟/ཿ؟ഒ ==> MyISAM • ཿѩؿۚ ==> InnoDB • Replicate for backup -
Python • षؿ • Battery Included • ֻٚ९Ӯඃ • ഠӮӉᇏ
• CPUG: http://python.cn/ -
Quixote • ࡥֆđਈđၞႿൌགྷRESTڄ֥۬URL • ֒ൈߎીႵDjango, TurboGears, Pylonsᆃུ࿊ᄴđᆺႵ၂ ۱Шᇗ֥ZOPE • http://www.douban.com/subject/1000001
# luz/subject/__init__.py def _q_lookup(request, name): subject = get_subject(name) return lambda req: subject_ui(req, subject) # luz/subject/subject_ui.ptl def subject_ui [html] (request, subject): site_header(request) “<h1>%s</h1>” % subject.title site_footer(request) -
Lighttpd • ޓݺ֥ބ࣡ྟି • ჰളSCGIᆦӻ • SCGI: ၂۱ࡥ߄ϱЧ֥FastCGIđႮ Quixoteषؿᆀषؿ •
෮Ⴕ֥౨๙ݖ80؊१֥lighttpdࣉӱ ٳؿđଽಸሼSCGI֞localhostഈ֥ Quixoteࣉӱb -
Memcache • ՖഈཌఏࣼᄝႨđႵིࡨMySQLڵք • ؓlibmemcacheቓਔpythonٿልčႨPyrexĎđྟି൞ Ղpythonϱ֥3x+ def get_subject(subject_id): subject =
mc.get(‘s:’+subject_id) if subject is None: store.farm.execute(“select xxx, xxx from subject where id=%s”, subject_id) subject = Subject(*store.farm.fetchone()) mc.set(‘s:’+subject_id, subject) return subject -
ࢳथٚσ • ܓઙਆ1Uڛༀఖ • pippin ބ meriadoc (ުڿmerry) • චނ,
4Gଽթđ250G SATA*3 • ၂ቔູႋႨڛༀఖđ၂ቔູඔऌ९ڛༀఖ • థ၍֞චཌචIPࠏٜđႨDNSࢳ༅҂ຩ؍ IP -_-b • ष؟ದླྀቔषؿđfrodoቓູषؿႨࠏ (subversion, trac, etc...) -
Internet DNS Lighttpd (!") Lighttpd (#$) App MySQL HTTP Proxy
Memcache Static Files FS SCGI -
ࢳथٚσ • ߐ֞ौ௶֥ࠏٜđ؟ཌֆIP(BGP) • ܓઙਔ၂ྍڛༀఖ (arwen) • 74G 1wሇ SATA
* 3 • ቓູඔऌ९ڛༀఖ • षႨህ֥ڛༀఖቔު࠹ෘ -
Internet Lighttpd App Memcache Data Mining MySQL Master MySQL Slave
Replicate Static Files SCGI read write -
ࢳथٚσ • ܓઙڛༀఖđචނđ4Gđ250G SATA*3 • ࡼோՖ၂۱նଢ్ٳӮ10000۱໓ࡱ၂۱ଢ • mod_rewriteЌӻURL҂э • ৫֥ோlighttpdࣉӱđఓႨmod_memcacheଆॶđ
ߏթཬோ • ࡨཬՈIO်ؓ૫٠໙֥႕ཙ • ࡼႋႨڛༀՖwebڛༀఖ৫ԛಀ • Ϝ۷؟֥ଽթٳ۳࣡໓ࡱڛༀ • ᄹࡆ၂۱ᆺඔऌ९ -
Web Service Internet Lighttpd App Memcache MySQL Master MySQL Slave
Replicate Static Files Lighttpd WebDAV WebDAV SCGI !"#$% Memcache Spiders Data Mining MySQL Slave Replicate Lighttpd (w/ mod_memcache) HTTP Proxy store.farm store.farmr write read -
ᆺඔऌ९ • storeᄹࡆfarmrඋྟđູ၂۱ॖႨ֥ᆺඔऌ९Ⴓѓ • ฮ֥replicate delay໙ี • ڣ९گᇅླေൈࡗ • ۷ྍᇶ९ުđ༯၂۱౨ສສࣼ൞ေඔऌč۷ྍඔऌު
ྍ်૫Ď • Ֆڣ९߶֝ᇁcacheթ٢֥൞ࣸඔऌ • ਲၳ൙ࡱĆ • ࢳथٚمğ۷ྍඔऌ९ުđᄝყ௹ॖି߶ઔഈႨ֥֞౦ঃ༯đᇶ ྍߏթ • ......҂ປૅđbut it works -
х૧replicate delayႄ ఏ֥ਲၳ൙ࡱ def get_subject(sid): sbj = mc.get(‘s:’+sid) if sbj
is None: sbj = flush_subject(sid, store.farmr) return sbj def flush_subject(sid, cursor=None): cursor = cursor or store.farm cursor.execute(“select ... from subject”) subject = Subject(*cursor.fetchone()) mc.set(‘s:’+sid, subject) return subject def update_subject(subject, props): store.farm.execute(“update subject ...”) store.farm.connection.commit() flush_subject(subject.id, store.farm) -
ࢳथٚσ • Scale Upđܓઙඹ1Uڛༀఖ • 16Gଽթđ147G SCSI *2 + 500G
SATA • SCSI ቓ RAID-0 • ႨMySQL SlaveটЌᆣჅ • ᄹࡆmemcachedࢫׄඔଢ • ෮Ⴕ֥MyISAMіڿູInnoDBі • ิۚଽթ০Ⴈིੱ • ࡼಆ໓ෆ෬၍ᇀSphinx -
Internet Lighttpd App Memcache MySQL Master MySQL Slave Replicate Static
Files Lighttpd WebDAV WebDAV SCGI Memcache Spiders Lighttpd (w/ mod_memcache) HTTP Proxy store.farm store.farmr Sphinx Web Service Memcache Memcache Web Service -
ࢳथٚσ • ฿ࣃ֥ࠏٜьၒ၂ུ :) • ӵքோੀਈ • ުඔऌड࠹ෘ • ಸᄙСٺ
• ܓઙ31Uڛༀఖğ4ނđ32Gଽթđ1T SATA * 3 • Ⴊ߄భ؊đఓႨ otho.douban.com ބ lotho.douban.com თ • lighttpd 1.5 with aio support • ҆ඇLVS • Scale Up: ႋႨڛༀఖଽթശࠩ 4G -> 8G -
Internet Lighttpd Static Files Lighttpd WebDAV Lighttpd HTTP Proxy Lighttpd
1.5 (w/ mod_cache) Lighttpd 1.5 (w/ mod_cache) LVS LB (Master) LVS LB (backup) Keepalived www.douban.com otho.douban.com -
MySQL Master MySQL Slave Replicate !"#$% Data Mining !"#$% Data
Mining MySQL Slave replicate read read write write -
໙ีԛགྷ • 6.4M౨/฿ (5M PV) • ႋႨڛༀఖӮູ࣠ • ଽթğᅝႨሹ൞ᄹӉđරެႵଽթ྄
• CPUğmemcacheؓའਙ߄/ّਙ߄ -
ࢳथٚσ • ֻؽႋႨڛༀఖഈཌ • lighttpd֥mod_scgiᆺିround-robin • lighttpd 1.5҂໗ק • mod_proxy
• proxy.balance = fair (load based, passive balancing) • ֒ࣉӱᅝႨଽթӑݖᚐᆴđ֒భ౨ປӮުሱ೪ • Ⴈspreadऊކರᆽ -
Lighttpd Static Files Lighttpd WebDAV Lighttpd HTTP Proxy App Memcache
Lighttpd App Memcache Lighttpd Internet SCGI SCGI HTTP Proxy HTTP Proxy Log Aggregator spread spread -
໙ีԛགྷ • 11M౨/฿č3ႋႨڛༀఖĎ • ॴࠏٜཿೆӮູު࠹ෘ֥࣠ • Sphinx֥קᇅྟ҂ܔ • ཌྷҨӁषؿᇏđླေࢳथோթԥ໙ ี
• ਲၳགྷའğ ຩᅟэતđࠒᄟਔնਈ৵ࢤ ֤҂֞ԩđloadಏ҂ۚ -
ࢳथٚσ • ඔऌ९ٳ९ • ࣴׄཌྷܱі৫ԛট • ඔऌडཌྷܱі৫ԛটđᇶ९٢ᄝ ฿ࣃđКࣘᆺ • Sphinx
-> Xapian • ႨMogileFS -
!" Master #$ Master %&'() Data Mining %&'() Data Mining
#$ Slave !" Slave *+,- Slave !" Slave #$ Slave *+,- Master replicate write write replicate replicate read read replicate replicate -
؟ඔऌ९৵ࢤ • іಆअື၂đົ၂۱іᇀඔऌ९֥႘ഝ • store.farm[r] -> store.get_cursor(table=‘xxx’, ro=True/False) def flush_subject(sid,
ro=False): cursor = store.get_cursor(table=‘subject’, ro=ro) cursor.execute(“select ... from subject”) subject = Subject(*cursor.fetchone()) mc.set(‘s:’+sid, subject) return subject • ᄝඔऌ९ࡗ୳іэ֤ಸၞđᄝᇶڣ९ࡗޙڵ ᄛ္э֤ಸၞ -
MogileFS Master Uploader MogileFS Node MogileFS Node nginx (w/ proxy_store)
MogileFS Tracker FileStorage Gateway Internet upload.douban.com otho.douban.com App signed POST form HTTP redirect -
ࢳथٚσč࿃Ď • libmemcache -> libmemcachedđႨconsistent hashࢆ֮memcacheטᆜսࡎ • ྩᆞlibmemcached֥consistent hashཌྷܱbug •
ႋႨڛༀఖശࠩᇀඹނCPU • ྩᆞlibmemcached֥failoverཌྷܱbug • Ⴈnginxูսlighttpdቓload balance • ቋުؿགྷቌঊࠋ൮൞spreadđ⯝ • ڿӮႨnginx࠺ರᆽ -
Lighttpd Static Files Lighttpd WebDAV Lighttpd HTTP Proxy App Lighttpd
Internet SCGI HTTP Proxy Nginx HTTP Proxy App Lighttpd SCGI HTTP Proxy App Lighttpd SCGI HTTP Proxy -
໙ีԛགྷ • 13M౨/฿ • ࠹߃ࡼ෮Ⴕ࣡ோ֝ೆMogileFS • ໓ࡱཬđඔਈնđTracker DBॖିӮູ ࣠ •
ཌྷҨӁޓ൳ߋ႒đթԥॢࡗषࣅᅦ -
ࢳथٚσ • ܓઙ8ྍڛༀఖ • 32GଽթđඹނCPU • (300G SCSI×2 + 1T
SATA) × 3 • (1T SATA × 3) × 5 • 6Кࣘđ2฿ࣃ • षؿDoubanFS -
DoubanFS • ીႵᇏྏඔऌ९đᆰࢤοᅶ໓ࡱhashҰᅳ෮ᄝࢫ ׄđॖഥ෪ྟ۷ݺ • οᅶhashᆴթӮଢඎđૄ۱ࢫׄڵᄳ၂ቆhashᆴ თ • קൈ҄đࠎႿ໓ࡱྩڿൈࡗ֥Merkle Tree
• ০Ⴈconsistent hashࡨഒᄹࢫׄջট֥ඔऌ၍ਈ • WebDAVቔູཿࢤ१ • ྟିູMogileFS֥3Пđཿྟିູ50П -
Uploader DoubanFS Node DoubanFS Node nginx (w/ proxy_store) FileStorage Gateway
Internet upload.douban.com otho.douban.com App signed POST form HTTP redirect -
ࢳथٚσ • षؿDoubanDB • ۷ݺ֥ഥ෪ྟ • ն໓Чሳ؍၍ԛުđMySQL֥ྟି֤֞ ᄹ఼ • MySQLචMasterٚσ
• failover۷ࡥֆ • ࢳथreplicate delay໙ี -
DoubanDB • ٳ҃ൔKey-Valueඔऌ९ • ՖAmazon Dynamoࠆ֤ਲۋđቓਔ၂ུࡥ߄ • ۱ࢤ१ğset(key, value), get(key),
delete(key) • memcacheླྀၰቔູཿࢤ१ • ᆇᆞ֥Merkle TreeđࠎႿଽթđުڿູࠎႿՈ໓ࡱ • Consistent Hash • ႨTokyoCabinetቓָູҪթԥ • ඔऌ९ᇏ֥ն໓Чሳ؍ࣉೆDoubanDB • DoubanFS 2.0 ࠎႿ DoubanDB ൌགྷ -
!" Master1 #$ Master1 %&'() Data Mining %&'() Data Mining
#$ Master2 !" Master2 *+,- Slave !" Slave #$ Slave *+,- Master replicate write write replicate replicate read read replicate replicate -
ቋྍ֥၂ུڿ • ࡼDoubanFSބඔऌडӱ၍߭Кࣘࠏٜ • ࡥ߄໓ࡱഈԮੀӱބս • ۷ݺ֥০Ⴈ႗ࡱሧჷ • Ⴈngnixቔູቋభ؊ •
www.douban.com္҆ඇLVS • ႨRabbitMQսูspread -
၂ུࣜဒ • Ϝఫᄝଽթഈ൞ᆴ֤֥ • ࡹ৫ਅݺ֥profile۽ऎđѩ০Ⴈᆭ • memcacheѩ҂৷ࡎđር༥॥ᇅcache֥ؓའնཬ ބ٠໙ٚൔ • х૧ඔऌ९֥joinҠቔ
• ᄝӁഈቓԛཋᇅၛх૧ݖત֥Ұ࿘ • ር༥ҷٳభުᄎෘ • Ⴈၘᅧ֥ඌࢳथ໙ี