The Technology behind pixiv Infrastructure

Slide 1

Slide 1 text

harukasan / ͸Δ͔͞Μ at #pyfes 2013.11 ͷΠϯϑϥΛࢧ͑Δٕज़ 2013

Slide 2

Slide 2 text

Harukasan@pixiv a.k.a MICHII Shunsuke ಓҪढ़հ 2003 Kurume National College of Tech.  - NHK ROBOCON  - ACM-ICPC 2008 Kyushu Inst. of Technology 2010 Tsukuba Univ.  - Computational Vision Science 2012 pixiv Inc.  - Infrastructure team

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

͜͜ʹpixivͷઆ໌͕ೖΔ

Slide 5

Slide 5 text

Server 400+ Trafﬁc 10Gbps+ Team member 6 Monthly PV 3.7 Billion

Slide 6

Slide 6 text

Bases of pixiv Infrastructure Ofﬁce IDCF DC Developments Testing Log Analytics Small Services Main Applications DB Image Cluster New DC Image Cluster

Slide 7

Slide 7 text

ISP Backbone Ofﬁce IDCF DC New DC 100M 1G 10G 1G 1G line 1G pixiv Network

Slide 8

Slide 8 text

ͷΠϯϑϥΛࢧ͑Δٕज़ 2013

Slide 9

Slide 9 text

http://www.slideshare.net/kamipo/pixiv @kamipo pixivͷΠϯϑϥΛࢧ͑Δٕज़ / ςΫηϛ2009

Slide 10

Slide 10 text

@cubicdaiya inside pixiv’s infrastructure / PHP Conference 2013 http://www.slideshare.net/cubicdaiya/inside-pixiv-infrastructure

Slide 11

Slide 11 text

Agenda pixiv Image Cluster Log Analysis Basis Management Tools

Slide 12

Slide 12 text

pixiv Image Cluster QJYJWͷը૾഑৴Ϋϥελʹ͍ͭͯ

Slide 13

Slide 13 text

pixiv Image Cluster • 2010೥͔Βӡ༻։࢝ • pixivͷϝΠϯίϯςϯπͰ͋ΔΠϥετΛ  ௒ߴ଎ʹॲཧ͢ΔͨΊʹ࠷దԽ • શτϥϑΟοΫͷ90%Ҏ্Λࡹ͍͍ͯΔ

Slide 14

Slide 14 text

Image Cluster nginx Front Cache DNS Round-Robin ATS Cache nginx Dispatch nginx Front Cache nginx Front Cache nginx Front Cache ATS Cache ATS Cache ATS Cache Consistent Hashing nginx Dispatch nginx Dispatch nginx Dispatch Apache Origin nginx Thumbnail DNS Round-Robin i1.pixiv.net i2.pixiv.net SmallLight

Slide 15

Slide 15 text

Cache strategy • ϝϞϦͱσΟεΫͷ2ஈΩϟογϡߏ੒ • τϥϑΟοΫ͕૿͑ΔʹͭΕͯ  εΠονؒτϥϑΟοΫ͕ແࢹͰ͖ͳ͘ͳͬͨ • ωοτϫʔΫτϥϑΟοΫΛ཈͑ͭͭ  Ωϟογϡ༰ྔͷ֬อ͢Δඞཁ

Slide 16

Slide 16 text

Cache strategy • ϝϞϦՁ֨ͷ௿Լ • SSDͷՁ͕֨๫མ • ߴIOPSͷSSD͕ొ৔ 2011 ¥40,000  2013 ¥20,000 256G READ WRITE PRICE ioDrive2 785GB MLC*1 215,000/230,000 IOPS I don’t know Intel 910 800GB 100,000/ 75,000 IOPS ¥400,000 SSD 256GBx3 RAID0 80,000/ 50,000 IOPS ¥60,000 *1: ioDrive2͸ެশ஋ Intel 910ɺ SSD RAID0ʹ͍ͭͯ͸ﬁoʹΑΔଌఆ 16G ECC RDIMM ¥20,000

Slide 17

Slide 17 text

Cache strategy ound-Robin nginx Front Cache nginx Front Cache nginx Front Cache ATS Cache ATS Cache ATS Cache Consistent Hashing nginx Dispatch nginx Dispatch nginx Dispatch Apache Origin nginx Thumbnail DNS Round-Robin i1.pixiv.net i2.pixiv.net 64GB Memory - nginx cache on tmpfs - cache hit rate: 50% - reduce network trafﬁc 256GB SSD x3 RAID0 - Apache Trafﬁc Server (standalone) - cache hit rate: 80-90% SSD SSD HDD Original & BIG Thumb. Small Thumbnails

Slide 18

Slide 18 text

Aggregate image domains • ը૾αʔό͸ϢʔβʔIDϕʔεͰ෼ࢄ  img01.pixiv.net - img1XX.pixiv.net • 1ϖʔδͰ40-60ճDNSϦΫΤετ͕ൃੜ  ‎Ոఉ༻ϧʔλ͸DNSղܾ͕Ͱ͖ͳ͘ͳΔ • શը૾ͷURIΛมߋͯ͠ରԠ OLD: http://img01.pixiv.net/img/****/*****.jpg NEW: http://i1.pixiv.net/img01/img/****/*****.jpg

Slide 19

Slide 19 text

New Image Store ৽͍͠ը૾ετϨʔδํࣜʹ͍ͭͯ

Slide 20

Slide 20 text

New image store • ࡞੒೔+IDϕʔεͷγʔέϯγϟϧͳURI • 1౓ॻ͖ࠐΜͩϑΝΠϧ͸Read Only • ࠶౤ߘॲཧ͸࡞੒೔Λߋ৽ • ngx_lua/OpenRestyΛ༻͍ͨ࿦ཧ࡟আ

Slide 21

Slide 21 text

Logical Delete ! Kyototycoon ngx_lua / nginx null 404 /img02.png 403 /img03.png 404 /img05.png 404 /img08.png 403 " " GET /img01.png GET /img03.png 404

Slide 22

Slide 22 text

Logical Delete local memcached = require "resty.memcached" local uri = ngx.var.request_uri local memc = memcached:new() . . . local val, flags, err = memc:get(request_uri) if val and val ~= "200" then exit(tonumber(val)) end logical_delete.lua location / { access_by_lua_file logical_delete.lua; }

Slide 23

Slide 23 text

Log Analysis Basis QJYJWͷϩάղੳج൫ʹ͍ͭͯ

Slide 24

Slide 24 text

Log Analysis Basis PHP Application MySQL/neoagent Front server - Error Log - Login Log - Activity Log - Slow Query - Access Log MongoDB Elasticsearch Fluentd File System

Slide 25

Slide 25 text

Error Viewer

Slide 26

Slide 26 text

Slow Query Viewer

Slide 27

Slide 27 text

Kibana 3

Slide 28

Slide 28 text

Output Log • JSONΛॻ͖ग़ͯ͠Fluentd͕tail͢Δ࢓૊Έ • ϓϩηε͕େྔʹىಈͯ͠΋εϧʔϓοτ͕མͪͳ͍ PHP Exception Handler Logger::write($type, $data) # JSON Fluentd in_tail

Slide 29

Slide 29 text

Fluentd conﬁg type tail path /var/tmp/log/activity.log pos_file /var/tmp/fluentd/activity.pos tag activity format json # JSONܗࣜΛಡΈࠐΉ type forward_with_hostname # HostΛೖΕͯforward flush_interval 1s # ඞͣ1ඵ ! buffer_type file # buffer type͸file/࠶ىಈͯ͠΋ফ͑ͳ͍ buffer_path /var/tmp/fluentd/buffer/activity.*.buffer buffer_chunk_limit 2m # chunkαΠζ͸খ͞Ί buffer_queue_limit 128 # Ͳͷ͘Β͍ফ͑ͪΌ͍͚ͳ͍͔ …

Slide 30

Slide 30 text

Management tools ؅ཧܥɺ؂ࢹܥπʔϧʹ͍ͭͯ

Slide 31

Slide 31 text

Monitoring servers/services • ͘͘͝͝Ұൠతͳ؂ࢹϓϩμΫτΛ࢖͍ͬͯΔ Ϧιʔεάϥϑ ϗετ/αʔϏε؂ࢹ ϗετ಺ϓϩηε؂ࢹ εΫϦϓτ Munin Nagios Monit Cron

Slide 32

Slide 32 text

Cluster Admin ϋʔυ΢ΣΞ৘ใ ϗετͷ༻్ ؂ࢹঢ়ଶ

Slide 33

Slide 33 text

Capistrano/Subversion • /etc/ҎԼͷઃఆϑΝΠϧ͕ͦͷ··subversionͷ  ؅ཧԼͷσΟϨΫτϦʹ • ઃఆ൓ө͸capistranoΛ࢖༻ͯ͠શ୆ʹσϓϩΠ • ϗετҰཡ͸APIܦ༝ͰऔಘͰ͖ΔΑ͏ʹ $cap dns:update $cap dns:check $cap dns:reload ex: update DNS Record

Slide 34

Slide 34 text

Management Tools • LVS؅ཧը໘ • MySQLͷ஗Ԇ؂ࢹ • αʔϏεͷϦιʔεϞχλϦϯά

Slide 35

Slide 35 text

Conclusion • pixivΛࢧ͑Δج൫γεςϜʹ͍ͭͯ঺հ • ࢖͍΍͍͢πʔϧΛࣗ෼͕ͨͪ࢖͍΍͍͢Α͏ʹ • ઑͬͨ͜ͱ͸ͤͣɺແཧͤͣӡ༻Ͱ͖Δঢ়ଶʹ  ΋͍ͬͯ͘