Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
The Technology behind pixiv Infrastructure
Search
Harukasan
PRO
November 30, 2013
Technology
10
4.1k
The Technology behind pixiv Infrastructure
pixivのインフラを支える技術2013
at Python Developers Festa 2013.11
Harukasan
PRO
November 30, 2013
Tweet
Share
More Decks by Harukasan
See All by Harukasan
Successor to PicoRabbit: Ruby Programming Envorinment / RubyKaigi 2025 follow up
harukasan
PRO
1
190
Write your own mrbgem, Create your own device
harukasan
PRO
1
110
PicoRabbit: a Tiny Presentation Device Powered by Ruby
harukasan
PRO
2
460
pixivを支える技術 / 技育CAMPアカデミア
harukasan
PRO
3
520
20240401 新卒研修 - ピクシブにおける技術領域
harukasan
PRO
1
840
ピクシブのコンテンツ配信基盤技術 / pixiv TECH SALON
harukasan
PRO
5
5.7k
Goにおける画像ファイル処理 / golang.tokyo #19
harukasan
PRO
7
6.7k
WebRTC動画をトランスコードする / Transcoding video streams from WebRTC
harukasan
PRO
5
1.6k
ImageFluxを支えるリモート開発 / 20171202
harukasan
PRO
2
1.9k
Other Decks in Technology
See All in Technology
Django's GeneratedField by example - DjangoCon US 2025
pauloxnet
0
150
Snowflake Intelligenceにはこうやって立ち向かう!クラシルが考えるAI Readyなデータ基盤と活用のためのDataOps
gappy50
0
270
ブロックテーマ時代における、テーマの CSS について考える Toro_Unit / 2025.09.13 @ Shinshu WordPress Meetup
torounit
0
130
DroidKaigi 2025 Androidエンジニアとしてのキャリア
mhidaka
2
380
dbt開発 with Claude Codeのためのガードレール設計
10xinc
2
1.3k
RSCの時代にReactとフレームワークの境界を探る
uhyo
10
3.5k
新アイテムをどう使っていくか?みんなであーだこーだ言ってみよう / 20250911-rpi-jam-tokyo
akkiesoft
0
320
LLMを搭載したプロダクトの品質保証の模索と学び
qa
0
1.1k
「Linux」という言葉が指すもの
sat
PRO
4
140
スマートファクトリーの第一歩 〜AWSマネージドサービスで 実現する予知保全と生成AI活用まで
ganota
2
280
「全員プロダクトマネージャー」を実現する、Cursorによる仕様検討の自動運転
applism118
22
12k
大「個人開発サービス」時代に僕たちはどう生きるか
sotarok
20
10k
Featured
See All Featured
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
35
3.1k
The Straight Up "How To Draw Better" Workshop
denniskardys
236
140k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
194
16k
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
eileencodes
26
3k
Agile that works and the tools we love
rasmusluckow
330
21k
Why Our Code Smells
bkeepers
PRO
339
57k
Embracing the Ebb and Flow
colly
87
4.8k
Rebuilding a faster, lazier Slack
samanthasiow
83
9.2k
Testing 201, or: Great Expectations
jmmastey
45
7.7k
Producing Creativity
orderedlist
PRO
347
40k
Being A Developer After 40
akosma
90
590k
A better future with KSS
kneath
239
17k
Transcript
harukasan / Δ͔͞Μ at #pyfes 2013.11 ͷΠϯϑϥΛࢧ͑Δٕज़ 2013
Harukasan@pixiv a.k.a MICHII Shunsuke ಓҪढ़հ 2003 Kurume National College of
Tech. - NHK ROBOCON - ACM-ICPC 2008 Kyushu Inst. of Technology 2010 Tsukuba Univ. - Computational Vision Science 2012 pixiv Inc. - Infrastructure team
None
͜͜ʹpixivͷઆ໌͕ೖΔ
Server 400+ Traffic 10Gbps+ Team member 6 Monthly PV 3.7
Billion
Bases of pixiv Infrastructure Office IDCF DC Developments Testing Log
Analytics Small Services Main Applications DB Image Cluster New DC Image Cluster
ISP Backbone Office IDCF DC New DC 100M 1G 10G
1G 1G line 1G pixiv Network
ͷΠϯϑϥΛࢧ͑Δٕज़ 2013
http://www.slideshare.net/kamipo/pixiv @kamipo pixivͷΠϯϑϥΛࢧ͑Δٕज़ / ςΫηϛ2009
@cubicdaiya inside pixiv’s infrastructure / PHP Conference 2013 http://www.slideshare.net/cubicdaiya/inside-pixiv-infrastructure
Agenda pixiv Image Cluster Log Analysis Basis Management Tools
pixiv Image Cluster QJYJWͷը૾৴Ϋϥελʹ͍ͭͯ
pixiv Image Cluster • 2010͔Βӡ༻։࢝ • pixivͷϝΠϯίϯςϯπͰ͋ΔΠϥετΛ ߴʹॲཧ͢ΔͨΊʹ࠷దԽ • શτϥϑΟοΫͷ90%Ҏ্Λࡹ͍͍ͯΔ
Image Cluster nginx Front Cache DNS Round-Robin ATS Cache nginx
Dispatch nginx Front Cache nginx Front Cache nginx Front Cache ATS Cache ATS Cache ATS Cache Consistent Hashing nginx Dispatch nginx Dispatch nginx Dispatch Apache Origin nginx Thumbnail DNS Round-Robin i1.pixiv.net i2.pixiv.net SmallLight
Cache strategy • ϝϞϦͱσΟεΫͷ2ஈΩϟογϡߏ • τϥϑΟοΫ͕૿͑ΔʹͭΕͯ εΠονؒτϥϑΟοΫ͕ແࢹͰ͖ͳ͘ͳͬͨ • ωοτϫʔΫτϥϑΟοΫΛ͑ͭͭ Ωϟογϡ༰ྔͷ֬อ͢Δඞཁ
Cache strategy • ϝϞϦՁ֨ͷԼ • SSDͷՁ͕֨མ • ߴIOPSͷSSD͕ొ 2011 ¥40,000
2013 ¥20,000 256G READ WRITE PRICE ioDrive2 785GB MLC*1 215,000/230,000 IOPS I don’t know Intel 910 800GB 100,000/ 75,000 IOPS ¥400,000 SSD 256GBx3 RAID0 80,000/ 50,000 IOPS ¥60,000 *1: ioDrive2ެশ Intel 910ɺ SSD RAID0ʹ͍ͭͯfioʹΑΔଌఆ 16G ECC RDIMM ¥20,000
Cache strategy ound-Robin nginx Front Cache nginx Front Cache nginx
Front Cache ATS Cache ATS Cache ATS Cache Consistent Hashing nginx Dispatch nginx Dispatch nginx Dispatch Apache Origin nginx Thumbnail DNS Round-Robin i1.pixiv.net i2.pixiv.net 64GB Memory - nginx cache on tmpfs - cache hit rate: 50% - reduce network traffic 256GB SSD x3 RAID0 - Apache Traffic Server (standalone) - cache hit rate: 80-90% SSD SSD HDD Original & BIG Thumb. Small Thumbnails
Aggregate image domains • ը૾αʔόϢʔβʔIDϕʔεͰࢄ img01.pixiv.net - img1XX.pixiv.net • 1ϖʔδͰ40-60ճDNSϦΫΤετ͕ൃੜ
Ոఉ༻ϧʔλDNSղܾ͕Ͱ͖ͳ͘ͳΔ • શը૾ͷURIΛมߋͯ͠ରԠ OLD: http://img01.pixiv.net/img/****/*****.jpg NEW: http://i1.pixiv.net/img01/img/****/*****.jpg
New Image Store ৽͍͠ը૾ετϨʔδํࣜʹ͍ͭͯ
New image store • ࡞+IDϕʔεͷγʔέϯγϟϧͳURI • 1ॻ͖ࠐΜͩϑΝΠϧRead Only • ࠶ߘॲཧ࡞Λߋ৽
• ngx_lua/OpenRestyΛ༻͍ͨཧআ
Logical Delete ! Kyototycoon ngx_lua / nginx null 404 /img02.png
403 /img03.png 404 /img05.png 404 /img08.png 403 " " GET /img01.png GET /img03.png 404
Logical Delete local memcached = require "resty.memcached" local uri =
ngx.var.request_uri local memc = memcached:new() . . . local val, flags, err = memc:get(request_uri) if val and val ~= "200" then exit(tonumber(val)) end logical_delete.lua location / { access_by_lua_file logical_delete.lua; }
Log Analysis Basis QJYJWͷϩάղੳج൫ʹ͍ͭͯ
Log Analysis Basis PHP Application MySQL/neoagent Front server - Error
Log - Login Log - Activity Log - Slow Query - Access Log MongoDB Elasticsearch Fluentd File System
Error Viewer
Slow Query Viewer
Kibana 3
Output Log • JSONΛॻ͖ग़ͯ͠Fluentd͕tail͢ΔΈ • ϓϩηε͕େྔʹىಈͯ͠εϧʔϓοτ͕མͪͳ͍ PHP Exception Handler Logger::write($type,
$data) # JSON Fluentd in_tail
Fluentd config <source> type tail path /var/tmp/log/activity.log pos_file /var/tmp/fluentd/activity.pos tag
activity format json # JSONܗࣜΛಡΈࠐΉ </source> <match activity> type forward_with_hostname # HostΛೖΕͯforward flush_interval 1s # ඞͣ1ඵ ! buffer_type file # buffer typefile/࠶ىಈͯ͠ফ͑ͳ͍ buffer_path /var/tmp/fluentd/buffer/activity.*.buffer buffer_chunk_limit 2m # chunkαΠζখ͞Ί buffer_queue_limit 128 # Ͳͷ͘Β͍ফ͑ͪΌ͍͚ͳ͍͔ … </match>
Management tools ཧܥɺࢹܥπʔϧʹ͍ͭͯ
Monitoring servers/services • ͘͘͝͝ҰൠతͳࢹϓϩμΫτΛ͍ͬͯΔ Ϧιʔεάϥϑ ϗετ/αʔϏεࢹ ϗετϓϩηεࢹ εΫϦϓτ Munin Nagios
Monit Cron
Cluster Admin ϋʔυΣΞใ ϗετͷ༻్ ࢹঢ়ଶ
Capistrano/Subversion • /etc/ҎԼͷઃఆϑΝΠϧ͕ͦͷ··subversionͷ ཧԼͷσΟϨΫτϦʹ • ઃఆөcapistranoΛ༻ͯ͠શʹσϓϩΠ • ϗετҰཡAPIܦ༝ͰऔಘͰ͖ΔΑ͏ʹ $cap dns:update
$cap dns:check $cap dns:reload ex: update DNS Record
Management Tools • LVSཧը໘ • MySQLͷԆࢹ • αʔϏεͷϦιʔεϞχλϦϯά
Conclusion • pixivΛࢧ͑Δج൫γεςϜʹ͍ͭͯհ • ͍͍͢πʔϧΛ͕͍͍ࣗͨͪ͢Α͏ʹ • ઑͬͨ͜ͱͤͣɺແཧͤͣӡ༻Ͱ͖Δঢ়ଶʹ ͍ͬͯ͘