Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
The Technology behind pixiv Infrastructure
Search
Harukasan
PRO
November 30, 2013
Technology
10
4k
The Technology behind pixiv Infrastructure
pixivのインフラを支える技術2013
at Python Developers Festa 2013.11
Harukasan
PRO
November 30, 2013
Tweet
Share
More Decks by Harukasan
See All by Harukasan
PicoRabbit: a Tiny Presentation Device Powered by Ruby
harukasan
PRO
2
360
pixivを支える技術 / 技育CAMPアカデミア
harukasan
PRO
3
490
20240401 新卒研修 - ピクシブにおける技術領域
harukasan
PRO
1
800
ピクシブのコンテンツ配信基盤技術 / pixiv TECH SALON
harukasan
PRO
5
5.6k
Goにおける画像ファイル処理 / golang.tokyo #19
harukasan
PRO
7
6.7k
WebRTC動画をトランスコードする / Transcoding video streams from WebRTC
harukasan
PRO
5
1.6k
ImageFluxを支えるリモート開発 / 20171202
harukasan
PRO
2
1.8k
YAPC::Fukuoka 前夜祭LT / Yet Another Pawoo Commit logs
harukasan
PRO
0
3k
YAPC::Fukuoka lunch session
harukasan
PRO
1
3.1k
Other Decks in Technology
See All in Technology
JavaのMCPサーバーで体験するAIエージェントの世界
tatsuya1bm
1
200
メルカリにおけるデータアナリティクス AI エージェント「Socrates」と ADK 活用事例
na0
14
6.6k
Digitization部 紹介資料
sansan33
PRO
1
4k
SwiftUI Transaction を徹底活用!ZOZOTOWN UI開発での活用事例
tsuzuki817
1
130
Oracle Cloud Infrastructure IaaS 新機能アップデート 2025/03 - 2025/05
oracle4engineer
PRO
1
110
DevOpsDays Taipei 2025 -- Creating Awesome Change in SmartNews!
martin_lover
0
200
うちの会社の評判は?SNSの投稿分析にAIを使ってみた
doumae
0
600
Information Architecture Recommoning: How Standardization Enables Differentiation
angioia
0
150
CSSの最新トレンド Ver.2025
tonkotsuboy_com
10
3.6k
データベースの引越しを Ora2Pg でスマートにやろう
jri_narita
0
170
Generational ZGCのメモリ運用改善 - その物理メモリ使用量、本当に正しい?
tabatad
0
260
ai bot got sick (abc 2025s version)
kojira
0
120
Featured
See All Featured
The Cult of Friendly URLs
andyhume
78
6.4k
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
irinanazarova
8
760
Art, The Web, and Tiny UX
lynnandtonic
298
21k
Docker and Python
trallard
44
3.4k
Why Our Code Smells
bkeepers
PRO
337
57k
The Illustrated Children's Guide to Kubernetes
chrisshort
48
50k
A better future with KSS
kneath
239
17k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
137
34k
Embracing the Ebb and Flow
colly
85
4.7k
Learning to Love Humans: Emotional Interface Design
aarron
273
40k
Fontdeck: Realign not Redesign
paulrobertlloyd
84
5.5k
Automating Front-end Workflow
addyosmani
1370
200k
Transcript
harukasan / Δ͔͞Μ at #pyfes 2013.11 ͷΠϯϑϥΛࢧ͑Δٕज़ 2013
Harukasan@pixiv a.k.a MICHII Shunsuke ಓҪढ़հ 2003 Kurume National College of
Tech. - NHK ROBOCON - ACM-ICPC 2008 Kyushu Inst. of Technology 2010 Tsukuba Univ. - Computational Vision Science 2012 pixiv Inc. - Infrastructure team
None
͜͜ʹpixivͷઆ໌͕ೖΔ
Server 400+ Traffic 10Gbps+ Team member 6 Monthly PV 3.7
Billion
Bases of pixiv Infrastructure Office IDCF DC Developments Testing Log
Analytics Small Services Main Applications DB Image Cluster New DC Image Cluster
ISP Backbone Office IDCF DC New DC 100M 1G 10G
1G 1G line 1G pixiv Network
ͷΠϯϑϥΛࢧ͑Δٕज़ 2013
http://www.slideshare.net/kamipo/pixiv @kamipo pixivͷΠϯϑϥΛࢧ͑Δٕज़ / ςΫηϛ2009
@cubicdaiya inside pixiv’s infrastructure / PHP Conference 2013 http://www.slideshare.net/cubicdaiya/inside-pixiv-infrastructure
Agenda pixiv Image Cluster Log Analysis Basis Management Tools
pixiv Image Cluster QJYJWͷը૾৴Ϋϥελʹ͍ͭͯ
pixiv Image Cluster • 2010͔Βӡ༻։࢝ • pixivͷϝΠϯίϯςϯπͰ͋ΔΠϥετΛ ߴʹॲཧ͢ΔͨΊʹ࠷దԽ • શτϥϑΟοΫͷ90%Ҏ্Λࡹ͍͍ͯΔ
Image Cluster nginx Front Cache DNS Round-Robin ATS Cache nginx
Dispatch nginx Front Cache nginx Front Cache nginx Front Cache ATS Cache ATS Cache ATS Cache Consistent Hashing nginx Dispatch nginx Dispatch nginx Dispatch Apache Origin nginx Thumbnail DNS Round-Robin i1.pixiv.net i2.pixiv.net SmallLight
Cache strategy • ϝϞϦͱσΟεΫͷ2ஈΩϟογϡߏ • τϥϑΟοΫ͕૿͑ΔʹͭΕͯ εΠονؒτϥϑΟοΫ͕ແࢹͰ͖ͳ͘ͳͬͨ • ωοτϫʔΫτϥϑΟοΫΛ͑ͭͭ Ωϟογϡ༰ྔͷ֬อ͢Δඞཁ
Cache strategy • ϝϞϦՁ֨ͷԼ • SSDͷՁ͕֨མ • ߴIOPSͷSSD͕ొ 2011 ¥40,000
2013 ¥20,000 256G READ WRITE PRICE ioDrive2 785GB MLC*1 215,000/230,000 IOPS I don’t know Intel 910 800GB 100,000/ 75,000 IOPS ¥400,000 SSD 256GBx3 RAID0 80,000/ 50,000 IOPS ¥60,000 *1: ioDrive2ެশ Intel 910ɺ SSD RAID0ʹ͍ͭͯfioʹΑΔଌఆ 16G ECC RDIMM ¥20,000
Cache strategy ound-Robin nginx Front Cache nginx Front Cache nginx
Front Cache ATS Cache ATS Cache ATS Cache Consistent Hashing nginx Dispatch nginx Dispatch nginx Dispatch Apache Origin nginx Thumbnail DNS Round-Robin i1.pixiv.net i2.pixiv.net 64GB Memory - nginx cache on tmpfs - cache hit rate: 50% - reduce network traffic 256GB SSD x3 RAID0 - Apache Traffic Server (standalone) - cache hit rate: 80-90% SSD SSD HDD Original & BIG Thumb. Small Thumbnails
Aggregate image domains • ը૾αʔόϢʔβʔIDϕʔεͰࢄ img01.pixiv.net - img1XX.pixiv.net • 1ϖʔδͰ40-60ճDNSϦΫΤετ͕ൃੜ
Ոఉ༻ϧʔλDNSղܾ͕Ͱ͖ͳ͘ͳΔ • શը૾ͷURIΛมߋͯ͠ରԠ OLD: http://img01.pixiv.net/img/****/*****.jpg NEW: http://i1.pixiv.net/img01/img/****/*****.jpg
New Image Store ৽͍͠ը૾ετϨʔδํࣜʹ͍ͭͯ
New image store • ࡞+IDϕʔεͷγʔέϯγϟϧͳURI • 1ॻ͖ࠐΜͩϑΝΠϧRead Only • ࠶ߘॲཧ࡞Λߋ৽
• ngx_lua/OpenRestyΛ༻͍ͨཧআ
Logical Delete ! Kyototycoon ngx_lua / nginx null 404 /img02.png
403 /img03.png 404 /img05.png 404 /img08.png 403 " " GET /img01.png GET /img03.png 404
Logical Delete local memcached = require "resty.memcached" local uri =
ngx.var.request_uri local memc = memcached:new() . . . local val, flags, err = memc:get(request_uri) if val and val ~= "200" then exit(tonumber(val)) end logical_delete.lua location / { access_by_lua_file logical_delete.lua; }
Log Analysis Basis QJYJWͷϩάղੳج൫ʹ͍ͭͯ
Log Analysis Basis PHP Application MySQL/neoagent Front server - Error
Log - Login Log - Activity Log - Slow Query - Access Log MongoDB Elasticsearch Fluentd File System
Error Viewer
Slow Query Viewer
Kibana 3
Output Log • JSONΛॻ͖ग़ͯ͠Fluentd͕tail͢ΔΈ • ϓϩηε͕େྔʹىಈͯ͠εϧʔϓοτ͕མͪͳ͍ PHP Exception Handler Logger::write($type,
$data) # JSON Fluentd in_tail
Fluentd config <source> type tail path /var/tmp/log/activity.log pos_file /var/tmp/fluentd/activity.pos tag
activity format json # JSONܗࣜΛಡΈࠐΉ </source> <match activity> type forward_with_hostname # HostΛೖΕͯforward flush_interval 1s # ඞͣ1ඵ ! buffer_type file # buffer typefile/࠶ىಈͯ͠ফ͑ͳ͍ buffer_path /var/tmp/fluentd/buffer/activity.*.buffer buffer_chunk_limit 2m # chunkαΠζখ͞Ί buffer_queue_limit 128 # Ͳͷ͘Β͍ফ͑ͪΌ͍͚ͳ͍͔ … </match>
Management tools ཧܥɺࢹܥπʔϧʹ͍ͭͯ
Monitoring servers/services • ͘͘͝͝ҰൠతͳࢹϓϩμΫτΛ͍ͬͯΔ Ϧιʔεάϥϑ ϗετ/αʔϏεࢹ ϗετϓϩηεࢹ εΫϦϓτ Munin Nagios
Monit Cron
Cluster Admin ϋʔυΣΞใ ϗετͷ༻్ ࢹঢ়ଶ
Capistrano/Subversion • /etc/ҎԼͷઃఆϑΝΠϧ͕ͦͷ··subversionͷ ཧԼͷσΟϨΫτϦʹ • ઃఆөcapistranoΛ༻ͯ͠શʹσϓϩΠ • ϗετҰཡAPIܦ༝ͰऔಘͰ͖ΔΑ͏ʹ $cap dns:update
$cap dns:check $cap dns:reload ex: update DNS Record
Management Tools • LVSཧը໘ • MySQLͷԆࢹ • αʔϏεͷϦιʔεϞχλϦϯά
Conclusion • pixivΛࢧ͑Δج൫γεςϜʹ͍ͭͯհ • ͍͍͢πʔϧΛ͕͍͍ࣗͨͪ͢Α͏ʹ • ઑͬͨ͜ͱͤͣɺແཧͤͣӡ༻Ͱ͖Δঢ়ଶʹ ͍ͬͯ͘