Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
The Technology behind pixiv Infrastructure
Search
Harukasan
PRO
November 30, 2013
Technology
10
4k
The Technology behind pixiv Infrastructure
pixivのインフラを支える技術2013
at Python Developers Festa 2013.11
Harukasan
PRO
November 30, 2013
Tweet
Share
More Decks by Harukasan
See All by Harukasan
pixivを支える技術 / 技育CAMPアカデミア
harukasan
PRO
3
430
20240401 新卒研修 - ピクシブにおける技術領域
harukasan
PRO
1
740
ピクシブのコンテンツ配信基盤技術 / pixiv TECH SALON
harukasan
PRO
5
5.5k
Goにおける画像ファイル処理 / golang.tokyo #19
harukasan
PRO
7
6.6k
WebRTC動画をトランスコードする / Transcoding video streams from WebRTC
harukasan
PRO
5
1.5k
ImageFluxを支えるリモート開発 / 20171202
harukasan
PRO
2
1.8k
YAPC::Fukuoka 前夜祭LT / Yet Another Pawoo Commit logs
harukasan
PRO
0
2.9k
YAPC::Fukuoka lunch session
harukasan
PRO
1
3k
マストドン会議: Pawoo / Mastodon Kaigi2
harukasan
PRO
2
460
Other Decks in Technology
See All in Technology
クラウドサービス事業者におけるOSS
tagomoris
1
770
リーダブルテストコード 〜メンテナンスしやすい テストコードを作成する方法を考える〜 #DevSumi #DevSumiB / Readable test code
nihonbuson
11
7.2k
PHPで印刷所に入稿できる名札データを作る / Generating Print-Ready Name Tag Data with PHP
tomzoh
0
100
モノレポ開発のエラー、誰が見る?Datadog で実現する適切なトリアージとエスカレーション
biwashi
6
810
The Future of SEO: The Impact of AI on Search
badams
0
200
抽象化をするということ - 具体と抽象の往復を身につける / Abstraction and concretization
soudai
16
5.9k
Helm , Kustomize に代わる !? 次世代 k8s パッケージマネージャー Glasskube 入門 / glasskube-entry
parupappa2929
0
250
「海外登壇」という 選択肢を与えるために 〜Gophers EX
logica0419
0
700
Data-centric AI入門第6章:Data-centric AIの実践例
x_ttyszk
1
400
関東Kaggler会LT: 人狼コンペとLLM量子化について
nejumi
3
580
SA Night #2 FinatextのSA思想/SA Night #2 Finatext session
satoshiimai
1
140
2024.02.19 W&B AIエージェントLT会 / AIエージェントが業務を代行するための計画と実行 / Algomatic 宮脇
smiyawaki0820
13
3.4k
Featured
See All Featured
It's Worth the Effort
3n
184
28k
Fireside Chat
paigeccino
34
3.2k
Keith and Marios Guide to Fast Websites
keithpitt
411
22k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
133
33k
RailsConf 2023
tenderlove
29
1k
Navigating Team Friction
lara
183
15k
Making the Leap to Tech Lead
cromwellryan
133
9.1k
Gamification - CAS2011
davidbonilla
80
5.1k
Imperfection Machines: The Place of Print at Facebook
scottboms
267
13k
Writing Fast Ruby
sferik
628
61k
Principles of Awesome APIs and How to Build Them.
keavy
126
17k
A Tale of Four Properties
chriscoyier
158
23k
Transcript
harukasan / Δ͔͞Μ at #pyfes 2013.11 ͷΠϯϑϥΛࢧ͑Δٕज़ 2013
Harukasan@pixiv a.k.a MICHII Shunsuke ಓҪढ़հ 2003 Kurume National College of
Tech. - NHK ROBOCON - ACM-ICPC 2008 Kyushu Inst. of Technology 2010 Tsukuba Univ. - Computational Vision Science 2012 pixiv Inc. - Infrastructure team
None
͜͜ʹpixivͷઆ໌͕ೖΔ
Server 400+ Traffic 10Gbps+ Team member 6 Monthly PV 3.7
Billion
Bases of pixiv Infrastructure Office IDCF DC Developments Testing Log
Analytics Small Services Main Applications DB Image Cluster New DC Image Cluster
ISP Backbone Office IDCF DC New DC 100M 1G 10G
1G 1G line 1G pixiv Network
ͷΠϯϑϥΛࢧ͑Δٕज़ 2013
http://www.slideshare.net/kamipo/pixiv @kamipo pixivͷΠϯϑϥΛࢧ͑Δٕज़ / ςΫηϛ2009
@cubicdaiya inside pixiv’s infrastructure / PHP Conference 2013 http://www.slideshare.net/cubicdaiya/inside-pixiv-infrastructure
Agenda pixiv Image Cluster Log Analysis Basis Management Tools
pixiv Image Cluster QJYJWͷը૾৴Ϋϥελʹ͍ͭͯ
pixiv Image Cluster • 2010͔Βӡ༻։࢝ • pixivͷϝΠϯίϯςϯπͰ͋ΔΠϥετΛ ߴʹॲཧ͢ΔͨΊʹ࠷దԽ • શτϥϑΟοΫͷ90%Ҏ্Λࡹ͍͍ͯΔ
Image Cluster nginx Front Cache DNS Round-Robin ATS Cache nginx
Dispatch nginx Front Cache nginx Front Cache nginx Front Cache ATS Cache ATS Cache ATS Cache Consistent Hashing nginx Dispatch nginx Dispatch nginx Dispatch Apache Origin nginx Thumbnail DNS Round-Robin i1.pixiv.net i2.pixiv.net SmallLight
Cache strategy • ϝϞϦͱσΟεΫͷ2ஈΩϟογϡߏ • τϥϑΟοΫ͕૿͑ΔʹͭΕͯ εΠονؒτϥϑΟοΫ͕ແࢹͰ͖ͳ͘ͳͬͨ • ωοτϫʔΫτϥϑΟοΫΛ͑ͭͭ Ωϟογϡ༰ྔͷ֬อ͢Δඞཁ
Cache strategy • ϝϞϦՁ֨ͷԼ • SSDͷՁ͕֨མ • ߴIOPSͷSSD͕ొ 2011 ¥40,000
2013 ¥20,000 256G READ WRITE PRICE ioDrive2 785GB MLC*1 215,000/230,000 IOPS I don’t know Intel 910 800GB 100,000/ 75,000 IOPS ¥400,000 SSD 256GBx3 RAID0 80,000/ 50,000 IOPS ¥60,000 *1: ioDrive2ެশ Intel 910ɺ SSD RAID0ʹ͍ͭͯfioʹΑΔଌఆ 16G ECC RDIMM ¥20,000
Cache strategy ound-Robin nginx Front Cache nginx Front Cache nginx
Front Cache ATS Cache ATS Cache ATS Cache Consistent Hashing nginx Dispatch nginx Dispatch nginx Dispatch Apache Origin nginx Thumbnail DNS Round-Robin i1.pixiv.net i2.pixiv.net 64GB Memory - nginx cache on tmpfs - cache hit rate: 50% - reduce network traffic 256GB SSD x3 RAID0 - Apache Traffic Server (standalone) - cache hit rate: 80-90% SSD SSD HDD Original & BIG Thumb. Small Thumbnails
Aggregate image domains • ը૾αʔόϢʔβʔIDϕʔεͰࢄ img01.pixiv.net - img1XX.pixiv.net • 1ϖʔδͰ40-60ճDNSϦΫΤετ͕ൃੜ
Ոఉ༻ϧʔλDNSղܾ͕Ͱ͖ͳ͘ͳΔ • શը૾ͷURIΛมߋͯ͠ରԠ OLD: http://img01.pixiv.net/img/****/*****.jpg NEW: http://i1.pixiv.net/img01/img/****/*****.jpg
New Image Store ৽͍͠ը૾ετϨʔδํࣜʹ͍ͭͯ
New image store • ࡞+IDϕʔεͷγʔέϯγϟϧͳURI • 1ॻ͖ࠐΜͩϑΝΠϧRead Only • ࠶ߘॲཧ࡞Λߋ৽
• ngx_lua/OpenRestyΛ༻͍ͨཧআ
Logical Delete ! Kyototycoon ngx_lua / nginx null 404 /img02.png
403 /img03.png 404 /img05.png 404 /img08.png 403 " " GET /img01.png GET /img03.png 404
Logical Delete local memcached = require "resty.memcached" local uri =
ngx.var.request_uri local memc = memcached:new() . . . local val, flags, err = memc:get(request_uri) if val and val ~= "200" then exit(tonumber(val)) end logical_delete.lua location / { access_by_lua_file logical_delete.lua; }
Log Analysis Basis QJYJWͷϩάղੳج൫ʹ͍ͭͯ
Log Analysis Basis PHP Application MySQL/neoagent Front server - Error
Log - Login Log - Activity Log - Slow Query - Access Log MongoDB Elasticsearch Fluentd File System
Error Viewer
Slow Query Viewer
Kibana 3
Output Log • JSONΛॻ͖ग़ͯ͠Fluentd͕tail͢ΔΈ • ϓϩηε͕େྔʹىಈͯ͠εϧʔϓοτ͕མͪͳ͍ PHP Exception Handler Logger::write($type,
$data) # JSON Fluentd in_tail
Fluentd config <source> type tail path /var/tmp/log/activity.log pos_file /var/tmp/fluentd/activity.pos tag
activity format json # JSONܗࣜΛಡΈࠐΉ </source> <match activity> type forward_with_hostname # HostΛೖΕͯforward flush_interval 1s # ඞͣ1ඵ ! buffer_type file # buffer typefile/࠶ىಈͯ͠ফ͑ͳ͍ buffer_path /var/tmp/fluentd/buffer/activity.*.buffer buffer_chunk_limit 2m # chunkαΠζখ͞Ί buffer_queue_limit 128 # Ͳͷ͘Β͍ফ͑ͪΌ͍͚ͳ͍͔ … </match>
Management tools ཧܥɺࢹܥπʔϧʹ͍ͭͯ
Monitoring servers/services • ͘͘͝͝ҰൠతͳࢹϓϩμΫτΛ͍ͬͯΔ Ϧιʔεάϥϑ ϗετ/αʔϏεࢹ ϗετϓϩηεࢹ εΫϦϓτ Munin Nagios
Monit Cron
Cluster Admin ϋʔυΣΞใ ϗετͷ༻్ ࢹঢ়ଶ
Capistrano/Subversion • /etc/ҎԼͷઃఆϑΝΠϧ͕ͦͷ··subversionͷ ཧԼͷσΟϨΫτϦʹ • ઃఆөcapistranoΛ༻ͯ͠શʹσϓϩΠ • ϗετҰཡAPIܦ༝ͰऔಘͰ͖ΔΑ͏ʹ $cap dns:update
$cap dns:check $cap dns:reload ex: update DNS Record
Management Tools • LVSཧը໘ • MySQLͷԆࢹ • αʔϏεͷϦιʔεϞχλϦϯά
Conclusion • pixivΛࢧ͑Δج൫γεςϜʹ͍ͭͯհ • ͍͍͢πʔϧΛ͕͍͍ࣗͨͪ͢Α͏ʹ • ઑͬͨ͜ͱͤͣɺແཧͤͣӡ༻Ͱ͖Δঢ়ଶʹ ͍ͬͯ͘