Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
The Technology behind pixiv Infrastructure
Search
Harukasan
PRO
November 30, 2013
Technology
10
4k
The Technology behind pixiv Infrastructure
pixivのインフラを支える技術2013
at Python Developers Festa 2013.11
Harukasan
PRO
November 30, 2013
Tweet
Share
More Decks by Harukasan
See All by Harukasan
pixivを支える技術 / 技育CAMPアカデミア
harukasan
PRO
3
430
20240401 新卒研修 - ピクシブにおける技術領域
harukasan
PRO
1
740
ピクシブのコンテンツ配信基盤技術 / pixiv TECH SALON
harukasan
PRO
5
5.5k
Goにおける画像ファイル処理 / golang.tokyo #19
harukasan
PRO
7
6.6k
WebRTC動画をトランスコードする / Transcoding video streams from WebRTC
harukasan
PRO
5
1.5k
ImageFluxを支えるリモート開発 / 20171202
harukasan
PRO
2
1.8k
YAPC::Fukuoka 前夜祭LT / Yet Another Pawoo Commit logs
harukasan
PRO
0
2.9k
YAPC::Fukuoka lunch session
harukasan
PRO
1
3k
マストドン会議: Pawoo / Mastodon Kaigi2
harukasan
PRO
2
460
Other Decks in Technology
See All in Technology
2.5Dモデルのすべて
yu4u
2
940
プロダクトエンジニア構想を立ち上げ、プロダクト志向な組織への成長を続けている話 / grow into a product-oriented organization
hiro_torii
1
300
Helm , Kustomize に代わる !? 次世代 k8s パッケージマネージャー Glasskube 入門 / glasskube-entry
parupappa2929
0
270
Apache Iceberg Case Study in LY Corporation
lycorptech_jp
PRO
0
140
ユーザーストーリーマッピングから始めるアジャイルチームと並走するQA / Starting QA with User Story Mapping
katawara
0
260
現場で役立つAPIデザイン
nagix
35
13k
ローカルLLMを活用したコード生成と、ローコード開発ツールへの応用
kazuhitoyokoi
0
140
転生CISOサバイバル・ガイド / CISO Career Transition Survival Guide
kanny
3
1.1k
コンピュータビジョンの社会実装について考えていたらゲームを作っていた話
takmin
1
480
ホワイトボードチャレンジ 説明&実行資料
ichimichi
0
130
Swiftの “private” を テストする / Testing Swift "private"
yutailang0119
0
140
The Future of SEO: The Impact of AI on Search
badams
0
240
Featured
See All Featured
Building Your Own Lightsaber
phodgson
104
6.2k
How GitHub (no longer) Works
holman
314
140k
Fashionably flexible responsive web design (full day workshop)
malarkey
406
66k
Improving Core Web Vitals using Speculation Rules API
sergeychernyshev
9
500
Side Projects
sachag
452
42k
RailsConf 2023
tenderlove
29
1k
Building Applications with DynamoDB
mza
93
6.2k
Building a Modern Day E-commerce SEO Strategy
aleyda
38
7.1k
[RailsConf 2023] Rails as a piece of cake
palkan
53
5.3k
We Have a Design System, Now What?
morganepeng
51
7.4k
Bash Introduction
62gerente
611
210k
For a Future-Friendly Web
brad_frost
176
9.5k
Transcript
harukasan / Δ͔͞Μ at #pyfes 2013.11 ͷΠϯϑϥΛࢧ͑Δٕज़ 2013
Harukasan@pixiv a.k.a MICHII Shunsuke ಓҪढ़հ 2003 Kurume National College of
Tech. - NHK ROBOCON - ACM-ICPC 2008 Kyushu Inst. of Technology 2010 Tsukuba Univ. - Computational Vision Science 2012 pixiv Inc. - Infrastructure team
None
͜͜ʹpixivͷઆ໌͕ೖΔ
Server 400+ Traffic 10Gbps+ Team member 6 Monthly PV 3.7
Billion
Bases of pixiv Infrastructure Office IDCF DC Developments Testing Log
Analytics Small Services Main Applications DB Image Cluster New DC Image Cluster
ISP Backbone Office IDCF DC New DC 100M 1G 10G
1G 1G line 1G pixiv Network
ͷΠϯϑϥΛࢧ͑Δٕज़ 2013
http://www.slideshare.net/kamipo/pixiv @kamipo pixivͷΠϯϑϥΛࢧ͑Δٕज़ / ςΫηϛ2009
@cubicdaiya inside pixiv’s infrastructure / PHP Conference 2013 http://www.slideshare.net/cubicdaiya/inside-pixiv-infrastructure
Agenda pixiv Image Cluster Log Analysis Basis Management Tools
pixiv Image Cluster QJYJWͷը૾৴Ϋϥελʹ͍ͭͯ
pixiv Image Cluster • 2010͔Βӡ༻։࢝ • pixivͷϝΠϯίϯςϯπͰ͋ΔΠϥετΛ ߴʹॲཧ͢ΔͨΊʹ࠷దԽ • શτϥϑΟοΫͷ90%Ҏ্Λࡹ͍͍ͯΔ
Image Cluster nginx Front Cache DNS Round-Robin ATS Cache nginx
Dispatch nginx Front Cache nginx Front Cache nginx Front Cache ATS Cache ATS Cache ATS Cache Consistent Hashing nginx Dispatch nginx Dispatch nginx Dispatch Apache Origin nginx Thumbnail DNS Round-Robin i1.pixiv.net i2.pixiv.net SmallLight
Cache strategy • ϝϞϦͱσΟεΫͷ2ஈΩϟογϡߏ • τϥϑΟοΫ͕૿͑ΔʹͭΕͯ εΠονؒτϥϑΟοΫ͕ແࢹͰ͖ͳ͘ͳͬͨ • ωοτϫʔΫτϥϑΟοΫΛ͑ͭͭ Ωϟογϡ༰ྔͷ֬อ͢Δඞཁ
Cache strategy • ϝϞϦՁ֨ͷԼ • SSDͷՁ͕֨མ • ߴIOPSͷSSD͕ొ 2011 ¥40,000
2013 ¥20,000 256G READ WRITE PRICE ioDrive2 785GB MLC*1 215,000/230,000 IOPS I don’t know Intel 910 800GB 100,000/ 75,000 IOPS ¥400,000 SSD 256GBx3 RAID0 80,000/ 50,000 IOPS ¥60,000 *1: ioDrive2ެশ Intel 910ɺ SSD RAID0ʹ͍ͭͯfioʹΑΔଌఆ 16G ECC RDIMM ¥20,000
Cache strategy ound-Robin nginx Front Cache nginx Front Cache nginx
Front Cache ATS Cache ATS Cache ATS Cache Consistent Hashing nginx Dispatch nginx Dispatch nginx Dispatch Apache Origin nginx Thumbnail DNS Round-Robin i1.pixiv.net i2.pixiv.net 64GB Memory - nginx cache on tmpfs - cache hit rate: 50% - reduce network traffic 256GB SSD x3 RAID0 - Apache Traffic Server (standalone) - cache hit rate: 80-90% SSD SSD HDD Original & BIG Thumb. Small Thumbnails
Aggregate image domains • ը૾αʔόϢʔβʔIDϕʔεͰࢄ img01.pixiv.net - img1XX.pixiv.net • 1ϖʔδͰ40-60ճDNSϦΫΤετ͕ൃੜ
Ոఉ༻ϧʔλDNSղܾ͕Ͱ͖ͳ͘ͳΔ • શը૾ͷURIΛมߋͯ͠ରԠ OLD: http://img01.pixiv.net/img/****/*****.jpg NEW: http://i1.pixiv.net/img01/img/****/*****.jpg
New Image Store ৽͍͠ը૾ετϨʔδํࣜʹ͍ͭͯ
New image store • ࡞+IDϕʔεͷγʔέϯγϟϧͳURI • 1ॻ͖ࠐΜͩϑΝΠϧRead Only • ࠶ߘॲཧ࡞Λߋ৽
• ngx_lua/OpenRestyΛ༻͍ͨཧআ
Logical Delete ! Kyototycoon ngx_lua / nginx null 404 /img02.png
403 /img03.png 404 /img05.png 404 /img08.png 403 " " GET /img01.png GET /img03.png 404
Logical Delete local memcached = require "resty.memcached" local uri =
ngx.var.request_uri local memc = memcached:new() . . . local val, flags, err = memc:get(request_uri) if val and val ~= "200" then exit(tonumber(val)) end logical_delete.lua location / { access_by_lua_file logical_delete.lua; }
Log Analysis Basis QJYJWͷϩάղੳج൫ʹ͍ͭͯ
Log Analysis Basis PHP Application MySQL/neoagent Front server - Error
Log - Login Log - Activity Log - Slow Query - Access Log MongoDB Elasticsearch Fluentd File System
Error Viewer
Slow Query Viewer
Kibana 3
Output Log • JSONΛॻ͖ग़ͯ͠Fluentd͕tail͢ΔΈ • ϓϩηε͕େྔʹىಈͯ͠εϧʔϓοτ͕མͪͳ͍ PHP Exception Handler Logger::write($type,
$data) # JSON Fluentd in_tail
Fluentd config <source> type tail path /var/tmp/log/activity.log pos_file /var/tmp/fluentd/activity.pos tag
activity format json # JSONܗࣜΛಡΈࠐΉ </source> <match activity> type forward_with_hostname # HostΛೖΕͯforward flush_interval 1s # ඞͣ1ඵ ! buffer_type file # buffer typefile/࠶ىಈͯ͠ফ͑ͳ͍ buffer_path /var/tmp/fluentd/buffer/activity.*.buffer buffer_chunk_limit 2m # chunkαΠζখ͞Ί buffer_queue_limit 128 # Ͳͷ͘Β͍ফ͑ͪΌ͍͚ͳ͍͔ … </match>
Management tools ཧܥɺࢹܥπʔϧʹ͍ͭͯ
Monitoring servers/services • ͘͘͝͝ҰൠతͳࢹϓϩμΫτΛ͍ͬͯΔ Ϧιʔεάϥϑ ϗετ/αʔϏεࢹ ϗετϓϩηεࢹ εΫϦϓτ Munin Nagios
Monit Cron
Cluster Admin ϋʔυΣΞใ ϗετͷ༻్ ࢹঢ়ଶ
Capistrano/Subversion • /etc/ҎԼͷઃఆϑΝΠϧ͕ͦͷ··subversionͷ ཧԼͷσΟϨΫτϦʹ • ઃఆөcapistranoΛ༻ͯ͠શʹσϓϩΠ • ϗετҰཡAPIܦ༝ͰऔಘͰ͖ΔΑ͏ʹ $cap dns:update
$cap dns:check $cap dns:reload ex: update DNS Record
Management Tools • LVSཧը໘ • MySQLͷԆࢹ • αʔϏεͷϦιʔεϞχλϦϯά
Conclusion • pixivΛࢧ͑Δج൫γεςϜʹ͍ͭͯհ • ͍͍͢πʔϧΛ͕͍͍ࣗͨͪ͢Α͏ʹ • ઑͬͨ͜ͱͤͣɺແཧͤͣӡ༻Ͱ͖Δঢ়ଶʹ ͍ͬͯ͘