Upgrade to Pro — share decks privately, control downloads, hide ads and more …

The Technology behind pixiv Infrastructure

Harukasan
November 30, 2013

The Technology behind pixiv Infrastructure

pixivのインフラを支える技術2013
at Python Developers Festa 2013.11

Harukasan

November 30, 2013
Tweet

More Decks by Harukasan

Other Decks in Technology

Transcript

  1. Harukasan@pixiv a.k.a MICHII Shunsuke ಓҪढ़հ 2003 Kurume National College of

    Tech.
 - NHK ROBOCON
 - ACM-ICPC 2008 Kyushu Inst. of Technology 2010 Tsukuba Univ.
 - Computational Vision Science 2012 pixiv Inc.
 - Infrastructure team
  2. Bases of pixiv Infrastructure Office IDCF DC Developments Testing Log

    Analytics Small Services Main Applications DB Image Cluster New DC Image Cluster
  3. Image Cluster nginx Front Cache DNS Round-Robin ATS Cache nginx

    Dispatch nginx Front Cache nginx Front Cache nginx Front Cache ATS Cache ATS Cache ATS Cache Consistent Hashing nginx Dispatch nginx Dispatch nginx Dispatch Apache Origin nginx Thumbnail DNS Round-Robin i1.pixiv.net i2.pixiv.net SmallLight
  4. Cache strategy • ϝϞϦՁ֨ͷ௿Լ • SSDͷՁ͕֨๫མ • ߴIOPSͷSSD͕ొ৔ 2011 ¥40,000


    2013 ¥20,000 256G READ WRITE PRICE ioDrive2 785GB MLC*1 215,000/230,000 IOPS I don’t know Intel 910 800GB 100,000/ 75,000 IOPS ¥400,000 SSD 256GBx3 RAID0 80,000/ 50,000 IOPS ¥60,000 *1: ioDrive2͸ެশ஋ Intel 910ɺ SSD RAID0ʹ͍ͭͯ͸fioʹΑΔଌఆ 16G ECC RDIMM ¥20,000
  5. Cache strategy ound-Robin nginx Front Cache nginx Front Cache nginx

    Front Cache ATS Cache ATS Cache ATS Cache Consistent Hashing nginx Dispatch nginx Dispatch nginx Dispatch Apache Origin nginx Thumbnail DNS Round-Robin i1.pixiv.net i2.pixiv.net 64GB Memory - nginx cache on tmpfs - cache hit rate: 50% - reduce network traffic 256GB SSD x3 RAID0 - Apache Traffic Server (standalone) - cache hit rate: 80-90% SSD SSD HDD Original & BIG Thumb. Small Thumbnails
  6. Aggregate image domains • ը૾αʔό͸ϢʔβʔIDϕʔεͰ෼ࢄ
 img01.pixiv.net - img1XX.pixiv.net • 1ϖʔδͰ40-60ճDNSϦΫΤετ͕ൃੜ


    ‎Ոఉ༻ϧʔλ͸DNSղܾ͕Ͱ͖ͳ͘ͳΔ • શը૾ͷURIΛมߋͯ͠ରԠ OLD: http://img01.pixiv.net/img/****/*****.jpg NEW: http://i1.pixiv.net/img01/img/****/*****.jpg
  7. Logical Delete ! Kyototycoon ngx_lua / nginx null 404 /img02.png

    403 /img03.png 404 /img05.png 404 /img08.png 403 " " GET /img01.png GET /img03.png 404
  8. Logical Delete local memcached = require "resty.memcached" local uri =

    ngx.var.request_uri local memc = memcached:new() . . . local val, flags, err = memc:get(request_uri) if val and val ~= "200" then exit(tonumber(val)) end logical_delete.lua location / { access_by_lua_file logical_delete.lua; }
  9. Log Analysis Basis PHP Application MySQL/neoagent Front server - Error

    Log - Login Log - Activity Log - Slow Query - Access Log MongoDB Elasticsearch Fluentd File System
  10. Fluentd config <source> type tail path /var/tmp/log/activity.log pos_file /var/tmp/fluentd/activity.pos tag

    activity format json # JSONܗࣜΛಡΈࠐΉ </source> <match activity> type forward_with_hostname # HostΛೖΕͯforward flush_interval 1s # ඞͣ1ඵ ! buffer_type file # buffer type͸file/࠶ىಈͯ͠΋ফ͑ͳ͍ buffer_path /var/tmp/fluentd/buffer/activity.*.buffer buffer_chunk_limit 2m # chunkαΠζ͸খ͞Ί buffer_queue_limit 128 # Ͳͷ͘Β͍ফ͑ͪΌ͍͚ͳ͍͔ … </match>