The Technology behind pixiv Infrastructure

124da56a613b15fa980427533e4e3839?s=47 Harukasan
November 30, 2013

The Technology behind pixiv Infrastructure

pixivのインフラを支える技術2013
at Python Developers Festa 2013.11

124da56a613b15fa980427533e4e3839?s=128

Harukasan

November 30, 2013
Tweet

Transcript

  1. harukasan / ͸Δ͔͞Μ at #pyfes 2013.11 ͷΠϯϑϥΛࢧ͑Δٕज़ 2013

  2. Harukasan@pixiv a.k.a MICHII Shunsuke ಓҪढ़հ 2003 Kurume National College of

    Tech.
 - NHK ROBOCON
 - ACM-ICPC 2008 Kyushu Inst. of Technology 2010 Tsukuba Univ.
 - Computational Vision Science 2012 pixiv Inc.
 - Infrastructure team
  3. None
  4. ͜͜ʹpixivͷઆ໌͕ೖΔ

  5. Server 400+ Traffic 10Gbps+ Team member 6 Monthly PV 3.7

    Billion
  6. Bases of pixiv Infrastructure Office IDCF DC Developments Testing Log

    Analytics Small Services Main Applications DB Image Cluster New DC Image Cluster
  7. ISP Backbone Office IDCF DC New DC 100M 1G 10G

    1G 1G line 1G pixiv Network
  8. ͷΠϯϑϥΛࢧ͑Δٕज़ 2013

  9. http://www.slideshare.net/kamipo/pixiv @kamipo pixivͷΠϯϑϥΛࢧ͑Δٕज़ / ςΫηϛ2009

  10. @cubicdaiya inside pixiv’s infrastructure / PHP Conference 2013 http://www.slideshare.net/cubicdaiya/inside-pixiv-infrastructure

  11. Agenda pixiv Image Cluster Log Analysis Basis Management Tools

  12. pixiv Image Cluster QJYJWͷը૾഑৴Ϋϥελʹ͍ͭͯ

  13. pixiv Image Cluster • 2010೥͔Βӡ༻։࢝ • pixivͷϝΠϯίϯςϯπͰ͋ΔΠϥετΛ
 ௒ߴ଎ʹॲཧ͢ΔͨΊʹ࠷దԽ • શτϥϑΟοΫͷ90%Ҏ্Λࡹ͍͍ͯΔ

  14. Image Cluster nginx Front Cache DNS Round-Robin ATS Cache nginx

    Dispatch nginx Front Cache nginx Front Cache nginx Front Cache ATS Cache ATS Cache ATS Cache Consistent Hashing nginx Dispatch nginx Dispatch nginx Dispatch Apache Origin nginx Thumbnail DNS Round-Robin i1.pixiv.net i2.pixiv.net SmallLight
  15. Cache strategy • ϝϞϦͱσΟεΫͷ2ஈΩϟογϡߏ੒ • τϥϑΟοΫ͕૿͑ΔʹͭΕͯ
 εΠονؒτϥϑΟοΫ͕ແࢹͰ͖ͳ͘ͳͬͨ • ωοτϫʔΫτϥϑΟοΫΛ཈͑ͭͭ
 Ωϟογϡ༰ྔͷ֬อ͢Δඞཁ

  16. Cache strategy • ϝϞϦՁ֨ͷ௿Լ • SSDͷՁ͕֨๫མ • ߴIOPSͷSSD͕ొ৔ 2011 ¥40,000


    2013 ¥20,000 256G READ WRITE PRICE ioDrive2 785GB MLC*1 215,000/230,000 IOPS I don’t know Intel 910 800GB 100,000/ 75,000 IOPS ¥400,000 SSD 256GBx3 RAID0 80,000/ 50,000 IOPS ¥60,000 *1: ioDrive2͸ެশ஋ Intel 910ɺ SSD RAID0ʹ͍ͭͯ͸fioʹΑΔଌఆ 16G ECC RDIMM ¥20,000
  17. Cache strategy ound-Robin nginx Front Cache nginx Front Cache nginx

    Front Cache ATS Cache ATS Cache ATS Cache Consistent Hashing nginx Dispatch nginx Dispatch nginx Dispatch Apache Origin nginx Thumbnail DNS Round-Robin i1.pixiv.net i2.pixiv.net 64GB Memory - nginx cache on tmpfs - cache hit rate: 50% - reduce network traffic 256GB SSD x3 RAID0 - Apache Traffic Server (standalone) - cache hit rate: 80-90% SSD SSD HDD Original & BIG Thumb. Small Thumbnails
  18. Aggregate image domains • ը૾αʔό͸ϢʔβʔIDϕʔεͰ෼ࢄ
 img01.pixiv.net - img1XX.pixiv.net • 1ϖʔδͰ40-60ճDNSϦΫΤετ͕ൃੜ


    ‎Ոఉ༻ϧʔλ͸DNSղܾ͕Ͱ͖ͳ͘ͳΔ • શը૾ͷURIΛมߋͯ͠ରԠ OLD: http://img01.pixiv.net/img/****/*****.jpg NEW: http://i1.pixiv.net/img01/img/****/*****.jpg
  19. New Image Store ৽͍͠ը૾ετϨʔδํࣜʹ͍ͭͯ

  20. New image store • ࡞੒೔+IDϕʔεͷγʔέϯγϟϧͳURI • 1౓ॻ͖ࠐΜͩϑΝΠϧ͸Read Only • ࠶౤ߘॲཧ͸࡞੒೔Λߋ৽

    • ngx_lua/OpenRestyΛ༻͍ͨ࿦ཧ࡟আ
  21. Logical Delete ! Kyototycoon ngx_lua / nginx null 404 /img02.png

    403 /img03.png 404 /img05.png 404 /img08.png 403 " " GET /img01.png GET /img03.png 404
  22. Logical Delete local memcached = require "resty.memcached" local uri =

    ngx.var.request_uri local memc = memcached:new() . . . local val, flags, err = memc:get(request_uri) if val and val ~= "200" then exit(tonumber(val)) end logical_delete.lua location / { access_by_lua_file logical_delete.lua; }
  23. Log Analysis Basis QJYJWͷϩάղੳج൫ʹ͍ͭͯ

  24. Log Analysis Basis PHP Application MySQL/neoagent Front server - Error

    Log - Login Log - Activity Log - Slow Query - Access Log MongoDB Elasticsearch Fluentd File System
  25. Error Viewer

  26. Slow Query Viewer

  27. Kibana 3

  28. Output Log • JSONΛॻ͖ग़ͯ͠Fluentd͕tail͢Δ࢓૊Έ • ϓϩηε͕େྔʹىಈͯ͠΋εϧʔϓοτ͕མͪͳ͍ PHP Exception Handler Logger::write($type,

    $data) # JSON Fluentd in_tail
  29. Fluentd config <source> type tail path /var/tmp/log/activity.log pos_file /var/tmp/fluentd/activity.pos tag

    activity format json # JSONܗࣜΛಡΈࠐΉ </source> <match activity> type forward_with_hostname # HostΛೖΕͯforward flush_interval 1s # ඞͣ1ඵ ! buffer_type file # buffer type͸file/࠶ىಈͯ͠΋ফ͑ͳ͍ buffer_path /var/tmp/fluentd/buffer/activity.*.buffer buffer_chunk_limit 2m # chunkαΠζ͸খ͞Ί buffer_queue_limit 128 # Ͳͷ͘Β͍ফ͑ͪΌ͍͚ͳ͍͔ … </match>
  30. Management tools ؅ཧܥɺ؂ࢹܥπʔϧʹ͍ͭͯ

  31. Monitoring servers/services • ͘͘͝͝Ұൠతͳ؂ࢹϓϩμΫτΛ࢖͍ͬͯΔ Ϧιʔεάϥϑ ϗετ/αʔϏε؂ࢹ ϗετ಺ϓϩηε؂ࢹ εΫϦϓτ Munin Nagios

    Monit Cron
  32. Cluster Admin ϋʔυ΢ΣΞ৘ใ ϗετͷ༻్ ؂ࢹঢ়ଶ

  33. Capistrano/Subversion • /etc/ҎԼͷઃఆϑΝΠϧ͕ͦͷ··subversionͷ
 ؅ཧԼͷσΟϨΫτϦʹ • ઃఆ൓ө͸capistranoΛ࢖༻ͯ͠શ୆ʹσϓϩΠ • ϗετҰཡ͸APIܦ༝ͰऔಘͰ͖ΔΑ͏ʹ $cap dns:update

    $cap dns:check $cap dns:reload ex: update DNS Record
  34. Management Tools • LVS؅ཧը໘ • MySQLͷ஗Ԇ؂ࢹ • αʔϏεͷϦιʔεϞχλϦϯά

  35. Conclusion • pixivΛࢧ͑Δج൫γεςϜʹ͍ͭͯ঺հ • ࢖͍΍͍͢πʔϧΛࣗ෼͕ͨͪ࢖͍΍͍͢Α͏ʹ • ઑͬͨ͜ͱ͸ͤͣɺແཧͤͣӡ༻Ͱ͖Δঢ়ଶʹ
 ΋͍ͬͯ͘