The Technology behind pixiv Infrastructure

Transcript

1. harukasan / ͸Δ͔͞Μ at #pyfes 2013.11 ͷΠϯϑϥΛࢧ͑Δٕज़ 2013

2. Harukasan@pixiv a.k.a MICHII Shunsuke ಓҪ
   ढ़հ 2003 Kurume National College of

   Tech.
 - NHK ROBOCON
 - ACM-ICPC 2008 Kyushu Inst. of Technology 2010 Tsukuba Univ.
 - Computational Vision Science 2012 pixiv Inc.
 - Infrastructure team
3. None

4. ͜͜ʹpixivͷઆ໌͕ೖΔ

5. Server 400+ Traffic 10Gbps+ Team member 6 Monthly PV 3.7

   Billion

6. Bases of pixiv Infrastructure Office IDCF DC Developments Testing Log

   Analytics Small Services Main Applications DB Image Cluster New DC Image Cluster

7. ISP Backbone Office IDCF DC New DC 100M 1G 10G

   1G 1G line 1G pixiv Network

8. ͷΠϯϑϥΛࢧ͑Δٕज़ 2013

9. http://www.slideshare.net/kamipo/pixiv @kamipo pixivͷΠϯϑϥΛࢧ͑Δٕज़ / ςΫηϛ2009

10. @cubicdaiya inside pixiv’s infrastructure / PHP Conference 2013 http://www.slideshare.net/cubicdaiya/inside-pixiv-infrastructure

11. Agenda pixiv Image Cluster Log Analysis Basis Management Tools

12. pixiv Image Cluster QJYJWͷը૾഑৴Ϋϥελʹ͍ͭͯ

13. pixiv Image Cluster • 2010೥͔Βӡ༻։࢝ • pixivͷϝΠϯίϯςϯπͰ͋ΔΠϥετΛ
 ௒ߴ଎ʹॲཧ͢ΔͨΊʹ࠷దԽ • શτϥϑΟοΫͷ90%Ҏ্Λࡹ͍͍ͯΔ

14. Image Cluster nginx Front Cache DNS Round-Robin ATS Cache nginx

    Dispatch nginx Front Cache nginx Front Cache nginx Front Cache ATS Cache ATS Cache ATS Cache Consistent Hashing nginx Dispatch nginx Dispatch nginx Dispatch Apache Origin nginx Thumbnail DNS Round-Robin i1.pixiv.net i2.pixiv.net SmallLight

15. Cache strategy • ϝϞϦͱσΟεΫͷ2ஈΩϟογϡߏ੒ • τϥϑΟοΫ͕૿͑ΔʹͭΕͯ
 εΠονؒτϥϑΟοΫ͕ແࢹͰ͖ͳ͘ͳͬͨ • ωοτϫʔΫτϥϑΟοΫΛ཈͑ͭͭ
 Ωϟογϡ༰ྔͷ֬อ͢Δඞཁ

16. Cache strategy • ϝϞϦՁ֨ͷ௿Լ • SSDͷՁ͕֨๫མ • ߴIOPSͷSSD͕ొ৔ 2011 ¥40,000


    2013 ¥20,000 256G READ WRITE PRICE ioDrive2 785GB MLC*1 215,000/230,000 IOPS I don’t know Intel 910 800GB 100,000/ 75,000 IOPS ¥400,000 SSD 256GBx3 RAID0 80,000/ 50,000 IOPS ¥60,000 *1: ioDrive2͸ެশ஋ Intel 910ɺ SSD RAID0ʹ͍ͭͯ͸fioʹΑΔଌఆ 16G ECC RDIMM ¥20,000

17. Cache strategy ound-Robin nginx Front Cache nginx Front Cache nginx

    Front Cache ATS Cache ATS Cache ATS Cache Consistent Hashing nginx Dispatch nginx Dispatch nginx Dispatch Apache Origin nginx Thumbnail DNS Round-Robin i1.pixiv.net i2.pixiv.net 64GB Memory - nginx cache on tmpfs - cache hit rate: 50% - reduce network traffic 256GB SSD x3 RAID0 - Apache Traffic Server (standalone) - cache hit rate: 80-90% SSD SSD HDD Original & BIG Thumb. Small Thumbnails

18. Aggregate image domains • ը૾αʔό͸ϢʔβʔIDϕʔεͰ෼ࢄ
 img01.pixiv.net - img1XX.pixiv.net • 1ϖʔδͰ40-60ճDNSϦΫΤετ͕ൃੜ


    ‎Ոఉ༻ϧʔλ͸DNSղܾ͕Ͱ͖ͳ͘ͳΔ • શը૾ͷURIΛมߋͯ͠ରԠ OLD: http://img01.pixiv.net/img/****/*****.jpg NEW: http://i1.pixiv.net/img01/img/****/*****.jpg

19. New Image Store ৽͍͠ը૾ετϨʔδํࣜʹ͍ͭͯ

20. New image store • ࡞੒೔+IDϕʔεͷγʔέϯγϟϧͳURI • 1౓ॻ͖ࠐΜͩϑΝΠϧ͸Read Only • ࠶౤ߘॲཧ͸࡞੒೔Λߋ৽

    • ngx_lua/OpenRestyΛ༻͍ͨ࿦ཧ࡟আ

21. Logical Delete ! Kyototycoon ngx_lua / nginx null 404 /img02.png

    403 /img03.png 404 /img05.png 404 /img08.png 403 " " GET /img01.png GET /img03.png 404

22. Logical Delete local memcached = require "resty.memcached" local uri =

    ngx.var.request_uri local memc = memcached:new() . . . local val, flags, err = memc:get(request_uri) if val and val ~= "200" then exit(tonumber(val)) end logical_delete.lua location / { access_by_lua_file logical_delete.lua; }

23. Log Analysis Basis QJYJWͷϩάղੳج൫ʹ͍ͭͯ

24. Log Analysis Basis PHP Application MySQL/neoagent Front server - Error

    Log - Login Log - Activity Log - Slow Query - Access Log MongoDB Elasticsearch Fluentd File System

25. Error Viewer

26. Slow Query Viewer

27. Kibana 3

28. Output Log • JSONΛॻ͖ग़ͯ͠Fluentd͕tail͢Δ࢓૊Έ • ϓϩηε͕େྔʹىಈͯ͠΋εϧʔϓοτ͕མͪͳ͍ PHP Exception Handler Logger::write($type,

    $data) # JSON Fluentd in_tail

29. Fluentd config <source> type tail path /var/tmp/log/activity.log pos_file /var/tmp/fluentd/activity.pos tag

    activity format json # JSONܗࣜΛಡΈࠐΉ </source> <match activity> type forward_with_hostname # HostΛೖΕͯforward flush_interval 1s # ඞͣ1ඵ ! buffer_type file # buffer type͸file/࠶ىಈͯ͠΋ফ͑ͳ͍ buffer_path /var/tmp/fluentd/buffer/activity.*.buffer buffer_chunk_limit 2m # chunkαΠζ͸খ͞Ί buffer_queue_limit 128 # Ͳͷ͘Β͍ফ͑ͪΌ͍͚ͳ͍͔ … </match>

30. Management tools ؅ཧܥɺ؂ࢹܥπʔϧʹ͍ͭͯ

31. Monitoring servers/services • ͘͘͝͝Ұൠతͳ؂ࢹϓϩμΫτΛ࢖͍ͬͯΔ Ϧιʔεάϥϑ ϗετ/αʔϏε؂ࢹ ϗετ಺ϓϩηε؂ࢹ εΫϦϓτ Munin Nagios

    Monit Cron

32. Cluster Admin ϋʔυ΢ΣΞ৘ใ ϗετͷ༻్ ؂ࢹঢ়ଶ

33. Capistrano/Subversion • /etc/ҎԼͷઃఆϑΝΠϧ͕ͦͷ··subversionͷ
 ؅ཧԼͷσΟϨΫτϦʹ • ઃఆ൓ө͸capistranoΛ࢖༻ͯ͠શ୆ʹσϓϩΠ • ϗετҰཡ͸APIܦ༝ͰऔಘͰ͖ΔΑ͏ʹ $cap dns:update

    $cap dns:check $cap dns:reload ex: update DNS Record

34. Management Tools • LVS؅ཧը໘ • MySQLͷ஗Ԇ؂ࢹ • αʔϏεͷϦιʔεϞχλϦϯά

35. Conclusion • pixivΛࢧ͑Δج൫γεςϜʹ͍ͭͯ঺հ • ࢖͍΍͍͢πʔϧΛࣗ෼͕ͨͪ࢖͍΍͍͢Α͏ʹ • ઑͬͨ͜ͱ͸ͤͣɺແཧͤͣӡ༻Ͱ͖Δঢ়ଶʹ
 ΋͍ͬͯ͘