The Technology behind pixiv Infrastructure

pixivのインフラを支える技術2013
at Python Developers Festa 2013.11

Transcript

1. harukasan / ͸Δ͔͞Μ
   
   at #pyfes 2013.11
   ͷΠϯϑϥΛࢧ͑Δٕज़
   2013
   

   View Slide

2. Harukasan@pixiv
   a.k.a MICHII Shunsuke ಓҪढ़հ
   2003 Kurume National College of Tech.

   - NHK ROBOCON

   - ACM-ICPC
   2008 Kyushu Inst. of Technology
   2010 Tsukuba Univ.

   - Computational Vision Science
   2012 pixiv Inc.

   - Infrastructure team
   

   View Slide

3. View Slide

4. ͜͜ʹpixivͷઆ໌͕ೖΔ
   

   View Slide

5. Server 400+
   Traffic 10Gbps+
   Team member 6
   Monthly PV 3.7 Billion
   

   View Slide

6. Bases of pixiv Infrastructure
   Office IDCF DC
   Developments
   Testing
   Log Analytics
   Small Services
   Main Applications
   DB
   Image Cluster
   New DC
   Image Cluster
   

   View Slide

7. ISP Backbone
   Office IDCF DC New DC
   100M 1G 10G 1G
   1G line
   1G
   pixiv Network
   

   View Slide

8. ͷΠϯϑϥΛࢧ͑Δٕज़
   2013
   

   View Slide

9. http://www.slideshare.net/kamipo/pixiv
   @kamipo pixivͷΠϯϑϥΛࢧ͑Δٕज़ / ςΫηϛ2009
   

   View Slide

10. @cubicdaiya inside pixiv’s infrastructure / PHP Conference 2013
    http://www.slideshare.net/cubicdaiya/inside-pixiv-infrastructure
    

    View Slide

11. Agenda
    pixiv Image Cluster
    Log Analysis Basis
    Management Tools
    

    View Slide

12. pixiv Image Cluster
    QJYJWͷը૾഑৴Ϋϥελʹ͍ͭͯ
    

    View Slide

13. pixiv Image Cluster
    • 2010೥͔Βӡ༻։࢝
    • pixivͷϝΠϯίϯςϯπͰ͋ΔΠϥετΛ

    ௒ߴ଎ʹॲཧ͢ΔͨΊʹ࠷దԽ
    • શτϥϑΟοΫͷ90%Ҏ্Λࡹ͍͍ͯΔ
    

    View Slide

14. Image Cluster
    nginx
    Front Cache
    DNS Round-Robin
    ATS
    Cache
    nginx
    Dispatch
    nginx
    Front Cache
    nginx
    Front Cache
    nginx
    Front Cache
    ATS
    Cache
    ATS
    Cache
    ATS
    Cache
    Consistent Hashing
    nginx
    Dispatch
    nginx
    Dispatch
    nginx
    Dispatch
    Apache
    Origin
    nginx
    Thumbnail
    DNS Round-Robin
    i1.pixiv.net i2.pixiv.net
    SmallLight
    

    View Slide

15. Cache strategy
    • ϝϞϦͱσΟεΫͷ2ஈΩϟογϡߏ੒
    • τϥϑΟοΫ͕૿͑ΔʹͭΕͯ

    εΠονؒτϥϑΟοΫ͕ແࢹͰ͖ͳ͘ͳͬͨ
    • ωοτϫʔΫτϥϑΟοΫΛ཈͑ͭͭ

    Ωϟογϡ༰ྔͷ֬อ͢Δඞཁ
    

    View Slide

16. Cache strategy
    • ϝϞϦՁ֨ͷ௿Լ
    • SSDͷՁ͕֨๫མ
    • ߴIOPSͷSSD͕ొ৔
    2011 ¥40,000

    2013 ¥20,000
    256G
    READ WRITE PRICE
    ioDrive2 785GB MLC*1 215,000/230,000 IOPS I don’t know
    Intel 910 800GB 100,000/ 75,000 IOPS ¥400,000
    SSD 256GBx3 RAID0 80,000/ 50,000 IOPS ¥60,000
    *1: ioDrive2͸ެশ஋
    Intel 910ɺ SSD RAID0ʹ͍ͭͯ͸fioʹΑΔଌఆ
    16G ECC RDIMM ¥20,000
    

    View Slide

17. Cache strategy
    ound-Robin
    nginx
    Front Cache
    nginx
    Front Cache
    nginx
    Front Cache
    ATS
    Cache
    ATS
    Cache
    ATS
    Cache
    Consistent Hashing
    nginx
    Dispatch
    nginx
    Dispatch
    nginx
    Dispatch
    Apache
    Origin
    nginx
    Thumbnail
    DNS Round-Robin
    i1.pixiv.net i2.pixiv.net
    64GB Memory
    - nginx cache on tmpfs
    - cache hit rate: 50%
    - reduce network traffic
    256GB SSD x3 RAID0
    - Apache Traffic Server (standalone)
    - cache hit rate: 80-90%
    SSD
    SSD
    HDD Original & BIG Thumb.
    Small Thumbnails
    

    View Slide

18. Aggregate image domains
    • ը૾αʔό͸ϢʔβʔIDϕʔεͰ෼ࢄ

    img01.pixiv.net - img1XX.pixiv.net
    • 1ϖʔδͰ40-60ճDNSϦΫΤετ͕ൃੜ

    ‎Ոఉ༻ϧʔλ͸DNSղܾ͕Ͱ͖ͳ͘ͳΔ
    • શը૾ͷURIΛมߋͯ͠ରԠ
    OLD: http://img01.pixiv.net/img/****/*****.jpg
    NEW: http://i1.pixiv.net/img01/img/****/*****.jpg
    

    View Slide

19. New Image Store
    ৽͍͠ը૾ετϨʔδํࣜʹ͍ͭͯ
    

    View Slide

20. New image store
    • ࡞੒೔+IDϕʔεͷγʔέϯγϟϧͳURI
    • 1౓ॻ͖ࠐΜͩϑΝΠϧ͸Read Only
    • ࠶౤ߘॲཧ͸࡞੒೔Λߋ৽
    • ngx_lua/OpenRestyΛ༻͍ͨ࿦ཧ࡟আ
    

    View Slide

21. Logical Delete
    !
    Kyototycoon
    ngx_lua / nginx
    null
    404
    /img02.png 403
    /img03.png 404
    /img05.png 404
    /img08.png 403
    "
    "
    GET /img01.png GET /img03.png
    404
    

    View Slide

22. Logical Delete
    local memcached = require "resty.memcached"
    local uri = ngx.var.request_uri
    local memc = memcached:new()
    . . .
    local val, flags, err = memc:get(request_uri)
    if val and val ~= "200" then
    exit(tonumber(val))
    end
    logical_delete.lua
    location / {
    access_by_lua_file logical_delete.lua;
    }
    

    View Slide

23. Log Analysis Basis
    QJYJWͷϩάղੳج൫ʹ͍ͭͯ
    

    View Slide

24. Log Analysis Basis
    PHP Application
    MySQL/neoagent
    Front server
    - Error Log
    - Login Log
    - Activity Log
    - Slow Query
    - Access Log
    MongoDB
    Elasticsearch
    Fluentd
    File System
    

    View Slide

25. Error Viewer
    

    View Slide

26. Slow Query Viewer
    

    View Slide

27. Kibana 3
    

    View Slide

28. Output Log
    • JSONΛॻ͖ग़ͯ͠Fluentd͕tail͢Δ࢓૊Έ
    • ϓϩηε͕େྔʹىಈͯ͠΋εϧʔϓοτ͕མͪͳ͍
    PHP
    Exception Handler
    Logger::write($type, $data)
    #
    JSON
    Fluentd
    in_tail
    

    View Slide

29. Fluentd config
    
    type tail
    path /var/tmp/log/activity.log
    pos_file /var/tmp/fluentd/activity.pos
    tag activity
    format json # JSONܗࣜΛಡΈࠐΉ
    
    
    type forward_with_hostname # HostΛೖΕͯforward
    flush_interval 1s # ඞͣ1ඵ
    !
    buffer_type file # buffer type͸file/࠶ىಈͯ͠΋ফ͑ͳ͍
    buffer_path /var/tmp/fluentd/buffer/activity.*.buffer
    buffer_chunk_limit 2m # chunkαΠζ͸খ͞Ί
    buffer_queue_limit 128 # Ͳͷ͘Β͍ফ͑ͪΌ͍͚ͳ͍͔
    …
    
    

    View Slide

30. Management tools
    ؅ཧܥɺ؂ࢹܥπʔϧʹ͍ͭͯ
    

    View Slide

31. Monitoring servers/services
    • ͘͘͝͝Ұൠతͳ؂ࢹϓϩμΫτΛ࢖͍ͬͯΔ
    Ϧιʔεάϥϑ
    ϗετ/αʔϏε؂ࢹ
    ϗετ಺ϓϩηε؂ࢹ
    εΫϦϓτ
    Munin
    Nagios
    Monit
    Cron
    

    View Slide

32. Cluster Admin
    ϋʔυ΢ΣΞ৘ใ
    ϗετͷ༻్
    ؂ࢹঢ়ଶ
    

    View Slide

33. Capistrano/Subversion
    • /etc/ҎԼͷઃఆϑΝΠϧ͕ͦͷ··subversionͷ

    ؅ཧԼͷσΟϨΫτϦʹ
    • ઃఆ൓ө͸capistranoΛ࢖༻ͯ͠શ୆ʹσϓϩΠ
    • ϗετҰཡ͸APIܦ༝ͰऔಘͰ͖ΔΑ͏ʹ
    $cap dns:update
    $cap dns:check
    $cap dns:reload
    ex: update DNS Record
    

    View Slide

34. Management Tools
    • LVS؅ཧը໘
    • MySQLͷ஗Ԇ؂ࢹ
    • αʔϏεͷϦιʔεϞχλϦϯά
    

    View Slide

35. Conclusion
    • pixivΛࢧ͑Δج൫γεςϜʹ͍ͭͯ঺հ
    • ࢖͍΍͍͢πʔϧΛࣗ෼͕ͨͪ࢖͍΍͍͢Α͏ʹ
    • ઑͬͨ͜ͱ͸ͤͣɺແཧͤͣӡ༻Ͱ͖Δঢ়ଶʹ

    ΋͍ͬͯ͘
    

    View Slide