$30 off During Our Annual Pro Sale. View Details »

The Technology behind pixiv Infrastructure

Harukasan
PRO
November 30, 2013

The Technology behind pixiv Infrastructure

pixivのインフラを支える技術2013
at Python Developers Festa 2013.11

Harukasan
PRO

November 30, 2013
Tweet

More Decks by Harukasan

Other Decks in Technology

Transcript

  1. harukasan / ͸Δ͔͞Μ

    at #pyfes 2013.11
    ͷΠϯϑϥΛࢧ͑Δٕज़
    2013

    View Slide

  2. Harukasan@pixiv
    a.k.a MICHII Shunsuke ಓҪढ़հ
    2003 Kurume National College of Tech.

    - NHK ROBOCON

    - ACM-ICPC
    2008 Kyushu Inst. of Technology
    2010 Tsukuba Univ.

    - Computational Vision Science
    2012 pixiv Inc.

    - Infrastructure team

    View Slide

  3. View Slide

  4. ͜͜ʹpixivͷઆ໌͕ೖΔ

    View Slide

  5. Server 400+
    Traffic 10Gbps+
    Team member 6
    Monthly PV 3.7 Billion

    View Slide

  6. Bases of pixiv Infrastructure
    Office IDCF DC
    Developments
    Testing
    Log Analytics
    Small Services
    Main Applications
    DB
    Image Cluster
    New DC
    Image Cluster

    View Slide

  7. ISP Backbone
    Office IDCF DC New DC
    100M 1G 10G 1G
    1G line
    1G
    pixiv Network

    View Slide

  8. ͷΠϯϑϥΛࢧ͑Δٕज़
    2013

    View Slide

  9. http://www.slideshare.net/kamipo/pixiv
    @kamipo pixivͷΠϯϑϥΛࢧ͑Δٕज़ / ςΫηϛ2009

    View Slide

  10. @cubicdaiya inside pixiv’s infrastructure / PHP Conference 2013
    http://www.slideshare.net/cubicdaiya/inside-pixiv-infrastructure

    View Slide

  11. Agenda
    pixiv Image Cluster
    Log Analysis Basis
    Management Tools

    View Slide

  12. pixiv Image Cluster
    QJYJWͷը૾഑৴Ϋϥελʹ͍ͭͯ

    View Slide

  13. pixiv Image Cluster
    • 2010೥͔Βӡ༻։࢝
    • pixivͷϝΠϯίϯςϯπͰ͋ΔΠϥετΛ

    ௒ߴ଎ʹॲཧ͢ΔͨΊʹ࠷దԽ
    • શτϥϑΟοΫͷ90%Ҏ্Λࡹ͍͍ͯΔ

    View Slide

  14. Image Cluster
    nginx
    Front Cache
    DNS Round-Robin
    ATS
    Cache
    nginx
    Dispatch
    nginx
    Front Cache
    nginx
    Front Cache
    nginx
    Front Cache
    ATS
    Cache
    ATS
    Cache
    ATS
    Cache
    Consistent Hashing
    nginx
    Dispatch
    nginx
    Dispatch
    nginx
    Dispatch
    Apache
    Origin
    nginx
    Thumbnail
    DNS Round-Robin
    i1.pixiv.net i2.pixiv.net
    SmallLight

    View Slide

  15. Cache strategy
    • ϝϞϦͱσΟεΫͷ2ஈΩϟογϡߏ੒
    • τϥϑΟοΫ͕૿͑ΔʹͭΕͯ

    εΠονؒτϥϑΟοΫ͕ແࢹͰ͖ͳ͘ͳͬͨ
    • ωοτϫʔΫτϥϑΟοΫΛ཈͑ͭͭ

    Ωϟογϡ༰ྔͷ֬อ͢Δඞཁ

    View Slide

  16. Cache strategy
    • ϝϞϦՁ֨ͷ௿Լ
    • SSDͷՁ͕֨๫མ
    • ߴIOPSͷSSD͕ొ৔
    2011 ¥40,000

    2013 ¥20,000
    256G
    READ WRITE PRICE
    ioDrive2 785GB MLC*1 215,000/230,000 IOPS I don’t know
    Intel 910 800GB 100,000/ 75,000 IOPS ¥400,000
    SSD 256GBx3 RAID0 80,000/ 50,000 IOPS ¥60,000
    *1: ioDrive2͸ެশ஋
    Intel 910ɺ SSD RAID0ʹ͍ͭͯ͸fioʹΑΔଌఆ
    16G ECC RDIMM ¥20,000

    View Slide

  17. Cache strategy
    ound-Robin
    nginx
    Front Cache
    nginx
    Front Cache
    nginx
    Front Cache
    ATS
    Cache
    ATS
    Cache
    ATS
    Cache
    Consistent Hashing
    nginx
    Dispatch
    nginx
    Dispatch
    nginx
    Dispatch
    Apache
    Origin
    nginx
    Thumbnail
    DNS Round-Robin
    i1.pixiv.net i2.pixiv.net
    64GB Memory
    - nginx cache on tmpfs
    - cache hit rate: 50%
    - reduce network traffic
    256GB SSD x3 RAID0
    - Apache Traffic Server (standalone)
    - cache hit rate: 80-90%
    SSD
    SSD
    HDD Original & BIG Thumb.
    Small Thumbnails

    View Slide

  18. Aggregate image domains
    • ը૾αʔό͸ϢʔβʔIDϕʔεͰ෼ࢄ

    img01.pixiv.net - img1XX.pixiv.net
    • 1ϖʔδͰ40-60ճDNSϦΫΤετ͕ൃੜ

    ‎Ոఉ༻ϧʔλ͸DNSղܾ͕Ͱ͖ͳ͘ͳΔ
    • શը૾ͷURIΛมߋͯ͠ରԠ
    OLD: http://img01.pixiv.net/img/****/*****.jpg
    NEW: http://i1.pixiv.net/img01/img/****/*****.jpg

    View Slide

  19. New Image Store
    ৽͍͠ը૾ετϨʔδํࣜʹ͍ͭͯ

    View Slide

  20. New image store
    • ࡞੒೔+IDϕʔεͷγʔέϯγϟϧͳURI
    • 1౓ॻ͖ࠐΜͩϑΝΠϧ͸Read Only
    • ࠶౤ߘॲཧ͸࡞੒೔Λߋ৽
    • ngx_lua/OpenRestyΛ༻͍ͨ࿦ཧ࡟আ

    View Slide

  21. Logical Delete
    !
    Kyototycoon
    ngx_lua / nginx
    null
    404
    /img02.png 403
    /img03.png 404
    /img05.png 404
    /img08.png 403
    "
    "
    GET /img01.png GET /img03.png
    404

    View Slide

  22. Logical Delete
    local memcached = require "resty.memcached"
    local uri = ngx.var.request_uri
    local memc = memcached:new()
    . . .
    local val, flags, err = memc:get(request_uri)
    if val and val ~= "200" then
    exit(tonumber(val))
    end
    logical_delete.lua
    location / {
    access_by_lua_file logical_delete.lua;
    }

    View Slide

  23. Log Analysis Basis
    QJYJWͷϩάղੳج൫ʹ͍ͭͯ

    View Slide

  24. Log Analysis Basis
    PHP Application
    MySQL/neoagent
    Front server
    - Error Log
    - Login Log
    - Activity Log
    - Slow Query
    - Access Log
    MongoDB
    Elasticsearch
    Fluentd
    File System

    View Slide

  25. Error Viewer

    View Slide

  26. Slow Query Viewer

    View Slide

  27. Kibana 3

    View Slide

  28. Output Log
    • JSONΛॻ͖ग़ͯ͠Fluentd͕tail͢Δ࢓૊Έ
    • ϓϩηε͕େྔʹىಈͯ͠΋εϧʔϓοτ͕མͪͳ͍
    PHP
    Exception Handler
    Logger::write($type, $data)
    #
    JSON
    Fluentd
    in_tail

    View Slide

  29. Fluentd config

    type tail
    path /var/tmp/log/activity.log
    pos_file /var/tmp/fluentd/activity.pos
    tag activity
    format json # JSONܗࣜΛಡΈࠐΉ


    type forward_with_hostname # HostΛೖΕͯforward
    flush_interval 1s # ඞͣ1ඵ
    !
    buffer_type file # buffer type͸file/࠶ىಈͯ͠΋ফ͑ͳ͍
    buffer_path /var/tmp/fluentd/buffer/activity.*.buffer
    buffer_chunk_limit 2m # chunkαΠζ͸খ͞Ί
    buffer_queue_limit 128 # Ͳͷ͘Β͍ফ͑ͪΌ͍͚ͳ͍͔


    View Slide

  30. Management tools
    ؅ཧܥɺ؂ࢹܥπʔϧʹ͍ͭͯ

    View Slide

  31. Monitoring servers/services
    • ͘͘͝͝Ұൠతͳ؂ࢹϓϩμΫτΛ࢖͍ͬͯΔ
    Ϧιʔεάϥϑ
    ϗετ/αʔϏε؂ࢹ
    ϗετ಺ϓϩηε؂ࢹ
    εΫϦϓτ
    Munin
    Nagios
    Monit
    Cron

    View Slide

  32. Cluster Admin
    ϋʔυ΢ΣΞ৘ใ
    ϗετͷ༻్
    ؂ࢹঢ়ଶ

    View Slide

  33. Capistrano/Subversion
    • /etc/ҎԼͷઃఆϑΝΠϧ͕ͦͷ··subversionͷ

    ؅ཧԼͷσΟϨΫτϦʹ
    • ઃఆ൓ө͸capistranoΛ࢖༻ͯ͠શ୆ʹσϓϩΠ
    • ϗετҰཡ͸APIܦ༝ͰऔಘͰ͖ΔΑ͏ʹ
    $cap dns:update
    $cap dns:check
    $cap dns:reload
    ex: update DNS Record

    View Slide

  34. Management Tools
    • LVS؅ཧը໘
    • MySQLͷ஗Ԇ؂ࢹ
    • αʔϏεͷϦιʔεϞχλϦϯά

    View Slide

  35. Conclusion
    • pixivΛࢧ͑Δج൫γεςϜʹ͍ͭͯ঺հ
    • ࢖͍΍͍͢πʔϧΛࣗ෼͕ͨͪ࢖͍΍͍͢Α͏ʹ
    • ઑͬͨ͜ͱ͸ͤͣɺແཧͤͣӡ༻Ͱ͖Δঢ়ଶʹ

    ΋͍ͬͯ͘

    View Slide