Ugoku Backend

124da56a613b15fa980427533e4e3839?s=47 Harukasan
August 02, 2014

Ugoku Backend

Ugoku "It's working" Backend
pixiv Ugoira Tech Talks / 2014-08-01

124da56a613b15fa980427533e4e3839?s=128

Harukasan

August 02, 2014
Tweet

Transcript

  1. Ugoku backend Ugoira Tech Talks / 2014-08-01 harukasan

  2. • Infrastructure Team in pixiv since 2012 • nginx /

    Apache Traffic Server • Fluentd / Kibana / BigQuery • Kosenconf Harukasan / Shunsuke MICHII !  ൃച @harukasan shunsuke.michii
  3. Ugoku Backend ͏͝ΠϥͷཪଆͰ

  4. Works in Ugoira Project • ͏͍͝Βͷ͚ͩͷͨΊʹؾΛݣͬͨ෦෼͸
 ͦΕ΄Ͳͳ͍ • ಛච͢΂͖ࣄฑ͸͋·Γͳ͍…… •

    ͲͪΒ͔ͱ͍͏ͱৗʹ͕Μ͹͍ͬͯΔ͔Μ͡
  5. Ugoku Backend

  6. None
  7. Statistics 3,700,000,000 PV/month 11,000,000 Users 45,000,000 Illustrations |

  8. Statistics 7K peak req/s (dynamic) 30K peak req/s (image) 12Gbps

    peak Traffic (image) 3 x 10Gbps backbone 40TB image files 400 servers
  9. Ugoku Backend ͏͝ΠϥͷཪଆͰ

  10. Ugoira Data Flow PHP nginx nginx Apache HTTPd nginx pixiv

    Contents Delivery Clusters uploading small light load balancing Store Images generating thumbnails Ý Creator Viewers PHP Redis queing
  11. Ugoira Data Flow PHP nginx nginx Apache HTTPd nginx pixiv

    Contents Delivery Clusters uploading small light load balancing Store Images generating thumbnails Ý Creator Viewers PHP Redis queing
  12. Upload System

  13. Upload System • ࠷େ150ຕͷը૾ͷม׵ॲཧΛߦ͏ඞཁ͕͋Δ • ϢʔβʔΛग़དྷΔ͚ͩ଴ͨͤͳ͍Α͏ʹ͍ͨ͠

  14. Asynchronous Uploading Upload View Information form View Complete View 1.

    ը૾ϑΝΠϧΛ
 Ξοϓϩʔυ 2. ৘ใೖྗ ◦ ׬ྃΛ଴ͭ Redis Uplaod Worker upload αϜωΠϧϚελը૾ੜ੒ Ý upload Ý GIFը૾෼ׂ active standby 3. Ξοϓϩʔυ׬ྃ
  15. Upload System • Ϣʔβʔ͕৘ใΛೖྗ͍ͯ͠Δؒʹ
 ը૾ϑΝΠϧΛ४උ • GIFը૾Λ෼ׂ Github: pixiv/gifsplit •

    ͏͝ΠϥͰ͸3छྨͷը૾Λ༻ҙ • WebDAVͰΞοϓϩʔυ͢Δ PHP nginx Redis php-resque nginx ϩʔυόϥϯα ΞϓϦέʔγϣϯ Ωϡʔ ϫʔΧʔ DAVετϨʔδ
  16. Ugoira Data Flow PHP nginx nginx Apache HTTPd nginx pixiv

    Contents Delivery Clusters uploading small light load balancing Store Images generating thumbnails Ý Creator Viewers PHP Redis queing
  17. Dynamic
 Thumbnail Generation

  18. Dynamic Thumb. Generation • Ξοϓϩʔυ࣌ʹͨ͘͞ΜͷαϜωΠϧΛ
 ͭ͘Δͷ͸େม • αΠζ͸σβΠϯมߋͱ͔Ͱม͍͑ͨ • ༧ΊϚελը૾Λੜ੒ͯͦ͠Ε͔Βੜ੒͢Δ

    ΦϦδφϧը૾ αϜωΠϧϚελը૾ αϜωΠϧը૾
  19. Thumbnail Master • ΞεϖΫτݻఆͱਖ਼ํܗΫϩοϓͷ2छྨ • JPEGը૾ʹม׵(PNG→JPEGͷม׵͸ॏ͍) 1200px 1200px 1200px 1200px

    1200px 1200px ඪ४αΠζ(1200x1200ʹऩ·ΔαΠζ) ਖ਼ํܗΫϩοϓ(ԣ௕ͱॎ௕ͰΫϩοϓํ๏͕ҟͳΔ) ԣ௕͸தԝ߹Θͤ ॎ௕͸্߹Θͤ
  20. • αϜωΠϧੜ੒͸mod_small_lightΛ࢖༻
 https://github.com/yamac/smalllight ! • ετϨʔδ͔ΒԿ౓΋Ϛελը૾Λ
 औಘ͠ͳ͍Α͏ʹΩϟογϡΛ͸͞Ή Dynamic Thumb. Generation

    Small Light ετϨʔδ Ωϟογϡ Ωϡʔ nginx Ý nginx ৼΓ෼͚
  21. Ugoira Data Flow PHP nginx nginx Apache HTTPd nginx pixiv

    Contents Delivery Cluster uploading small light load balancing Store Images generating thumbnails Ý Creator Viewers PHP Redis queing
  22. Contents Delivery Cluster

  23. • εέʔϥϒϧ • ௿஗Ԇ (< 100ms) • ߴ͍ώοτ཰ (> 95%)

    • ઃఆ͕มߋ͠΍͍͢ Storage pixiv Contents Delivery Clusters Contents Delivery Cluster
  24. Inside of pixiv CDC 10Gbps 10Gbps nginx nginx nginx ATS

    ATS ATS nginx nginx nginx Hashing 64GB Memory Caching (60%) 3x 256GB SSD Caching (90%) 1Gbps
  25. Ugoira Compatibility • ZIPϑΝΠϧͷ഑৴ • CORS Non-simpleϦΫΤετ΁ͷԠ౴

  26. CORS • ผυϝΠϯ΁ͷϦιʔε΁ͷϦΫΤετΛڐՄ͢Δ • simple requests / non-simple requests ͷ2͕ͭ͋Δ

    • Ϟμϯϒϥ΢β͸͍͍ͩͨରԠ
  27. CORS Non-Simple Request someone.example.com www.pixiv.net/api var req = new XMLHTTPRequest();

    req.open(“GET”, “//www.pixiv.net/api”); req.setRequestHeader(“Range”, “0-10000”); req.send(); GET /api HTTP/1.1 Origin:someone.example.com Range: 0-10000 ϒϥ΢βͰϒϩοΫ͞ΕΔ
  28. someone.example.com www.pixiv.net/api OPTIONS /api HTTP/1.1 Origin:someone.example.com Access-Control-Request-Headers: range Access-Control-Request-Method: GET

    Preflight request HTTP/1.1 200 OK Access-Control-Allow-Headers: Range Access-Control-Allow-Methods: GET, POST, OPTIONS Access-Control-Allow-Origin:* Access-Control-Expose-Headers: Content-Length Access-Control-Max-Age: 2592000 ڐՄ͢ΔϦΫΤετϔομ ڐՄ͢ΔϦΫΤετϝιου ڐՄ͢ΔϦΫΤετݩΦϦδϯ Script͔Βར༻Ͱ͖ΔϨεϙϯεϔομ ΩϟογϡՄೳ࣌ؒ ڐՄΛٻΊΔϦΫΤετϔομ ڐՄΛٻΊΔϦΫΤετϝιου GET /api HTTP/1.1 Origin:someone.example.com Range: 0-10000
  29. Ugoira Data Flow PHP nginx nginx Apache HTTPd nginx pixiv

    Contents Delivery Cluster uploading small light load balancing Store Images generating thumbnails Ý Creator Viewers PHP Redis queing
  30. Complete!

  31. Conclusion • ͏͝ΠϥͷόοΫΤϯυ͸ී௨ʹ͏͍͍ͯ͝Δ • ៉ྷͳγεςϜ΋ྑ͍͚Ͳ͏͘͝ͷ͸લఏ • ͏্͘͝ʹ֦ு͠΍͍͢ͷ͸ॏཁ