Ugoku Backend

Ugoku "It's working" Backend
pixiv Ugoira Tech Talks / 2014-08-01

Transcript

1. Ugoku backend Ugoira Tech Talks / 2014-08-01 harukasan
2. • Infrastructure Team in pixiv since 2012 • nginx / Apache Traffic Server • Fluentd / Kibana / BigQuery • Kosenconf Harukasan / Shunsuke MICHII ! 
   ൃച @harukasan shunsuke.michii
3. Ugoku Backend ͏͝ΠϥͷཪଆͰ
4. Works in Ugoira Project • ͏͍͝Βͷ͚ͩͷͨΊʹؾΛݣͬͨ෦෼͸
 ͦΕ΄Ͳͳ͍ • ಛච͢΂͖ࣄฑ͸͋·Γͳ͍…… • ͲͪΒ͔ͱ͍͏ͱৗʹ͕Μ͹͍ͬͯΔ͔Μ͡
5. Ugoku Backend
7. Statistics 3,700,000,000 PV/month 11,000,000 Users 45,000,000 Illustrations |
8. Statistics 7K peak req/s (dynamic) 30K peak req/s (image) 12Gbps peak Traffic (image) 3 x 10Gbps backbone 40TB image files 400 servers
9. Ugoku Backend ͏͝ΠϥͷཪଆͰ
10. Ugoira Data Flow PHP nginx nginx Apache HTTPd nginx pixiv Contents Delivery Clusters uploading small light load balancing Store Images generating thumbnails Ý Creator Viewers PHP Redis queing
11. Ugoira Data Flow PHP nginx nginx Apache HTTPd nginx pixiv Contents Delivery Clusters uploading small light load balancing Store Images generating thumbnails Ý Creator Viewers PHP Redis queing
12. Upload System
13. Upload System • ࠷େ150ຕͷը૾ͷม׵ॲཧΛߦ͏ඞཁ͕͋Δ • ϢʔβʔΛग़དྷΔ͚ͩ଴ͨͤͳ͍Α͏ʹ͍ͨ͠
14. Asynchronous Uploading Upload View Information form View Complete View 1. ը૾ϑΝΠϧΛ
 Ξοϓϩʔυ 2. ৘ใೖྗ ◦ ׬ྃΛ଴ͭ Redis Uplaod Worker upload αϜωΠϧϚελը૾ੜ੒ Ý upload Ý GIFը૾෼ׂ active standby 3. Ξοϓϩʔυ׬ྃ
15. Upload System • Ϣʔβʔ͕৘ใΛೖྗ͍ͯ͠Δؒʹ
 ը૾ϑΝΠϧΛ४උ • GIFը૾Λ෼ׂ Github: pixiv/gifsplit • ͏͝ΠϥͰ͸3छྨͷը૾Λ༻ҙ • WebDAVͰΞοϓϩʔυ͢Δ PHP nginx Redis php-resque nginx ϩʔυόϥϯα ΞϓϦέʔγϣϯ Ωϡʔ ϫʔΧʔ DAVετϨʔδ
16. Ugoira Data Flow PHP nginx nginx Apache HTTPd nginx pixiv Contents Delivery Clusters uploading small light load balancing Store Images generating thumbnails Ý Creator Viewers PHP Redis queing
17. Dynamic
 Thumbnail Generation
18. Dynamic Thumb. Generation • Ξοϓϩʔυ࣌ʹͨ͘͞ΜͷαϜωΠϧΛ
 ͭ͘Δͷ͸େม • αΠζ͸σβΠϯมߋͱ͔Ͱม͍͑ͨ • ༧ΊϚελը૾Λੜ੒ͯͦ͠Ε͔Βੜ੒͢Δ ΦϦδφϧը૾ αϜωΠϧϚελը૾ αϜωΠϧը૾
19. Thumbnail Master • ΞεϖΫτݻఆͱਖ਼ํܗΫϩοϓͷ2छྨ • JPEGը૾ʹม׵(PNG→JPEGͷม׵͸ॏ͍) 1200px 1200px 1200px 1200px 1200px 1200px ඪ४αΠζ(1200x1200ʹऩ·ΔαΠζ) ਖ਼ํܗΫϩοϓ(ԣ௕ͱॎ௕ͰΫϩοϓํ๏͕ҟͳΔ) ԣ௕͸தԝ߹Θͤ ॎ௕͸্߹Θͤ
20. • αϜωΠϧੜ੒͸mod_small_lightΛ࢖༻
 https://github.com/yamac/smalllight ! • ετϨʔδ͔ΒԿ౓΋Ϛελը૾Λ
 औಘ͠ͳ͍Α͏ʹΩϟογϡΛ͸͞Ή Dynamic Thumb. Generation Small Light ετϨʔδ Ωϟογϡ Ωϡʔ nginx Ý nginx ৼΓ෼͚
21. Ugoira Data Flow PHP nginx nginx Apache HTTPd nginx pixiv Contents Delivery Cluster uploading small light load balancing Store Images generating thumbnails Ý Creator Viewers PHP Redis queing
22. Contents Delivery Cluster
23. • εέʔϥϒϧ • ௿஗Ԇ (< 100ms) • ߴ͍ώοτ཰ (> 95%) • ઃఆ͕มߋ͠΍͍͢ Storage pixiv Contents Delivery Clusters Contents Delivery Cluster
24. Inside of pixiv CDC 10Gbps 10Gbps nginx nginx nginx ATS ATS ATS nginx nginx nginx Hashing 64GB Memory Caching (60%) 3x 256GB SSD Caching (90%) 1Gbps
25. Ugoira Compatibility • ZIPϑΝΠϧͷ഑৴ • CORS Non-simpleϦΫΤετ΁ͷԠ౴
26. CORS • ผυϝΠϯ΁ͷϦιʔε΁ͷϦΫΤετΛڐՄ͢Δ • simple requests / non-simple requests ͷ2͕ͭ͋Δ • Ϟμϯϒϥ΢β͸͍͍ͩͨରԠ
27. CORS Non-Simple Request someone.example.com www.pixiv.net/api var req = new XMLHTTPRequest(); req.open(“GET”, “//www.pixiv.net/api”); req.setRequestHeader(“Range”, “0-10000”); req.send(); GET /api HTTP/1.1 Origin:someone.example.com Range: 0-10000 ϒϥ΢βͰϒϩοΫ͞ΕΔ
28. someone.example.com www.pixiv.net/api OPTIONS /api HTTP/1.1 Origin:someone.example.com Access-Control-Request-Headers: range Access-Control-Request-Method: GET Preflight request HTTP/1.1 200 OK Access-Control-Allow-Headers: Range Access-Control-Allow-Methods: GET, POST, OPTIONS Access-Control-Allow-Origin:* Access-Control-Expose-Headers: Content-Length Access-Control-Max-Age: 2592000 ڐՄ͢ΔϦΫΤετϔομ ڐՄ͢ΔϦΫΤετϝιου ڐՄ͢ΔϦΫΤετݩΦϦδϯ Script͔Βར༻Ͱ͖ΔϨεϙϯεϔομ ΩϟογϡՄೳ࣌ؒ ڐՄΛٻΊΔϦΫΤετϔομ ڐՄΛٻΊΔϦΫΤετϝιου GET /api HTTP/1.1 Origin:someone.example.com Range: 0-10000
29. Ugoira Data Flow PHP nginx nginx Apache HTTPd nginx pixiv Contents Delivery Cluster uploading small light load balancing Store Images generating thumbnails Ý Creator Viewers PHP Redis queing
30. Complete!
31. Conclusion • ͏͝ΠϥͷόοΫΤϯυ͸ී௨ʹ͏͍͍ͯ͝Δ • ៉ྷͳγεςϜ΋ྑ͍͚Ͳ͏͘͝ͷ͸લఏ • ͏্͘͝ʹ֦ு͠΍͍͢ͷ͸ॏཁ