Ugoku Backend

Transcript

1. Ugoku backend Ugoira Tech Talks / 2014-08-01 harukasan

2. • Infrastructure Team in pixiv since 2012 • nginx /

   Apache Traffic Server • Fluentd / Kibana / BigQuery • Kosenconf Harukasan / Shunsuke MICHII ! 
   ൃച @harukasan shunsuke.michii

3. Ugoku Backend ͏͝ΠϥͷཪଆͰ

4. Works in Ugoira Project • ͏͍͝Βͷ͚ͩͷͨΊʹؾΛݣͬͨ෦෼͸
 ͦΕ΄Ͳͳ͍ • ಛච͢΂͖ࣄฑ͸͋·Γͳ͍…… •

   ͲͪΒ͔ͱ͍͏ͱৗʹ͕Μ͹͍ͬͯΔ͔Μ͡

5. Ugoku Backend

6. None

7. Statistics 3,700,000,000 PV/month 11,000,000 Users 45,000,000 Illustrations |

8. Statistics 7K peak req/s (dynamic) 30K peak req/s (image) 12Gbps

   peak Traffic (image) 3 x 10Gbps backbone 40TB image files 400 servers

9. Ugoku Backend ͏͝ΠϥͷཪଆͰ

10. Ugoira Data Flow PHP nginx nginx Apache HTTPd nginx pixiv

    Contents Delivery Clusters uploading small light load balancing Store Images generating thumbnails Ý Creator Viewers PHP Redis queing

11. Ugoira Data Flow PHP nginx nginx Apache HTTPd nginx pixiv

    Contents Delivery Clusters uploading small light load balancing Store Images generating thumbnails Ý Creator Viewers PHP Redis queing

12. Upload System

13. Upload System • ࠷େ150ຕͷը૾ͷม׵ॲཧΛߦ͏ඞཁ͕͋Δ • ϢʔβʔΛग़དྷΔ͚ͩ଴ͨͤͳ͍Α͏ʹ͍ͨ͠

14. Asynchronous Uploading Upload View Information form View Complete View 1.

    ը૾ϑΝΠϧΛ
 Ξοϓϩʔυ 2. ৘ใೖྗ ◦ ׬ྃΛ଴ͭ Redis Uplaod Worker upload αϜωΠϧϚελը૾ੜ੒ Ý upload Ý GIFը૾෼ׂ active standby 3. Ξοϓϩʔυ׬ྃ

15. Upload System • Ϣʔβʔ͕৘ใΛೖྗ͍ͯ͠Δؒʹ
 ը૾ϑΝΠϧΛ४උ • GIFը૾Λ෼ׂ Github: pixiv/gifsplit •

    ͏͝ΠϥͰ͸3छྨͷը૾Λ༻ҙ • WebDAVͰΞοϓϩʔυ͢Δ PHP nginx Redis php-resque nginx ϩʔυόϥϯα ΞϓϦέʔγϣϯ Ωϡʔ ϫʔΧʔ DAVετϨʔδ

16. Ugoira Data Flow PHP nginx nginx Apache HTTPd nginx pixiv

    Contents Delivery Clusters uploading small light load balancing Store Images generating thumbnails Ý Creator Viewers PHP Redis queing

17. Dynamic
 Thumbnail Generation

18. Dynamic Thumb. Generation • Ξοϓϩʔυ࣌ʹͨ͘͞ΜͷαϜωΠϧΛ
 ͭ͘Δͷ͸େม • αΠζ͸σβΠϯมߋͱ͔Ͱม͍͑ͨ • ༧ΊϚελը૾Λੜ੒ͯͦ͠Ε͔Βੜ੒͢Δ

    ΦϦδφϧը૾ αϜωΠϧϚελը૾ αϜωΠϧը૾

19. Thumbnail Master • ΞεϖΫτݻఆͱਖ਼ํܗΫϩοϓͷ2छྨ • JPEGը૾ʹม׵(PNG→JPEGͷม׵͸ॏ͍) 1200px 1200px 1200px 1200px

    1200px 1200px ඪ४αΠζ(1200x1200ʹऩ·ΔαΠζ) ਖ਼ํܗΫϩοϓ(ԣ௕ͱॎ௕ͰΫϩοϓํ๏͕ҟͳΔ) ԣ௕͸தԝ߹Θͤ ॎ௕͸্߹Θͤ

20. • αϜωΠϧੜ੒͸mod_small_lightΛ࢖༻
 https://github.com/yamac/smalllight ! • ετϨʔδ͔ΒԿ౓΋Ϛελը૾Λ
 औಘ͠ͳ͍Α͏ʹΩϟογϡΛ͸͞Ή Dynamic Thumb. Generation

    Small Light ετϨʔδ Ωϟογϡ Ωϡʔ nginx Ý nginx ৼΓ෼͚

21. Ugoira Data Flow PHP nginx nginx Apache HTTPd nginx pixiv

    Contents Delivery Cluster uploading small light load balancing Store Images generating thumbnails Ý Creator Viewers PHP Redis queing

22. Contents Delivery Cluster

23. • εέʔϥϒϧ • ௿஗Ԇ (< 100ms) • ߴ͍ώοτ཰ (> 95%)

    • ઃఆ͕มߋ͠΍͍͢ Storage pixiv Contents Delivery Clusters Contents Delivery Cluster

24. Inside of pixiv CDC 10Gbps 10Gbps nginx nginx nginx ATS

    ATS ATS nginx nginx nginx Hashing 64GB Memory Caching (60%) 3x 256GB SSD Caching (90%) 1Gbps

25. Ugoira Compatibility • ZIPϑΝΠϧͷ഑৴ • CORS Non-simpleϦΫΤετ΁ͷԠ౴

26. CORS • ผυϝΠϯ΁ͷϦιʔε΁ͷϦΫΤετΛڐՄ͢Δ • simple requests / non-simple requests ͷ2͕ͭ͋Δ

    • Ϟμϯϒϥ΢β͸͍͍ͩͨରԠ

27. CORS Non-Simple Request someone.example.com www.pixiv.net/api var req = new XMLHTTPRequest();

    req.open(“GET”, “//www.pixiv.net/api”); req.setRequestHeader(“Range”, “0-10000”); req.send(); GET /api HTTP/1.1 Origin:someone.example.com Range: 0-10000 ϒϥ΢βͰϒϩοΫ͞ΕΔ

28. someone.example.com www.pixiv.net/api OPTIONS /api HTTP/1.1 Origin:someone.example.com Access-Control-Request-Headers: range Access-Control-Request-Method: GET

    Preflight request HTTP/1.1 200 OK Access-Control-Allow-Headers: Range Access-Control-Allow-Methods: GET, POST, OPTIONS Access-Control-Allow-Origin:* Access-Control-Expose-Headers: Content-Length Access-Control-Max-Age: 2592000 ڐՄ͢ΔϦΫΤετϔομ ڐՄ͢ΔϦΫΤετϝιου ڐՄ͢ΔϦΫΤετݩΦϦδϯ Script͔Βར༻Ͱ͖ΔϨεϙϯεϔομ ΩϟογϡՄೳ࣌ؒ ڐՄΛٻΊΔϦΫΤετϔομ ڐՄΛٻΊΔϦΫΤετϝιου GET /api HTTP/1.1 Origin:someone.example.com Range: 0-10000

29. Ugoira Data Flow PHP nginx nginx Apache HTTPd nginx pixiv

    Contents Delivery Cluster uploading small light load balancing Store Images generating thumbnails Ý Creator Viewers PHP Redis queing

30. Complete!

31. Conclusion • ͏͝ΠϥͷόοΫΤϯυ͸ී௨ʹ͏͍͍ͯ͝Δ • ៉ྷͳγεςϜ΋ྑ͍͚Ͳ͏͘͝ͷ͸લఏ • ͏্͘͝ʹ֦ு͠΍͍͢ͷ͸ॏཁ