Ugoku backend Ugoira Tech Talks / 2014-08-01 harukasan

• Infrastructure Team in pixiv since 2012 • nginx / Apache Traffic Server • Fluentd / Kibana / BigQuery • Kosenconf Harukasan / Shunsuke MICHII ! ൃച @harukasan shunsuke.michii

Ugoku Backend ͏͝ΠϥͷཪଆͰ

Works in Ugoira Project • ͏͍͝Βͷ͚ͩͷͨΊʹؾΛݣͬͨ෦෼͸
 ͦΕ΄Ͳͳ͍ • ಛච͢΂͖ࣄฑ͸͋·Γͳ͍…… • ͲͪΒ͔ͱ͍͏ͱৗʹ͕Μ͹͍ͬͯΔ͔Μ͡

Ugoku Backend

No content

Statistics 3,700,000,000 PV/month 11,000,000 Users 45,000,000 Illustrations |

Statistics 7K peak req/s (dynamic) 30K peak req/s (image) 12Gbps peak Traffic (image) 3 x 10Gbps backbone 40TB image files 400 servers

Ugoku Backend ͏͝ΠϥͷཪଆͰ

Ugoira Data Flow PHP nginx nginx Apache HTTPd nginx pixiv Contents Delivery Clusters uploading small light load balancing Store Images generating thumbnails Ý Creator Viewers PHP Redis queing

Ugoira Data Flow PHP nginx nginx Apache HTTPd nginx pixiv Contents Delivery Clusters uploading small light load balancing Store Images generating thumbnails Ý Creator Viewers PHP Redis queing

Upload System

Upload System • ࠷େ150ຕͷը૾ͷม׵ॲཧΛߦ͏ඞཁ͕͋Δ • ϢʔβʔΛग़དྷΔ͚ͩ଴ͨͤͳ͍Α͏ʹ͍ͨ͠

Asynchronous Uploading Upload View Information form View Complete View 1. ը૾ϑΝΠϧΛ
 Ξοϓϩʔυ 2. ৘ใೖྗ ○ ׬ྃΛ଴ͭ Redis Uplaod Worker upload αϜωΠϧϚελը૾ੜ੒ Ý upload Ý GIFը૾෼ׂ active standby 3. Ξοϓϩʔυ׬ྃ

Upload System • Ϣʔβʔ͕৘ใΛೖྗ͍ͯ͠Δؒʹ
 ը૾ϑΝΠϧΛ४උ • GIFը૾Λ෼ׂ Github: pixiv/gifsplit • ͏͝ΠϥͰ͸3छྨͷը૾Λ༻ҙ • WebDAVͰΞοϓϩʔυ͢Δ PHP nginx Redis php-resque nginx ϩʔυόϥϯα ΞϓϦέʔγϣϯ Ωϡʔ ϫʔΧʔ DAVετϨʔδ

Ugoira Data Flow PHP nginx nginx Apache HTTPd nginx pixiv Contents Delivery Clusters uploading small light load balancing Store Images generating thumbnails Ý Creator Viewers PHP Redis queing

Dynamic
 Thumbnail Generation

Dynamic Thumb. Generation • Ξοϓϩʔυ࣌ʹͨ͘͞ΜͷαϜωΠϧΛ
 ͭ͘Δͷ͸େม • αΠζ͸σβΠϯมߋͱ͔Ͱม͍͑ͨ • ༧ΊϚελը૾Λੜ੒ͯͦ͠Ε͔Βੜ੒͢Δ ΦϦδφϧը૾ αϜωΠϧϚελը૾ αϜωΠϧը૾

Thumbnail Master • ΞεϖΫτݻఆͱਖ਼ํܗΫϩοϓͷ2छྨ • JPEGը૾ʹม׵(PNG→JPEGͷม׵͸ॏ͍) 1200px 1200px 1200px 1200px 1200px 1200px ඪ४αΠζ(1200x1200ʹऩ·ΔαΠζ) ਖ਼ํܗΫϩοϓ(ԣ௕ͱॎ௕ͰΫϩοϓํ๏͕ҟͳΔ) ԣ௕͸தԝ߹Θͤ ॎ௕͸্߹Θͤ

• αϜωΠϧੜ੒͸mod_small_lightΛ࢖༻
 https://github.com/yamac/smalllight ! • ετϨʔδ͔ΒԿ౓΋Ϛελը૾Λ
 औಘ͠ͳ͍Α͏ʹΩϟογϡΛ͸͞Ή Dynamic Thumb. Generation Small Light ετϨʔδ Ωϟογϡ Ωϡʔ nginx Ý nginx ৼΓ෼͚

Ugoira Data Flow PHP nginx nginx Apache HTTPd nginx pixiv Contents Delivery Cluster uploading small light load balancing Store Images generating thumbnails Ý Creator Viewers PHP Redis queing

Contents Delivery Cluster

• εέʔϥϒϧ • ௿஗Ԇ (< 100ms) • ߴ͍ώοτ཰ (> 95%) • ઃఆ͕มߋ͠΍͍͢ Storage pixiv Contents Delivery Clusters Contents Delivery Cluster

Inside of pixiv CDC 10Gbps 10Gbps nginx nginx nginx ATS ATS ATS nginx nginx nginx Hashing 64GB Memory Caching (60%) 3x 256GB SSD Caching (90%) 1Gbps

Ugoira Compatibility • ZIPϑΝΠϧͷ഑৴ • CORS Non-simpleϦΫΤετ΁ͷԠ౴

CORS • ผυϝΠϯ΁ͷϦιʔε΁ͷϦΫΤετΛڐՄ͢Δ • simple requests / non-simple requests ͷ2͕ͭ͋Δ • Ϟμϯϒϥ΢β͸͍͍ͩͨରԠ

CORS Non-Simple Request someone.example.com www.pixiv.net/api var req = new XMLHTTPRequest(); req.open(“GET”, “//www.pixiv.net/api”); req.setRequestHeader(“Range”, “0-10000”); req.send(); GET /api HTTP/1.1 Origin:someone.example.com Range: 0-10000 ϒϥ΢βͰϒϩοΫ͞ΕΔ

someone.example.com www.pixiv.net/api OPTIONS /api HTTP/1.1 Origin:someone.example.com Access-Control-Request-Headers: range Access-Control-Request-Method: GET Preflight request HTTP/1.1 200 OK Access-Control-Allow-Headers: Range Access-Control-Allow-Methods: GET, POST, OPTIONS Access-Control-Allow-Origin:* Access-Control-Expose-Headers: Content-Length Access-Control-Max-Age: 2592000 ڐՄ͢ΔϦΫΤετϔομ ڐՄ͢ΔϦΫΤετϝιου ڐՄ͢ΔϦΫΤετݩΦϦδϯ Script͔Βར༻Ͱ͖ΔϨεϙϯεϔομ ΩϟογϡՄೳ࣌ؒ ڐՄΛٻΊΔϦΫΤετϔομ ڐՄΛٻΊΔϦΫΤετϝιου GET /api HTTP/1.1 Origin:someone.example.com Range: 0-10000

Ugoira Data Flow PHP nginx nginx Apache HTTPd nginx pixiv Contents Delivery Cluster uploading small light load balancing Store Images generating thumbnails Ý Creator Viewers PHP Redis queing

Complete!

Conclusion • ͏͝ΠϥͷόοΫΤϯυ͸ී௨ʹ͏͍͍ͯ͝Δ • ៉ྷͳγεςϜ΋ྑ͍͚Ͳ͏͘͝ͷ͸લఏ • ͏্͘͝ʹ֦ு͠΍͍͢ͷ͸ॏཁ