Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up for free
Ugoku Backend
Harukasan
August 02, 2014
Technology
1
1.5k
Ugoku Backend
Ugoku "It's working" Backend
pixiv Ugoira Tech Talks / 2014-08-01
Harukasan
August 02, 2014
Tweet
Share
More Decks by Harukasan
See All by Harukasan
ピクシブのコンテンツ配信基盤技術 / pixiv TECH SALON
harukasan
5
4.3k
Goにおける画像ファイル処理 / golang.tokyo #19
harukasan
7
5.5k
WebRTC動画をトランスコードする / Transcoding video streams from WebRTC
harukasan
5
1.1k
ImageFluxを支えるリモート開発 / 20171202
harukasan
2
1.4k
YAPC::Fukuoka 前夜祭LT / Yet Another Pawoo Commit logs
harukasan
0
2.5k
YAPC::Fukuoka lunch session
harukasan
1
2.6k
マストドン会議: Pawoo / Mastodon Kaigi2
harukasan
2
320
大規模Mastodonインスタンスを運用するコツ / Inside Pawoo Mastodon infrastructure
harukasan
0
2.4k
JPEG streaming in Go
harukasan
2
2k
Other Decks in Technology
See All in Technology
マネージャーからみたスクラムと自己管理化
shibe23
0
930
トランザクションスクリプトはどこから来たのか トランザクションスクリプトは何者か トランザクションスクリプトはどこへ行くのか #sekkeinight
a_suenami
7
2.1k
公式版Scratchやtoio DoでIoT(ブラウザ上でのJavaScript実行で) / ビジュアルプログラミングIoTLT vol.11
you
0
150
miisan's career talk
mii3king
0
210
2022年度新卒技術研修「 ソフトウェアテスト」講義
excitejp
PRO
0
320
1人目SETとして入社して2ヶ月の間におこなったこと
tarappo
3
390
FoodTechにおける商流・金流・物流の進化/Evolution of Commercial, Financial, and Logistics in FoodTech
dskst
0
340
Camp Digital 2022: tailored advice
kyliehavelock
0
120
MRTK3 - DataBinding and Theming 入門
futo23
0
120
Biz・経営・Dev+PMがOne Teamであり続けるために効果的だったこと / CTO-PM_vol2_presentation_LMI_220609
lmi
2
460
JDK Flight Recorder入門
chiroito
1
470
Microsoft Build2022 最新アップデートAI/ML
shisyu_gaku
1
390
Featured
See All Featured
Writing Fast Ruby
sferik
612
57k
Building Flexible Design Systems
yeseniaperezcruz
310
34k
Automating Front-end Workflow
addyosmani
1351
200k
A better future with KSS
kneath
225
15k
Put a Button on it: Removing Barriers to Going Fast.
kastner
56
2.3k
Building Better People: How to give real-time feedback that sticks.
wjessup
344
17k
Java REST API Framework Comparison - PWX 2021
mraible
PRO
11
4.7k
How to Ace a Technical Interview
jacobian
265
21k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
15
940
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
237
19k
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
498
130k
The Illustrated Children's Guide to Kubernetes
chrisshort
14
36k
Transcript
Ugoku backend Ugoira Tech Talks / 2014-08-01 harukasan
• Infrastructure Team in pixiv since 2012 • nginx /
Apache Traffic Server • Fluentd / Kibana / BigQuery • Kosenconf Harukasan / Shunsuke MICHII ! ൃച @harukasan shunsuke.michii
Ugoku Backend ͏͝ΠϥͷཪଆͰ
Works in Ugoira Project • ͏͍͝Βͷ͚ͩͷͨΊʹؾΛݣͬͨ෦ ͦΕ΄Ͳͳ͍ • ಛච͖͢ࣄฑ͋·Γͳ͍…… •
ͲͪΒ͔ͱ͍͏ͱৗʹ͕Μ͍ͬͯΔ͔Μ͡
Ugoku Backend
None
Statistics 3,700,000,000 PV/month 11,000,000 Users 45,000,000 Illustrations |
Statistics 7K peak req/s (dynamic) 30K peak req/s (image) 12Gbps
peak Traffic (image) 3 x 10Gbps backbone 40TB image files 400 servers
Ugoku Backend ͏͝ΠϥͷཪଆͰ
Ugoira Data Flow PHP nginx nginx Apache HTTPd nginx pixiv
Contents Delivery Clusters uploading small light load balancing Store Images generating thumbnails Ý Creator Viewers PHP Redis queing
Ugoira Data Flow PHP nginx nginx Apache HTTPd nginx pixiv
Contents Delivery Clusters uploading small light load balancing Store Images generating thumbnails Ý Creator Viewers PHP Redis queing
Upload System
Upload System • ࠷େ150ຕͷը૾ͷมॲཧΛߦ͏ඞཁ͕͋Δ • ϢʔβʔΛग़དྷΔ͚ͩͨͤͳ͍Α͏ʹ͍ͨ͠
Asynchronous Uploading Upload View Information form View Complete View 1.
ը૾ϑΝΠϧΛ Ξοϓϩʔυ 2. ใೖྗ ◦ ྃΛͭ Redis Uplaod Worker upload αϜωΠϧϚελը૾ੜ Ý upload Ý GIFը૾ׂ active standby 3. Ξοϓϩʔυྃ
Upload System • Ϣʔβʔ͕ใΛೖྗ͍ͯ͠Δؒʹ ը૾ϑΝΠϧΛ४උ • GIFը૾Λׂ Github: pixiv/gifsplit •
͏͝ΠϥͰ3छྨͷը૾Λ༻ҙ • WebDAVͰΞοϓϩʔυ͢Δ PHP nginx Redis php-resque nginx ϩʔυόϥϯα ΞϓϦέʔγϣϯ Ωϡʔ ϫʔΧʔ DAVετϨʔδ
Ugoira Data Flow PHP nginx nginx Apache HTTPd nginx pixiv
Contents Delivery Clusters uploading small light load balancing Store Images generating thumbnails Ý Creator Viewers PHP Redis queing
Dynamic Thumbnail Generation
Dynamic Thumb. Generation • Ξοϓϩʔυ࣌ʹͨ͘͞ΜͷαϜωΠϧΛ ͭ͘Δͷେม • αΠζσβΠϯมߋͱ͔Ͱม͍͑ͨ • ༧ΊϚελը૾Λੜͯͦ͠Ε͔Βੜ͢Δ
ΦϦδφϧը૾ αϜωΠϧϚελը૾ αϜωΠϧը૾
Thumbnail Master • ΞεϖΫτݻఆͱਖ਼ํܗΫϩοϓͷ2छྨ • JPEGը૾ʹม(PNG→JPEGͷมॏ͍) 1200px 1200px 1200px 1200px
1200px 1200px ඪ४αΠζ(1200x1200ʹऩ·ΔαΠζ) ਖ਼ํܗΫϩοϓ(ԣͱॎͰΫϩοϓํ๏͕ҟͳΔ) ԣதԝ߹Θͤ ॎ্߹Θͤ
• αϜωΠϧੜmod_small_lightΛ༻ https://github.com/yamac/smalllight ! • ετϨʔδ͔ΒԿϚελը૾Λ औಘ͠ͳ͍Α͏ʹΩϟογϡΛ͞Ή Dynamic Thumb. Generation
Small Light ετϨʔδ Ωϟογϡ Ωϡʔ nginx Ý nginx ৼΓ͚
Ugoira Data Flow PHP nginx nginx Apache HTTPd nginx pixiv
Contents Delivery Cluster uploading small light load balancing Store Images generating thumbnails Ý Creator Viewers PHP Redis queing
Contents Delivery Cluster
• εέʔϥϒϧ • Ԇ (< 100ms) • ߴ͍ώοτ (> 95%)
• ઃఆ͕มߋ͍͢͠ Storage pixiv Contents Delivery Clusters Contents Delivery Cluster
Inside of pixiv CDC 10Gbps 10Gbps nginx nginx nginx ATS
ATS ATS nginx nginx nginx Hashing 64GB Memory Caching (60%) 3x 256GB SSD Caching (90%) 1Gbps
Ugoira Compatibility • ZIPϑΝΠϧͷ৴ • CORS Non-simpleϦΫΤετͷԠ
CORS • ผυϝΠϯͷϦιʔεͷϦΫΤετΛڐՄ͢Δ • simple requests / non-simple requests ͷ2͕ͭ͋Δ
• Ϟμϯϒϥβ͍͍ͩͨରԠ
CORS Non-Simple Request someone.example.com www.pixiv.net/api var req = new XMLHTTPRequest();
req.open(“GET”, “//www.pixiv.net/api”); req.setRequestHeader(“Range”, “0-10000”); req.send(); GET /api HTTP/1.1 Origin:someone.example.com Range: 0-10000 ϒϥβͰϒϩοΫ͞ΕΔ
someone.example.com www.pixiv.net/api OPTIONS /api HTTP/1.1 Origin:someone.example.com Access-Control-Request-Headers: range Access-Control-Request-Method: GET
Preflight request HTTP/1.1 200 OK Access-Control-Allow-Headers: Range Access-Control-Allow-Methods: GET, POST, OPTIONS Access-Control-Allow-Origin:* Access-Control-Expose-Headers: Content-Length Access-Control-Max-Age: 2592000 ڐՄ͢ΔϦΫΤετϔομ ڐՄ͢ΔϦΫΤετϝιου ڐՄ͢ΔϦΫΤετݩΦϦδϯ Script͔Βར༻Ͱ͖ΔϨεϙϯεϔομ ΩϟογϡՄೳ࣌ؒ ڐՄΛٻΊΔϦΫΤετϔομ ڐՄΛٻΊΔϦΫΤετϝιου GET /api HTTP/1.1 Origin:someone.example.com Range: 0-10000
Ugoira Data Flow PHP nginx nginx Apache HTTPd nginx pixiv
Contents Delivery Cluster uploading small light load balancing Store Images generating thumbnails Ý Creator Viewers PHP Redis queing
Complete!
Conclusion • ͏͝ΠϥͷόοΫΤϯυී௨ʹ͏͍͍ͯ͝Δ • ៉ྷͳγεςϜྑ͍͚Ͳ͏͘͝ͷલఏ • ͏্͘͝ʹ֦ு͍͢͠ͷॏཁ