ピクシブのコンテンツ配信基盤技術 / pixiv TECH SALON

ピクシブのコンテンツ配信基盤技術
pixiv TECH SALON / 2018-03-06
https://techsalon.pixiv.co.jp/

Transcript

1. ϐΫγϒͷ഑৴ج൫ٕज़  pixiv Technologies Inc. Harukasan / ಓҪ ढ़հ
2. Harukasan ಓҪ ढ़հ 2012೥౓৽ଔೖࣾ
 Πϯϑϥ෦ˠImageFluxࣄۀ্ཱͪ͛ˠ഑৴ٕज़෦Ϛωʔδϟʔ
 ϐΫγϒςΫϊϩδʔζגࣜձࣾࣥߦ໾һ
 γχΞςοΫϦʔυ
3. ϐΫγϒͷ੒௕Λࢧ͖͑ͯͨ഑৴ٕज़
4. Infrastructure ImageFlux Live Streaming Future ϐΫγϒͷ੒௕Λࢧ͖͑ͯͨ഑৴ٕज़
5. Infrastructure
6. B28 ϕχϠػ
8. ऩ·Βͳ͍ճઢ ٧·Δہࣷ ᫔᫓͢ΔωοτϫʔΫ ৴པੑ
9. スパゲッティーコード（笑）
10. 1G private lines ! SERVER ROOM SHINJUKU DC 1G x 8 Internet Cache Origin
11. ! FUKUSHIMA
 Shirakawa DC 10G x 8 Internet
12. pixiv͚ͩͰͳ͘શαʔϏεڞ௨Ͱ࢖͑Δը૾഑৴ج൫ ߴूੵαʔόΛಋೖ (2U 24SSD) αʔϏε͔Βಠཱͨ͠ߏ੒ ϑϧ10GωοτϫʔΫ ༰ྔޮ཰UP PXIMG
13. ImageFlux
14. PC iOS Android SP FP 20+ images
15. ॲཧ͕ෳࡶͰख͕෇͚ΒΕͳ͍
 ౤ߘʹ͕͔͔࣌ؒΔ ը૾αΠζ͕খ͍͞ ಈతม׵͢Δʹ΋CPUϦιʔε͕ਏ͍ ैདྷͷը૾ม׵ͷ໰୊఺
16. go-thumber Gopher image was created by Takuya Ueda (https://twitter.com/tenntenn).
 Licensed under the Creative Commons 3.0 Attributions license. https://github.com/pixiv/go-thumber αϜωΠϧม׵ϓϩΩγ
17. ͏·͍
    ΍͍͢
    ͸΍͍
18. ߴ඼࣭
    লϝϞϦ
    ௿஗Ԇ
19. Faster JPEG resizing Decode Y'CbCr→RGB RGB→Y'CbCr Encode Resize Buffer ImageMagick
20. Faster JPEG resizing Decode Y'CbCr→RGB RGB→Y'CbCr Encode Resize Buffer ImageMagick Decode Encode Resize go-thumber Buffer
21. Faster JPEG resizing Decode Y'CbCr→RGB RGB→Y'CbCr Encode Resize Buffer ImageMagick Decode Encode Resize go-thumber Buffer 3x
 faster
22. αϜωΠϧϚελͷੜ੒ ΦϦδφϧը૾ αϜωΠϧϚελJPEG Ξοϓϩʔυ࣌ੜ੒ go-thumberͰੜ੒
23. pixivͷશαϜωΠϧΛಈతੜ੒ʹஔ͖׵͑ ޷͖ͳαΠζͷαϜωΠϧΛੜ੒Ͱ͖ΔΑ͏ʹ ଞαʔϏεʹಋೖ͢Δʹ͸αϜωΠϧϚελ͕ඞཁ go-thumberʹΑΔը૾ม׵
24. શαʔϏεԣஅͰ഑৴ج൫͕ར༻Ͱ͖Δঢ়ଶͰ͸ͳ͍ ը૾ϑΥʔϚοτʹର͢Δਖ਼֬ͳ஌͕ࣝཁٻ͞ΕΔ αʔϏεΛ࡞Δͨͼʹը૾ʹ͍ͭͯߟ͑Δͷ͸ਏ͍ ͦ΋ͦ΋ը૾ม׵͕΍Γ͍ͨΘ͚Ͱ͸ͳ͍ go-thumberͷ໰୊఺
25. ϐΫγϒ͕ը૾ม׵ٕज़ʹ౤ࢿ͠ଓ͚Δʹ͸ શαʔϏεԣஅͰར༻Ͱ͖Δ഑৴ج൫Λ։ൃ͢Δʹ͸ ը૾ม׵͸ΠϯϑϥαʔϏεͰ͋Δ΂͖Ͱ͸
26. ։ൃαΠΫϧΛଅਐ͢ΔΫϥ΢υը૾ม׵αʔϏε
27. ։ൃαΠΫϧΛଅਐ͢ΔΫϥ΢υը૾ม׵αʔϏε Base:
 go-thumber
 pixiv഑৴Ϋϥελ Format:
 JPEG, PNG, GIF
 and WebP Origin:
 HTTPS and S3 Infrastructure: ͘͞ΒΠϯλʔωοτ
 ίϯςϯπ഑৴ج൫ JPEG PNG WebP HTTPS
28. γεςϜ։ൃ Ӧۀ/αϙʔτ Πϯϑϥӡ༻ " ͓٬༷
29. ImageFluxΛ͝ར༻ͷ͓٬༷
30. PXIMG Ϋϥ΢υαʔϏε ࣾ಺഑৴ج൫ ίʔυϕʔεɾϊ΢ϋ΢Λڞ༗ ͘͞ΒΠϯλʔωοτӡ༻ ࣾ಺Πϯϑϥ෦ӡ༻ ߴՄ༻ੑ ௿ίετ ؅ཧίϯιʔϧ ϨΨγʔͳઃఆϑΝΠϧ
31. JPEGɺPNGɺGIFʹՃ͑ͯWebPग़ྗʹରԠ ύϥϝʔλॺ໊ػೳ ύε͝ͱͷσϑΥϧτύϥϝʔλ ΧϥʔϓϩϑΝΠϧ΁ͷରԠɺద༻ํ๏ݟ௚͠ ෳ਺ຕͷΦʔόʔϨΠʹରԠ ϦϦʔεޙͷػೳ௥Ճ
32. JPEGɺPNGͷຒΊࠐΈΧϥʔϓϩϑΝΠϧʹରԠ ೖྗઐ༻ɺຒΊࠐΈग़ྗͰ͖ͳ͍৔߹͸sRGBʹม׵ Color ProfileରԠ ICC ICC ICC ICC ICC LittleCMS2
33. type ScanLineReader interface {
 
 // Config returns the image configuration. Config() *Config
 
 // ReadScanLines reads scan lines into the p for each component. ReadScanLines(ctx context.Context, p [][]uint8) (int, error)
 } ֤ը૾ม׵ίϯϙʔωϯτΛؔ਺ͱ࣮ͯ͠૷ ը૾ม׵ύΠϓϥΠϯ func Scale(src ScanLineReader, width, height int) ScanLineReader
34. c := Decode(src) c = Clip(c, image.Rect(10,10,410,310)) c = Scale(c, 200, 150) Encode(rw, c) ը૾ม׵ύΠϓϥΠϯ Decode Clip Scale Encode ෳ਺ͷը૾ॲཧΛ࿈݁
35. Decode Scaling ඞཁͳॲཧΛඞཁͳ͚ͩͭͳ͛Δ Decode Y'CbCr→RGB Compositing Scaling RGB→Y'CbCr Encode Color transform
36. ImageFluxࣄۀ෦ͷൃ଍ʢݱ഑৴ٕज़෦ʣ ը૾഑৴ٕज़୯ମͰ։ൃαΠΫϧΛ·ΘͤΔΑ͏ʹ ར༻Ϣʔβ͕֦େ͢Δ͜ͱͰɺΑΓ඼࣭ɺػೳ͕޲্ ߴ౓ͳը૾ม׵ٕज़ʹ௅ઓͰ͖ΔΑ͏ʹ ImageFluxޙͷੈք
37. Live Streaming
38. pixiv Sketch LIVE ͓ֆ͔͖ΛΈΜͳͰϦΞϧλΠϜϥΠϒ഑৴
39. ϦϦʔε౰ॳͷ໰୊ WebRTC SFU͔Β௚઀ࢹௌऀʹ഑৴ ਺ઍਓͷಉ࣌ࢹௌΛ࣮ݱͰ͖ͳ͍ WebRTC - HLSม׵Λ࣮૷Ͱ͖ͳ͚Ε͹࣍ʹਐΊͳ͍
40. WebRTC - HLS WebRTC SFU Sora Streamer Transcoder RTP forwarding spawn GPU MPEG-TS PIPE M3U8 # " Cache ഑৴ऀ ࢹௌऀ
41. Live Streaming ߴ඼࣭ͳϥΠϒ഑৴Λखܰʹ࣮ݱ͢Δ
 ϦΞϧλΠϜಈըม׵
42. ϐΫγϒΛࢧ͖͑ͯͨ഑৴ٕज़ ௿ίετɺߴ඼࣭ͳ഑৴Λ࣮ݱͨ͠Πϯϑϥ؀ڥ ࣗࣾ։ൃΛଓ͚͖ͯͨը૾ɺಈըม׵ٕज़
43. Future
44. ίϯςϯπ͸ΑΓϦονʹɺΑΓߴ඼࣭ʹ ΑΓԕ͘ͷɺΑΓͨ͘͞ΜͷϢʔβʔʹಧ͚ΒΕΔΑ͏ʹ
45. ߴ଎ɺେ༰ྔ഑৴ͷݚڀ։ൃ Format:
 VP9 HEIF (HEVC) AV1 Protocol:
 WebRTC LHLS Infrastructure:
 100G network ෳ਺Ϧʔδϣϯ CDNͱͷ౷߹ Research:
 ػցֶश ετϦʔϜม׵
46. Fluent Delivery ͳΊΒ͔ͳ഑৴
47. 1ຕ͚ͩը૾͕ग़ͳͯ͘
 ΨΫοͱͳΔͷ͸͍͢͝ετϨε pixivΛӾཡͨ͠ͱ͖ʹ
 ετϨεͳ͘ӾཡͰ͖Δ ੈքதͰۉҰͳ඼࣭Λఏڙ͢Δ ͳΊΒ͔ͳ഑৴
48. ߴ଎
    େ༰ྔ
    ͳΊΒ͔
49. Infrastructure ImageFlux Live Streaming Future
50. ಧ͚͍ͨ΋ͷΛɺ͋ͨΓ·͑ʹಧ͚Δ
51. ͳΊΒ͔ͳ഑৴ٕज़ ը૾ɾಈըม׵ ϥΠϒετϦʔϛϯά ΤϯδχΞΛืू͍ͯ͠·͢ʂ