LINE LIVE's Development Challenge - 2021 early

TECH STAND #4 常時接続SNS~ライブ配信基盤を支える技術~

「LINE LIVEで経験してきた課題や解決策などをぶっちゃける」
LINE株式会社 開発3センター サービス開発1室 室長 大沢和宏 (@Yappo : https://twitter.com/Yappo )

Transcript

1. LINE LIVEͰܦݧ͖ͯͨ͠ ՝୊΍ղܾࡦͳͲΛͿͬͪΌ͚Δ > Kazuhiro Osawa / @Yappo TECH STAND #4 ৗ࣌઀ଓSNS~ϥΠϒ഑৴ج൫Λࢧ͑Δٕज़~ 2021/05/14
2. ࣗݾ঺հ › ։ൃ3ηϯλʔ αʔϏε։ൃ1ࣨ › LINEͷ֤छαʔϏε։ൃ୲౰ޙʹ 2015೥ͷLINE LIVE։ൃ։࢝࣌ΑΓ LINE LIVEͷ։ൃΛ୲౰ › ݱࡏ͸ଞͷΤϯλϝܥαʔϏεͷ։ൃ΋୲౰ ͢Δ෦໳ͷࣨ௕Λͯ͠Δ
3. Agenda › LINE LIVEʹ͍ͭͯ › LINE LIVEͳΒͰ͸ͷ೉͍͠ͱ͜Ζ › ՝୊ղܾͷͨΊͷऔΓ૊Έ
4. #01 LINE LIVEʹ͍ͭͯ
5. αʔϏε֓ཁ › ϦϦʔεલ: LINE Appͷதʹੜ഑৴ΛࢹௌͰ͖Δػೳ͕ఏڙ͞Ε͍ͯͨ › 2015೥: ܳೳਓ΍ΞʔςΟετͳͲͷੜ഑৴ΛࢹௌͰ͖Δ LINE LIVE AppΛϦϦʔε › 2016೥: ݸਓͰ΋ಈը഑৴ΛՄೳʹ͢Δ഑৴ػೳΛఏڙ › LINE AppͷதͷLIVEϓϨΠϠʔͱLINE LIVE AppͷϓϨΠϠʔͷ ιʔείʔυΛڞ௨ʹ͢Δ։ൃମ੍੔උ(࠷ۙ͸৭ʑߟ͑ͯ෼཭͢Δ͜ͱʹͨ͠) › 2020೥: ༗ྉΠϕϯτͳͲͷΦϯϥΠϯ഑৴Λߦ͏LINE LIVE-VIEWINGΛ։࢝
6. Tech Stack › Java, Kotlin, Spring Boot, Akka › Private Cloud › MySQL, Redis, Elasticsearch, Kafka › WebSocket, RTMP, HLS › Native app͸Swift, Kotlin › ※ಈըαʔό͸ࣗࣾ։ൃ
7. Object Storage Media Servers Host CDN LINE Talk Server LIVE Billing CMS API Server LINE LIVE Architecture RTMP Chat Server JSON API WebSocket Upload HLS files Cache
8. Object Storage Media Servers Host CDN LINE Talk Server Billing CMS API Server LINE LIVE Architecture (LINE app) RTMP Chat Server JSON API Upload HLS files Cache LINE App WebSocket
9. ࣮͸਺ଟ͘ͷPivotΛ͍ͯ͠Δ
10. ࣮͸਺ଟ͘ͷPivotΛ͍ͯ͠Δ › গ਺ͷஶ໊ͳํ͕େن໛ͳࢹௌऀʹ഑৴Λఏڙ͢ΔαʔϏε͔Βɺଟ਺ͷҰൠ഑৴ऀ͕খத ن໛ͷࢹௌऀʹ഑৴Λఏڙ͢ΔαʔϏε΁ › LINE LIVE appͷΈࢹௌͰ͖͍͕ͯͨLINE app͔Β΋ࢹௌՄೳʹ › LINE appͷ୆࿷ɺλΠɺΠϯυωγΞͷϢʔβʹରͯ͠ػೳఏڙ › ݸਓ഑৴ऀ͕ΑΓ׆༂͕ՄೳʹͳΔαʔϏεڧԽΛ࣮ࢪ › LINE NEWSͳͲͱͷϦΞϧλΠϜ࿈ܞ › ༗ྉ഑৴։࢝ › (ٕज़తʹ͸Web ϑϨʔϜϫʔΫ΍ࣾ಺Πϯϑϥͷมߋ࡞ۀͳͲͳͲ΋͋Γ)
11. #02 LINE LIVEͳΒͰ͸ͷ೉͠͞
12. 5೥Ҏ্ͷྺ࢙ͷதͰ มԽ͖ͯͨ͜͠ͱʹΑΔෳࡶੑ › Pivot͔ͨ͠Βͱ͍ͬͯ΋ɺաڈʹఏڙ͍ͯͨ͠αʔϏε಺༰͸ফ͑ͳ͍ › ৗʹAdd-on͞Ε͍ͯ͘ͷͰγεςϜ͕ͲΜͲΜେ͖͘ͳΔ (ͱ͸͍͑ɺফ͑ͨػೳ͸͋Δ͜ͱ͸͋Δ) › ࣾ಺Ͱͷ࿈ܞઌαʔϏε΋૿͑ଓ͚Δ ྫ͑͹Web্Ͱఏڙ͍ͯ͠ΔLINE LIVE-VIEWINGͷ։࢝࣌ʹ͸ LINE STOREͱLINEνέοτͱ࿈ܞͯ͠ΦϯϥΠϯνέοτൢചΛ࣮ࢪ
13. ө૾඼࣭΋؆୯ʹܾ·Βͳ͍ › ίϯςϯπʹΑͬͯը࣭ͷڐ༰౓͕มΘͬͯ͘Δ › େܕΞʔςΟετͷίετΛ͔͚ͨεςʔδઃඋͱࣗ୐ͳͲ͔Β഑৴͢Δ ݸਓ഑৴Ͱ͸ө૾Λ៉ྷʹఏڙ͢ΔͨΊͷϏοτϨʔτ΋มΘͬͯ͘Δ › ө૾ͷ஗Ԇ଎౓ʹର͢Δҙࣝͷ࣋ͪํͷҧ͍ › ࢹௌऀʹ޲͚ͨίϯςϯπσϦόϦʔͷ඼࣭΍ίετײ΋͔ΘΔ
14. APIαʔό΁ͷෛՙ΁ͷߟ͑ํ › ΍͸Γɺ༑ͩͪ਺͕ଟ͍LINEެࣜΞΧ΢ϯτͰͷ഑৴͸ෛՙ͸ଟ͍ ਺஋͸ඇެදͰ͕͢ɺগͳ͘ॻ͍ͯ΋ಉ࣌઀ଓ਺͕ສ୯Ґͷ഑৴͸ൃੜ͢Δ › ௒ਓؾΞʔςΟετͷ഑৴ͳͲ͕͋Δͱɺඵؒͷϝοηʔδసૹ਺΋݁ߏଟ͍ Chat Server΁ͷ઀ଓ਺ * ඵؒ͋ͨΓͷॻ͖ࠐΈ਺ʹͳΔͷͰɻɻ › ৗ࣌େن໛഑৴͕ൃੜ͢ΔΘ͚Ͱ΋ແ͍ͷͰɺՋͳ࣌ͷαʔόϦιʔεͷແବ ݣ͍ײ͕՝୊
15. ࿈ܞઌͷ։ൃνʔϜͷଟ͞ › LINEͷαʔϏε։ൃશൠతʹɺଞͷ։ൃνʔϜͱͷ࿈ܞ͸ଟ͘ͳΓ͕ͪ ྑ͍ѱ͍Ͱ͍͏ͱɺ͜Ε͸ྑ͍ࣄͰͯ͠ɺಛఆͷ෼໺ͷ։ൃͷઐ໳νʔϜ ͕͋Δ͜ͱͰɺϢʔβ͕௚઀৮ΕΔαʔϏεΛ։ൃ͢ΔΤϯδχΞ͸Ϣʔ βͷ͜ͱ͚ͩΛߟ͑ͯ։ൃʹूதͰ͖Δ › ྑ͍͜ͱͰ͸͋Δ͕ɺίϛϡχέʔγϣϯ͸૿͑ͯ͠·͏ͷͰɺ͜ͷลΓ Λޮ཰తʹڠۀ͍ͯ͘͜͠ͱ΋ߟ͑ͳ͍ͱͳΒͳ͍
16. #03 ՝୊ղܾͷͨΊͷऔΓ૊Έ
17. มԽ͕͋Δ͜ͱΛ೔ৗԽ͢Δ › ଟ͘ͷਓ͸ݱঢ়͔ΒԿ͔มԽ͢Δ͜ͱ͸ۤखͳΜͩͱࢥ͍ͬͯΔ › খ͞ͳϦϑΝΫλϦϯάۀ຿Ͱ΋ྑ͍ͷͰɺৗ೔ࠒ͔ΒԿ͔͕มΘ͍ͬͯ͘ͱ ͍͏ঢ়گʹͳ͍ͬͯΔͷ͸ྑ͍ͷ͔΋ͱࢥ͏ › มԽʹ׳Εͨঢ়ଶʹͳΔͱɺ͋Δ೔ಥવαʔϏε͕มΘΖ͏ͱͯ͠΋ॗʑͱର ԠͰ͖ΔΑ͏ʹͳΔͷ͔ͳɻͱ › ※ແཧͯ͠ෆཁͳมߋ͸͔͚ͳ͍Ͱɺඞཁͦ͏ͳมߋ͚ͩ΍ͬͯΔͭ΋Γɻɻ
18. #03 ՝୊ղܾͷͨΊͷऔΓ૊Έ ։ൃͱؔ܎ͳ͍࿩ʹͳͬͨͷͰ࢓੾Γ௚͠
19. ଞͷ։ൃνʔϜͱͷ࿈ܞ › LIVEͷ഑৴։࢝ʹ߹Θͤͯίϯςϯπߋ৽͢ΔܥͷαʔϏεͳͲͷ࿈ܞ΋ ग़͖͍ͯͯΔͷͰɺKafkaΛ࢖ͬͯ͋·Γਂ͘ߟ͑ͣʹૄ݁߹͢Δํ਑
20. ߴը࣭Խ › LINE LIVE-VIEWING͕࢝·Δ·Ͱ͸720p͕࠷ߴը࣭ͩͬͨ › PCͳͲͰϥΠϒϏϡʔΠϯάΛݟΑ͏ͱͨ࣌͠ʹ720pͰ͸ඍົͳͨΊ 1080pͰࢹௌ͕ՄೳͳΑ͏ʹมߋ › εςʔδ্ͷԋग़ʹΑͬͯ͸ը࣭͕མͪΔέʔε͕͋ΔͷͰɺ࠷దͳϏο τϨʔτͷௐࠪݚڀΛ͠ɺϏοτϨʔτͳͲͷҾ͖༲͛Λ͠ը࣭Λ্͛Δ › ͨͩ͠ɺLINE LIVE app͔Βͷݸਓ഑৴Ͱɺ͜͜·Ͱͷը࣭ʹͯ͠͠·͏ͱ ίετ͕େ෯ʹ௓Ͷ্͕ΔͷͰɺͦͷลΓͷௐ੔͸͍ͯ͠Δ
21. APIαʔόͷෛՙରԠ › K8s͍ΕͯΦʔτεέʔϦϯάͰɻɻɻͱݴ͍͍ͨͱ͜ΖͰ͕͢ɺLINE LIVEͷτϥϑΟοΫ܏޲ͩͱΦʔτεέʔϧ͕ؒʹ߹ΘΜͷͰ࣮௚ʹɻɻ
22. APIαʔόͷෛՙରԠ › K8s͍ΕͯΦʔτεέʔϦϯάͰɻɻɻͱݴ͍͍ͨͱ͜ΖͰ͕͢ɺLINE LIVEͷτϥϑΟοΫ܏޲ͩͱΦʔτεέʔϧ͕ؒʹ߹ΘΜͷͰ࣮௚ʹɻɻ › k6ͱ͍͏΋ͷΛ࢖ͬͯޮ཰తʹϕϯνϚʔΫΛऔΕΔ؀ڥΛ࡞ͬͯӡ༻։ ࢝͢Δ͜ͱʹ
23. ϕϯνϚʔΫγφϦΦྫ IUUQTLJP
24. Load Testing Architecture Slack Bot API Lode Test Manager GitHub Sync scenario files Load Test Node Load Test Node Load Test Node Target Servers with exporters MySQL Redis .stress and scenario ID Alert Manager Detect dashboard urls and send images Datasource
25. › Engineers can execute test via slack › Engineers can specify “load variable” like › simultaneous viewers num “.stress” for Load Testing Repeatability
26. › Abstract of the result is notified to Slack › Important server metrics is also notified via Bot Result Notification Understandability
27. ࣮͸ࠓ͸΋͏গ͠ҧ͏΋ͷΛ࢖ͬͯΔ › ࣾ಺Ͱk6Λϕʔεʹͯ͠Stampedeͱ͍͏΋ͷΛ࡞ͬͯΔ › ෛՙςετΛ࣮ߦ͢ΔͨΊͷAPI͕੔උ͞Ε͍ͯͨΓ › ෛՙςετͷ݁Ռ͕GrafanaͰ࣌ܥྻͰݟΕͨΓͱ͔ › ޮ཰Α͘ϕϯνϚʔΫ͕ӡ༻Ͱ͖ΔΑ͏ʹ੔උத
28. #04 ࠓޙ΍͍͖͍ͬͯͨ͜ͱ
29. αʔόϦιʔεͷޮ཰Խ › ࣮͸֤ࠃͷେن໛഑৴Λ؅ཧ͢ΔΧϨϯμʔγεςϜΛ࡞ͬͯӡ༻͍ͯ͠Δ › ͜ͷεέδϡʔϧ͞Εͨ৘ใΛ΋ͱʹͯ͠K8sͷPodͷ؅ཧΛࣗಈԽ͢Δ࢓ ૊Έ͕࡞ΕΔΜ͡Όͳ͍͔ͱࢥͬͯΔ › Chat Server΋ৗʹશྗτϥϑΟοΫΛड͚ΔͨΊͷߏ੒ʹͳ͍ͬͯΔͷ Ͱɺϝοηʔδ഑ૹΛ୲౰͢ΔαʔόͷόϥϯγϯάͳͲΛޮ཰తʹ͢Ε ͹ίετ͸ݮΒͤͦ͏
30. ΋͏ͪΐͬͱͪΌΜͱʹmicroservices › ͬ͘͟Γͱͨ͠microservicesʹͳ͍ͬͯΔ͕ɺ΋͏গ͠໾ׂʹԠͯ͡αʔ όΛ෼཭͍ͨ͠ › LINE LIVEͩͱɺαϒαʔϏεͷछྨ΋ଟ͘ɺͦΕΒαϒαʔϏε͝ͱʹ ظ଴͞ΕͯΔϐʔΫύϑΥʔϚϯε΍ݎ࿚ੑͳͲ΋มΘ͍ͬͯΔͷͰɺ୲౰ ྖҬͰαʔόΛ෼͚Δ͜ͱͰޮՌతͳӡ༻Λߦͳ͍͖͍ͬͯͨ
31. ଞʹ΋΍Γ͍ͨ͜ͱ͸ଟ͍ › ͕ɺݱঢ়֎ͩ͠ग़དྷΔ/࿩͠΍͍͓͢࿩͸͜͜·Ͱ
32. ҰॹʹαʔϏεΛਐԽͤͯ͘͞ΕΔ ஥ؒΛઈࢍٸืதͰ͢ʂ › αʔόʔαΠυΤϯδχΞ / LINEϑΝϛϦʔαʔϏε https://linecorp.com/ja/career/position/241 › iOS/Android / LINEϑΝϛϦʔαʔϏε https://linecorp.com/ja/career/position/859 › ςΫχΧϧϓϩμΫτϚωʔδϟʔ / LINEϑΝϛϦʔαʔϏε https://linecorp.com/ja/career/position/2246
33. Thank You!