LINE LIVE's Development Challenge - 2021 early

by LINE Developers

Slide 1

Slide 1 text

LINE LIVEͰܦݧ͖ͯͨ͠ ՝୊΍ղܾࡦͳͲΛͿͬͪΌ͚Δ > Kazuhiro Osawa / @Yappo TECH STAND #4 ৗ࣌઀ଓSNS~ϥΠϒ഑৴ج൫Λࢧ͑Δٕज़~ 2021/05/14

Slide 2

Slide 2 text

ࣗݾ঺հ › ։ൃ3ηϯλʔ αʔϏε։ൃ1ࣨ › LINEͷ֤छαʔϏε։ൃ୲౰ޙʹ 2015೥ͷLINE LIVE։ൃ։࢝࣌ΑΓ LINE LIVEͷ։ൃΛ୲౰ › ݱࡏ͸ଞͷΤϯλϝܥαʔϏεͷ։ൃ΋୲౰ ͢Δ෦໳ͷࣨ௕Λͯ͠Δ

Slide 3

Slide 3 text

Agenda › LINE LIVEʹ͍ͭͯ › LINE LIVEͳΒͰ͸ͷ೉͍͠ͱ͜Ζ › ՝୊ղܾͷͨΊͷऔΓ૊Έ

Slide 4

Slide 4 text

#01 LINE LIVEʹ͍ͭͯ

Slide 5

Slide 5 text

αʔϏε֓ཁ › ϦϦʔεલ: LINE Appͷதʹੜ഑৴ΛࢹௌͰ͖Δػೳ͕ఏڙ͞Ε͍ͯͨ › 2015೥: ܳೳਓ΍ΞʔςΟετͳͲͷੜ഑৴ΛࢹௌͰ͖Δ LINE LIVE AppΛϦϦʔε › 2016೥: ݸਓͰ΋ಈը഑৴ΛՄೳʹ͢Δ഑৴ػೳΛఏڙ › LINE AppͷதͷLIVEϓϨΠϠʔͱLINE LIVE AppͷϓϨΠϠʔͷ ιʔείʔυΛڞ௨ʹ͢Δ։ൃମ੍੔උ(࠷ۙ͸৭ʑߟ͑ͯ෼཭͢Δ͜ͱʹͨ͠) › 2020೥: ༗ྉΠϕϯτͳͲͷΦϯϥΠϯ഑৴Λߦ͏LINE LIVE-VIEWINGΛ։࢝

Slide 6

Slide 6 text

Tech Stack › Java, Kotlin, Spring Boot, Akka › Private Cloud › MySQL, Redis, Elasticsearch, Kafka › WebSocket, RTMP, HLS › Native app͸Swift, Kotlin › ※ಈըαʔό͸ࣗࣾ։ൃ

Slide 7

Slide 7 text

Object Storage Media Servers Host CDN LINE Talk Server LIVE Billing CMS API Server LINE LIVE Architecture RTMP Chat Server JSON API WebSocket Upload HLS files Cache

Slide 8

Slide 8 text

Object Storage Media Servers Host CDN LINE Talk Server Billing CMS API Server LINE LIVE Architecture (LINE app) RTMP Chat Server JSON API Upload HLS files Cache LINE App WebSocket

Slide 9

Slide 9 text

࣮͸਺ଟ͘ͷPivotΛ͍ͯ͠Δ

Slide 10

Slide 10 text

࣮͸਺ଟ͘ͷPivotΛ͍ͯ͠Δ › গ਺ͷஶ໊ͳํ͕େن໛ͳࢹௌऀʹ഑৴Λఏڙ͢ΔαʔϏε͔Βɺଟ਺ͷҰൠ഑৴ऀ͕খத ن໛ͷࢹௌऀʹ഑৴Λఏڙ͢ΔαʔϏε΁ › LINE LIVE appͷΈࢹௌͰ͖͍͕ͯͨLINE app͔Β΋ࢹௌՄೳʹ › LINE appͷ୆࿷ɺλΠɺΠϯυωγΞͷϢʔβʹରͯ͠ػೳఏڙ › ݸਓ഑৴ऀ͕ΑΓ׆༂͕ՄೳʹͳΔαʔϏεڧԽΛ࣮ࢪ › LINE NEWSͳͲͱͷϦΞϧλΠϜ࿈ܞ › ༗ྉ഑৴։࢝ › (ٕज़తʹ͸Web ϑϨʔϜϫʔΫ΍ࣾ಺Πϯϑϥͷมߋ࡞ۀͳͲͳͲ΋͋Γ)

Slide 11

Slide 11 text

#02 LINE LIVEͳΒͰ͸ͷ೉͠͞

Slide 12

Slide 12 text

5೥Ҏ্ͷྺ࢙ͷதͰ มԽ͖ͯͨ͜͠ͱʹΑΔෳࡶੑ › Pivot͔ͨ͠Βͱ͍ͬͯ΋ɺաڈʹఏڙ͍ͯͨ͠αʔϏε಺༰͸ফ͑ͳ͍ › ৗʹAdd-on͞Ε͍ͯ͘ͷͰγεςϜ͕ͲΜͲΜେ͖͘ͳΔ (ͱ͸͍͑ɺফ͑ͨػೳ͸͋Δ͜ͱ͸͋Δ) › ࣾ಺Ͱͷ࿈ܞઌαʔϏε΋૿͑ଓ͚Δ ྫ͑͹Web্Ͱఏڙ͍ͯ͠ΔLINE LIVE-VIEWINGͷ։࢝࣌ʹ͸ LINE STOREͱLINEνέοτͱ࿈ܞͯ͠ΦϯϥΠϯνέοτൢചΛ࣮ࢪ

Slide 13

Slide 13 text

ө૾඼࣭΋؆୯ʹܾ·Βͳ͍ › ίϯςϯπʹΑͬͯը࣭ͷڐ༰౓͕มΘͬͯ͘Δ › େܕΞʔςΟετͷίετΛ͔͚ͨεςʔδઃඋͱࣗ୐ͳͲ͔Β഑৴͢Δ ݸਓ഑৴Ͱ͸ө૾Λ៉ྷʹఏڙ͢ΔͨΊͷϏοτϨʔτ΋มΘͬͯ͘Δ › ө૾ͷ஗Ԇ଎౓ʹର͢Δҙࣝͷ࣋ͪํͷҧ͍ › ࢹௌऀʹ޲͚ͨίϯςϯπσϦόϦʔͷ඼࣭΍ίετײ΋͔ΘΔ

Slide 14

Slide 14 text

APIαʔό΁ͷෛՙ΁ͷߟ͑ํ › ΍͸Γɺ༑ͩͪ਺͕ଟ͍LINEެࣜΞΧ΢ϯτͰͷ഑৴͸ෛՙ͸ଟ͍ ਺஋͸ඇެදͰ͕͢ɺগͳ͘ॻ͍ͯ΋ಉ࣌઀ଓ਺͕ສ୯Ґͷ഑৴͸ൃੜ͢Δ › ௒ਓؾΞʔςΟετͷ഑৴ͳͲ͕͋Δͱɺඵؒͷϝοηʔδసૹ਺΋݁ߏଟ͍ Chat Server΁ͷ઀ଓ਺ * ඵؒ͋ͨΓͷॻ͖ࠐΈ਺ʹͳΔͷͰɻɻ › ৗ࣌େن໛഑৴͕ൃੜ͢ΔΘ͚Ͱ΋ແ͍ͷͰɺՋͳ࣌ͷαʔόϦιʔεͷແବ ݣ͍ײ͕՝୊

Slide 15

Slide 15 text

࿈ܞઌͷ։ൃνʔϜͷଟ͞ › LINEͷαʔϏε։ൃશൠతʹɺଞͷ։ൃνʔϜͱͷ࿈ܞ͸ଟ͘ͳΓ͕ͪ ྑ͍ѱ͍Ͱ͍͏ͱɺ͜Ε͸ྑ͍ࣄͰͯ͠ɺಛఆͷ෼໺ͷ։ൃͷઐ໳νʔϜ ͕͋Δ͜ͱͰɺϢʔβ͕௚઀৮ΕΔαʔϏεΛ։ൃ͢ΔΤϯδχΞ͸Ϣʔ βͷ͜ͱ͚ͩΛߟ͑ͯ։ൃʹूதͰ͖Δ › ྑ͍͜ͱͰ͸͋Δ͕ɺίϛϡχέʔγϣϯ͸૿͑ͯ͠·͏ͷͰɺ͜ͷลΓ Λޮ཰తʹڠۀ͍ͯ͘͜͠ͱ΋ߟ͑ͳ͍ͱͳΒͳ͍

Slide 16

Slide 16 text

#03 ՝୊ղܾͷͨΊͷऔΓ૊Έ

Slide 17

Slide 17 text

มԽ͕͋Δ͜ͱΛ೔ৗԽ͢Δ › ଟ͘ͷਓ͸ݱঢ়͔ΒԿ͔มԽ͢Δ͜ͱ͸ۤखͳΜͩͱࢥ͍ͬͯΔ › খ͞ͳϦϑΝΫλϦϯάۀ຿Ͱ΋ྑ͍ͷͰɺৗ೔ࠒ͔ΒԿ͔͕มΘ͍ͬͯ͘ͱ ͍͏ঢ়گʹͳ͍ͬͯΔͷ͸ྑ͍ͷ͔΋ͱࢥ͏ › มԽʹ׳Εͨঢ়ଶʹͳΔͱɺ͋Δ೔ಥવαʔϏε͕มΘΖ͏ͱͯ͠΋ॗʑͱର ԠͰ͖ΔΑ͏ʹͳΔͷ͔ͳɻͱ › ※ແཧͯ͠ෆཁͳมߋ͸͔͚ͳ͍Ͱɺඞཁͦ͏ͳมߋ͚ͩ΍ͬͯΔͭ΋Γɻɻ

Slide 18

Slide 18 text

#03 ՝୊ղܾͷͨΊͷऔΓ૊Έ ։ൃͱؔ܎ͳ͍࿩ʹͳͬͨͷͰ࢓੾Γ௚͠

Slide 19

Slide 19 text

ଞͷ։ൃνʔϜͱͷ࿈ܞ › LIVEͷ഑৴։࢝ʹ߹Θͤͯίϯςϯπߋ৽͢ΔܥͷαʔϏεͳͲͷ࿈ܞ΋ ग़͖͍ͯͯΔͷͰɺKafkaΛ࢖ͬͯ͋·Γਂ͘ߟ͑ͣʹૄ݁߹͢Δํ਑

Slide 20

Slide 20 text

ߴը࣭Խ › LINE LIVE-VIEWING͕࢝·Δ·Ͱ͸720p͕࠷ߴը࣭ͩͬͨ › PCͳͲͰϥΠϒϏϡʔΠϯάΛݟΑ͏ͱͨ࣌͠ʹ720pͰ͸ඍົͳͨΊ 1080pͰࢹௌ͕ՄೳͳΑ͏ʹมߋ › εςʔδ্ͷԋग़ʹΑͬͯ͸ը࣭͕མͪΔέʔε͕͋ΔͷͰɺ࠷దͳϏο τϨʔτͷௐࠪݚڀΛ͠ɺϏοτϨʔτͳͲͷҾ͖༲͛Λ͠ը࣭Λ্͛Δ › ͨͩ͠ɺLINE LIVE app͔Βͷݸਓ഑৴Ͱɺ͜͜·Ͱͷը࣭ʹͯ͠͠·͏ͱ ίετ͕େ෯ʹ௓Ͷ্͕ΔͷͰɺͦͷลΓͷௐ੔͸͍ͯ͠Δ

Slide 21

Slide 21 text

APIαʔόͷෛՙରԠ › K8s͍ΕͯΦʔτεέʔϦϯάͰɻɻɻͱݴ͍͍ͨͱ͜ΖͰ͕͢ɺLINE LIVEͷτϥϑΟοΫ܏޲ͩͱΦʔτεέʔϧ͕ؒʹ߹ΘΜͷͰ࣮௚ʹɻɻ

Slide 22

Slide 22 text

APIαʔόͷෛՙରԠ › K8s͍ΕͯΦʔτεέʔϦϯάͰɻɻɻͱݴ͍͍ͨͱ͜ΖͰ͕͢ɺLINE LIVEͷτϥϑΟοΫ܏޲ͩͱΦʔτεέʔϧ͕ؒʹ߹ΘΜͷͰ࣮௚ʹɻɻ › k6ͱ͍͏΋ͷΛ࢖ͬͯޮ཰తʹϕϯνϚʔΫΛऔΕΔ؀ڥΛ࡞ͬͯӡ༻։ ࢝͢Δ͜ͱʹ

Slide 23

Slide 23 text

ϕϯνϚʔΫγφϦΦྫ IUUQTLJP

Slide 24

Slide 24 text

Load Testing Architecture Slack Bot API Lode Test Manager GitHub Sync scenario files Load Test Node Load Test Node Load Test Node Target Servers with exporters MySQL Redis .stress and scenario ID Alert Manager Detect dashboard urls and send images Datasource

Slide 25

Slide 25 text

› Engineers can execute test via slack › Engineers can specify “load variable” like › simultaneous viewers num “.stress” for Load Testing Repeatability

Slide 26

Slide 26 text

› Abstract of the result is notified to Slack › Important server metrics is also notified via Bot Result Notification Understandability

Slide 27

Slide 27 text

࣮͸ࠓ͸΋͏গ͠ҧ͏΋ͷΛ࢖ͬͯΔ › ࣾ಺Ͱk6Λϕʔεʹͯ͠Stampedeͱ͍͏΋ͷΛ࡞ͬͯΔ › ෛՙςετΛ࣮ߦ͢ΔͨΊͷAPI͕੔උ͞Ε͍ͯͨΓ › ෛՙςετͷ݁Ռ͕GrafanaͰ࣌ܥྻͰݟΕͨΓͱ͔ › ޮ཰Α͘ϕϯνϚʔΫ͕ӡ༻Ͱ͖ΔΑ͏ʹ੔උத

Slide 28

Slide 28 text

#04 ࠓޙ΍͍͖͍ͬͯͨ͜ͱ

Slide 29

Slide 29 text

αʔόϦιʔεͷޮ཰Խ › ࣮͸֤ࠃͷେن໛഑৴Λ؅ཧ͢ΔΧϨϯμʔγεςϜΛ࡞ͬͯӡ༻͍ͯ͠Δ › ͜ͷεέδϡʔϧ͞Εͨ৘ใΛ΋ͱʹͯ͠K8sͷPodͷ؅ཧΛࣗಈԽ͢Δ࢓ ૊Έ͕࡞ΕΔΜ͡Όͳ͍͔ͱࢥͬͯΔ › Chat Server΋ৗʹશྗτϥϑΟοΫΛड͚ΔͨΊͷߏ੒ʹͳ͍ͬͯΔͷ Ͱɺϝοηʔδ഑ૹΛ୲౰͢ΔαʔόͷόϥϯγϯάͳͲΛޮ཰తʹ͢Ε ͹ίετ͸ݮΒͤͦ͏

Slide 30

Slide 30 text

΋͏ͪΐͬͱͪΌΜͱʹmicroservices › ͬ͘͟Γͱͨ͠microservicesʹͳ͍ͬͯΔ͕ɺ΋͏গ͠໾ׂʹԠͯ͡αʔ όΛ෼཭͍ͨ͠ › LINE LIVEͩͱɺαϒαʔϏεͷछྨ΋ଟ͘ɺͦΕΒαϒαʔϏε͝ͱʹ ظ଴͞ΕͯΔϐʔΫύϑΥʔϚϯε΍ݎ࿚ੑͳͲ΋มΘ͍ͬͯΔͷͰɺ୲౰ ྖҬͰαʔόΛ෼͚Δ͜ͱͰޮՌతͳӡ༻Λߦͳ͍͖͍ͬͯͨ

Slide 31

Slide 31 text

ଞʹ΋΍Γ͍ͨ͜ͱ͸ଟ͍ › ͕ɺݱঢ়֎ͩ͠ग़དྷΔ/࿩͠΍͍͓͢࿩͸͜͜·Ͱ

Slide 32

Slide 32 text

ҰॹʹαʔϏεΛਐԽͤͯ͘͞ΕΔ ஥ؒΛઈࢍٸืதͰ͢ʂ › αʔόʔαΠυΤϯδχΞ / LINEϑΝϛϦʔαʔϏε https://linecorp.com/ja/career/position/241 › iOS/Android / LINEϑΝϛϦʔαʔϏε https://linecorp.com/ja/career/position/859 › ςΫχΧϧϓϩμΫτϚωʔδϟʔ / LINEϑΝϛϦʔαʔϏε https://linecorp.com/ja/career/position/2246

Slide 33

Slide 33 text

Thank You!