2022/07/21 Developers Summit 2022 Summer
イベントURL : https://event.shoeisha.jp/devsumi/20220721
গ৫Ͱ։ൃ͢ΔେنαʔϏεגࣜձࣾ TVer / Ճզ وࢤ (SRE)גࣜձࣾ TVer / ւ ܙհ (Backend) 2022/07/21 Developers Summit 2022 Summer
View Slide
͜ͷηογϣϯʹ͍ͭͯຊ͓͢Δ͜ͱ● TVerϦχϡʔΞϧʹ͓͚ΔϞχλϦϯάվળ● গਓͰ։ൃͱվળΛߦ͏ͨΊͷ New Relic׆༻ํ๏
ࣗݾհɾՃզ وࢤ (@TAKA_0411)ɾגࣜձࣾTVer ɹαʔϏεࣄۀຊ෦ SREɾझຯαφͱήʔϜͱVTuberɾϗʔϜαφߔޱԹઘ تָཬ
ΞδΣϯμ● TVerͷհ● αʔϏεϦχϡʔΞϧʹ͍ͭͯ● ϦϦʔε४උฤ● ϦϦʔεޙͷվળ● ·ͱΊ
TVerͷհ
TVerͷհ[TVer] 2022 3 ݄ͷಈը࠶ੜ͕ 2 ԯ 5 ઍສճ/݄Λಥഁ! https://tver.co.jp/news/20220426-1.html
TVerͷհTVerͷٕज़ελοΫ
TVerͷٕज़ελοΫ※ϑϩϯτΤϯυɺTVΞϓϦɺϞόΠϧΞϓϦέʔγϣϯ ֎෦ͷڠྗձࣾͷ͓ੈʹͳ͓ͬͯΓ·͢
TVerͷհTVerͷΠϯϑϥετϥΫνϟ
TVerͷΠϯϑϥετϥΫνϟWebαʔϏεͱͯ͠ͷTVer൪࿈ಈγεςϜͰ Քಇ͍ͯ͠ΔϚΠΫϩαʔϏε
TVer (WebαʔϏε) ͷΠϯϑϥετϥΫνϟ
αʔϏεϦχϡʔΞϧʹ͍ͭͯ
αʔϏεϦχϡʔΞϧʹ͍ͭͯ[TVer] ςϨϏίϯςϯπͷࢹௌମݧࠂମݧͷ্Λࢦ͢ ʮTVer IDʯͷఏڙΛ։࢝ https://tver.co.jp/news/20220404-1.html
αʔϏεϦχϡʔΞϧʹ͍ͭͯγεςϜͷมߋ● ΠϯϑϥͷԽ● όοΫΤϯυͷ৽● ϞχλϦϯάͷ৽
αʔϏεϦχϡʔΞϧʹ͍ͭͯγεςϜͷมߋ● ΠϯϑϥͷԽ● όοΫΤϯυͷ৽● ϞχλϦϯάͷ৽ɹ←ࠓճ͜͜ͷϦχϡʔΞϧΛػʹNew Relic OneΛϑϧ׆༻
ϞχλϦϯάͷ৽ϞχλϦϯάͷ৽● ͷཧ● ಋೖ͢Δػೳͷऔࣺબ● μογϡϘʔυͱσʔλͷ׆༻
ϞχλϦϯάͷ৽Before (ʙ202112݄)ʮNew RelicΛ ยखؒ ͰݟΔͷ͕ͭΒ͍ʂʯBackend γεΠϯϑϥNew RelicपΓͷରԠNew RelicपΓͷରԠNew RelicपΓͷରԠAPIͷ։ൃ FargateݕূෛՙςετΞΧϯτઃܭͱඋίετࢼࢉ͓Αͼܖ֤෦ॺ͔Βͷ͍߹ΘͤରԠνϡʔχϯάCI/CDඋཁ݅ཧΞϓϦϑϩϯτΤϯυͱͷௐΠϯϑϥߏஙͱCDKपΓͷඋ
After (20221݄ʙ)ϞχλϦϯάͷ৽SRE (ࢲ)New RelicपΓͷλεΫ·Δ͝ͱҾ͖ड͚·͢ͶΈΜͳ͡Ό͋ϦʔυΑΖ͘͠ʂ ଞͷλεΫਐΊΔͶ
࠷ۙϞχλϦϯάͷ৽Backend SREμογϡϘʔυͰ ͜ΜͳϝτϦΫε͕ݟ͍ͨΜ͚ͩͲʙ͜ͷ߹ʹΞϥʔτඈͯ͠ཉ͍͠ΑͶΓ͍ͨࣄΛԕྀͤͣʹ૬ஊɾ࣮ݱͰ͖Δମ੍○○ͱ͍͏ΛΩʔʹͯ͠ΤϥʔΛௐ͍ࠪͨ͠ͱࢥͬͯΔΘ͚ΑΓ·͢
ϞχλϦϯάͷ৽NRU101 New Relic One ೖ https://newrelic.com/jp/resources/presentations/nru101_20211027
ϞχλϦϯάͷ৽ࣗͷཧ● MELTΛશͯຬ͍ͨͨ͠○ Metrics, Events, Logs, Traces● ΤϥʔௐࠪͷͨΊʹTraces(APM)Λಋೖ͍ͨ͠
ϞχλϦϯάͷ৽ݱ࣮● New RelicͷGo Agent(APM)ͷಋೖ͕एׯ໘● APMͷΦʔόʔϔου͕ෆ҆● pprofͰྑ͘ͳ͍ʁ● ͍ͬͯ͏͔ࠓඞཁͳΜ͚ͩͬʁϦϦʔε·Ͱ͋ͱ2ϲ݄ɾɾɾ
ϞχλϦϯάͷ৽ͱΓ͋͑ͣ New Relic Go Agent Λඞཁ࠷খݶͷՕॴʹ͚ͩಋೖͨ͠
ϞχλϦϯάͷ৽
ϞχλϦϯάͷ৽Dev (Backend)ɹɹOps (SRE)ಋೖ݁Ռ● EchoͷIntegrationΛઃఆ͢Δ͚ͩͰେ෯ʹՄࢹԽͰ͖ͨ● Τϥʔൃੜ࣌ͷτϦΞʔδ͕ Մೳʹͳͬͨ● ࠷খݶͷ࿑ྗͰ࠷େͷޮՌΛಘΔ͜ͱ͕Ͱ͖ͨ
ϞχλϦϯάͷ৽গ৫ × New Relic One● ػೳ͕๛ͰΦʔϧΠϯϫϯ● ಋೖͷϋʔυϧ͕͍● υΩϡϝϯτɾαϙʔτɾίϛϡχςΟ͕ॆ࣮● ͱΓ͋͑ͣಋೖ͢Δ͚ͩͰେ෯ʹՄࢹԽͰ͖Δಋೖɾӡ༻ͷίετ͕࠷খݶͰࡁΜͩ
ϞχλϦϯάͷ৽ϞχλϦϯάͷ৽● ͷׂ● ಋೖ͢Δػೳͷऔࣺબ● μογϡϘʔυͱσʔλͷ׆༻
ϞχλϦϯάͷ৽ΫϥΠΞϯτͷಉ࣌ଓΫϥογϡͷϝτϦΫε
ϞχλϦϯάͷ৽ि1ͰϞόΠϧΞϓϦͷΫϥογϡωοτϫʔΫΤϥʔΛௐࠪ͠ɺࣾ͘ʹڞ༗
ϞχλϦϯάͷ৽μογϡϘʔυͱσʔλͷ׆༻● ඞཁͳϝτϦΫεͰμογϡϘʔυΛඋ● ϑϩϯτΤϯυ͔ΒόοΫΤϯυ·ͰΧόʔ● ΤϥʔΛௐࠪͯࣾ͠ʹڞ༗● APIͷվળʹAPMΛ׆༻
ࣗݾհɾւ ܙհ (@voidofglans)ɾגࣜձࣾTVer αʔϏεࣄۀຊ෦ ɹϦʔυΤϯδχΞɾ͖ͳͷ:ՐͰম͍ͨϗϧϞϯɾಸྑࡏॅͷԕִۈதɾNFCຒΊࠐΜͩΓ͢Δͷ͕͖
ϦϦʔε४උฤ● ϑΣʔζʹΑͬͯNew Relicͷ͍͚● গਓͰεϐʔυײΛམͱ͞ͳ͍ͨΊʹ● ύϑΥʔϚϯεɾνϡʔχϯά/ෛՙରࡦ
ϑΣʔζʹΑͬͯNew Relicͷ͍͚APMಋೖޙͷϦϦʔε·ͰͷϑΣʔζ● 2ϲ݄લ● 1ϲ݄લ● 2िؒલ
ϑΣʔζʹΑͬͯNew Relicͷ͍͚(2ϲ݄લ● Δ͖͜ͱόά௵͠● ओʹར༻ͨ͠ػೳ○ Errors Inbox○ Errors○ NRQL● 4xx/5xxܥΤϥʔͷ༰Λੳ○ 4xx: ΫϥΠΞϯτ͕ఆ͍ͯ͠ͳ͍ύϥϝʔλ͕ͳ͍͔○ 5xx: ఆ͍ͯ͠ͳ͍ॲཧλΠϜΞτ͕ͳ͍͔
ϑΣʔζʹΑͬͯNew Relicͷ͍͚(1ϲ݄લ● Δ͖͜ͱෛՙࢼݧʹΑΔϘτϧωοΫௐࠪ● ओʹར༻ͨ͠ػೳ○ Transactions○ Go Runtime○ CloudWatch Metrics● ߴෛՙ࣌ͷΞϓϦέʔγϣϯશମͷڍಈΛੳ○ TransactionsͰͷSlowest95percentileͳͲΛࢦඪʹ○ ϝϞϦϦʔΫɺ์ஔ͞Εͨgoroutineͷଘࡏ○ GC AllocationʹΑΔஅଓతͳੑೳྼԽ
ϑΣʔζʹΑͬͯNew Relicͷ͍͚(2िؒલ● Δ͖͜ͱཁٻੑೳͷୡ● ओʹར༻ͨ͠ػೳ○ Transactions○ CloudWatch Metrics● શମΛ௨ͨ͠ෛՙࢼݧͰͷ҆ఆύϑΥʔϚϯε○ 10msҎ্͔͔ΔAPI○ ֤ϛυϧΣΞͷΫΤϦଓͷ○ ॻ͖ࠐΈܥAPIͰͷLock,Waitͷ
গਓͰεϐʔυײΛམͱ͞ͳ͍ͨΊʹૣ͘ߦ͖͚ͨΕɺͻͱΓͰߦ͚ɻԕ͘·Ͱߦ͖͚ͨΕɺΈΜͳͰߦ͚ɻIf you want to go fast, go alone. If you want to go far, go together.
গਓͰεϐʔυײΛམͱ͞ͳ͍ͨΊʹ͓͚ͬʂ͡Ό͋ͪΐͬͱ1ਓͰߦͬͯ͘Δʂ
গਓͰεϐʔυײΛམͱ͞ͳ͍ͨΊʹϦϦʔε·ͰʹΈΜͳݱू߹Ͱʂ
● ͦΕͧΕ͕ͦΕͧΕͷ࠷Ͱతʹ● ͷ໌֬ͳղ● ໌֬ͳతͷڞ༗গਓͰεϐʔυײΛམͱ͞ͳ͍ͨΊʹ
● ίέͳ͍͜ͱ͕ॏཁ● Backend/Infra/SREͦΕͧΕ͕ઐੑΛ࣋ͬͯಠཱͨ͠ϝϯόʔ● ίϛϡχέʔγϣϯίετ͕࠷ݶʹ͑ΒΕͨগਓͰεϐʔυײΛམͱ͞ͳ͍ͨΊʹ
গਓͰεϐʔυײΛམͱ͞ͳ͍ͨΊʹ● ൫ͬͨ● ࣍ΈΜͳͰԕ͘ߦ͜͏
● Α͋͘ΔಥൃతͳεύΠΫ○ ্࿈ಈ์ૹΕʹΑΔྲྀೖ■ 1minޙ͖ͬ͞·Ͱͷ10ഒҎ্ͷϦΫΤετ■ ૿͑Δಉ࣌ଓ○ ΞΠυϧࣄॴλϨϯτ༷ग़ԋ൪○ ϦΞϧλΠϜ৴൪ऴྃ࣌■ Ϣʔβʔ͕Ұ੪ʹβοϐϯά■ ಉظͷͱΕͨສʙेສͷϢʔβʔΞΫγϣϯύϑΥʔϚϯεɾνϡʔχϯά/ෛՙରࡦ
ύϑΥʔϚϯεɾνϡʔχϯά/ෛՙରࡦ
● ϦϦʔεલʹSRE͕͔ͬ͠Γઐͯ͘͠Εͨ○ New RelicΛݟ͓͚ͯҰൠతͳࢦඪશ෦ݟΕΔ○ ఆ͞ΕΔΞΫηεύλʔϯʹ͓͚ΔMetricsͷมԽΛࣄલʹೝࣝ͢Δ○ ෛՙࢼݧ/ରࡦʹूத͢Δ͜ͱ͕Ͱ͖ͨύϑΥʔϚϯεɾνϡʔχϯά/ෛՙରࡦ
ϦϦʔεޙͷվળ● ԣஅతʹՄࢹԽ͞ΕͨMetricsͷԸܙ● ఆظతʹ࣮ࢪ͢ΔμογϡϘʔυோΊΔձ● ͷʹΑΔରԠͷ୲อ
ԣஅతʹՄࢹԽ͞ΕͨMetricsͷԸܙ● ҆ఆՔಇ͍ͯ͠Δ͜ͱ͕ԕͰΘ͔Δ҆৺ײ
ԣஅతʹՄࢹԽ͞ΕͨMetricsͷԸܙ● σόΠε͝ͱͷ࣮ʹΑΔࠩҟͷݕ○ σόΠεछผΛCustomAttributeʹ֨ೲ͢Δ○ ϦΫΤετΤϥʔͷѲ■ iOSΞϓϦ͚ͩಛఆͷϖʔδͰͷAPIϦΫΤετ͕ҟৗʹଟ͍⿌ ϦετදࣔͰཁૉ͝ͱʹAPIΛݺͼग़͍ͯͨ͠⿌ 30minΒͣͰݪҼಛఆˠ։ൃϕϯμʔͷڞ༗⿌ 3ޙͷϦϦʔεʹแؚ͠ɺແࣄղফ🎉
ԣஅతʹՄࢹԽ͞ΕͨMetricsͷԸܙ● ։ൃॳఆ͞Ε͍ͯͳ͍ϢʔβʔγφϦΦ○ ࢥͬͨͷͷ3ഒ͘Β͍ݕࡧػೳ͕ΘΕ͍ͯΔʂʂ○ ࢥͬͨΑΓxxͷػೳ͕ΘΕͯͳ͍ʂʂఆͱӡ༻ဃ͢Δͷ
● աͳϦιʔε͕͙͢Θ͔Δ○ ϦϦʔεॳظ҆৺ΛۚͰങ͍ͬͯͨͷͰ५ͳϦιʔεΛ༻ҙ͍ͯͨ͠○ ίετΠϯύΫτΛ࣠ʹॱ࣍Ϧιʔεͷ࠷దԽΛߦ͑ͨ■ ͪΖΜ͜ΕՄࢹԽ͞Ε͍ͯΔͷͰμϝͳ߹ͷݕਝԣஅతʹՄࢹԽ͞ΕͨMetricsͷԸܙ
● ࡢࠓؾʹͳͬͨࢦඪΛμογϡϘʔυʹՃ● Metricsͷҧײʹؾͮ͘ᄿ֮ͷ܇࿅● োޙτϥϒϧൃੜ࣌ͷݟͷूੵఆظతʹ࣮ࢪ͢ΔμογϡϘʔυோΊΔձ
ͷʹΑΔରԠͷ୲อ● ਖ਼͘͠ઐʹͤΔ○ த్ʹ͍݉ͯ͠ΔͱɺԱଌओ؍͕ೖͬͯ͠·͍͕ͪ○ ໌֬ͳͷ■ SREࣄ࣮ͱϩάΛ͢ͱ͜Ζʹઐ೦
·ͱΊ● 1ਓͰ๊͑ࠐΉͷΑ͘ͳ͍● ૣ͘ΔͨΊʹΛ໌֬ʹղ͢Δඞཁ͕͋Δ● ͰͬͺΓΈΜͳͰૣ͘ԕ͘ߦ͖͍ͨʂʂ
·ͱΊ● ແࣄϦϦʔεˍ҆ఆӡ༻͕࢝·ͬͨ🎉● NewRelicΛ։ൃڥͰ׆༻○ ։ൃ࣌ͷόάτϥοΩϯά○ ՝ཧͱͷ࿈ܞ○ QAνʔϜʹΑΔNewRelicͰͷࣄ࣮ڞ༗● ։ൃ৫ͷϦϑΝΫλϦϯάΛ࢝Ί͍͖ͯ·͢○ ࠓ·Ͱͷ୲ΛਓνʔϜͰ୲○ ઐνʔϜʹΑΔεϧʔϓοτΛ୲อͨ͠։ൃ৫
techtver
hosimesi11
kadogen_0527
tbshiki
tkmnzm
leekwangsoo1995
bun913
niryuu
sat
takmin
yumayabe
nenonaninu
trycycle
afnizarnur
quramy
zakiwarfel
danielanewman
aarron
sferik
matthewcrist
hatefulcrawdad
davidbonilla
swwweet
kevinliebholz
lara