Tweet
Tweet

Slide 1

Slide 1 text

গ਺૊৫Ͱ։ൃ͢Δେن໛αʔϏε גࣜձࣾ TVer / Ճզ وࢤ (SRE) גࣜձࣾ TVer / ಺ւ ܙհ (Backend)
 2022/07/21
 Developers Summit 2022 Summer

Slide 2

Slide 2 text

͜ͷηογϣϯʹ͍ͭͯ ຊ೔͓࿩͢Δ͜ͱ ● TVerϦχϡʔΞϧʹ͓͚ΔϞχλϦϯάվળ ● গਓ਺Ͱ։ൃͱվળΛߦ͏ͨΊͷ
 New Relic׆༻ํ๏

Slide 3

Slide 3 text

ࣗݾ঺հ ɾՃզ وࢤ (@TAKA_0411) ɾגࣜձࣾTVer
 ɹαʔϏεࣄۀຊ෦ SRE ɾझຯ͸α΢φͱήʔϜͱVTuber ɾϗʔϜα΢φ͸ߔޱԹઘ تָཬ

Slide 4

Slide 4 text

ΞδΣϯμ ● TVerͷ঺հ ● αʔϏεϦχϡʔΞϧʹ͍ͭͯ ● ϦϦʔε४උฤ ● ϦϦʔεޙͷվળ ● ·ͱΊ

Slide 5

Slide 5 text

ΞδΣϯμ ● TVerͷ঺հ ● αʔϏεϦχϡʔΞϧʹ͍ͭͯ ● ϦϦʔε४උฤ ● ϦϦʔεޙͷվળ ● ·ͱΊ

Slide 6

Slide 6 text

TVerͷ঺հ

Slide 7

Slide 7 text

TVerͷ঺հ [TVer] 2022 ೥ 3 ݄ͷಈը࠶ੜ਺͕ 2 ԯ 5 ઍສճ/݄Λಥഁ!
 https://tver.co.jp/news/20220426-1.html

Slide 8

Slide 8 text

TVerͷ঺հ TVerͷ ٕज़ελοΫ

Slide 9

Slide 9 text

TVerͷٕज़ελοΫ ※ϑϩϯτΤϯυɺTVΞϓϦɺϞόΠϧΞϓϦέʔγϣϯ͸
 ֎෦ͷڠྗձࣾͷ͓ੈ࿩ʹͳ͓ͬͯΓ·͢

Slide 10

Slide 10 text

TVerͷ঺հ TVerͷ ΠϯϑϥετϥΫνϟ

Slide 11

Slide 11 text

TVerͷΠϯϑϥετϥΫνϟ WebαʔϏεͱͯ͠ͷTVer΍ ൪૊࿈ಈγεςϜ౳Ͱ 
 Քಇ͍ͯ͠ΔϚΠΫϩαʔϏε

Slide 12

Slide 12 text

TVer (WebαʔϏε) ͷΠϯϑϥετϥΫνϟ

Slide 13

Slide 13 text

ΞδΣϯμ ● TVerͷ঺հ ● αʔϏεϦχϡʔΞϧʹ͍ͭͯ ● ϦϦʔε४උฤ ● ϦϦʔεޙͷվળ ● ·ͱΊ

Slide 14

Slide 14 text

αʔϏεϦχϡʔΞϧʹ͍ͭͯ

Slide 15

Slide 15 text

αʔϏεϦχϡʔΞϧʹ͍ͭͯ [TVer] ςϨϏίϯςϯπͷࢹௌମݧ΍޿ࠂମݧͷ޲্Λ໨ࢦ͢ ʮTVer IDʯͷఏڙΛ։࢝
 https://tver.co.jp/news/20220404-1.html

Slide 16

Slide 16 text

αʔϏεϦχϡʔΞϧʹ͍ͭͯ γεςϜͷมߋ఺ ● Πϯϑϥͷ಺੡Խ ● όοΫΤϯυͷ࡮৽ ● ϞχλϦϯάͷ࡮৽

Slide 17

Slide 17 text

αʔϏεϦχϡʔΞϧʹ͍ͭͯ γεςϜͷมߋ఺ ● Πϯϑϥͷ಺੡Խ ● όοΫΤϯυͷ࡮৽ ● ϞχλϦϯάͷ࡮৽ɹ←ࠓճ͸͜͜ͷ࿩ ϦχϡʔΞϧΛػʹ New Relic OneΛϑϧ׆༻

Slide 18

Slide 18 text

ϞχλϦϯάͷ࡮৽ ϞχλϦϯάͷ࡮৽ ● ੹຿ͷ੔ཧ ● ಋೖ͢Δػೳͷऔࣺબ୒ ● μογϡϘʔυͱσʔλͷ׆༻

Slide 19

Slide 19 text

ϞχλϦϯάͷ࡮৽ ϞχλϦϯάͷ࡮৽ ● ੹຿ͷ੔ཧ ● ಋೖ͢Δػೳͷऔࣺબ୒ ● μογϡϘʔυͱσʔλͷ׆༻

Slide 20

Slide 20 text

ϞχλϦϯάͷ࡮৽ Before (ʙ2021೥12݄) ʮNew RelicΛ ยखؒ ͰݟΔͷ͕ͭΒ͍ʂʯ Backend ৘γε Πϯϑϥ New Relic पΓͷରԠ New Relic पΓͷରԠ New Relic पΓͷରԠ APIͷ։ൃ Fargateݕূ ෛՙςετ ΞΧ΢ϯτ ઃܭͱ੔උ ίετࢼࢉ ͓Αͼܖ໿ ֤෦ॺ͔Β ͷ໰͍߹Θ ͤରԠ νϡʔχϯά CI/CD੔උ ཁ݅੔ཧ ΞϓϦ΍ϑϩ ϯτΤϯυͱ ͷௐ੔ Πϯϑϥߏங ͱCDKपΓͷ ੔උ

Slide 21

Slide 21 text

After (2022೥1݄ʙ) ϞχλϦϯάͷ࡮৽ SRE (ࢲ) New RelicपΓͷλεΫ ·Δ͝ͱҾ͖ड͚·͢Ͷ ΈΜͳ ͡Ό͋ϦʔυΑΖ͘͠ʂ 
 ଞͷλεΫਐΊΔͶ

Slide 22

Slide 22 text

࠷ۙ ϞχλϦϯάͷ࡮৽ Backend SRE μογϡϘʔυͰ 
 ͜ΜͳϝτϦΫε͕ݟ ͍ͨΜ͚ͩͲʙ ͜ͷ৔߹ʹ͸Ξϥʔτ ඈ͹ͯ͠ཉ͍͠ΑͶ ΍Γ͍ͨࣄΛԕྀͤͣʹ૬ஊɾ࣮ݱͰ͖Δମ੍΁ ○○ͱ͍͏஋ΛΩʔʹ͠ ͯΤϥʔΛௐ͍ࠪͨ͠ ͱࢥͬͯΔΘ͚Α ΍Γ·͢

Slide 23

Slide 23 text

ϞχλϦϯάͷ࡮৽ ϞχλϦϯάͷ࡮৽ ● ੹຿ͷ੔ཧ ● ಋೖ͢Δػೳͷऔࣺબ୒ ● μογϡϘʔυͱσʔλͷ׆༻

Slide 24

Slide 24 text

ϞχλϦϯάͷ࡮৽ NRU101 New Relic One ೖ໳
 https://newrelic.com/jp/resources/presentations/nru101_20211027

Slide 25

Slide 25 text

ϞχλϦϯάͷ࡮৽ ࣗ෼ͷཧ૝ ● MELTΛશͯຬ͍ͨͨ͠ ○ Metrics, Events, Logs, Traces ● ΤϥʔௐࠪͷͨΊʹ Traces(APM)Λಋೖ͍ͨ͠

Slide 26

Slide 26 text

ϞχλϦϯάͷ࡮৽ ݱ࣮ ● New RelicͷGo Agent(APM)ͷ ಋೖ͕एׯ໘౗ ● APMͷΦʔόʔϔου͕ෆ҆ ● pprofͰྑ͘ͳ͍ʁ ● ͍ͬͯ͏͔ࠓඞཁͳΜ͚ͩͬʁ ϦϦʔε·Ͱ͋ͱ2ϲ݄ɾɾɾ

Slide 27

Slide 27 text

ϞχλϦϯάͷ࡮৽ ͱΓ͋͑ͣ New Relic Go Agent Λ ඞཁ࠷খݶͷՕॴʹ͚ͩಋೖͨ͠

Slide 28

Slide 28 text

ϞχλϦϯάͷ࡮৽

Slide 29

Slide 29 text

ϞχλϦϯάͷ࡮৽ Dev (Backend)ɹɹOps (SRE) ಋೖ݁Ռ ● EchoͷIntegrationΛઃఆ͢Δͩ ͚Ͱେ෯ʹՄࢹԽͰ͖ͨ ● Τϥʔൃੜ࣌ͷτϦΞʔδ͕
 Մೳʹͳͬͨ ● ࠷খݶͷ࿑ྗͰ࠷େͷޮՌΛಘ Δ͜ͱ͕Ͱ͖ͨ

Slide 30

Slide 30 text

ϞχλϦϯάͷ࡮৽ গ਺૊৫ × New Relic One ● ػೳ͕๛෋ͰΦʔϧΠϯϫϯ ● ಋೖͷϋʔυϧ͕௿͍ ● υΩϡϝϯτɾαϙʔτɾίϛϡχςΟ͕ॆ࣮ ● ͱΓ͋͑ͣಋೖ͢Δ͚ͩͰେ෯ʹՄࢹԽͰ͖Δ ಋೖɾӡ༻ͷίετ͕࠷খݶͰࡁΜͩ

Slide 31

Slide 31 text

ϞχλϦϯάͷ࡮৽ ϞχλϦϯάͷ࡮৽ ● ੹຿ͷ෼ׂ ● ಋೖ͢Δػೳͷऔࣺબ୒ ● μογϡϘʔυͱσʔλͷ׆༻

Slide 32

Slide 32 text

ϞχλϦϯάͷ࡮৽ ΫϥΠΞϯτͷಉ࣌઀ଓ਺΍Ϋϥογϡ཰ͷϝτϦΫε

Slide 33

Slide 33 text

ϞχλϦϯάͷ࡮৽ ि1ͰϞόΠϧΞϓϦͷΫϥογϡ΍ωοτϫʔΫΤϥʔΛௐࠪ͠ɺ޿ࣾ͘಺ʹڞ༗

Slide 34

Slide 34 text

ϞχλϦϯάͷ࡮৽ μογϡϘʔυͱσʔλͷ׆༻ ● ඞཁͳϝτϦΫεͰμογϡϘʔυΛ੔උ ● ϑϩϯτΤϯυ͔ΒόοΫΤϯυ·ͰΧόʔ ● Τϥʔ܏޲Λௐࠪͯࣾ͠಺ʹڞ༗ ● APIͷվળʹAPMΛ׆༻

Slide 35

Slide 35 text

ࣗݾ঺հ ɾ಺ւ ܙհ (@voidofglans) ɾגࣜձࣾTVer αʔϏεࣄۀຊ෦ ɹϦʔυ ΤϯδχΞ ɾ޷͖ͳ΋ͷ:୸ՐͰম͍ͨϗϧϞϯ ɾಸྑࡏॅͷԕִۈ຿த ɾNFCຒΊࠐΜͩΓ͢Δͷ͕޷͖

Slide 36

Slide 36 text

ΞδΣϯμ ● TVerͷ঺հ ● αʔϏεϦχϡʔΞϧʹ͍ͭͯ ● ϦϦʔε४උฤ ● ϦϦʔεޙͷվળ ● ·ͱΊ

Slide 37

Slide 37 text

ϦϦʔε४උฤ ● ϑΣʔζʹΑͬͯNew Relicͷ࢖͍෼͚ ● গਓ਺ͰεϐʔυײΛམͱ͞ͳ͍ͨΊʹ ● ύϑΥʔϚϯεɾνϡʔχϯά/ෛՙରࡦ

Slide 38

Slide 38 text

ϦϦʔε४උฤ ● ϑΣʔζʹΑͬͯNew Relicͷ࢖͍෼͚ ● গਓ਺ͰεϐʔυײΛམͱ͞ͳ͍ͨΊʹ ● ύϑΥʔϚϯεɾνϡʔχϯά/ෛՙରࡦ

Slide 39

Slide 39 text

ϑΣʔζʹΑͬͯNew Relicͷ࢖͍෼͚ APMಋೖޙͷϦϦʔε·ͰͷϑΣʔζ ● 2ϲ݄લ ● 1ϲ݄લ ● 2िؒલ

Slide 40

Slide 40 text

ϑΣʔζʹΑͬͯNew Relicͷ࢖͍෼͚(2ϲ݄લ ● ΍Δ΂͖͜ͱ͸όά௵͠ ● ओʹར༻ͨ͠ػೳ ○ Errors Inbox ○ Errors ○ NRQL ● 4xx/5xxܥΤϥʔͷ಺༰Λ෼ੳ ○ 4xx: ΫϥΠΞϯτ͕૝ఆ͍ͯ͠ͳ͍ύϥϝʔλ͕ͳ͍͔ ○ 5xx: ૝ఆ͍ͯ͠ͳ͍ॲཧ΍λΠϜΞ΢τ͕ͳ͍͔

Slide 41

Slide 41 text

ϑΣʔζʹΑͬͯNew Relicͷ࢖͍෼͚(1ϲ݄લ ● ΍Δ΂͖͜ͱ͸ෛՙࢼݧʹΑΔϘτϧωοΫௐࠪ ● ओʹར༻ͨ͠ػೳ ○ Transactions ○ Go Runtime ○ CloudWatch Metrics ● ߴෛՙ࣌ͷΞϓϦέʔγϣϯશମͷڍಈΛ෼ੳ ○ TransactionsͰͷSlowest΍95percentileͳͲΛࢦඪʹ ○ ϝϞϦϦʔΫ΍ɺ์ஔ͞Εͨgoroutineͷଘࡏ ○ GC AllocationʹΑΔஅଓతͳੑೳྼԽ

Slide 42

Slide 42 text

ϑΣʔζʹΑͬͯNew Relicͷ࢖͍෼͚(2िؒલ ● ΍Δ΂͖͜ͱ͸ཁٻੑೳͷୡ੒ ● ओʹར༻ͨ͠ػೳ ○ Transactions ○ CloudWatch Metrics ● શମΛ௨ͨ͠ෛՙࢼݧͰͷ҆ఆύϑΥʔϚϯε ○ 10msҎ্͔͔ΔAPI ○ ֤ϛυϧ΢ΣΞ΁ͷΫΤϦ΍઀ଓͷ਺ ○ ॻ͖ࠐΈܥAPIͰͷLock,Waitͷ෼෍

Slide 43

Slide 43 text

ϦϦʔε४උฤ ● ϑΣʔζʹΑͬͯNew Relicͷ࢖͍෼͚ ● গਓ਺ͰεϐʔυײΛམͱ͞ͳ͍ͨΊʹ ● ύϑΥʔϚϯεɾνϡʔχϯά/ෛՙରࡦ

Slide 44

Slide 44 text

গਓ਺ͰεϐʔυײΛམͱ͞ͳ͍ͨΊʹ ૣ͘ߦ͖͚ͨΕ͹ɺͻͱΓͰߦ͚ɻ ԕ͘·Ͱߦ͖͚ͨΕ͹ɺΈΜͳͰߦ͚ɻ If you want to go fast, go alone. If you want to go far, go together.

Slide 45

Slide 45 text

গਓ਺ͰεϐʔυײΛམͱ͞ͳ͍ͨΊʹ ͓͚ͬʂ ͡Ό͋ͪΐͬͱ1ਓͰߦͬͯ͘Δʂ

Slide 46

Slide 46 text

গਓ਺ͰεϐʔυײΛམͱ͞ͳ͍ͨΊʹ ϦϦʔε೔·Ͱʹ ΈΜͳݱ஍ू߹Ͱʂ

Slide 47

Slide 47 text

● ͦΕͧΕ͕ͦΕͧΕͷ࠷଎Ͱ໨త஍ʹ ● ੹຿ͷ໌֬ͳ෼ղ ● ໌֬ͳ໨తͷڞ༗ গਓ਺ͰεϐʔυײΛམͱ͞ͳ͍ͨΊʹ

Slide 48

Slide 48 text

● ίέͳ͍͜ͱ͕ॏཁ ● Backend/Infra/SREͦΕͧΕ͕ઐ೚ੑΛ࣋ͬͯಠཱ͠ ͨϝϯόʔ ● ίϛϡχέʔγϣϯίετ͕࠷௿ݶʹ཈͑ΒΕͨ গਓ਺ͰεϐʔυײΛམͱ͞ͳ͍ͨΊʹ

Slide 49

Slide 49 text

গਓ਺ͰεϐʔυײΛམͱ͞ͳ͍ͨΊʹ ● ஍൫͸੔ͬͨ ● ࣍͸ΈΜͳͰԕ͘΁ߦ͜͏

Slide 50

Slide 50 text

ϦϦʔε४උฤ ● ϑΣʔζʹΑͬͯNew Relicͷ࢖͍෼͚ ● গਓ਺ͰεϐʔυײΛམͱ͞ͳ͍ͨΊʹ ● ύϑΥʔϚϯεɾνϡʔχϯά/ෛՙରࡦ

Slide 51

Slide 51 text

● Α͋͘ΔಥൃతͳεύΠΫ ○ ஍্೾࿈ಈ΍์ૹ࿮੾ΕʹΑΔྲྀೖ ■ 1minޙ͸͖ͬ͞·Ͱͷ10ഒҎ্ͷϦΫΤετ਺ ■ ૿͑Δಉ࣌઀ଓ ○ ๭ΞΠυϧࣄ຿ॴλϨϯτ༷ग़ԋ൪૊ ○ ϦΞϧλΠϜ഑৴൪૊ऴྃ࣌ ■ Ϣʔβʔ͕Ұ੪ʹβοϐϯά ■ ಉظͷͱΕͨ਺ສʙ਺ेສͷϢʔβʔΞΫγϣϯ ύϑΥʔϚϯεɾνϡʔχϯά/ෛՙରࡦ

Slide 52

Slide 52 text

ύϑΥʔϚϯεɾνϡʔχϯά/ෛՙରࡦ

Slide 53

Slide 53 text

● ϦϦʔεલʹSRE͕͔ͬ͠Γઐ೚ͯ͘͠Εͨ ○ New RelicΛݟ͓͚ͯ͹Ұൠతͳࢦඪ͸શ෦ݟΕΔ ○ ૝ఆ͞ΕΔΞΫηεύλʔϯʹ͓͚ΔMetricsͷมԽΛࣄ લʹೝࣝ͢Δ ○ ෛՙࢼݧ/ରࡦʹूத͢Δ͜ͱ͕Ͱ͖ͨ ύϑΥʔϚϯεɾνϡʔχϯά/ෛՙରࡦ

Slide 54

Slide 54 text

ΞδΣϯμ ● TVerͷ঺հ ● αʔϏεϦχϡʔΞϧʹ͍ͭͯ ● ϦϦʔε४උฤ ● ϦϦʔεޙͷվળ ● ·ͱΊ

Slide 55

Slide 55 text

ϦϦʔεޙͷվળ ● ԣஅతʹՄࢹԽ͞ΕͨMetricsͷԸܙ ● ఆظతʹ࣮ࢪ͢ΔμογϡϘʔυோΊΔձ ● ੹຿ͷ෼཭ʹΑΔରԠ଎౓ͷ୲อ

Slide 56

Slide 56 text

ԣஅతʹՄࢹԽ͞ΕͨMetricsͷԸܙ ● ҆ఆՔಇ͍ͯ͠Δ͜ͱ͕ԕ໨ͰΘ͔Δ҆৺ײ

Slide 57

Slide 57 text

ԣஅతʹՄࢹԽ͞ΕͨMetricsͷԸܙ ● σόΠε͝ͱͷ࣮૷ʹΑΔࠩҟͷݕ஌ ○ σόΠεछผΛCustomAttributeʹ֨ೲ͢Δ ○ ϦΫΤετ܏޲΍Τϥʔ܏޲ͷ೺Ѳ ■ iOSΞϓϦ͚ͩಛఆͷϖʔδͰͷAPIϦΫΤετ͕ҟৗʹଟ͍ ⿌ Ϧετදࣔ಺Ͱཁૉ͝ͱʹAPIΛݺͼग़͍ͯͨ͠ ⿌ 30min଍ΒͣͰݪҼಛఆˠ։ൃϕϯμʔ΁ͷڞ༗ ⿌ 3೔ޙͷϦϦʔεʹแؚ͠ɺແࣄղফ🎉

Slide 58

Slide 58 text

ԣஅతʹՄࢹԽ͞ΕͨMetricsͷԸܙ ● ։ൃ౰ॳ૝ఆ͞Ε͍ͯͳ͍ϢʔβʔγφϦΦ ○ ࢥͬͨͷͷ3ഒ͘Β͍ݕࡧػೳ͕࢖ΘΕ͍ͯΔʂʂ ○ ࢥͬͨΑΓ΋xxͷػೳ͕࢖ΘΕͯͳ͍ʂʂ ૝ఆͱӡ༻͸ဃ཭͢Δ΋ͷ

Slide 59

Slide 59 text

● ա৒ͳϦιʔε͕͙͢Θ͔Δ ○ ϦϦʔεॳظ͸҆৺ΛۚͰങ͍ͬͯͨͷͰ५୔ͳϦιʔε Λ༻ҙ͍ͯͨ͠ ○ ίετΠϯύΫτΛ࣠ʹॱ࣍Ϧιʔεͷ࠷దԽΛߦ͑ͨ ■ ΋ͪΖΜ͜Ε΋ՄࢹԽ͞Ε͍ͯΔͷͰμϝͳ৔߹ͷݕ஌΋ਝ଎ ԣஅతʹՄࢹԽ͞ΕͨMetricsͷԸܙ

Slide 60

Slide 60 text

● ࡢ೔ࠓ೔ؾʹͳͬͨࢦඪΛμογϡϘʔυʹ௥Ճ ● Metricsͷҧ࿨ײʹؾͮ͘ᄿ֮ͷ܇࿅ ● ো֐ޙ΍τϥϒϧൃੜ࣌ͷ஌ݟͷूੵ ఆظతʹ࣮ࢪ͢ΔμογϡϘʔυோΊΔձ

Slide 61

Slide 61 text

੹຿ͷ෼཭ʹΑΔରԠ଎౓ͷ୲อ ● ਖ਼͘͠ઐ೚ʹ೚ͤΔ ○ த్൒୺ʹ݉຿͍ͯ͠ΔͱɺԱଌ΍ओ؍͕ೖͬͯ͠·͍͕ ͪ ○ ໌֬ͳ੹຿ͷ෼཭ ■ SRE͸ࣄ࣮ͱϩάΛ౉͢ͱ͜Ζʹઐ೦

Slide 62

Slide 62 text

ΞδΣϯμ ● TVerͷ঺հ ● αʔϏεϦχϡʔΞϧʹ͍ͭͯ ● ϦϦʔε४උฤ ● ϦϦʔεޙͷվળ ● ·ͱΊ

Slide 63

Slide 63 text

·ͱΊ ● 1ਓͰ๊͑ࠐΉͷ͸Α͘ͳ͍ ● ૣ͘΍ΔͨΊʹ͸੹຿Λ໌֬ʹ෼ղ͢Δඞཁ͕͋Δ ● Ͱ΋΍ͬͺΓΈΜͳͰૣ͘ԕ͘΁ߦ͖͍ͨʂʂ

Slide 64

Slide 64 text

·ͱΊ ● ແࣄϦϦʔεˍ҆ఆӡ༻͕࢝·ͬͨ🎉 ● NewRelicΛ։ൃ؀ڥͰ΋׆༻ ○ ։ൃ࣌ͷόάτϥοΩϯά ○ ՝୊؅ཧͱͷ࿈ܞ ○ QAνʔϜʹΑΔNewRelicͰͷࣄ࣮ڞ༗ ● ։ൃ૊৫ͷϦϑΝΫλϦϯάΛ࢝Ί͍͖ͯ·͢ ○ ࠓ·Ͱͷ୲຿Λ਺ਓνʔϜͰ୲౰ ○ ઐ೚νʔϜʹΑΔεϧʔϓοτΛ୲อͨ͠։ൃ૊৫

Slide 65

Slide 65 text

No content

Slide 66

Slide 66 text

No content