Upgrade to Pro — share decks privately, control downloads, hide ads and more …

少数組織で開発する大規模サービス

 少数組織で開発する大規模サービス

2022/07/21
Developers Summit 2022 Summer

イベントURL :
https://event.shoeisha.jp/devsumi/20220721

TVer Inc.
PRO

July 21, 2022
Tweet

More Decks by TVer Inc.

Other Decks in Technology

Transcript

  1. গ਺૊৫Ͱ։ൃ͢Δେن໛αʔϏε
    גࣜձࣾ TVer / Ճզ وࢤ (SRE)
    גࣜձࣾ TVer / ಺ւ ܙհ (Backend)

    2022/07/21

    Developers Summit 2022 Summer

    View Slide

  2. ͜ͷηογϣϯʹ͍ͭͯ
    ຊ೔͓࿩͢Δ͜ͱ
    ● TVerϦχϡʔΞϧʹ͓͚ΔϞχλϦϯάվળ
    ● গਓ਺Ͱ։ൃͱվળΛߦ͏ͨΊͷ

    New Relic׆༻ํ๏

    View Slide

  3. ࣗݾ঺հ
    ɾՃզ وࢤ (@TAKA_0411)
    ɾגࣜձࣾTVer

    ɹαʔϏεࣄۀຊ෦ SRE
    ɾझຯ͸α΢φͱήʔϜͱVTuber
    ɾϗʔϜα΢φ͸ߔޱԹઘ تָཬ

    View Slide

  4. ΞδΣϯμ
    ● TVerͷ঺հ
    ● αʔϏεϦχϡʔΞϧʹ͍ͭͯ
    ● ϦϦʔε४උฤ
    ● ϦϦʔεޙͷվળ
    ● ·ͱΊ

    View Slide

  5. ΞδΣϯμ
    ● TVerͷ঺հ
    ● αʔϏεϦχϡʔΞϧʹ͍ͭͯ
    ● ϦϦʔε४උฤ
    ● ϦϦʔεޙͷվળ
    ● ·ͱΊ

    View Slide

  6. TVerͷ঺հ

    View Slide

  7. TVerͷ঺հ
    [TVer] 2022 ೥ 3 ݄ͷಈը࠶ੜ਺͕ 2 ԯ 5 ઍສճ/݄Λಥഁ!

    https://tver.co.jp/news/20220426-1.html

    View Slide

  8. TVerͷ঺հ
    TVerͷ
    ٕज़ελοΫ

    View Slide

  9. TVerͷٕज़ελοΫ
    ※ϑϩϯτΤϯυɺTVΞϓϦɺϞόΠϧΞϓϦέʔγϣϯ͸

    ֎෦ͷڠྗձࣾͷ͓ੈ࿩ʹͳ͓ͬͯΓ·͢

    View Slide

  10. TVerͷ঺հ
    TVerͷ
    ΠϯϑϥετϥΫνϟ

    View Slide

  11. TVerͷΠϯϑϥετϥΫνϟ
    WebαʔϏεͱͯ͠ͷTVer΍


    ൪૊࿈ಈγεςϜ౳Ͱ

    Քಇ͍ͯ͠ΔϚΠΫϩαʔϏε

    View Slide

  12. TVer (WebαʔϏε) ͷΠϯϑϥετϥΫνϟ

    View Slide

  13. ΞδΣϯμ
    ● TVerͷ঺հ
    ● αʔϏεϦχϡʔΞϧʹ͍ͭͯ
    ● ϦϦʔε४උฤ
    ● ϦϦʔεޙͷվળ
    ● ·ͱΊ

    View Slide

  14. αʔϏεϦχϡʔΞϧʹ͍ͭͯ

    View Slide

  15. αʔϏεϦχϡʔΞϧʹ͍ͭͯ
    [TVer] ςϨϏίϯςϯπͷࢹௌମݧ΍޿ࠂମݧͷ޲্Λ໨ࢦ͢ ʮTVer IDʯͷఏڙΛ։࢝

    https://tver.co.jp/news/20220404-1.html

    View Slide

  16. αʔϏεϦχϡʔΞϧʹ͍ͭͯ
    γεςϜͷมߋ఺
    ● Πϯϑϥͷ಺੡Խ
    ● όοΫΤϯυͷ࡮৽
    ● ϞχλϦϯάͷ࡮৽

    View Slide

  17. αʔϏεϦχϡʔΞϧʹ͍ͭͯ
    γεςϜͷมߋ఺
    ● Πϯϑϥͷ಺੡Խ
    ● όοΫΤϯυͷ࡮৽
    ● ϞχλϦϯάͷ࡮৽ɹ←ࠓճ͸͜͜ͷ࿩
    ϦχϡʔΞϧΛػʹ
    New Relic OneΛϑϧ׆༻

    View Slide

  18. ϞχλϦϯάͷ࡮৽
    ϞχλϦϯάͷ࡮৽
    ● ੹຿ͷ੔ཧ
    ● ಋೖ͢Δػೳͷऔࣺબ୒
    ● μογϡϘʔυͱσʔλͷ׆༻

    View Slide

  19. ϞχλϦϯάͷ࡮৽
    ϞχλϦϯάͷ࡮৽
    ● ੹຿ͷ੔ཧ
    ● ಋೖ͢Δػೳͷऔࣺબ୒
    ● μογϡϘʔυͱσʔλͷ׆༻

    View Slide

  20. ϞχλϦϯάͷ࡮৽
    Before (ʙ2021೥12݄)
    ʮNew RelicΛ ยखؒ ͰݟΔͷ͕ͭΒ͍ʂʯ
    Backend ৘γε
    Πϯϑϥ
    New Relic


    पΓͷରԠ
    New Relic


    पΓͷରԠ
    New Relic


    पΓͷରԠ
    APIͷ։ൃ Fargateݕূ
    ෛՙςετ
    ΞΧ΢ϯτ
    ઃܭͱ੔උ


    ίετࢼࢉ
    ͓Αͼܖ໿
    ֤෦ॺ͔Β
    ͷ໰͍߹Θ
    ͤରԠ
    νϡʔχϯά
    CI/CD੔උ
    ཁ݅੔ཧ
    ΞϓϦ΍ϑϩ
    ϯτΤϯυͱ
    ͷௐ੔
    Πϯϑϥߏங
    ͱCDKपΓͷ
    ੔උ


    View Slide

  21. After (2022೥1݄ʙ)
    ϞχλϦϯάͷ࡮৽
    SRE (ࢲ)
    New RelicपΓͷλεΫ


    ·Δ͝ͱҾ͖ड͚·͢Ͷ
    ΈΜͳ
    ͡Ό͋ϦʔυΑΖ͘͠ʂ

    ଞͷλεΫਐΊΔͶ

    View Slide

  22. ࠷ۙ
    ϞχλϦϯάͷ࡮৽
    Backend SRE
    μογϡϘʔυͰ

    ͜ΜͳϝτϦΫε͕ݟ
    ͍ͨΜ͚ͩͲʙ
    ͜ͷ৔߹ʹ͸Ξϥʔτ


    ඈ͹ͯ͠ཉ͍͠ΑͶ
    ΍Γ͍ͨࣄΛԕྀͤͣʹ૬ஊɾ࣮ݱͰ͖Δମ੍΁
    ○○ͱ͍͏஋ΛΩʔʹ͠
    ͯΤϥʔΛௐ͍ࠪͨ͠


    ͱࢥͬͯΔΘ͚Α
    ΍Γ·͢

    View Slide

  23. ϞχλϦϯάͷ࡮৽
    ϞχλϦϯάͷ࡮৽
    ● ੹຿ͷ੔ཧ
    ● ಋೖ͢Δػೳͷऔࣺબ୒
    ● μογϡϘʔυͱσʔλͷ׆༻

    View Slide

  24. ϞχλϦϯάͷ࡮৽
    NRU101 New Relic One ೖ໳

    https://newrelic.com/jp/resources/presentations/nru101_20211027

    View Slide

  25. ϞχλϦϯάͷ࡮৽
    ࣗ෼ͷཧ૝
    ● MELTΛશͯຬ͍ͨͨ͠
    ○ Metrics, Events, Logs, Traces
    ● ΤϥʔௐࠪͷͨΊʹ
    Traces(APM)Λಋೖ͍ͨ͠

    View Slide

  26. ϞχλϦϯάͷ࡮৽
    ݱ࣮
    ● New RelicͷGo Agent(APM)ͷ
    ಋೖ͕एׯ໘౗
    ● APMͷΦʔόʔϔου͕ෆ҆
    ● pprofͰྑ͘ͳ͍ʁ
    ● ͍ͬͯ͏͔ࠓඞཁͳΜ͚ͩͬʁ
    ϦϦʔε·Ͱ͋ͱ2ϲ݄ɾɾɾ

    View Slide

  27. ϞχλϦϯάͷ࡮৽
    ͱΓ͋͑ͣ New Relic Go Agent Λ
    ඞཁ࠷খݶͷՕॴʹ͚ͩಋೖͨ͠

    View Slide

  28. ϞχλϦϯάͷ࡮৽

    View Slide

  29. ϞχλϦϯάͷ࡮৽
    Dev (Backend)ɹɹOps (SRE)
    ಋೖ݁Ռ
    ● EchoͷIntegrationΛઃఆ͢Δͩ
    ͚Ͱେ෯ʹՄࢹԽͰ͖ͨ
    ● Τϥʔൃੜ࣌ͷτϦΞʔδ͕

    Մೳʹͳͬͨ
    ● ࠷খݶͷ࿑ྗͰ࠷େͷޮՌΛಘ
    Δ͜ͱ͕Ͱ͖ͨ

    View Slide

  30. ϞχλϦϯάͷ࡮৽
    গ਺૊৫ × New Relic One
    ● ػೳ͕๛෋ͰΦʔϧΠϯϫϯ
    ● ಋೖͷϋʔυϧ͕௿͍
    ● υΩϡϝϯτɾαϙʔτɾίϛϡχςΟ͕ॆ࣮
    ● ͱΓ͋͑ͣಋೖ͢Δ͚ͩͰେ෯ʹՄࢹԽͰ͖Δ
    ಋೖɾӡ༻ͷίετ͕࠷খݶͰࡁΜͩ

    View Slide

  31. ϞχλϦϯάͷ࡮৽
    ϞχλϦϯάͷ࡮৽
    ● ੹຿ͷ෼ׂ
    ● ಋೖ͢Δػೳͷऔࣺબ୒
    ● μογϡϘʔυͱσʔλͷ׆༻

    View Slide

  32. ϞχλϦϯάͷ࡮৽
    ΫϥΠΞϯτͷಉ࣌઀ଓ਺΍Ϋϥογϡ཰ͷϝτϦΫε

    View Slide

  33. ϞχλϦϯάͷ࡮৽
    ि1ͰϞόΠϧΞϓϦͷΫϥογϡ΍ωοτϫʔΫΤϥʔΛௐࠪ͠ɺ޿ࣾ͘಺ʹڞ༗

    View Slide

  34. ϞχλϦϯάͷ࡮৽
    μογϡϘʔυͱσʔλͷ׆༻
    ● ඞཁͳϝτϦΫεͰμογϡϘʔυΛ੔උ
    ● ϑϩϯτΤϯυ͔ΒόοΫΤϯυ·ͰΧόʔ
    ● Τϥʔ܏޲Λௐࠪͯࣾ͠಺ʹڞ༗
    ● APIͷվળʹAPMΛ׆༻

    View Slide

  35. ࣗݾ঺հ
    ɾ಺ւ ܙհ (@voidofglans)
    ɾגࣜձࣾTVer αʔϏεࣄۀຊ෦ ɹϦʔυ
    ΤϯδχΞ
    ɾ޷͖ͳ΋ͷ:୸ՐͰম͍ͨϗϧϞϯ
    ɾಸྑࡏॅͷԕִۈ຿த
    ɾNFCຒΊࠐΜͩΓ͢Δͷ͕޷͖

    View Slide

  36. ΞδΣϯμ
    ● TVerͷ঺հ
    ● αʔϏεϦχϡʔΞϧʹ͍ͭͯ
    ● ϦϦʔε४උฤ
    ● ϦϦʔεޙͷվળ
    ● ·ͱΊ

    View Slide

  37. ϦϦʔε४උฤ
    ● ϑΣʔζʹΑͬͯNew Relicͷ࢖͍෼͚
    ● গਓ਺ͰεϐʔυײΛམͱ͞ͳ͍ͨΊʹ
    ● ύϑΥʔϚϯεɾνϡʔχϯά/ෛՙରࡦ

    View Slide

  38. ϦϦʔε४උฤ
    ● ϑΣʔζʹΑͬͯNew Relicͷ࢖͍෼͚
    ● গਓ਺ͰεϐʔυײΛམͱ͞ͳ͍ͨΊʹ
    ● ύϑΥʔϚϯεɾνϡʔχϯά/ෛՙରࡦ

    View Slide

  39. ϑΣʔζʹΑͬͯNew Relicͷ࢖͍෼͚
    APMಋೖޙͷϦϦʔε·ͰͷϑΣʔζ
    ● 2ϲ݄લ
    ● 1ϲ݄લ
    ● 2िؒલ

    View Slide

  40. ϑΣʔζʹΑͬͯNew Relicͷ࢖͍෼͚(2ϲ݄લ
    ● ΍Δ΂͖͜ͱ͸όά௵͠
    ● ओʹར༻ͨ͠ػೳ
    ○ Errors Inbox
    ○ Errors
    ○ NRQL
    ● 4xx/5xxܥΤϥʔͷ಺༰Λ෼ੳ
    ○ 4xx: ΫϥΠΞϯτ͕૝ఆ͍ͯ͠ͳ͍ύϥϝʔλ͕ͳ͍͔
    ○ 5xx: ૝ఆ͍ͯ͠ͳ͍ॲཧ΍λΠϜΞ΢τ͕ͳ͍͔

    View Slide

  41. ϑΣʔζʹΑͬͯNew Relicͷ࢖͍෼͚(1ϲ݄લ
    ● ΍Δ΂͖͜ͱ͸ෛՙࢼݧʹΑΔϘτϧωοΫௐࠪ
    ● ओʹར༻ͨ͠ػೳ
    ○ Transactions
    ○ Go Runtime
    ○ CloudWatch Metrics
    ● ߴෛՙ࣌ͷΞϓϦέʔγϣϯશମͷڍಈΛ෼ੳ
    ○ TransactionsͰͷSlowest΍95percentileͳͲΛࢦඪʹ
    ○ ϝϞϦϦʔΫ΍ɺ์ஔ͞Εͨgoroutineͷଘࡏ
    ○ GC AllocationʹΑΔஅଓతͳੑೳྼԽ

    View Slide

  42. ϑΣʔζʹΑͬͯNew Relicͷ࢖͍෼͚(2िؒલ
    ● ΍Δ΂͖͜ͱ͸ཁٻੑೳͷୡ੒
    ● ओʹར༻ͨ͠ػೳ
    ○ Transactions
    ○ CloudWatch Metrics
    ● શମΛ௨ͨ͠ෛՙࢼݧͰͷ҆ఆύϑΥʔϚϯε
    ○ 10msҎ্͔͔ΔAPI
    ○ ֤ϛυϧ΢ΣΞ΁ͷΫΤϦ΍઀ଓͷ਺
    ○ ॻ͖ࠐΈܥAPIͰͷLock,Waitͷ෼෍

    View Slide

  43. ϦϦʔε४උฤ
    ● ϑΣʔζʹΑͬͯNew Relicͷ࢖͍෼͚
    ● গਓ਺ͰεϐʔυײΛམͱ͞ͳ͍ͨΊʹ
    ● ύϑΥʔϚϯεɾνϡʔχϯά/ෛՙରࡦ

    View Slide

  44. গਓ਺ͰεϐʔυײΛམͱ͞ͳ͍ͨΊʹ
    ૣ͘ߦ͖͚ͨΕ͹ɺͻͱΓͰߦ͚ɻ
    ԕ͘·Ͱߦ͖͚ͨΕ͹ɺΈΜͳͰߦ͚ɻ
    If you want to go fast, go alone. If you want to go far, go together.

    View Slide

  45. গਓ਺ͰεϐʔυײΛམͱ͞ͳ͍ͨΊʹ
    ͓͚ͬʂ
    ͡Ό͋ͪΐͬͱ1ਓͰߦͬͯ͘Δʂ

    View Slide

  46. গਓ਺ͰεϐʔυײΛམͱ͞ͳ͍ͨΊʹ
    ϦϦʔε೔·Ͱʹ
    ΈΜͳݱ஍ू߹Ͱʂ

    View Slide

  47. ● ͦΕͧΕ͕ͦΕͧΕͷ࠷଎Ͱ໨త஍ʹ
    ● ੹຿ͷ໌֬ͳ෼ղ
    ● ໌֬ͳ໨తͷڞ༗
    গਓ਺ͰεϐʔυײΛམͱ͞ͳ͍ͨΊʹ

    View Slide

  48. ● ίέͳ͍͜ͱ͕ॏཁ
    ● Backend/Infra/SREͦΕͧΕ͕ઐ೚ੑΛ࣋ͬͯಠཱ͠
    ͨϝϯόʔ
    ● ίϛϡχέʔγϣϯίετ͕࠷௿ݶʹ཈͑ΒΕͨ
    গਓ਺ͰεϐʔυײΛམͱ͞ͳ͍ͨΊʹ

    View Slide

  49. গਓ਺ͰεϐʔυײΛམͱ͞ͳ͍ͨΊʹ
    ● ஍൫͸੔ͬͨ
    ● ࣍͸ΈΜͳͰԕ͘΁ߦ͜͏

    View Slide

  50. ϦϦʔε४උฤ
    ● ϑΣʔζʹΑͬͯNew Relicͷ࢖͍෼͚
    ● গਓ਺ͰεϐʔυײΛམͱ͞ͳ͍ͨΊʹ
    ● ύϑΥʔϚϯεɾνϡʔχϯά/ෛՙରࡦ

    View Slide

  51. ● Α͋͘ΔಥൃతͳεύΠΫ
    ○ ஍্೾࿈ಈ΍์ૹ࿮੾ΕʹΑΔྲྀೖ
    ■ 1minޙ͸͖ͬ͞·Ͱͷ10ഒҎ্ͷϦΫΤετ਺
    ■ ૿͑Δಉ࣌઀ଓ
    ○ ๭ΞΠυϧࣄ຿ॴλϨϯτ༷ग़ԋ൪૊
    ○ ϦΞϧλΠϜ഑৴൪૊ऴྃ࣌
    ■ Ϣʔβʔ͕Ұ੪ʹβοϐϯά
    ■ ಉظͷͱΕͨ਺ສʙ਺ेສͷϢʔβʔΞΫγϣϯ
    ύϑΥʔϚϯεɾνϡʔχϯά/ෛՙରࡦ

    View Slide

  52. ύϑΥʔϚϯεɾνϡʔχϯά/ෛՙରࡦ

    View Slide

  53. ● ϦϦʔεલʹSRE͕͔ͬ͠Γઐ೚ͯ͘͠Εͨ
    ○ New RelicΛݟ͓͚ͯ͹Ұൠతͳࢦඪ͸શ෦ݟΕΔ
    ○ ૝ఆ͞ΕΔΞΫηεύλʔϯʹ͓͚ΔMetricsͷมԽΛࣄ
    લʹೝࣝ͢Δ
    ○ ෛՙࢼݧ/ରࡦʹूத͢Δ͜ͱ͕Ͱ͖ͨ
    ύϑΥʔϚϯεɾνϡʔχϯά/ෛՙରࡦ

    View Slide

  54. ΞδΣϯμ
    ● TVerͷ঺հ
    ● αʔϏεϦχϡʔΞϧʹ͍ͭͯ
    ● ϦϦʔε४උฤ
    ● ϦϦʔεޙͷվળ
    ● ·ͱΊ

    View Slide

  55. ϦϦʔεޙͷվળ
    ● ԣஅతʹՄࢹԽ͞ΕͨMetricsͷԸܙ
    ● ఆظతʹ࣮ࢪ͢ΔμογϡϘʔυோΊΔձ
    ● ੹຿ͷ෼཭ʹΑΔରԠ଎౓ͷ୲อ

    View Slide

  56. ԣஅతʹՄࢹԽ͞ΕͨMetricsͷԸܙ
    ● ҆ఆՔಇ͍ͯ͠Δ͜ͱ͕ԕ໨ͰΘ͔Δ҆৺ײ

    View Slide

  57. ԣஅతʹՄࢹԽ͞ΕͨMetricsͷԸܙ
    ● σόΠε͝ͱͷ࣮૷ʹΑΔࠩҟͷݕ஌
    ○ σόΠεछผΛCustomAttributeʹ֨ೲ͢Δ
    ○ ϦΫΤετ܏޲΍Τϥʔ܏޲ͷ೺Ѳ
    ■ iOSΞϓϦ͚ͩಛఆͷϖʔδͰͷAPIϦΫΤετ͕ҟৗʹଟ͍
    ⿌ Ϧετදࣔ಺Ͱཁૉ͝ͱʹAPIΛݺͼग़͍ͯͨ͠
    ⿌ 30min଍ΒͣͰݪҼಛఆˠ։ൃϕϯμʔ΁ͷڞ༗
    ⿌ 3೔ޙͷϦϦʔεʹแؚ͠ɺແࣄղফ🎉

    View Slide

  58. ԣஅతʹՄࢹԽ͞ΕͨMetricsͷԸܙ
    ● ։ൃ౰ॳ૝ఆ͞Ε͍ͯͳ͍ϢʔβʔγφϦΦ
    ○ ࢥͬͨͷͷ3ഒ͘Β͍ݕࡧػೳ͕࢖ΘΕ͍ͯΔʂʂ
    ○ ࢥͬͨΑΓ΋xxͷػೳ͕࢖ΘΕͯͳ͍ʂʂ
    ૝ఆͱӡ༻͸ဃ཭͢Δ΋ͷ

    View Slide

  59. ● ա৒ͳϦιʔε͕͙͢Θ͔Δ
    ○ ϦϦʔεॳظ͸҆৺ΛۚͰങ͍ͬͯͨͷͰ५୔ͳϦιʔε
    Λ༻ҙ͍ͯͨ͠
    ○ ίετΠϯύΫτΛ࣠ʹॱ࣍Ϧιʔεͷ࠷దԽΛߦ͑ͨ
    ■ ΋ͪΖΜ͜Ε΋ՄࢹԽ͞Ε͍ͯΔͷͰμϝͳ৔߹ͷݕ஌΋ਝ଎
    ԣஅతʹՄࢹԽ͞ΕͨMetricsͷԸܙ

    View Slide

  60. ● ࡢ೔ࠓ೔ؾʹͳͬͨࢦඪΛμογϡϘʔυʹ௥Ճ
    ● Metricsͷҧ࿨ײʹؾͮ͘ᄿ֮ͷ܇࿅
    ● ো֐ޙ΍τϥϒϧൃੜ࣌ͷ஌ݟͷूੵ
    ఆظతʹ࣮ࢪ͢ΔμογϡϘʔυோΊΔձ

    View Slide

  61. ੹຿ͷ෼཭ʹΑΔରԠ଎౓ͷ୲อ
    ● ਖ਼͘͠ઐ೚ʹ೚ͤΔ
    ○ த్൒୺ʹ݉຿͍ͯ͠ΔͱɺԱଌ΍ओ؍͕ೖͬͯ͠·͍͕
    ͪ
    ○ ໌֬ͳ੹຿ͷ෼཭
    ■ SRE͸ࣄ࣮ͱϩάΛ౉͢ͱ͜Ζʹઐ೦

    View Slide

  62. ΞδΣϯμ
    ● TVerͷ঺հ
    ● αʔϏεϦχϡʔΞϧʹ͍ͭͯ
    ● ϦϦʔε४උฤ
    ● ϦϦʔεޙͷվળ
    ● ·ͱΊ

    View Slide

  63. ·ͱΊ
    ● 1ਓͰ๊͑ࠐΉͷ͸Α͘ͳ͍
    ● ૣ͘΍ΔͨΊʹ͸੹຿Λ໌֬ʹ෼ղ͢Δඞཁ͕͋Δ
    ● Ͱ΋΍ͬͺΓΈΜͳͰૣ͘ԕ͘΁ߦ͖͍ͨʂʂ

    View Slide

  64. ·ͱΊ
    ● ແࣄϦϦʔεˍ҆ఆӡ༻͕࢝·ͬͨ🎉
    ● NewRelicΛ։ൃ؀ڥͰ΋׆༻
    ○ ։ൃ࣌ͷόάτϥοΩϯά
    ○ ՝୊؅ཧͱͷ࿈ܞ
    ○ QAνʔϜʹΑΔNewRelicͰͷࣄ࣮ڞ༗
    ● ։ൃ૊৫ͷϦϑΝΫλϦϯάΛ࢝Ί͍͖ͯ·͢
    ○ ࠓ·Ͱͷ୲຿Λ਺ਓνʔϜͰ୲౰
    ○ ઐ೚νʔϜʹΑΔεϧʔϓοτΛ୲อͨ͠։ൃ૊৫

    View Slide

  65. View Slide

  66. View Slide