Bitkey_SRE_急成長するプロダクトを支えるためのSRE

Transcript

1. SREνʔϜϚωʔδϟʔ ๺ଜ Bitkey × SRE ʙٸ੒௕͢ΔϓϩμΫτΛࢧ͑ΔͨΊͷSREʙ

2. Outline 2 2 1. ͦ΋ͦ΋SREͱ͸ 2. BitkeyͰSRE͕ඞཁʹͳͬͨཧ༝ 3. SREνʔϜ্ཱͪ͛Ͱ΍ͬͨ͜ͱ 4.

   3ϲ݄ৼΓฦͬͯΈͯ 5. SREνʔϜͷ͜Ε͔Β

3. ๺ଜ ॱ Jun Kitamura 2015 2016 גࣜձࣾϑϩϜεΫϥονʢݱɾגࣜձࣾσʔλ ̭ʣʹೖࣾ SaaSϚʔέςΟϯάπʔϧͷಋೖΤϯδχΞΛ୲౰ 2019

   ಋೖۀ຿͔ΒΠϯϑϥΤϯδχΞ΁ҟಈ ϓϩμΫτશମͷAWSɾGCPͷ਺ඦ~1000୆ͷαʔ όʔ؅ཧɾอक͠ͳ͕Βɺ։ൃνʔϜͷεΫϥϜϚ ελʔ΋݉຿ ϏοτΩʔ΁ࢀը ࠷ॳͷ੡඼ϦϦʔεͱ΄΅ಉ࣌ظʹೖࣾ
 “ΧΪ” ͷϓϥοτϑΥʔϜ: bitkey platform ΍إೝূ ϓϩμΫτͷ։ൃʹैࣄ 2021 νʔϜϚωʔδϟʔͱͯ͠SREͷ্ཱͪ͛ https://github.com/jkkitakita

4. 4 4 1. SRE νʔϜͷ্ཱͪ͛ͬͯ۩ମతʹԿ͢Ε͹͍͍ͷʁ 2. Bitkey ͷ SRE ͬͯͲΜͳײ͡ͳͷʁ

   ࠓճ఻͍͑ͨ͜ͱ

5. 1. ͦ΋ͦ΋SREͱ͸ 5 5

6. 1. ͦ΋ͦ΋SREͱ͸ SREʢSite Reliability Engineeringʣ Google͕ࣾఏএɺ࣮ફ͍ͯ͠Δ γεςϜ؅ཧͱαʔϏεӡ༻ͷํ๏࿦ 6 6 ࢀߟ:

   ʮαΠτϦϥΠΞϏϦςΟΤϯδχΞϦϯάʯʢ2021೥1݄10೔ (೔) 10:29ʣʰ΢ΟΩϖσΟΞ೔ຊޠ൛ʱ

7. 2. BitkeyͰSRE͕ඞཁʹɹɹɹ ͳͬͨཧ༝ 7 7

8. 8 ΤϯδχΞ 2. ͳͥSREνʔϜ͕ඞཁʹͳͬͨͷ͔

9. 9 ΤϯδχΞ ࠓ·ͰҎ্ʹ ৴པੑΛ ҡ࣋ɾ޲্͍͖͍ͯͨ͠ʂ 2. ͳͥSREνʔϜ͕ඞཁʹͳͬͨͷ͔

10. 10 ΤϯδχΞ τϥϑΟοΫ૿Ճͯ͠Δ͔ ΒɺܭըతʹΩϟύγςΟ ϓϥϯχϯά͍ͨ͠ʂ εέʔϥϏϦςΟ޲্ͤ͞ ͍͖͍ͯͨʂ 2. ͳͥSREνʔϜ͕ඞཁʹͳͬͨͷ͔ ࠓ·ͰҎ্ʹ

    ৴པੑΛ ҡ࣋ɾ޲্͍͖͍ͯͨ͠ʂ

11. 11 ΤϯδχΞ ୯ମςετ΋E2Eςετ΋ ֦ॆ͍͖͍͚ͯͨ͠Ͳɺͦ ͷͨΊʹ৭ʑ४උ͠ͳ͍ ͱ͍͚ͳͯ͘େมʂ 2. ͳͥSREνʔϜ͕ඞཁʹͳͬͨͷ͔ τϥϑΟοΫ૿Ճͯ͠Δ͔ ΒɺܭըతʹΩϟύγςΟ

    ϓϥϯχϯά͍ͨ͠ʂ εέʔϥϏϦςΟ޲্ͤ͞ ͍͖͍ͯͨʂ ࠓ·ͰҎ্ʹ ৴པੑΛ ҡ࣋ɾ޲্͍͖͍ͯͨ͠ʂ

12. 12 ΤϯδχΞ ৭ʑ΍Βͳ͖Ό͚ͩͲ ΞϓϦέʔγϣϯϨΠϠʔ ͷ৽ػೳ։ൃʹूதͨ͠ ͍ʂ 2. ͳͥSREνʔϜ͕ඞཁʹͳͬͨͷ͔ τϥϑΟοΫ૿Ճͯ͠Δ͔ ΒɺܭըతʹΩϟύγςΟ

    ϓϥϯχϯά͍ͨ͠ʂ εέʔϥϏϦςΟ޲্ͤ͞ ͍͖͍ͯͨʂ ୯ମςετ΋E2Eςετ΋ ֦ॆ͍͖͍͚ͯͨ͠Ͳɺͦ ͷͨΊʹ৭ʑ४උ͠ͳ͍ ͱ͍͚ͳͯ͘େมʂ ࠓ·ͰҎ্ʹ ৴པੑΛ ҡ࣋ɾ޲্͍͖͍ͯͨ͠ʂ

13. 13 ΤϯδχΞ 2. ͳͥSREνʔϜ͕ඞཁʹͳͬͨͷ͔ ૊৫ɾϓϩμΫτͷٸ֦େ ʹΑΓɺࠓ·ͰҎ্ʹɺ ΑΓ৴པੑΛҡ࣋ɾ޲্ ͍ͨ͠ʂ τϥϑΟοΫ૿Ճͯ͠Δ͔ ΒɺܭըతʹΩϟύϓϥ͠

    ͍ͨʂ εέʔϥϏϦςΟ޲্ͤ͞ ͍͖͍ͯͨʂ ୯ମςετ΋e2eςετ΋ ֦ॆ͍͖͍͚ͯͨ͠Ͳɺͦ ͷͨΊʹ৭ʑ४උ͠ͳ͍ ͱ͍͚ͳͯ͘େมʂ ΞϓϦέʔγϣϯϨΠ Ϡʔͷ৽ػೳ։ൃʹूத ͍ͨ͠ʂ ࠓ·Ͱ͸ɺ֤νʔϜͷߴ͍ΦʔφʔγοϓʹΑͬͯ ։ൃεϐʔυͱϓϩμΫτ৴པੑͷཱ྆Λ࣮ݱ͍ͯͨ͠ɻ ͔͠͠ɺ૊৫ͷٸ֦େʹΑͬͯ ֤νʔϜͷαΠϩԽɾඇޮ཰ͳۀ຿͕ൃੜ͖ͯͨ͠🥺

14. 14 ΤϯδχΞ 2. ͳͥSREνʔϜ͕ඞཁʹͳͬͨͷ͔ ૊৫ɾϓϩμΫτͷٸ֦େ ʹΑΓɺࠓ·ͰҎ্ʹɺ ΑΓ৴པੑΛҡ࣋ɾ޲্ ͍ͨ͠ʂ τϥϑΟοΫ૿Ճͯ͠Δ͔ ΒɺܭըతʹΩϟύϓϥ͠

    ͍ͨʂ εέʔϥϏϦςΟ޲্ͤ͞ ͍͖͍ͯͨʂ ୯ମςετ΋e2eςετ΋ ֦ॆ͍͖͍͚ͯͨ͠Ͳɺͦ ͷͨΊʹ৭ʑ४උ͠ͳ͍ ͱ͍͚ͳͯ͘େมʂ ΞϓϦέʔγϣϯϨΠ Ϡʔͷ৽ػೳ։ൃʹूத ͍ͨ͠ʂ ɹɹɹɹɹɾઃཱ͔Β3೥ ɹɹɹɹɹɾΤϯδχΞ਺70௒ ɹɹɹɹɹɾϓϩμΫτνʔϜ਺30௒ʢSW/FW/HWʣ ɹɹɹɹɹɾϓϩμΫτϩʔϯν਺72௒ʢ2021೥9݄຤࣌఺ʣ

15. 15 ΤϯδχΞ 2. ͳͥSREνʔϜ͕ඞཁʹͳͬͨͷ͔ ૊৫ɾϓϩμΫτͷٸ֦େ ʹΑΓɺࠓ·ͰҎ্ʹɺ ΑΓ৴པੑΛҡ࣋ɾ޲্ ͍ͨ͠ʂ τϥϑΟοΫ૿Ճͯ͠Δ͔ ΒɺܭըతʹΩϟύϓϥ͠

    ͍ͨʂ εέʔϥϏϦςΟ޲্ͤ͞ ͍͖͍ͯͨʂ ୯ମςετ΋e2eςετ΋ ֦ॆ͍͖͍͚ͯͨ͠Ͳɺͦ ͷͨΊʹ৭ʑ४උ͠ͳ͍ ͱ͍͚ͳͯ͘େมʂ ΞϓϦέʔγϣϯϨΠ Ϡʔͷ৽ػೳ։ൃʹूத ͍ͨ͠ʂ ࢀߟ: Bitkey ϓϩμΫτ/ٕज़ମ੍ ঺հࢿྉ

16. 16 ΤϯδχΞ 2. ͳͥSREνʔϜ͕ඞཁʹͳͬͨͷ͔ ૊৫ɾϓϩμΫτͷٸ֦େ ʹΑΓɺࠓ·ͰҎ্ʹɺ ΑΓ৴པੑΛҡ࣋ɾ޲্ ͍ͨ͠ʂ τϥϑΟοΫ૿Ճͯ͠Δ͔ ΒɺܭըతʹΩϟύϓϥ͠

    ͍ͨʂ εέʔϥϏϦςΟ޲্ͤ͞ ͍͖͍ͯͨʂ ୯ମςετ΋e2eςετ΋ ֦ॆ͍͖͍͚ͯͨ͠Ͳɺͦ ͷͨΊʹ৭ʑ४උ͠ͳ͍ ͱ͍͚ͳͯ͘େมʂ ΞϓϦέʔγϣϯϨΠ Ϡʔͷ৽ػೳ։ൃʹूத ͍ͨ͠ʂ ͜ΕΒͷཁٻʹରͯ͠ Ξϓϩʔν͍ͯ͘͠ԣஅ૊৫ͷ ඞཁੑ͕ߴ·͖ͬͯͨʂʂ

17. 17 ΤϯδχΞ 2. ͳͥSREνʔϜ͕ඞཁʹͳͬͨͷ͔ ૊৫ɾϓϩμΫτͷٸ֦େ ʹΑΓɺࠓ·ͰҎ্ʹɺ ΑΓ৴པੑΛҡ࣋ɾ޲্ ͍ͨ͠ʂ τϥϑΟοΫ૿Ճͯ͠Δ͔ ΒɺܭըతʹΩϟύϓϥ͠

    ͍ͨʂ εέʔϥϏϦςΟ޲্ͤ͞ ͍͖͍ͯͨʂ ୯ମςετ΋e2eςετ΋ ֦ॆ͍͖͍͚ͯͨ͠Ͳɺͦ ͷͨΊʹ৭ʑ४උ͠ͳ͍ ͱ͍͚ͳͯ͘େมʂ ΞϓϦέʔγϣϯϨΠ Ϡʔͷ৽ػೳ։ൃʹूத ͍ͨ͠ʂ 2021೥7݄ɺSREνʔϜര஀ʂʂʂ

18. 3. SREνʔϜ্ཱͪ͛ ɹɹɹͰ΍ͬͨ͜ͱ 18 18

19. 3. SREνʔϜ্ཱͪ͛Ͱ΍ͬͨ͜ͱ 19 19 ര஀ͨ͠ͷ͸͍͍͚Ͳ ·ͣԿΛ͢Ε͹...

20. 3. SREνʔϜ্ཱͪ͛Ͱ΍ͬͨ͜ͱ 20 20 㙽ʹ΋֯ʹ΋ SRE νʔϜͱ͸Կऀͳͷ͔ ΛޠΕΔΑ͏ʹͳͬͯ

21. 3. SREνʔϜ্ཱͪ͛Ͱ΍ͬͨ͜ͱ 21 21 ͪΌΜͱ ϓϩμΫτ૊৫ʹ͓͚Δ ଘࡏײΛग़͍͔ͯ͠ͳ͚Ε͹ʂ

22. Bitkey ͷ SRE νʔϜ্ཱͪ͛Ͱɺओʹ΍ͬͨ͜ͱ 1. Bitkey ͷ SRE νʔϜͱͯ͠΍Γ͍ͨ͜ͱͷݴޠԽ 2.

    νʔϜߏ੒ݕ౼ 3. ਓһܭըɾ࠾༻׆ಈ 4. ྠಡձ 5. ϓϩμΫτνʔϜ΁SREࢪࡦͷಋೖਪਐ 22 22 3. SREνʔϜ্ཱͪ͛Ͱ΍ͬͨ͜ͱ

23. 1. SREνʔϜͱͯ͠΍Γ͍ͨ͜ͱͷݴޠԽ 23 23 3. SREνʔϜ্ཱͪ͛Ͱ΍ͬͨ͜ͱ

24. 3. SREνʔϜ্ཱͪ͛Ͱ΍ͬͨ͜ͱ 24 24 SREʁ ͳΔ΄ͲɻΘ͔ΒΜɻ

25. ϏοτΩʔʹ͓͚Δ SRE νʔϜͱͯ͠΍Γ͍ͨ͜ͱ 25 25 3. SREνʔϜ্ཱͪ͛Ͱ΍ͬͨ͜ͱ

26. ϏοτΩʔʹ͓͚Δ SRE νʔϜͱͯ͠΍Γ͍ͨ͜ͱ 1. ࣗಈԽͳͲͷࢪࡦΛ௨ͯ͠ɺγεςϜͷ։ൃ଎౓ͱ඼࣭Λڞʹ޲্ͤ͞Δ͜ͱ 2. γεςϜͷ৴པੑ΍ύϑΥʔϚϯεɺεέʔϥϏϦςΟͷվળ 3. ΑΓ࣋ଓՄೳͳ։ൃɾӡ༻ମ੍ʹ޲͚ͨDeveloper Experienceͷվળ

    4. ϓϩμΫτνʔϜ΁ͷΞʔΩςΫνϟઃܭࢧԉ΍ΩϟύγςΟϓϥχϯά౳Λ௨ ͯ͠ɺΑΓ҆ఆͨ͠ϦϦʔεମݧΛఏڙ͢Δ͜ͱ 5. ηΩϡϦςΟνʔϜ΁ͷٕज़ࢧԉɾϨϏϡʔ౳Λ௨ͯ͠ɺΑΓηΩϡΞͳࣾ಺؀ ڥ੔උ 6. ޮ཰తͳϦάϨογϣϯςετɺٴͼɺͦͷࣗಈԽɺ؀ڥ੔උΛ௨ͨ͡ϓϩμΫ τͷ඼࣭޲্ 26 26 3. SREνʔϜ্ཱͪ͛Ͱ΍ͬͨ͜ͱ

27. 1. SREνʔϜͱͯ͠ͷ੹຿ͷ໌֬Խ͕Ͱ͖ͨ 2. ࣾ಺֎΁ͷίϛϡχέʔγϣϯίετͷ௿ԼͰ͖ͨ 3. SREνʔϜͱͯ͠ɺ٬؍తʹৼΓฦΓ͕Ͱ͖Δ 27 27 3. SREνʔϜ্ཱͪ͛Ͱ΍ͬͨ͜ͱ

    🌞 Good ☔ Not Good 1. ͳ͠

28. 2. νʔϜߏ੒ݕ౼ 28 28 3. SREνʔϜ্ཱͪ͛Ͱ΍ͬͨ͜ͱ

29. 29 1. SREνʔϜઐ೚ a. ౰ͨΓલͱࢥ͏͔΋͠Εͳ͍͕ɺ্ཱͪ͛ϑΣʔζʹ͓͍ͯ͸ɺಛʹॏཁɻ 2. Embedded SRE ͱ Central

    SRE ͷߏ੒ a. ։ൃ଎౓ʹదͨ͠ࡉ΍͔ͳࢧԉ(Embedded SRE)ͱԣஅతͳSREશମઓུ (Central SRE)ͷ྆ྠ͕ͳ͚Ε͹ʮSREͱͯ͠΍Γ͍ͨ͜ͱʯ͸࣮ݱͰ͖ͳ͍ɻ ໨ࢦ͍ͯ͠ΔνʔϜߏ੒ ϓϩμΫτνʔϜ Embedded SRE Central SRE SREνʔϜ 3. SREνʔϜ্ཱͪ͛Ͱ΍ͬͨ͜ͱ

30. 3. ਓһܭըɾ࠾༻׆ಈ 30 30 3. SREνʔϜ্ཱͪ͛Ͱ΍ͬͨ͜ͱ

31. 1. ਓһܭըɾϩʔυϚοϓ࡞੒ a. ʮ΍Γ͍ͨ͜ͱʯͱʮ໨ࢦ͍ͯ͠ΔνʔϜߏ੒ʯ͔Βٯࢉ 2. SREνʔϜͱͯ͠࠾༻਺஋໨ඪΛઃఆ a. Կ΋Θ͔Βͳ͍தͰͷ࠾༻਺஋໨ඪ͸ෆ࣮֬ੑ͕ߴ͍ɻʢ໨ඪະୡͷՄೳੑ͕ߴ͍ b. ͔͠͠ɺνʔϜ໨ඪͱͯ͠࠾༻໨ඪΛઃఆ͠ͳ͍ͱɺ௨ৗۀ຿ͷํʹ޻਺࢖ͬͯ͠·

    ͍ͦ͏ͳ͜ͱ͕༰қʹ૝૾Ͱ͖ͨɻ c. ໨ඪະୡΑΓɺSREνʔϜ͕த్൒୺ʹͳͬͯɺՁ஋ఏڙͰ͖ͳ͍͜ͱͷํ͕ݏͩɻ ۩ମతʹ΍ͬͨ͜ͱ 31 31 3. SREνʔϜ্ཱͪ͛Ͱ΍ͬͨ͜ͱ

32. 🌞 Good 32 32 1. ϦΫϧʔςΟϯάνʔϜͱSRE࠾༻ʹ͓͚Δೝࣝͷ͢Γ߹Θ͕ͤͰ͖ͨɻ a. e.g. ຖ೔30෼ SRE࠾༻ఆྫΛϦΫϧʔςΟϯάνʔϜͱ࣮ࢪ

    2. ࠾༻׆ಈʹܞΘΔલͱޙͰɺ࠾༻ϓϩηεʹ͓͚Δվળ͕ΈΒΕͨɻ ☔ Not Good 1. ࠾༻׆ಈʹ͓͚ΔܧଓతͳධՁɾվળͷ࢓૊Έ͕ͳ͍ 3. SREνʔϜ্ཱͪ͛Ͱ΍ͬͨ͜ͱ

33. 4. ྠಡձ 33 33 3. SREνʔϜ্ཱͪ͛Ͱ΍ͬͨ͜ͱ

34. 34 34 ࢀߟ: https://sre.google/books/ ໨త 1. SREͱͯ͠ݪཧݪଇΛཧղ্ͨ͠Ͱɺ૊৫΁࠷దԽ͍ͯͨ͘͠Ί 2. SREνʔϜͱͯ͠ͷڞ௨ೝࣝɾڞ௨ݴޠΛ֫ಘ͢ΔͨΊ ର৅ॻ੶

    1. Site Reliability Engineering: ݪཧݪଇ 2. The Site Reliability Workbook: ۩ମతͳํ๏࿦ 3. SREνʔϜ্ཱͪ͛Ͱ΍ͬͨ͜ͱ

35. 35 35 3. SREνʔϜ্ཱͪ͛Ͱ΍ͬͨ͜ͱ ྠಡձͷਐΊํ 1. ςʔϚ୯Ґʢe.g. SLO/SLIɾΠϯγσϯτ etc...ʣʹԣ࣠Ͱɺݪଇͱ࣮ફΛ߹Θͤ ͯಡΉ

    a. ݪଇ: Site Reliability Engineering b. ࣮ફ: The Site Reliability Workbook 2. ૊৫ͷঢ়گʹԠͯ͡ɺςʔϚʹ༏ઌॱҐΛ͚ͭΔ 3. ຖि༵ۚ೔։࠵ʢ1࣌ؒʣ 4. ಉ͡ςʔϚΛϛʔςΟϯά·ͰʹಡΜͰɺཁ໿ɾҙ༁ͯ͠·ͱΊΔ 5. ղऍͷ࢓ํ͕೉͍͠෦෼౳ɺσΟεΧογϣϯͯ͠ೝࣝ߹Θͤ

36. 36 36 1. SREͷݪଇͱ࣮ફํ๏Λಉ͡λΠϛϯάͰֶͿ͜ͱͰɺͦΕͧΕಠཱͯ͠ಡΉΑΓ΋ɺ஌ࣝͱͯ͠ఆண͠ ͍ͯΔؾ͕͢Δɻ 2. ࣗ෼ͳΓͷҙ༁ɾղऍ΋߹Θͤͯ੔ཧ͢Δ͜ͱͰɺࣗ෼ͷݴ༿Ͱઆ໌Ͱ͖ΔͷͰɺೝࣝ߹Θ͕ͤ͠΍͔ͬ͢ ͨɻ 3. ੔ཧͨ͠಺༰Λͦͷ··ɺEmbedded

    SRE ͱͯ͠ͷ֤ϓϩμΫτνʔϜ΁ల։Ͱ͖ɺઆ໌ίετ͕௿Լ ͨ͠ɻ 1. ४උʹ͕͔͔࣌ؒΔɻɻɻ 3. SREνʔϜ্ཱͪ͛Ͱ΍ͬͨ͜ͱ 🌞 Good ☔ Not Good

37. ҰൠతͳSRE࿦͸ओʹSWྖҬʢՄ༻ੑɾՔಇ཰ɾϨΠςϯγʔ...ʣ Bitkey͸SW͚ͩͷձࣾͰ͸ͳ͘ɺSW×FW×HW͕͋Δ → PHYSICAL ʴ DIGITAL = PHYSITAL ͷձࣾ SREͷࢹ఺ɾ࣮૷ΛHW/FWʹ΋֦ு͢Δඞཁ͕͋Δ

    → ͜Ε͕BitkeyͷSRE͕೉͍͠ཧ༝Ͱ͸ɾɾɾʁ 37 37 3. SREνʔϜ্ཱͪ͛Ͱ΍ͬͨ͜ͱ ؾ෇͖ ࢀߟ: https://bitkey.co.jp/about/connect-vision/

38. 5. ϓϩμΫτνʔϜ΁SREࢪࡦͷಋೖਪਐ 38 38 3. SREνʔϜ্ཱͪ͛Ͱ΍ͬͨ͜ͱ

39. • ։ൃΨΠυϥΠϯɾϧʔϧࡦఆ • Πϯγσϯτج४ɾϓϩηεͷ੔උ͔Βӡ༻·Ͱ • Ϗϧυߴ଎Խʢ50%Ҏ্ͷ଎౓վળ • ΠϯϑϥߏஙɺٴͼɺΞʔΩςΫνϟϨϏϡʔ • Datadog

    ಋೖ • E2Eςετج൫ߏங • Ϋϥ΢υηΩϡϦςΟ਍அ • ΦϯϓϨϛεαʔόʔߏ੒؅ཧಋೖ etc... ۩ମతʹ࣮ࢪͨ͠SREࢪࡦ 39 39 3. SREνʔϜ্ཱͪ͛Ͱ΍ͬͨ͜ͱ

40. 3ϲ݄ৼΓฦͬͯΈͯ 40 40

41. 41 41 1. ʮSREνʔϜ্ཱͪ͛Ͱ΍ͬͨ͜ͱʯΛ࣮ફ͢Δ͜ͱͰɺΤϯδχΞ૊৫ʹ͓͚Δ SREνʔϜͷଘࡏײ͸૿͖ͯͨ͠ͱײ͍ͯ͡Δ 2. ϓϩμΫτνʔϜͷ৭Μͳํʑͱ࢓ࣄͰ͖Δͷ͸ָ͍͠ʂ 3. PHYSITAL ΊͬͪΌ೉͍͜͠ͱ͕Θ͔ͬͨɻɻɻ

    1. Embedded SRE ͱ Central SRE Λ݉຿͍ͯͨ͠ 4. 3ϲ݄ৼΓฦͬͯΈͯ 🌞 Good ☔ Not Good

42. SREνʔϜͷ͜Ε͔Β 42 42

43. 43 43 ·ͩ·ͩ΍Βͳ͖Ό͍͚ͳ͍͜ͱ͸ͨ͘͞Μʂ 3. SREνʔϜͷ͜Ε͔Β 1. Embedded SRE ͱ Central

    SRE ͦΕͧΕઐ೚΁ 2. Embedded SRE ͷϩʔςʔγϣϯ 3. SW͚ͩͰ͸ͳ͘ɺFW/HW΋ؚΊͨSLOɾSLIͷࡦఆ 4. Ҿ͖ଓ͖ɺ࠾༻׆ಈؤுΔʂʂ etc...

44. Bitkey ͷ ϓϩμΫτνʔϜɾSRE νʔϜ͸ ·ͩ·ͩ՝୊͕͍ͬͺ͍ͳͷͰ 44 44

45. Ұॹʹ৴པੑͷߴ͍ ϓϩμΫτΛ ࡞͍͖ͬͯ·͠ΐ͏ʂ 45 45

46. 46