はてなでの サービス信頼性向上のための 取り組み事例

Transcript

1. ͸ͯͳͰͷ αʔϏε৴པੑ޲্ͷͨΊͷ औΓ૊Έࣄྫ SRE Tech Talks גࣜձࣾ͸ͯͳ id:dekokun 

2. ࣗݾ঺հ 2

3. ࣗݾ঺հ • id:dekokun • ͸ͯͳͰΠϯϑϥ@౦ژ • WebΦϖϨʔγϣϯΤϯδχΞ • ͸ͯͳͷ͍Ζ͍ΖͳαʔϏεͷΠϯϑϥ୲౰ •

   ͸ͯͳϒϩάͱଞʹ΋͍͔ͭ͘ • ೖࣾ1೥ऑ • લ৬͸PHPͱJSॻ͍ͯ·ͨ͠ 3

4. 4 • SREͬͯͦ΋ͦ΋ͳʹʁ • ͸ͯͳͷWebΦϖϨʔγϣϯΤϯδχΞͱ
 WebΞϓϦέʔγϣϯΤϯδχΞͷڠྗମ੍ • DBసૹྔ࡟ݮ΁ͷऔΓ૊Έ • ίʔυͱݕূʹΑͬͯαʔϏεͷ৴པੑΛ޲্ͤ͞Δ࿩

   • AWSʹ͓͚ΔRedisͷ৑௕ߏ੒ߏங • ໨ࢦ͢΂͖γεςϜͷ࢟ʹ͍ͭͯ • མͪͯ΋མͪͯ΋ࣗಈͰىಈͯ͘͠Δαʔό ຊ೔ͷൃද಺༰

5. 5 • SREͬͯͦ΋ͦ΋ͳʹʁ • ͸ͯͳͷWebΦϖϨʔγϣϯΤϯδχΞͱ
 WebΞϓϦέʔγϣϯΤϯδχΞͷڠྗମ੍ • DBసૹྔ࡟ݮ΁ͷऔΓ૊Έ • ίʔυͱݕূʹΑͬͯαʔϏεͷ৴པੑΛ޲্ͤ͞Δ࿩

   • AWSʹ͓͚ΔRedisͷ৑௕ߏ੒ߏங • ໨ࢦ͢΂͖γεςϜͷ࢟ʹ͍ͭͯ • མͪͯ΋མͪͯ΋ࣗಈͰىಈͯ͘͠Δαʔό ຊ೔ͷൃද಺༰

6. SREͬͯͦ΋ͦ΋Կʁ 6

7. SREͱ͸ • SRE Tech Talksͱ͍͏໊લͷΠϕϯτͰ
 ൃද͍ͤͯͨͩ͘͜͞ͱʹͳ͕ͬͨ
 ͦ΋ͦ΋SRE͕Կ͔Λৄ͘͠஌Βͳ͔ͬͨ • ࢲͷ৬छ΋”SRE”ͱ͍͏໊લͰ͸ͳ͍ •

   SLAΛܾΊͯͦͷதͰ௅ઓ͍ͯ͘͠Πϝʔδ • ίʔυΛΑ͘ॻ͘Πϝʔδ • DevOpsͱ͸ҧ͏ͷ͔ͳʁ 7

8. SREͱ͸ • Site Reliability EngineeringಡΈ࢝Ί·ͨ͠ • ࣾ಺ྠಡձ࢝·Γ·ͨ͠ • ·ͩchapter 1,2,4͔͠ಡΜͰ͍ͳ͍

   • ࠓճ͸ɺࢲͷൃදʹؔ܎͋Γͦ͏ͳ෦෼͚ͩ
 ܰ͘આ໌͓ͭͭ͠࿩Λ͠·͢ • ҎԼɺ”SREຊ”ͱݺͼ·͢ 8 Site Reliability Engineering: http://shop.oreilly.com/product/0636920041528.do

9. SREͱ͸ • ͸ͯͳʹ͸SREͱ͍͏ݞॻ͸͋Γ·ͤΜ͕ɺ
 ͸ͯͳͰ΋αʔϏεͷ৴པੑ޲্ͷͨΊʹ
 ೔ʑ༷ʑͳ͜ͱΛ͍ͯ͠·͢ • ຊ೔͸ͦΕΒͷҰ෦Λ͝঺հ͍ͨ͠·͢ 9

10. ༻ޠղઆ 10

11. ༻ޠղઆ • WebΦϖϨʔγϣϯΤϯδχΞ • ͍ΘΏΔΠϯϑϥ୲౰ • ͜ͷࢿྉͰ͸”OpsΤϯδχΞ” • WebΞϓϦέʔγϣϯΤϯδχΞ •

    ͍ΘΏΔ։ൃΤϯδχΞ • ͜ͷࢿྉͰ͸”DevΤϯδχΞ” • ͸ͯͳʹ͓͍ͯ྆ऀ͕ߦ͏ࣄͷڥ໨͸͋Δఔ౓ᐆດ 11

12. 12 • SREͬͯͦ΋ͦ΋ͳʹʁ • ͸ͯͳͷWebΦϖϨʔγϣϯΤϯδχΞͱ
 WebΞϓϦέʔγϣϯΤϯδχΞͷڠྗମ੍ • DBసૹྔ࡟ݮ΁ͷऔΓ૊Έ • ίʔυͱݕূʹΑͬͯαʔϏεͷ৴པੑΛ޲্ͤ͞Δ࿩

    • AWSʹ͓͚ΔRedisͷ৑௕ߏ੒ߏங • ໨ࢦ͢΂͖γεςϜͷ࢟ʹ͍ͭͯ • མͪͯ΋མͪͯ΋ࣗಈͰىಈͯ͘͠Δαʔό ຊ೔ͷൃද಺༰

13. OpsΤϯδχΞͱDevΤϯδχΞͷ ڠྗମ੍ DBͷసૹྔ࡟ݮ 13

14. SREͱ͸(࠶ܝ) • “In SRE we bring this conflict to the

    fore, and then resolve it with the introduction of an error budget” • “Site Reliability Engineering: How Google Runs Production Systems” 2016/3/23 Betsy Beyerɺ Chris Jones Chapter 1 ΑΓ 14

15. opsΤϯδχΞͷαʔϏε΁ͷؔΘΓํ • SREຊʹΑΕ͹ɺSREͰ͸ͳ͍Ops͸Devͱ͸
 ໨ඪ͕ҟͳΔ͜ͱʹΑΓ໰୊͕ൃੜ͕ͪͩ͠ͱͷ͜ͱ • ͸ͯͳͰ͸͋·Γͳ͍Α͏ʹࢥ͑Δ • ͸ͯͳͰͷOpsΤϯδχΞͱDevΤϯδχΞͷ
 αʔϏε΁ͷؔΘΓํΛ঺հ͠·͢ 15

16. OpsΤϯδχΞͷαʔϏε΁ͷؔΘΓํ WebΦϖϨʔγϣϯ ΤϯδχΞνʔϜ ֤αʔϏε ͸ͯͳϒϩά ͸ͯͳϒοΫϚʔΫ etc… • ֤αʔϏεʹOpsΤϯδχΞ͕
 ୲౰ͱͯͭ͘͠

    16

17. OpsΤϯδχΞͷαʔϏε΁ͷؔΘΓํ • ͸ͯͳʹ͓͍ͯ྆ऀ͕ߦ͏ࣄͷڥ໨͸͋Δఔ౓ᐆດ • OpsΤϯδχΞ͕αʔϏεͷίʔυʹpull requestΛ
 ग़ͨ͠Γ • DevΤϯδχΞ͕chefΛ͍ͬͨ͡Γ͢Δ 17

18. ڠྗମ੍ࣄྫɿ DBͷసૹྔ࡟ݮ 18

19. DBసૹྔ࡟ݮࣄྫ • ͋Δ೔ɺmaster DB(MySQL)ͷసૹྔ͕
 ݶքۙ͘ͳ͍ͬͯΔ͜ͱʹؾ෇͘ • ೔ʹΑͬͯ͸ϐʔΫ࣌ʹ͸
 ”ޙ10%૿͑ͨΒݶքಥഁͩͳ”
 ͱ͍͏ײ͡ʹͳ͍ͬͯΔ৔߹΋… •

    खΛଧͭඞཁ͕͋Δ 19 ཧ࿦্ͷݶք஋ ΪϦΪϦͩ… Πϝʔδਤ

20. DBసૹྔ࡟ݮࣄྫ • CPU࢖༻཰͸ΊͪΌͪ͘Ό༨͍ͬͯͨͷͰMySQLͷ௨ ৴ѹॖ͕ྑͦ͞͏ͱ͍͏͜ͱͰݕূΛߦͬͨ • DSNʹ’mysql_compression=1’Λࢦఆ • ѹॖʹΑΓCPU͸৯͏͕ϚϧνίΞΛ࢖ͬͯ͘ΕΔ͠ ҆৺ 20

21. DBసૹྔ࡟ݮࣄྫ • ݕূ݁Ռ্͕ʑͩͬͨͷͰpull requestΛग़ͯ͠
 OpsΤϯδχΞʹϨϏϡʔͯ͠΋ΒͬͨΓ
 मਖ਼ͯ͠΋ΒͬͨΓͭͭ͠ϦϦʔε 21

22. DBసૹྔ࡟ݮࣄྫ • సૹྔܹݮ(໿1/4) ΊͰ͍ͨ • DBͷCPU࢖༻཰͕໰୊ͳ͍ൣғͰܹ૿ 22 సૹྔܹݮ CPU࢖༻཰ܹ૿

23. DBసૹྔ࡟ݮࣄྫ • సૹྔ͕ݮগ҆͠৺͍ͯͨ͋͠Δ໷ɺ͍͖ͳΓCPU࢖ ༻཰͕ܹ૿͠DB͕٧·Γ͔͚Δ • “Writing to net”ͷDBεϨου͕େྔʹଘࡏ • సૹྔ͕૿͑͗ͯ͢ѹॖʹΑͬͯCPUΛ৯͍ͭͿ͠ѹॖ

    ͕௥͍͍͍ͭͯͳ͍ • ѹॖʹΑΔ΋ͷ͕ͩɺ͔ͱ͍ͬͯΞϓϦέʔγϣϯαʔ όશ୆ͰѹॖΛղআ͢Δͱࠓ౓͸సૹྔͰ٧·Δ 23

24. DBసૹྔ࡟ݮࣄྫ Ұ࣍ରԠ • Ұ࣍ରԠͱͯ͠ΞϓϦέʔγϣϯαʔόͷҰ෦ͷΈͰ
 ѹॖΛղআ͠೉ΛಀΕͨ 24 ѹॖ ѹॖ ඇѹॖ ඇѹॖ

25. DBసૹྔ࡟ݮࣄྫ Ұ࣍ରԠ • ཌ೔ɺࣄଶ͸·ͩ༧அΛڐ͞ͳ͍ঢ়گͳͷͰ
 Ұ୴ΠϯελϯελΠϓΛ্͛ͯmasterDB੾Γସ͑Λ ߦ͍Ұ҆৺ • AWS͞ΜMHA͞Μ͋Γ͕ͱ͏ 25

26. DBసૹྔ࡟ݮࣄྫ ࠜຊରॲ ௐࠪ • ٸʹCPU࢖༻཰্͕͕Δͷ͸ௐ͕ࠪඞཁ • DevΤϯδχΞͱڞʹkibana౳͔ΒݪҼΛௐࠪ͠ɺ
 ͋ΔΤϯυϙΠϯτ͕ϢʔβͷߦಈʹΑͬͯ
 ܹ͘͠DBͷసૹྔΛফඅ͢ΔΑ͏ʹͳΔ͜ͱΛൃݟɾ
 DevΤϯδχΞͱڞ༗

    26

27. DBసૹྔ࡟ݮࣄྫ ࠜຊରॲ • ཌ೔ʹ͸DevΤϯδχΞͷखʹΑͬͯରॲ൛͕
 ϦϦʔε͞ΕɺDBͷసૹྔ΋ݮΓΊͰͨ͠ΊͰͨ͠ • ͦͷޙDevΤϯδχΞͷखͰɺߋʹDBͷసૹྔΛݮΒ ͢ϦϦʔε͕Կ౓͔ߦΘΕͨ 27

28. OpsΤϯδχΞͱDevΤϯδχΞ • SREຊͰ͸“(SREͰ͸ͳ͍چདྷͷ)Ops͸αΠτͷՄ༻ੑ ΛकΔͨΊʹૉૣ͍σϓϩΠͳͲʹରཱ͢Δ”ͱ͍͏Α ͏ͳ࿩΋͋Γ·ͨ͠ɻ͕ɺ • ࠓճݟͨΑ͏ʹɺڍಈ͕Ϣʔβͷߦಈʹґଘ͢Δ
 ҰൠͷWebαʔϏεʹ͓͍ͯͦ΋ͦ΋”αΠτͷՄ༻ੑ ΛकΔ”ͱ͍͏จ຺Ͱ΋Ospଆ΋ૉૣ͍มԽΛ๬·ͳ͍ ͱࢮ͵ͷͰ͸

    • ΋ͪΖΜαʔϏεͷ੒௕͕Ұ൪ 28

29. 29 • ΤϯδχΞ͕ࢀՃ͢ΔɺຖिߦΘΕΔࣾ಺ษڧձͰ͸
 ͦΕͧΕͷ࢓ࣄͷ಺༰Λൃද͍ͯ͘͠ͳͲ
 ૬ޓͷཧղͷॿ͚ʹͳ͍ͬͯ·͢ • ۀ຿࣌ؒ֎ͷো֐ରԠΛݮΒͨ͢Ίɺ
 ҎԼͷΑ͏ͳ։ൃνʔϜͱͷ߹ҙ΋ߦ͍ͬͯ·͢ • ༵ۚ೔ʹϦϦʔεΛ͠ͳ͍

    • ༦ํҎ߱ʹϦϦʔεΛ͠ͳ͍ OpsΤϯδχΞͱDevΤϯδχΞ ଞʹ΋͍Ζ͍Ζ

30. 30 • ͜ͷΑ͏ʹɺ͸ͯͳͰ͸೔ʑOpsΤϯδχΞͱ
 DevΤϯδχΞ͕ڠྗͭͭ͠
 αʔϏεͷ৴པੑ޲্΁ͷऔΓ૊ΈΛ
 ߦ͍͍ͬͯͬͯ·͢ OpsΤϯδχΞͱDevΤϯδχΞ ·ͱΊ

31. 31 • SREͬͯͦ΋ͦ΋ͳʹʁ • ͸ͯͳͷWebΦϖϨʔγϣϯΤϯδχΞͱ
 WebΞϓϦέʔγϣϯΤϯδχΞͷڠྗମ੍ • DBసૹྔ࡟ݮ΁ͷऔΓ૊Έ • ίʔυͱݕূʹΑͬͯαʔϏεͷ৴པੑΛ޲্ͤ͞Δ࿩

    • AWSʹ͓͚ΔRedisͷ৑௕ߏ੒ߏங • ໨ࢦ͢΂͖γεςϜͷ࢟ʹ͍ͭͯ • མͪͯ΋མͪͯ΋ࣗಈͰىಈͯ͘͠Δαʔό ຊ೔ͷൃද಺༰

32. ίʔυͱݕূʹΑͬͯ αʔϏεͷ৴པੑΛ޲্ͤ͞Δ࿩ AWS؀ڥͷRedisͷ৑௕ߏ੒ߏங 32

33. SREͱ͸(࠶ܝ) • “Google caps operational work for SREs at 50%

    of their time. Their remaining time should be spent using their coding skills on project work.” • “Site Reliability Engineering: How Google Runs Production Systems” 2016/3/23 Betsy Beyerɺ Chris Jones Chapter 1 ΑΓ 33

34. AWS؀ڥͷRedisͷ৑௕ߏ੒ߏங • զʑ͸50%΋ίʔυ͸ॻ͍͍ͯͳ͍ʂ • ͔͠͠ɺ΋ͪΖΜίʔυ͸ॻ͍͍ͯ·͢ • ίʔυΛॻ͖ͭͭɺ໰୊ղܾΛߦͬͨ࿩Λ͠·͢ 34

35. AWS؀ڥͷRedisͷ৑௕ߏ੒ߏங • ͸ͯͳͰ͸σʔληϯλʔͱAWSͷ྆ํͰαʔϏεΛ ӡӦ͍ͯ͠·͢ • Redisͷ৑௕ߏ੒͸͜Ε·ͰσʔληϯλʔͰ͸
 ߦ͍ͬͯ·͕ͨ͠ɺAWS্Ͱ͸Ͱ͖͍ͯ·ͤΜͰͨ͠ • AWS্Ͱ΋Redisͷ৑௕ߏ੒Λߏங͓ͨ͠࿩Λ͠·͢ 35

36. σʔληϯλʔͰͷRedisͷ৑௕ߏ੒ 36 Redis (master) keepalived IP: x.x.x.x Redis (slave) keepalived

    IP:y.y.y.y VRRP /w broadcast replication health check health check VIP: z.z.z.z VIPͰ઀ଓ

37. σʔληϯλʔͰͷRedisͷ৑௕ߏ੒ • keepalived + VRRPʹΑΔ৑௕ߏ੒ • ΞϓϦέʔγϣϯ͔Β͸VIPͰRedisʹ઀ଓ • Redisʹ໰୊͕͋ͬͨ৔߹͸VIP͕Ҡಈ •

    VIPͷҠಈͱಉ࣌ʹɺslaveͷmasterঢ֨/masterͷslave ߱֨Λ࣮ࢪ 37 Redis (master) keepalived IP: x.x.x.x Redis (slave) keepalived IP:y.y.y.y VRRP /w broadcast replication health check VIP: z.z.z.z VIPͰ઀ଓ health check

38. Redis (master) keepalived IP: x.x.x.x Redis (slave) keepalived IP:y.y.y.y VRRP

    /w broadcast replication health check health check VIP: z.z.z.z VIPͰ઀ଓ AWSͰσʔληϯλʔͱಉ༷ͷߏ੒ΛͱΔͨΊͷ໰୊ 38 VRRP /w broadcast VIP: z.z.z.z

39. Redis (master) keepalived IP: x.x.x.x Redis (slave) keepalived IP:y.y.y.y VRRP

    /w broadcast replication health check health check VIP: z.z.z.z VIPͰ઀ଓ AWSͰσʔληϯλʔͱಉ༷ͷߏ੒ΛͱΔͨΊͷ໰୊఺ 1 39 VRRP /w broadcast AWSͰ͸ broadcast͕࢖͑ͳ͍

40. 40 • AWS VPCͰ͸IPͷbroadcast/multicast͸࢖༻Ͱ͖ͳ͍ ͷͰunicastΛ࢖༻ͨ͠VRRPΛߦ͏ඞཁ͕͋Δ • ͸ͯͳͰ࢖͍ͬͯΔkeepalivedͷߏจνΣοΧ͸ൺֱ త৽͍͠ػೳͰ͋ΔunicastʹରԠ͍ͯ͠ͳ͍ • keepalived͸ඪ४ͰߏจνΣοΧ͕͍͓ͭͯΒͣɺ

    ޡͬͨߏจͰreload͢Δͱਖ਼ৗʹಈ࡞͠ͳ͘ͳΔͷ ͰߏจνΣοΧͳ͠Ͱ͸ාͯ͘࢖͑ͳ͍ Redisͷ৑௕ߏ੒ unicastͱߏจνΣοΫ

41. 41 • ͸ͯͳͷ࢖͍ͬͯͨߏจνΣοΧ͸Haskell੡ • ࢲ͕unicastରԠͷHaskellίʔυΛॻͨ͘Ίʹ
 Ϗϧυ؀ڥΛ੔͑Α͏ͱ࢛ۤീ͍ۤͯ͠Δ಺ʹɺ
 ผͷDevΤϯδχΞͷid:y_uuki ͞Μ͕
 GoݴޠͰߏจνΣοΧΛ࡞ͬͯ͘Εͯղܾ Redisͷ৑௕ߏ੒

    unicastͱߏจνΣοΫ

42. Redisͷ৑௕ߏ੒ unicastͱߏจνΣοΫ 42

43. Redis (master) keepalived IP: x.x.x.x Redis (slave) keepalived IP:y.y.y.y VRRP

    /w broadcast replication health check health check VIP: z.z.z.z VIPͰ઀ଓ AWSͰσʔληϯλʔͱಉ༷ͷߏ੒ΛͱΔͨΊͷ໰୊఺ 2 43 VIP: z.z.z.z VIPͷҠಈΛͲ͏͢Δʁ

44. 44 • σʔληϯλʔ಺Ͱ͸VIPͷҠಈ͸
 GARPʹΑͬͯαϒωοτʹप஌͞ΕΔ͕
 AWS VPCͰ͸ͦͷํ๏͸࢖͑ͳ͍ • VIPͷҠಈ͸ENIΛҠಈͤ͞Δ͜ͱͰ࣮ݱ • ͜͜Ͱ΋id:y_uuki͞Μͷ࡞ͬͨ


    Goݴޠ੡ͷιϑτ΢ΣΞɺgrabeniΛ࢖༻ Redisͷ৑௕ߏ੒ VIPͷҠಈ

45. Redisͷ৑௕ߏ੒ VIPͷҠಈ 45

46. 46 • ͍ͭͰʹݕূதʹݟ͔༷ͭͬͨʑͳখ͍͞໰୊ΛຒΊΔ • keepalivedͷݱࡏͷstatusΛϑΝΠϧग़ྗ • keepalivedͷMASTER/BACKUPߏ੒͔Β
 BACKUP/BACKUPߏ੒΁ • keepalivedͷstatus੾Γସ͑௨஌ϝʔϧͷ


    λΠτϧʹϗετ໊͕ग़ྗ͞ΕΔΑ͏ʹ • Redisͷfailover࣌ʹconfig rewriteΛߦ͍ɺfailoverޙʹ Redis͕࠶ىಈͯ͠΋ઃఆ͕ר͖໭Βͳ͍Α͏ʹ Redisͷ৑௕ߏ੒ ͦͷଞ

47. 47 • ͱ͍͏༷ʑͳιϑτ΢ΣΞ։ൃ΍ݕূʹΑͬͯɺAWS ʹ͓͚ΔRedisͷ৑௕ߏ੒͸࣮ݱ͠·ͨ͠ Redisͷ৑௕ߏ੒ ·ͱΊ

48. 48 • SREͬͯͦ΋ͦ΋ͳʹʁ • ͸ͯͳͷWebΦϖϨʔγϣϯΤϯδχΞͱ
 WebΞϓϦέʔγϣϯΤϯδχΞͷڠྗମ੍ • DBసૹྔ࡟ݮ΁ͷऔΓ૊Έ • ίʔυͱݕূʹΑͬͯαʔϏεͷ৴པੑΛ޲্ͤ͞Δ࿩

    • AWSʹ͓͚ΔRedisͷ৑௕ߏ੒ߏங • ໨ࢦ͢΂͖γεςϜͷ࢟ʹ͍ͭͯ • མͪͯ΋མͪͯ΋ࣗಈͰىಈͯ͘͠Δαʔό ຊ೔ͷൃද಺༰

49. ໨ࢦ͢΂͖γεςϜͷ࢟ʹ͍ͭͯ མͪͯ΋མͪͯ΋ࣗಈͰ
 ىಈͯ͘͠Δαʔό 49

50. SREͱ͸(࠶ܝ) • “Monitoring should never require a human to interpret

    any part of the alerting domain. Instead, software should do the interpreting, and humans should be notified only when they need to take action. ” • “Site Reliability Engineering: How Google Runs Production Systems” 2016/3/23 Betsy Beyerɺ Chris Jones Chapter 1 ΑΓ 50

51. 51 • ਓؒ͸ਓؒͷͰ͖Δ͜ͱʹूத͢΂͖Ͱɺ
 ػց͕Ͱ͖Δ͜ͱ͸ػցʹ೚ͤΔ΂͖ • ͸ͯͳʹೖͬͯ໘ന͍ͳͱײͨ͡
 ࣗ཯෮چγεςϜʹ͍ͭͯ • ΋ͪΖΜଞʹ΋͍Ζ͍Ζͳ
 ࣗಈγεςϜ͸͋ΔͷͰ͕͢ɺ


    Ұ൪͓΋͠Ζ͔ͬͨ΋ͷΛ ໨ࢦ͢΂͖γεςϜͷ࢟

52. 52 • ͸ͯͳͷσʔληϯλʔͰ͸
 ෺ཧαʔόΛԾ૝Խͯ͠࢖༻ • Ծ૝Խج൫ʹ͸XenΛ࢖༻͠ɺ
 1୆ͷ෺ཧαʔόʹෳ਺ͷOSΛฒྻʹಈ࡞͍ͤͯ͞Δ Xenʹ͍ͭͯ

53. Xenʹ͍ͭͯ(༻ޠ) • DomU: ͍ΘΏΔී௨ͷαʔόͱͯ͠࢖༻ • Dom0: DomUͷىಈ౳΋ؚΊͨXenͷ؅ཧ΍ϋʔυ΢ΣΞΞΫηεΛߦ͏ 53 ϋʔυ΢ΣΞ Dom0(Xenͷ؅ཧ)

    DomU(Webαʔό) DomU(Webαʔό) DomU(DBαʔό)

54. 54 • Xenͷ্Ͱ࡞ΒΕͨɺੲͷαʔϏεʹΑ͘࢖ΘΕ͍ͯΔ ࣗ཯෮چγεςϜ ໨ࢦ͢΂͖γεςϜͷ࢟ ࣗ཯෮چ

55. ໨ࢦ͢΂͖γεςϜͷ࢟ ࣗ཯෮چ • Dom0͕DomU্Ͱಈ͍͍ͯΔHTTPΞϓϦέʔγϣϯ ΛHTTPͰ؂ࢹ(monit࢖༻) 55 ؂ࢹ ؂ࢹ ϋʔυ΢ΣΞ Dom0(Xenͷ؅ཧ)

    DomU(Webαʔό) DomU(Webαʔό) DomU(DBαʔό)

56. ໨ࢦ͢΂͖γεςϜͷ࢟ ࣗ཯෮چ • DomUͷHTTPαʔό͕ࢮΜͩ͜ͱΛݕ஌͢Δͱ 56 ؂ࢹ ࢮ๢ݕ஌ ϋʔυ΢ΣΞ Dom0(Xenͷ؅ཧ) DomU(Webαʔό)

    DomU(Webαʔό) DomU(DBαʔό)

57. ໨ࢦ͢΂͖γεςϜͷ࢟ ࣗ཯෮چ • monit͕ͦͷDomUΛOS͝ͱstop/start • xm destroy && xm create

    57 ؂ࢹ DomUΛOS͝ͱ stop/start ϋʔυ΢ΣΞ Dom0(Xenͷ؅ཧ) DomU(Webαʔό) DomU(Webαʔό) DomU(DBαʔό)

58. ໨ࢦ͢΂͖γεςϜͷ࢟ ࣗ཯෮چ • উखʹ෮چ׬ྃ 58 ؂ࢹ ؂ࢹ ϋʔυ΢ΣΞ Dom0(Xenͷ؅ཧ) DomU(Webαʔό)

    DomU(Webαʔό) DomU(DBαʔό)

59. ໨ࢦ͢΂͖γεςϜͷ࢟ • ͋Δ೔ߴෛՙͰαʔό͕མͪͯ͸ࣗಈ෮چ͍ͯ͠ΔͷΛ ݟͯײಈͨ࣌͠ͷൃݴ 59

60. 60 • ਓؒ͸ਓؒʹ͔͠Ͱ͖ͳ͍͜ͱʹ
 ूத͢Δ؀ڥΛ࡞͍ͬͯ͘ ໨ࢦ͢΂͖γεςϜͷ࢟

61. 61 • ͸ͯͳͰͷࣄྫ঺հΛ͠·ͨ͠ • ͜Ε͔Β΋DevΤϯδχΞͱڞʹྑ͍αʔϏεΛ࡞ͬͯ ͍͖·͢ • ͜Ε͔Β΋೪Γڧࣗ͘ಈԽΛਐΊ͍͖ͯ·͢ • ػցͷͰ͖Δ͜ͱ͸શͯػցʹ೚ͤΔੈք΁…

    ·ͱΊ

62. 62 • ͸ͯͳͰ͸OpsΤϯδχΞΛੵۃ࠾༻͍ͯ͠·͢ʂ • ౦ژͰ΋ژ౎Ͱ΋ʂ • ࢲ͸౦ژͰژ౎ͷDev/OpsΤϯδχΞͱ
 Ұॹʹ࢓ࣄ͍ͯ͠·͢ʂ • ࠓ೔ͷൃද΍౦ژͱژ౎ͷϦϞʔτϫʔΫͷ࿩ͳͲͰ΋


    ฉ͖͍ͨ͜ͱ͋Γ·ͨ͠Βੋඇ࠙਌ձͰʂ ࠷ޙʹ