Share
Embed
はてなでの サービス信頼性向上のための 取り組み事例
SRE Tech Talks ( http://connpass.com/event/34825/ ) でお話した際の資料です
dekokun
July 25, 2016
Transcript
62 slides
-
ࣗݾհ • id:dekokun • ͯͳͰΠϯϑϥ@౦ژ • WebΦϖϨʔγϣϯΤϯδχΞ • ͯͳͷ͍Ζ͍ΖͳαʔϏεͷΠϯϑϥ୲ •
ͯͳϒϩάͱଞʹ͍͔ͭ͘ • ೖࣾ1ऑ • લ৬PHPͱJSॻ͍ͯ·ͨ͠ 3 -
4 • SREͬͯͦͦͳʹʁ • ͯͳͷWebΦϖϨʔγϣϯΤϯδχΞͱ WebΞϓϦέʔγϣϯΤϯδχΞͷڠྗମ੍ • DBసૹྔݮͷऔΓΈ • ίʔυͱݕূʹΑͬͯαʔϏεͷ৴པੑΛ্ͤ͞Δ
• AWSʹ͓͚ΔRedisͷߏߏங • ࢦ͖͢γεςϜͷ࢟ʹ͍ͭͯ • མͪͯམͪͯࣗಈͰىಈͯ͘͠Δαʔό ຊͷൃද༰ -
5 • SREͬͯͦͦͳʹʁ • ͯͳͷWebΦϖϨʔγϣϯΤϯδχΞͱ WebΞϓϦέʔγϣϯΤϯδχΞͷڠྗମ੍ • DBసૹྔݮͷऔΓΈ • ίʔυͱݕূʹΑͬͯαʔϏεͷ৴པੑΛ্ͤ͞Δ
• AWSʹ͓͚ΔRedisͷߏߏங • ࢦ͖͢γεςϜͷ࢟ʹ͍ͭͯ • མͪͯམͪͯࣗಈͰىಈͯ͘͠Δαʔό ຊͷൃද༰ -
SREͱ • SRE Tech Talksͱ͍͏໊લͷΠϕϯτͰ ൃද͍ͤͯͨͩ͘͜͞ͱʹͳ͕ͬͨ ͦͦSRE͕Կ͔Λৄ͘͠Βͳ͔ͬͨ • ࢲͷ৬छ”SRE”ͱ͍͏໊લͰͳ͍ •
SLAΛܾΊͯͦͷதͰઓ͍ͯ͘͠Πϝʔδ • ίʔυΛΑ͘ॻ͘Πϝʔδ • DevOpsͱҧ͏ͷ͔ͳʁ 7 -
SREͱ • Site Reliability EngineeringಡΈ࢝Ί·ͨ͠ • ࣾྠಡձ࢝·Γ·ͨ͠ • ·ͩchapter 1,2,4͔͠ಡΜͰ͍ͳ͍
• ࠓճɺࢲͷൃදʹؔ͋Γͦ͏ͳ෦͚ͩ ܰ͘આ໌͓ͭͭ͠Λ͠·͢ • ҎԼɺ”SREຊ”ͱݺͼ·͢ 8 Site Reliability Engineering: http://shop.oreilly.com/product/0636920041528.do -
༻ޠղઆ • WebΦϖϨʔγϣϯΤϯδχΞ • ͍ΘΏΔΠϯϑϥ୲ • ͜ͷࢿྉͰ”OpsΤϯδχΞ” • WebΞϓϦέʔγϣϯΤϯδχΞ •
͍ΘΏΔ։ൃΤϯδχΞ • ͜ͷࢿྉͰ”DevΤϯδχΞ” • ͯͳʹ͓͍ͯ྆ऀ͕ߦ͏ࣄͷڥ͋Δఔᐆດ 11 -
12 • SREͬͯͦͦͳʹʁ • ͯͳͷWebΦϖϨʔγϣϯΤϯδχΞͱ WebΞϓϦέʔγϣϯΤϯδχΞͷڠྗମ੍ • DBసૹྔݮͷऔΓΈ • ίʔυͱݕূʹΑͬͯαʔϏεͷ৴པੑΛ্ͤ͞Δ
• AWSʹ͓͚ΔRedisͷߏߏங • ࢦ͖͢γεςϜͷ࢟ʹ͍ͭͯ • མͪͯམͪͯࣗಈͰىಈͯ͘͠Δαʔό ຊͷൃද༰ -
SREͱ(࠶ܝ) • “In SRE we bring this conflict to the
fore, and then resolve it with the introduction of an error budget” • “Site Reliability Engineering: How Google Runs Production Systems” 2016/3/23 Betsy Beyerɺ Chris Jones Chapter 1 ΑΓ 14 -
DBసૹྔݮࣄྫ • ͋Δɺmaster DB(MySQL)ͷసૹྔ͕ ݶքۙ͘ͳ͍ͬͯΔ͜ͱʹؾ͘ • ʹΑͬͯϐʔΫ࣌ʹ ”ޙ10%૿͑ͨΒݶքಥഁͩͳ” ͱ͍͏ײ͡ʹͳ͍ͬͯΔ߹… •
खΛଧͭඞཁ͕͋Δ 19 ཧ্ͷݶք ΪϦΪϦͩ… Πϝʔδਤ -
DBసૹྔݮࣄྫ • సૹྔ͕ݮগ҆͠৺͍ͯͨ͋͠Δɺ͍͖ͳΓCPU ༻͕ܹ૿͠DB͕٧·Γ͔͚Δ • “Writing to net”ͷDBεϨου͕େྔʹଘࡏ • సૹྔ͕૿͑͗ͯ͢ѹॖʹΑͬͯCPUΛ৯͍ͭͿ͠ѹॖ
͕͍͍͍ͭͯͳ͍ • ѹॖʹΑΔͷ͕ͩɺ͔ͱ͍ͬͯΞϓϦέʔγϣϯαʔ όશͰѹॖΛղআ͢ΔͱࠓసૹྔͰ٧·Δ 23 -
29 • ΤϯδχΞ͕ࢀՃ͢ΔɺຖिߦΘΕΔࣾษڧձͰ ͦΕͧΕͷࣄͷ༰Λൃද͍ͯ͘͠ͳͲ ૬ޓͷཧղͷॿ͚ʹͳ͍ͬͯ·͢ • ۀ࣌ؒ֎ͷোରԠΛݮΒͨ͢Ίɺ ҎԼͷΑ͏ͳ։ൃνʔϜͱͷ߹ҙߦ͍ͬͯ·͢ • ༵ۚʹϦϦʔεΛ͠ͳ͍
• ༦ํҎ߱ʹϦϦʔεΛ͠ͳ͍ OpsΤϯδχΞͱDevΤϯδχΞ ଞʹ͍Ζ͍Ζ -
31 • SREͬͯͦͦͳʹʁ • ͯͳͷWebΦϖϨʔγϣϯΤϯδχΞͱ WebΞϓϦέʔγϣϯΤϯδχΞͷڠྗମ੍ • DBసૹྔݮͷऔΓΈ • ίʔυͱݕূʹΑͬͯαʔϏεͷ৴པੑΛ্ͤ͞Δ
• AWSʹ͓͚ΔRedisͷߏߏங • ࢦ͖͢γεςϜͷ࢟ʹ͍ͭͯ • མͪͯམͪͯࣗಈͰىಈͯ͘͠Δαʔό ຊͷൃද༰ -
SREͱ(࠶ܝ) • “Google caps operational work for SREs at 50%
of their time. Their remaining time should be spent using their coding skills on project work.” • “Site Reliability Engineering: How Google Runs Production Systems” 2016/3/23 Betsy Beyerɺ Chris Jones Chapter 1 ΑΓ 33 -
σʔληϯλʔͰͷRedisͷߏ 36 Redis (master) keepalived IP: x.x.x.x Redis (slave) keepalived
IP:y.y.y.y VRRP /w broadcast replication health check health check VIP: z.z.z.z VIPͰଓ -
σʔληϯλʔͰͷRedisͷߏ • keepalived + VRRPʹΑΔߏ • ΞϓϦέʔγϣϯ͔ΒVIPͰRedisʹଓ • Redisʹ͕͋ͬͨ߹VIP͕Ҡಈ •
VIPͷҠಈͱಉ࣌ʹɺslaveͷmasterঢ֨/masterͷslave ߱֨Λ࣮ࢪ 37 Redis (master) keepalived IP: x.x.x.x Redis (slave) keepalived IP:y.y.y.y VRRP /w broadcast replication health check VIP: z.z.z.z VIPͰଓ health check -
Redis (master) keepalived IP: x.x.x.x Redis (slave) keepalived IP:y.y.y.y VRRP
/w broadcast replication health check health check VIP: z.z.z.z VIPͰଓ AWSͰσʔληϯλʔͱಉ༷ͷߏΛͱΔͨΊͷ 38 VRRP /w broadcast VIP: z.z.z.z -
Redis (master) keepalived IP: x.x.x.x Redis (slave) keepalived IP:y.y.y.y VRRP
/w broadcast replication health check health check VIP: z.z.z.z VIPͰଓ AWSͰσʔληϯλʔͱಉ༷ͷߏΛͱΔͨΊͷ 1 39 VRRP /w broadcast AWSͰ broadcast͕͑ͳ͍ -
40 • AWS VPCͰIPͷbroadcast/multicast༻Ͱ͖ͳ͍ ͷͰunicastΛ༻ͨ͠VRRPΛߦ͏ඞཁ͕͋Δ • ͯͳͰ͍ͬͯΔkeepalivedͷߏจνΣοΧൺֱ త৽͍͠ػೳͰ͋ΔunicastʹରԠ͍ͯ͠ͳ͍ • keepalivedඪ४ͰߏจνΣοΧ͕͍͓ͭͯΒͣɺ
ޡͬͨߏจͰreload͢Δͱਖ਼ৗʹಈ࡞͠ͳ͘ͳΔͷ ͰߏจνΣοΧͳ͠Ͱාͯ͑͘ͳ͍ Redisͷߏ unicastͱߏจνΣοΫ -
Redis (master) keepalived IP: x.x.x.x Redis (slave) keepalived IP:y.y.y.y VRRP
/w broadcast replication health check health check VIP: z.z.z.z VIPͰଓ AWSͰσʔληϯλʔͱಉ༷ͷߏΛͱΔͨΊͷ 2 43 VIP: z.z.z.z VIPͷҠಈΛͲ͏͢Δʁ -
44 • σʔληϯλʔͰVIPͷҠಈ GARPʹΑͬͯαϒωοτʹप͞ΕΔ͕ AWS VPCͰͦͷํ๏͑ͳ͍ • VIPͷҠಈENIΛҠಈͤ͞Δ͜ͱͰ࣮ݱ • ͜͜Ͱid:y_uuki͞Μͷ࡞ͬͨ
GoݴޠͷιϑτΣΞɺgrabeniΛ༻ Redisͷߏ VIPͷҠಈ -
46 • ͍ͭͰʹݕূதʹݟ͔༷ͭͬͨʑͳখ͍͞ΛຒΊΔ • keepalivedͷݱࡏͷstatusΛϑΝΠϧग़ྗ • keepalivedͷMASTER/BACKUPߏ͔Β BACKUP/BACKUPߏ • keepalivedͷstatusΓସ͑௨ϝʔϧͷ
λΠτϧʹϗετ໊͕ग़ྗ͞ΕΔΑ͏ʹ • Redisͷfailover࣌ʹconfig rewriteΛߦ͍ɺfailoverޙʹ Redis͕࠶ىಈͯ͠ઃఆ͕ר͖Βͳ͍Α͏ʹ Redisͷߏ ͦͷଞ -
48 • SREͬͯͦͦͳʹʁ • ͯͳͷWebΦϖϨʔγϣϯΤϯδχΞͱ WebΞϓϦέʔγϣϯΤϯδχΞͷڠྗମ੍ • DBసૹྔݮͷऔΓΈ • ίʔυͱݕূʹΑͬͯαʔϏεͷ৴པੑΛ্ͤ͞Δ
• AWSʹ͓͚ΔRedisͷߏߏங • ࢦ͖͢γεςϜͷ࢟ʹ͍ͭͯ • མͪͯམͪͯࣗಈͰىಈͯ͘͠Δαʔό ຊͷൃද༰ -
SREͱ(࠶ܝ) • “Monitoring should never require a human to interpret
any part of the alerting domain. Instead, software should do the interpreting, and humans should be notified only when they need to take action. ” • “Site Reliability Engineering: How Google Runs Production Systems” 2016/3/23 Betsy Beyerɺ Chris Jones Chapter 1 ΑΓ 50 -
Xenʹ͍ͭͯ(༻ޠ) • DomU: ͍ΘΏΔී௨ͷαʔόͱͯ͠༻ • Dom0: DomUͷىಈؚΊͨXenͷཧϋʔυΣΞΞΫηεΛߦ͏ 53 ϋʔυΣΞ Dom0(Xenͷཧ)
DomU(Webαʔό) DomU(Webαʔό) DomU(DBαʔό) -
ࢦ͖͢γεςϜͷ࢟ ࣗ෮چ • Dom0͕DomU্Ͱಈ͍͍ͯΔHTTPΞϓϦέʔγϣϯ ΛHTTPͰࢹ(monit༻) 55 ࢹ ࢹ ϋʔυΣΞ Dom0(Xenͷཧ)
DomU(Webαʔό) DomU(Webαʔό) DomU(DBαʔό) -
ࢦ͖͢γεςϜͷ࢟ ࣗ෮چ • DomUͷHTTPαʔό͕ࢮΜͩ͜ͱΛݕ͢Δͱ 56 ࢹ ࢮݕ ϋʔυΣΞ Dom0(Xenͷཧ) DomU(Webαʔό)
DomU(Webαʔό) DomU(DBαʔό) -
ࢦ͖͢γεςϜͷ࢟ ࣗ෮چ • monit͕ͦͷDomUΛOS͝ͱstop/start • xm destroy && xm create
57 ࢹ DomUΛOS͝ͱ stop/start ϋʔυΣΞ Dom0(Xenͷཧ) DomU(Webαʔό) DomU(Webαʔό) DomU(DBαʔό) -
ࢦ͖͢γεςϜͷ࢟ ࣗ෮چ • উखʹ෮چྃ 58 ࢹ ࢹ ϋʔυΣΞ Dom0(Xenͷཧ) DomU(Webαʔό)
DomU(Webαʔό) DomU(DBαʔό)