Share
Embed
開発と運用でサービスの信頼性を高める 「SRE」の実践/Mercari SRE in practice Enterprise Development Conference
開発と運用でサービスの信頼性を高める
「SRE」の実践
Enterprise Development Conference
kazeburo
September 01, 2017
Transcript
45 slides
-
ࣗݾհ • Masahiro Nagano / խ • @kazeburo (twitter/github) •
גࣜձࣾϝϧΧϦ ϓϦϯγύϧΤϯδχΞ Site Reliability Engineering (SRE) νʔϜ • BASE, Inc ٕज़ΞυόΠβʔ -
ࣗݾհ(ܦྺɾ׆ಈ) • ܦྺ • 2006 mixi - ΞϓϦӡ༻νʔϜ • 2010
livedoor (LINE) - ։ൃࢧԉνʔϜ • 2015 ݱ৬ - SRE • 15Ҏ্ WebαʔϏεΛΠϯϑϥ͔Βࢧ͑Δۀ • ొஃʗࣥච • AWS Dev Day Tokyo 2017 ొஃ • WEB+DB PRESS Vol. 100 هࣄࣥච -
ϝϧΧϦ • ࠃ࠷େڃͷϑϦϚΞϓϦ • 3Ͱ؆୯ʹग़ 1) ࣸਅΛࡱΔ 2) ใΛهೖ 3)
ग़ϘλϯΛԡ͢ • ҆৺҆શͳܾࡁɾऔҾ • ΤεΫϩʔ • ͓ۚͷΓͱΓ͕ࣾؒʹհࡏ • ಗ໊ૹ -
ϝϧΧϦγεςϜ֓ཁ ©2011 Amazon Web Services LLC or its affiliates. All
rights reserved. Client Multimedia Corporate data center Traditional server Mobile Client IAM Add-on Example: IAM Add-on ence ) Assignment/ Task Requester Workers ग़! DB Search 5-දࣔ ݕࡧө ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific େྔͷϦΫΤετ ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corp data c Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ϦΫΤετԠ DB Search ߪೖ! ඵʙ30ඵ ඵʙ ߴʹฒߦͯ͠େྔͷτϥϯβΫγϣϯΛѻ͏ ը૾ ܾࡁ AI ϑΟʔυόοΫ -
ΠϯϑϥετϥΫνϟ JP US UK DNS: Amazon Route53 CDN: Akamai, Fastly,
ImageFlux Storage: Amazon S3 Analysis: Google BigQuery ܾࡁ/ྲྀαʔϏε ܾࡁ/ྲྀαʔϏε ܾࡁ/ྲྀαʔϏε -
SREͱ • Site Reliability Engineering/Engineer ͷུ • Reliability = ৴པੑ
• γεςϜཧͱαʔϏεӡ༻ͷํ๏ͱͯ͠Googleͷӡ༻νʔϜΛ͍͍ͯͨ Ben Treynor͕ఏএ • USΛத৺ʹେنͳITΠϯϑϥΛӡ༻͢Δ֤ࣾʹ·Δ • ໌֬ͳఆٛͳ͍͕ʮιϑτΣΞΤϯδχΞϦϯάʹΑͬͯɺΠϯϑϥετϥΫ νϟɾαʔϏεશମͷՄ༻ੑɺੑೳɺηΩϡϦςΟΛվળ͢ΔʯΤϯδχΞ/νʔϜ -
Google SRE • ιϑτΣΞΤϯδχΞϦϯάʹՃ͑ɺγεςϜɾӡ༻ͷೳྗ͕ٻΊΒΕ Δɻ·ͨ৫ɾνʔϜͷ͋ΓํؚΜͰ͍Δ • ιϑτΣΞΤϯδχΞϦϯάʮࣗಈԽʯʹϑΥʔΧε • SREͷਓαʔϏεͷنʹൺྫͤ͞ͳ͍(ݱ࣮తʹͰ͖ͳ͍) •
ʮτΠϧʯͷ໓ • ख࡞ۀͰߦΘΕɺࣗಈԽՄೳͰ܁Γฦ͢͜ͱʹՁΛ࣋ͨͳ͍ -
Google SRE • ۀ࣌ؒͷ50%ιϑτΣΞΤϯδχΞϦϯάΛߦ͏ • ࣗಈԽ(ࣗԽ)ɺ৴པੑ্ʹ͋ͯΔ • 50%Λ͑Δ͜ͱ͕͋Εۀͷେ෯ͳݟ͠ΛഭΒΕΔ • SLAɺΤϥʔόδΣοτ(༧ࢉ)ʹΑΔ։ൃऀͷརௐ
• ։ൃऀνʔϜͱՄ༻ੑͷඪΛαʔϏε͝ͱʹઃఆ • ΤϥʔόδΣοτʹ͋Δͱ͖։ൃऀੵۃతͳϦϦʔεΛߦ͍ɺ༧ࢉΛ ͑Δ߹৴པੑճ෮ͷͨΊͷ։ൃʹઐ೦͢Δ͜ͱ͕ٻΊΒΕΔ -
ຊࠃͰͷSRE • 201511݄ ϝϧΧϦٕज़blogͰSREΛհ • RettyɺαΠϘζɺCookPadɺMixiɺͯͳͳͲWebܥاۀΛத৺ʹSREͷ࠾༻͕ਐΜͰ͍Δ • ΠϯϑϥνʔϜ/ΠϯϑϥΤϯδχΞ͔ΒͷҠߦ͕ଟ͍ • ΫϥυԽΛഎܠʹΠϯϑϥΤϯδχΞͱ͍͏ݺͼํ͕దͰͳ͘ͳ͖ͬͯͨ
• SRE Tech Talk։࠵ • ୈҰճ: 20166݄ɻୈೋճ: 20171݄ • 100໊Ҏ্ͷࢀՃऀ -
SREͷظͷߴ·Γ • ॻ੶/ࡶࢽ • ΦϥΠϦʔʮSRE αΠτϦϥΠΞϏϦςΟΤϯδχΞϦϯάʯ • ܦBPʮܦSYSTEM 2017/7ʯ •
Πϯλʔωοτ্ͷಛूهࣄ • ITPro - άʔάϧൃͷ৽ख๏ʮSREʯɺຊͰ֦େ • http://itpro.nikkeibp.co.jp/atcl/column/14/346926/030600869/ • @IT - ಛूɿγεʹٻΊΒΕΔʮSREʯͱ͍͏৽ͨͳׂ • http://www.atmarkit.co.jp/ait/series/4503/ -
ϝϧΧϦ SRE • ͍ͭͰշద͔ͭ҆શʹར༻Ͱ͖Δʮ৴པੑͷߴ͍ʯαʔϏεͷ࣮ݱ • ʮ৽نαʔϏεͷ։ൃҎ֎ͷιϑτΣΞΤϯδχΞϦϯάશ෦Δʯ • ݱࡏϝϯόʔʮ10ਓʯ • શϝϯόʔ౦ژۈɻग़ுͰถࠃʹ͍͘͜ͱ
• େنͳWebαʔϏεͰͷܦݧ͕͋Δத్͕ଟ͍͕ɺ৽ଔϝϯόʔࡏ੶ • ࣾπʔϧͷӡ༻վળɺAIܥͷۀʹܞΘΔϝϯόʔ͍Δ -
ϝϧΧϦͰSREΛ࠾༻ͨ͠ཧ༝ • ϝϧΧϦΛ͓٬͞·ʹͬͯ͘Β͏ʹʮ͍ͭͰշదʹ҆શʹ ͑Δʯ৴པੑ͕ॏཁ • ΠϯϑϥͰͳ͘ɺιϑτΣΞؚΊ৴པੑʹΛ࣋ͭ͜ͱΛ໌Β ͔ʹ͢Δ • ւ֎Ͱ௨͡Δ໊শ •
JP/US/UKͰͷల։ɻάϩʔόϧͰͷ࠾༻ৗʹҙࣝ • ઌਐతͳऔΓΈͱͯ͠ͷૂ͍ -
Mercari SRE ͷۀൣғ Operations Software Eng. ج൫ߏங OnCall (োରԠ) Automation
εέʔϥϏϦςΟɾՄ༻ੑվળ DBAɺϛυϧΣΞߏங ΞϓϦέʔγϣϯͷઃܭϨϏϡʔ ϩάऩूɾੳج൫ͷߏஙɺӡ༻ αʔόϓϩϏδϣχϯάɾσϓϩΠɺϚΠΫϩαʔϏεج൫ͷඋ ηΩϡϦςΟʗෆਖ਼ར༻ݕग़ γεςϜӡ༻ΛʮΈʯͱͯ͠ ࡞Γ্͛Δ͜ͱ͕ٻΊΒΕ͍ͯΔ -
SRE൪/OnCall • ΞϥʔτରԠ • ΞϥʔτΛड͚औΔεϚϗΛৗʹOnʹ͠ɺҰ࣍ରԠΛߦ͏ • ి൪ • SlackΛ௨ͨ͡ۓٸిͷड৴ •
ʮোͷݕ͔ΒରԠ։࢝·Ͱͷ࣌ؒͷॖʯʮଞϦʔδϣϯ ͔ΒͷۓٸґཔͷରԠʯ͕త -
൪/OnCall Λࢧ͑Δٕज़ • ࢹ • Mackerel • ௨/Scheduling • Slack,
PagerDuty • BotʹΑΔిݺͼग़͠ -
mackerel: ΫϥυܕࢹαʔϏε • גࣜձࣾͯͳ ఏڙͷࢹαʔϏε • ͯͳࣾͰͷαʔόӡ༻ϊϋ • ֤छAPI͕༻ҙ͞ΕDevOpsͱͷ૬ੑྑ͍ •
PluginͰࢹ߲ͷ֦ு͕Մೳ • 40ݸҎ্ͷSREνʔϜ։ൃͷPluginΛར༻ • αʔόͷঢ়ଶ͚ͩͰͳ͘ɺ֎ܗࢹɺαʔϏεʹؔΘΔͷՄࢹԽɺΞϥʔτઃఆՄೳ • Ξϥʔτͷ௨SlackΛ࢝Ί֤छαʔϏε࿈ܞ͕༻ҙ -
slackͱΞϥʔτͷ • SlackϏδωε͚νϟοταʔϏε • OSSձࣾؒͷίϛϡχέʔγϣϯʹΘΕΔ • Chat botΛར༻ͨ͠ChatOpsͳͲߦ͏ • Ξϥʔτͷ௨ઌͱͯ͠దͰͳ͍߹
• ௨ԻͷΧελϚΠζͰ͖ͳ͍ • 1ճ͔͠௨Ͱ͖ͳ͍ɻྲྀΕ͕ͪ • ετϨε⤴ / ӡ༻ʹͳΓ͕ͪ -
PagerDutyʹΑΔ௨ • ༷ʑͳखஈͰ௨Λߦ͏͜ͱ͕Ͱ͖Δɻ Ԡ͢Δ·Ͱଓ͘ • mail • SMS • App
(iOS, Android) • ి • ʮ10Λ͑ͨͱ͜ΖͰҰిΛೖ ΕΔʯϧʔϧͰӡ༻ -
BotʹΑΔۓٸి • SlackͰbotʹର໋ͯ͠ྩ͢Δ͜ͱͰɺ൪ʹి͕ ͔͔ΔΈ • SRE͕։ൃɾӡ༻ • PagerDutyͷAPIΛར༻͠ɺ൪ͷ࿈བྷઌΛऔಘ • TwilioΛͬͯࣗಈͰి
• US/UKؚΊɺؾܰʹͬͯΒ͑ΔΑ͏ࣾࠂ ࠷ۙͰSREνʔϜͰΤεΧϨʔγϣϯతͰ ར༻ ☎ -
༷ʑͳ ChatOps • ൪ͷిͷଞʹbotΛར༻ͨ͠ӡ༻Λߦͳͬͯ ͍·͢ • ൪ใ • CDNͷΩϟογϡΫϦΞ •
ຊ൪ڥͰͷSQLͷ࣮ߦܭը֬ೝ • σϓϩΠ -
ύεϫʔυϦετ߈ܸͷ؇ • ൺֱత୯७ͳ߈ܸΞϓϦέʔγϣϯͰࣗಈతʹޚ • ಉҰͷϝʔϧΞυϨεʹΑΔෳճͷϩάΠϯࢼߦ • ಉҰͷIPΞυϨεʹΑΔෳΞΧϯτͷϩάΠϯࢼߦ • ߈ܸͱஅ͞Εͨ߹֘IPΛҰఆظؒڋ൱ •
ϩάΠϯ͞ΕͨͱࢥΘ͖͠ɺ͓٬͞·ʹύεϫʔυͷϦηοτଅ͢ -
େنύεϫʔυϦετ߈ܸࣄྫ • 2016ʹ࣮ࡍʹى͖ͨ߈ܸͷΞΫ ηεݩͷࠃ • ࣍ʑʹIPΛมߋ͠ɺͦΕͧΕͷIPͰ ճ͔͠ϩάΠϯࢼߦͤͣɺࣗಈ Ͱ͙͜ͱ͕͍͠ ͦͷଞ 18%
Armenia 2% Azerbaijan 2% Bahrain 2% Georgia 2% Japan 2% Russian 2% Indonesia 3% Nepal 3% Pakistan 5% Thailand 5% Taiwan 6% Viet Nam 6% Brazil 10% India 30% -
େنύεϫʔυϦετ߈ܸͷඋ͑ • ߈ܸ͞Ε͍͢Web൛ͰreCAPTCHAΛಋೖ • Client Reputation ͷར༻ • GeoIP: IPΞυϨε͔Βࠃఆ
• ಗ໊ϓϩΩγͷར༻ͷ༗ແ • IP Reputation -
Client reputation/GeoIP • ֎෦ͷIPσʔλϕʔεΛར༻ • ࠃ/भ/ݝ/ࢢϨϕϧ·Ͱਪఆ • σʔληϯλʔ͕ར༻͍ͯ͠ΔIP • ಗ໊ԽϓϩΩγͷఆ
• ࠃෆಠࣗʹऩू https://www.ip2location.com/demo -
Client reputation/IP reputation • ओʹSPAMϑΟϧλͳͲʹΘΕΔٕज़ • աڈʹSPAMͷૹ৴Λ͓͜ͳͬͨIPɺBotnetͱ ͯ͠ΘΕͨIPΛݕग़ • ௐࠪͷ݁ՌɺύεϫʔυϦετ߈ܸʹΘΕ
͍ͯΔ͜ͱ͕໌ • ϦεΫͷߴ͍IP͔ΒͷΞΫηεڋ൱͢Δ http://www.cyren.com/security-center/ip-reputation-check -
߈ܸݩIP Reputation • 2016ͷ߈ܸݩௐࠪͷ݁Ռ • 4ׂ͙͜ͱ͕Ͱ͖ͨՄೳੑ͕͋Δ* • ݱࡏͰɺ༷ʑͳใϦιʔεΛΈ߹Θͤͯ αʔϏεͷ҆શੑΛߴΊ͍ͯ·͢ -PX
.JEEMF )JHI * ߈ܸͷ͋ͬͨλΠϛϯάͷreputationͰͳ͍ͷͰଟগζϨ͕͋Γ·͢ -
·ͱΊ • SREͱ • Googleʹ͓͚ΔγεςϜӡ༻ͷ࣮ફ • ιϑτΣΞΤϯδχΞϦϯάʹΑͬͯ৴པੑΛ্ͤ͞Δ • ϝϧΧϦͷSREͷࣄྫհ •
OnCall/ӡ༻൪ • ύεϫʔυϦετ߈ܸ͔Βͷޚࣄྫ