開発と運用でサービスの信頼性を高める 「SRE」の実践/Mercari SRE in practice Enterprise Development Conference

開発と運用でサービスの信頼性を高める
「SRE」の実践
Enterprise Development Conference

Transcript

1. ։ൃͱӡ༻ͰαʔϏεͷ৴པੑΛߴΊΔ
 ʮSREʯͷ࣮ફ Enterprise Development Conference 2017/09/01 גࣜձࣾϝϧΧϦ Masahiro Nagano @kazeburo
2. ࣗݾ঺հ • Masahiro Nagano / ௕໺խ޿ • @kazeburo (twitter/github) • גࣜձࣾϝϧΧϦ
 ϓϦϯγύϧΤϯδχΞ
 Site Reliability Engineering (SRE) νʔϜ • BASE, Inc ٕज़ΞυόΠβʔ
3. ࣗݾ঺հ(ܦྺɾ׆ಈ) • ܦྺ • 2006೥ mixi - ΞϓϦӡ༻νʔϜ • 2010೥ livedoor (LINE) - ։ൃࢧԉνʔϜ • 2015೥ ݱ৬ - SRE • 15೥Ҏ্ WebαʔϏεΛΠϯϑϥ͔Βࢧ͑Δۀ຿ • ొஃʗࣥච • AWS Dev Day Tokyo 2017 ొஃ • WEB+DB PRESS Vol. 100 هࣄࣥච
4. ΞδΣϯμ • ࣗݾ঺հ • ϝϧΧϦʹ͍ͭͯ • SREͱ͸ • ϝϧΧϦͷSREͷࣄྫ঺հ
5. ϝϧΧϦʹ͍ͭͯ
6. ϝϧΧϦ • ࠃ಺࠷େڃͷϑϦϚΞϓϦ • 3෼Ͱ؆୯ʹग़඼ 1) ࣸਅΛࡱΔ 2) ঎඼৘ใΛهೖ 3) ग़඼ϘλϯΛԡ͢ • ҆৺҆શͳܾࡁɾऔҾ • ΤεΫϩʔ • ͓ۚͷ΍ΓͱΓ͸౰͕ࣾؒʹհࡏ • ಗ໊഑ૹ
7. ถࠃ/ӳࠃ ΁ͷల։ JP US UK
8. KPI μ΢ϯϩʔυ਺ GMV(૯औҾֹ) 7500ສDL(JP+US) ݄ؒ100ԯԁҎ্ ग़඼਺ 1೔100ສ඼Ҏ্ (ϐʔΫ࣌ؒଳ͸1෼ؒʹ1000ग़඼Ҏ্)
9. ϝϧΧϦγεςϜ֓ཁ ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. Client Multimedia Corporate data center Traditional server Mobile Client IAM Add-on Example: IAM Add-on ence ) Assignment/ Task Requester Workers ग़඼! DB Search 5-දࣔ ݕࡧ൓ө ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific େྔͷϦΫΤετ ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corp data c Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ϦΫΤετԠ౴ DB Search ߪೖ! ਺ඵʙ30ඵ ਺ඵʙ ߴ଎ʹฒߦͯ͠େྔͷτϥϯβΫγϣϯΛѻ͏ ը૾ ܾࡁ AI ϑΟʔυόοΫ
10. ΠϯϑϥετϥΫνϟ JP US UK DNS: Amazon Route53 CDN: Akamai, Fastly, ImageFlux Storage: Amazon S3 Analysis: Google BigQuery ܾࡁ/෺ྲྀαʔϏε ܾࡁ/෺ྲྀαʔϏε ܾࡁ/෺ྲྀαʔϏε
11. SREͱ͸
12. SREͱ͸ • Site Reliability Engineering/Engineer ͷུ • Reliability = ৴པੑ • γεςϜ؅ཧͱαʔϏεӡ༻ͷํ๏࿦ͱͯ͠Googleͷӡ༻νʔϜΛ཰͍͍ͯͨ Ben Treynor͕ఏএ • USΛத৺ʹେن໛ͳITΠϯϑϥΛӡ༻͢Δ֤ࣾʹ޿·Δ • ໌֬ͳఆٛ͸ͳ͍͕ʮιϑτ΢ΣΞΤϯδχΞϦϯάʹΑͬͯɺΠϯϑϥετϥΫ νϟɾαʔϏεશମͷՄ༻ੑɺੑೳɺηΩϡϦςΟΛվળ͢ΔʯΤϯδχΞ/νʔϜ
13. Google SRE • ιϑτ΢ΣΞΤϯδχΞϦϯάʹՃ͑ɺγεςϜɾӡ༻ͷೳྗ͕ٻΊΒΕ Δɻ·ͨ૊৫ɾνʔϜͷ͋Γํ΋ؚΜͰ͍Δ • ιϑτ΢ΣΞΤϯδχΞϦϯά͸ʮࣗಈԽʯʹϑΥʔΧε • SREͷਓ਺͸αʔϏεͷن໛ʹൺྫͤ͞ͳ͍(ݱ࣮తʹͰ͖ͳ͍) • ʮτΠϧʯͷ๾໓ • ख࡞ۀͰߦΘΕɺࣗಈԽՄೳͰ܁Γฦ͢͜ͱʹՁ஋Λ࣋ͨͳ͍
14. Google SRE • ۀ຿࣌ؒͷ50%͸ιϑτ΢ΣΞΤϯδχΞϦϯάΛߦ͏ • ࣗಈԽ(ࣗ཯Խ)ɺ৴པੑ޲্ʹ͋ͯΔ • 50%Λ௒͑Δ͜ͱ͕͋Ε͹ۀ຿ͷେ෯ͳݟ௚͠ΛഭΒΕΔ • SLAɺΤϥʔόδΣοτ(༧ࢉ)ʹΑΔ։ൃऀͷར֐ௐ੔ • ։ൃऀνʔϜͱՄ༻ੑͷ໨ඪΛαʔϏε͝ͱʹઃఆ • ΤϥʔόδΣοτ಺ʹ͋Δͱ͖͸։ൃऀ͸ੵۃతͳϦϦʔεΛߦ͍ɺ༧ࢉΛ௒ ͑Δ৔߹͸৴པੑճ෮ͷͨΊͷ։ൃʹઐ೦͢Δ͜ͱ͕ٻΊΒΕΔ
15. ೔ຊࠃ಺ͰͷSRE • 2015೥11݄ ϝϧΧϦٕज़blogͰSREΛ঺հ • RettyɺαΠϘ΢ζɺCookPadɺMixiɺ͸ͯͳͳͲWebܥاۀΛத৺ʹSREͷ࠾༻͕ਐΜͰ͍Δ • ΠϯϑϥνʔϜ/ΠϯϑϥΤϯδχΞ͔ΒͷҠߦ͕ଟ͍ • Ϋϥ΢υԽΛഎܠʹΠϯϑϥΤϯδχΞͱ͍͏ݺͼํ͕ద੾Ͱ͸ͳ͘ͳ͖ͬͯͨ • SRE Tech Talk։࠵ • ୈҰճ: 2016೥6݄ɻୈೋճ: 2017೥1݄ • 100໊Ҏ্ͷࢀՃऀ
16. SRE΁ͷظ଴ͷߴ·Γ • ॻ੶/ࡶࢽ • ΦϥΠϦʔʮSRE αΠτϦϥΠΞϏϦςΟΤϯδχΞϦϯάʯ • ೔ܦBPʮ೔ܦSYSTEM 2017/7ʯ • Πϯλʔωοτ্ͷಛूهࣄ • ITPro - άʔάϧൃͷ৽ख๏ʮSREʯɺ೔ຊͰ֦େ • http://itpro.nikkeibp.co.jp/atcl/column/14/346926/030600869/ • @IT - ಛूɿ৘γεʹٻΊΒΕΔʮSREʯͱ͍͏৽ͨͳ໾ׂ • http://www.atmarkit.co.jp/ait/series/4503/
17. ϝϧΧϦ SRE
18. ϝϧΧϦ SRE • ͍ͭͰ΋շద͔ͭ҆શʹར༻Ͱ͖Δʮ৴པੑͷߴ͍ʯαʔϏεͷ࣮ݱ • ʮ৽نαʔϏεͷ։ൃҎ֎ͷιϑτ΢ΣΞΤϯδχΞϦϯά͸શ෦΍Δʯ • ݱࡏϝϯόʔ͸ʮ10ਓʯ • શϝϯόʔ౦ژۈ຿ɻग़ுͰถࠃʹ͍͘͜ͱ΋ • େن໛ͳWebαʔϏεͰͷܦݧ͕͋Δத్͕ଟ͍͕ɺ৽ଔϝϯόʔ΋ࡏ੶ • ࣾ಺πʔϧͷӡ༻վળɺAIܥͷۀ຿ʹܞΘΔϝϯόʔ΋͍Δ
19. ϝϧΧϦͰSREΛ࠾༻ͨ͠ཧ༝ • ϝϧΧϦΛ͓٬͞·ʹ௕͘࢖ͬͯ΋Β͏ʹ͸ʮ͍ͭͰ΋շదʹ҆શʹ ࢖͑Δʯ৴པੑ͕ॏཁ • ΠϯϑϥͰ͸ͳ͘ɺιϑτ΢ΣΞؚΊ৴པੑʹ੹೚Λ࣋ͭ͜ͱΛ໌Β ͔ʹ͢Δ • ւ֎Ͱ΋௨͡Δ໊শ • JP/US/UKͰͷల։ɻάϩʔόϧͰͷ࠾༻͸ৗʹҙࣝ • ઌਐతͳऔΓ૊Έͱͯ͠ͷૂ͍
20. Mercari SRE ͷۀ຿ൣғ Operations Software Eng. ج൫ߏங OnCall (ো֐ରԠ) Automation εέʔϥϏϦςΟɾՄ༻ੑվળ DBAɺϛυϧ΢ΣΞߏங ΞϓϦέʔγϣϯͷઃܭϨϏϡʔ ϩάऩूɾ෼ੳج൫ͷߏஙɺӡ༻
    αʔόϓϩϏδϣχϯάɾσϓϩΠɺϚΠΫϩαʔϏεج൫ͷ੔උ
    ηΩϡϦςΟʗෆਖ਼ར༻ݕग़ γεςϜӡ༻Λʮ࢓૊Έʯͱͯ͠
 ࡞Γ্͛Δ͜ͱ͕ٻΊΒΕ͍ͯΔ
21. ϝϧΧϦ SREͷ࣮ફࣄྫ
22. ϝϧΧϦSRE࣮ફࣄྫ • OnCall/౰൪ରԠ • ηΩϡϦςΟͷऔΓ૊Έ • ύεϫʔυϦετ߈ܸͷࣄྫ
23. OnCall/౰൪ରԠ
24. SRE౰൪/OnCall • ΞϥʔτରԠ • ΞϥʔτΛड͚औΔεϚϗΛৗʹOnʹ͠ɺҰ࣍ରԠΛߦ͏ • ి࿩౰൪ • SlackΛ௨ͨ͡ۓٸి࿩ͷड৴ • ʮো֐ͷݕ஌͔ΒରԠ։࢝·Ͱͷ࣌ؒͷ୹ॖʯʮଞϦʔδϣϯ ͔Βͷۓٸґཔ΁ͷରԠʯ͕໨త
25. SRE౰൪/OnCall • ೔༵೔0͔࣌Β౔༵೔24࣌·Ͱ1िؒͰަ୅ • ฏ೔͸νʔϜϝϯόʔ͕ग़ࣾ͢Δ·Ͱࣗ୐଴ػ • 9͔࣌Βࣗ୐଴ػ͠ɺUS͔ΒͷґཔͳͲʹରԠ • ٳ೔΋15-20෼Ҏ಺ʹରԠ։࢝Ͱ͖Δ͜ͱ͕๬·͘͠ɺߦಈʹ੍ ݶ΋͋Δ • ਂ໷ٳ೔ͷରԠͳͲɺՈ଒ͷڠྗ΋ඞཁ
26. ౰൪/OnCall Λࢧ͑Δٕज़ • ؂ࢹ • Mackerel • ௨஌/Scheduling • Slack, PagerDuty • BotʹΑΔి࿩ݺͼग़͠
27. mackerel: Ϋϥ΢υܕ؂ࢹαʔϏε • גࣜձࣾ͸ͯͳ ఏڙͷ؂ࢹαʔϏε • ͸ͯͳࣾͰͷαʔόӡ༻ϊ΢ϋ΢ • ֤छAPI͕༻ҙ͞ΕDevOpsͱͷ૬ੑ΋ྑ͍ • PluginͰ؂ࢹ߲໨ͷ֦ு͕Մೳ • 40ݸҎ্ͷSREνʔϜ։ൃͷPluginΛར༻ • αʔόͷঢ়ଶ͚ͩͰ͸ͳ͘ɺ֎ܗ؂ࢹɺαʔϏεʹؔΘΔ਺஋ͷՄࢹԽɺΞϥʔτઃఆՄೳ • Ξϥʔτͷ௨஌͸SlackΛ࢝Ί֤छαʔϏε࿈ܞ͕༻ҙ
28. slackͱΞϥʔτͷ໰୊఺ • Slack͸Ϗδωε޲͚νϟοταʔϏε • OSS΍ձࣾؒͷίϛϡχέʔγϣϯʹ΋࢖ΘΕΔ • Chat botΛར༻ͨ͠ChatOpsͳͲ΋ߦ͏ • Ξϥʔτͷ௨஌ઌͱͯ͠͸ద੾Ͱ͸ͳ͍৔߹΋ • ௨஌ԻͷΧελϚΠζͰ͖ͳ͍ • 1ճ͔͠௨஌Ͱ͖ͳ͍ɻྲྀΕ͕ͪ • ετϨε⤴ / ӡ༻ʹͳΓ͕ͪ
29. PagerDuty: Πϯγσϯτ؅ཧ • Πϯγσϯτ؅ཧͷΫϥ΢υαʔϏε • ༷ʑͳखஈʹΑΔ௨஌ • ௨஌ͷϩʔςʔγϣϯ؅ཧ • ΤεΧϨʔγϣϯϧʔϧͷઃఆ
30. PagerDutyʹΑΔ௨஌ • ༷ʑͳखஈͰ௨஌Λߦ͏͜ͱ͕Ͱ͖Δɻ ൓Ԡ͢Δ·Ͱଓ͘ • mail • SMS • App (iOS, Android) • ి࿩ • ʮ10෼Λ௒͑ͨͱ͜ΖͰҰ౓ి࿩Λೖ ΕΔʯϧʔϧͰӡ༻
31. BotʹΑΔۓٸి࿩ • SlackͰbotʹର໋ͯ͠ྩ͢Δ͜ͱͰɺ౰൪ʹి࿩͕ ͔͔Δ࢓૊Έ • SRE͕։ൃɾӡ༻ • PagerDutyͷAPIΛར༻͠ɺ౰൪ͷ࿈བྷઌΛऔಘ • TwilioΛ࢖ͬͯࣗಈͰి࿩ • US/UKؚΊɺؾܰʹ࢖ͬͯ΋Β͑ΔΑ͏ࣾ಺ࠂ஌
 ࠷ۙͰ͸SREνʔϜ಺Ͱ΋ΤεΧϨʔγϣϯ໨తͰ ར༻ ☎
32. ༷ʑͳ ChatOps • ౰൪΁ͷి࿩ͷଞʹ΋botΛར༻ͨ͠ӡ༻Λߦͳͬͯ ͍·͢ • ౰൪৘ใ • CDNͷΩϟογϡΫϦΞ • ຊ൪؀ڥͰͷSQLͷ࣮ߦܭը֬ೝ • σϓϩΠ
33. ηΩϡϦςΟͷऔΓ૊Έ ύεϫʔυϦετ߈ܸͷࣄྫ
34. ύεϫʔυϦετ߈ܸ • ͓٬͞·ͷΞΧ΢ϯτʹରͯ͠ɺϥϯμϜͳύεϫʔυ΋͘͠͸ผͰ࿙Ӯ ͨ͠ύεϫʔυจࣈྻΛ࢖͍ϩάΠϯΛࢼߦɺෆਖ਼ϩάΠϯΛߦ͏ • ͞·͟·ͳن໛ͷ߈ܸ͕ߦΘΕɺଞͷαʔϏεͰͷࣄྫ΋ଟ͍
35. ύεϫʔυϦετ߈ܸ΁ͷରԠ • Ϧετ߈ܸͷݕ஌ • ߈ܸ͕ߦΘΕ͍ͯΔ͜ͱΛͰ͖ΔݶΓૣ͘ݕ஌͠ɺΞϥʔτΛ͋͛Δ • ߈ܸ؇࿨ࡦͷ࣮ࢪ • ΞΫηεύλʔϯ͔ΒࣗಈͰ๷ޚ • ߈ܸΛड͚ͨΞΧ΢ϯτͷύεϫʔυϦηοτ
36. ύεϫʔυϦετ߈ܸͷݕ஌ • ϩάΠϯࣦഊΛAPIͷϩάͱͯ͠ه࿥ • ϩάΛϦΞϧλΠϜͰूܭɺ
 mackerelͰՄࢹԽͱ؂ࢹΛߦ͏ • ϝϧΧϦ͕TVʹऔΓ্͛ΒΕΔͱ
 Ξϥʔτ͕དྷΔ͜ͱ΋
37. ύεϫʔυϦετ߈ܸͷ؇࿨ • ൺֱత୯७ͳ߈ܸ͸ΞϓϦέʔγϣϯ಺Ͱࣗಈతʹ๷ޚ • ಉҰͷϝʔϧΞυϨεʹΑΔෳ਺ճͷϩάΠϯࢼߦ • ಉҰͷIPΞυϨεʹΑΔෳ਺ΞΧ΢ϯτ΁ͷϩάΠϯࢼߦ • ߈ܸͱ൑அ͞Εͨ৔߹͸֘౰IPΛҰఆظؒڋ൱ • ϩάΠϯ͞ΕͨͱࢥΘ͖͠ɺ͓٬͞·ʹ͸ύεϫʔυͷϦηοτଅ͢
38. େن໛ͳύεϫʔυϦετ߈ܸࣄྫ • ւ֎޲͚αʔϏε͸ࠃ಺޲͚αʔϏεΑΓ΋߈ܸΛड͚Δճ਺͕ଟ͍ • ߈ܸͷن໛͕େ͖͘ͳΓɺखޱ΋ෳࡶʹͳΔ܏޲
39. େن໛ύεϫʔυϦετ߈ܸࣄྫ • 2016೥ʹ࣮ࡍʹى͖ͨ߈ܸͷΞΫ ηεݩͷࠃ • ࣍ʑʹIPΛมߋ͠ɺͦΕͧΕͷIPͰ ͸਺ճ͔͠ϩάΠϯࢼߦͤͣɺࣗಈ Ͱ๷͙͜ͱ͕೉͍͠ ͦͷଞ 18% Armenia 2% Azerbaijan 2% Bahrain 2% Georgia 2% Japan 2% Russian 2% Indonesia 3% Nepal 3% Pakistan 5% Thailand 5% Taiwan 6% Viet Nam 6% Brazil 10% India 30%
40. େن໛ύεϫʔυϦετ߈ܸ΁ͷඋ͑ • ߈ܸ͞Ε΍͍͢Web൛Ͱ͸reCAPTCHAΛಋೖ • Client Reputation ͷར༻ • GeoIP: IPΞυϨε͔Βࠃ൑ఆ • ಗ໊ϓϩΩγͷར༻ͷ༗ແ • IP Reputation
41. Client reputation/GeoIP • ֎෦ͷIPσʔλϕʔεΛར༻ • ࠃ/भ/ݝ/ࢢϨϕϧ·Ͱਪఆ • σʔληϯλʔ͕ར༻͍ͯ͠ΔIP • ಗ໊ԽϓϩΩγͷ൑ఆ • ࠃ಺ෆ଍෼͸ಠࣗʹऩू https://www.ip2location.com/demo
42. Client reputation/IP reputation • ओʹSPAMϑΟϧλͳͲʹ࢖ΘΕΔٕज़ • աڈʹSPAMͷૹ৴Λ͓͜ͳͬͨIPɺBotnetͱ ͯ͠࢖ΘΕͨIPΛݕग़ • ௐࠪͷ݁ՌɺύεϫʔυϦετ߈ܸʹ΋࢖ΘΕ ͍ͯΔ͜ͱ͕൑໌ • ϦεΫͷߴ͍IP͔ΒͷΞΫηε͸ڋ൱͢Δ http://www.cyren.com/security-center/ip-reputation-check
43. ߈ܸݩIP Reputation • 2016೥ͷ߈ܸݩௐࠪͷ݁Ռ • 4ׂ͸๷͙͜ͱ͕Ͱ͖ͨՄೳੑ͕͋Δ* • ݱࡏͰ͸ɺ༷ʑͳ৘ใϦιʔεΛ૊Έ߹Θͤͯ
 αʔϏεͷ҆શੑΛߴΊ͍ͯ·͢ -PX  .JEEMF  )JHI  * ߈ܸͷ͋ͬͨλΠϛϯάͷreputationͰ͸ͳ͍ͷͰଟগζϨ͕͋Γ·͢
44. ·ͱΊ • SREͱ͸ • Googleʹ͓͚ΔγεςϜӡ༻ͷ࣮ફ • ιϑτ΢ΣΞΤϯδχΞϦϯάʹΑͬͯ৴པੑΛ޲্ͤ͞Δ • ϝϧΧϦͷSREͷࣄྫ঺հ • OnCall/ӡ༻౰൪ • ύεϫʔυϦετ߈ܸ͔Βͷ๷ޚࣄྫ
45. ͝੩ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠ ຊεϥΠυ͸speakerdeckʹͯެ։͠·͢
 https://speakerdeck.com/kazeburo/