Share
Embed
インフラチームからSREへ / SRE in Mercari Developers Summit 2018
インフラチームからSREへ
〜メルカリを支える新しいインフラのあり方
Developers Summit 2018/2/16
kazeburo
February 16, 2018
Transcript
41 slides
-
Me • Masahiro Nagano / խ • @kazeburo (twitter/github) •
גࣜձࣾϝϧΧϦ ϓϦϯγύϧΤϯδχΞ Site Reliability Engineering (SRE) νʔϜ • BASE, Inc ٕज़ΞυόΠβʔ -
Me • ~ 2006: ژͰελʔτΞοϓࢀՃ • ΤϯδχΞ໊ • ։ൃΛ͠ͳ͕ΒΠϯϑϥͷ໘ΛݟΔɻDC࡞ۀͬͨ •
ΞϓϦέʔγϣϯͷνϡʔχϯάɺۭ͍ͨϦιʔεͰ৽ػೳͷՃͱ͍͏αΠΫϧ • 2006 ~: mixi • ʮΞϓϦέʔγϣϯӡ༻νʔϜʯDCʹߦ͔ͳ͍ΦϖϨʔγϣϯ • େنը૾৴ΞϓϦέʔγϣϯͷνϡʔχϯά -
Me • 2010 ~: livedoor (NHN Japan => LINE) •
livedoorLINEϑΝϛϦʔͷαʔϏεΛԣஅͯ͠ΠϯϑϥετϥΫνϟ ύϑΥʔϚϯεͷվળ • livedoor Blog ͷMySQLνϡʔχϯά / Plack࠷దԽ • 2015/02 ~ : mercari -
࠷ۙͷ׆ಈ • ొஃ • AWS Dev Day Tokyo 2017 •
YAPC::Fukuoka 2017, YAPC::Hokkaido 2016 • YAPC::Okinawa 2018, Manabiya Teratail Developer Days ొஃ༧ఆ • هࣄ • WEB+DB PRESS Vol.88, Vol.92-97 ࿈ࡌ, Vol.100 • ܦSYSTEMS 2017 7݄߸, ITPro -
AGENDA • ࣗݾհ • SREͱͷग़ձ͍ • ϝϧΧϦʹ͍ͭͯ • SREͱ •
ϝϧΧϦSREͷࣄྫͱ͜Ε͔Β -
SREͱͷग़ձ͍ • 2012/7 ༑ਓͱͷIRCͰͷձ͔Βڭ͑ͯΒ͏ • GoogleͷڊେͳΠϯϑϥͱαʔϏεͷՔಇɺ҆ఆੑΛ୲͢ΔνʔϜ͕SRE • https://research.googleblog.com/2012/07/site-reliability-engineers-solving-most.html ʮSite Reliability
Engineers: “solving the most interesting problems”ʯ͜ͷهࣄ͕ެ։͞Εͨࠒ • twitter ͷbioൃදεϥΠυʹʮSite ReliabilityʯΛՃͯ͠ҙࣝ • https://www.slideshare.net/kazeburo/yapc2102mysql/2 (2012/9) • 2015/11 ϝϧΧϦʹͯνʔϜ໊ͱͯ͠ఏҊ -
ϝϧΧϦ • ࠃ࠷େڃͷϑϦϚΞϓϦ • 3Ͱ؆୯ʹग़ 1) ࣸਅΛࡱΔ 2) ใΛهೖ 3)
ग़ϘλϯΛԡ͢ • ҆৺҆શͳܾࡁɾऔҾ • ΤεΫϩʔ(͓ۚͷΓͱΓ͕ࣾؒʹհࡏ) • ಗ໊ૹ -
ϝϧΧϦγεςϜ֓ཁ ©2011 Amazon Web Services LLC or its affiliates. All
rights reserved. Client Multimedia Corporate data center Traditional server Mobile Client IAM Add-on Example: IAM Add-on ence ) Assignment/ Task Requester Workers ग़! DB Search 5-දࣔ ݕࡧө ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific େྔͷϦΫΤετ ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corp data c Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ϦΫΤετԠ DB Search ߪೖ! ඵʙ30ඵ ඵʙ ը૾ ܾࡁ AI ߴʹฒߦͯ͠େྔͷτϥϯβΫγϣϯΛѻ͏ -
ΠϯϑϥετϥΫνϟ DNS: Amazon Route53 CDN: Akamai, Fastly, ImageFlux Storage: Amazon
S3 Analysis: Google BigQuery / Monitoring: Mackerel, DataDog JP UK US + + -
ϚΠΫϩαʔϏεج൫ API Gateway ©2011 Amazon Web Services LLC or its
affiliates. All rights reserved. Users Client Multimedia Corporate data center Traditional server Mobile Client Management onsole IAM Add-on Example: IAM Add-on man Intelligence Tasks (HIT) Assignment/ Task Requester Workers search backend service offer JP US • طଘAPI(ϞϊϦεAPI)ΛWrap͢Δ API Gateway Λ։ൃ͠ɺGCP(GKE)Ͱߏங • ϞϊϦεAPI֎Ͱͷ৽ػೳ։ൃ • αʔϏεΛஈ֊తʹϚΠΫϩαʔϏεͱ͠ ͯղ • ϞϊϦεAPIɾϚΠΫϩαʔϏε͔Βݺͼ ग़͞ΕΔBackendαʔϏεGKE্Ͱಈ࡞ ϞϊϦεAPI -
Google SRE • GoogleͷSREʹιϑτΣΞΤϯδχΞϦϯάʹՃ͑ɺγεςϜɾӡ༻ ͷೳྗ͕ٻΊΒΕΔ • ιϑτΣΞΤϯδχΞϦϯάʮࣗಈԽʯʹಛʹྗ • SREͷਓαʔϏεͷنʹൺྫͤ͞ͳ͍(Googleʹ͓͍ͯݱ࣮తʹͰ͖ͳ͍) •
ʮτΠϧ(ख࡞ۀͰߦΘΕɺࣗಈԽՄೳͰ܁Γฦ͢͜ͱʹՁΛ࣋ͨͳ͍)ʯͷ໓ -
Google SRE • ۀ࣌ؒͷ50%ιϑτΣΞΤϯδχΞϦϯάΛߦ͏ • ࣗಈԽ(ࣗԽ)ɺ৴པੑ্ʹ͋ͯΔ • 50%Λ͑Δ͜ͱ͕͋Εۀͷݟ͠ΛഭΒΕΔ • SLAɺΤϥʔόδΣοτ(༧ࢉ)ʹΑΔ։ൃऀͷརௐ
• ։ൃऀνʔϜͱՄ༻ੑͷඪ(SLA)ΛαʔϏε͝ͱʹઃఆɻߴ͗͢Δઃఆ͠ͳ͍ • ΤϥʔόδΣοτʹ͋Δͱ͖։ൃऀੵۃతͳϦϦʔεΛߦ͍ɺ༧ࢉΛ͑Δ ߹৴པੑճ෮ͷͨΊͷ։ൃʹઐ೦͢Δ͜ͱ͕ٻΊΒΕΔ -
ຊࠃͰͷSRE • 201511݄ ϝϧΧϦٕज़blogͰSREΛհ • RettyɺαΠϘζɺCookPadɺMixiɺͯͳͳͲWebܥاۀΛத৺ʹSRE ͷ࠾༻͕ਐΉ • SRE Tech
Talk։࠵ • ୈҰճ: 20166݄ɻୈೋճ: 20171݄ • 100໊Ҏ্ͷࢀՃऀΛूΊΔ -
ϝϧΧϦSRE • 2018/2 ࣌Ͱϝϯόʔ10໊ • ϚΠΫϩαʔϏεج൫ߏஙɺSys-ML(MLOps)ʹܞΘΔΤϯδχΞ • େنͳWebαʔϏεͷӡ༻ܦݧ͕͋Δத్͕ଟ͍͕ɺ৽ଔϝϯόʔࡏ੶ • ݸʑͷϝϯόʔ͕ೳಈతʹΛൃݟ͠ɺղܾ͍ͯ͘͠
• SlackɾGitHubͰͷٞɺJiraͰͷνέοτཧΛ௨ͯ͠ใڞ༗ -
ϝϧΧϦ SRE ͷۀൣғ Operations Software Eng. ج൫ߏங OnCall (োରԠ) ґཔରԠ
εέʔϥϏϦςΟɾՄ༻ੑվળ ࣗಈԽɺDBAɺϛυϧΣΞߏங ΞϓϦέʔγϣϯͷઃܭϨϏϡʔ ϩάऩूɾੳج൫ͷߏஙɺӡ༻ αʔόϓϩϏδϣχϯάɾσϓϩΠɺϚΠΫϩαʔϏεɾ.-ج൫ͷඋ ηΩϡϦςΟʗෆਖ਼ར༻ݕग़ γεςϜӡ༻ΛʮΈʯͱͯ͠ ࡞Γ্͛Δ͜ͱ͕ٻΊΒΕ͍ͯΔ -
Worker/QueueγεςϜͷ • Jobͷॲཧ༷ʑͳཁҼͰมԽ͢Δ • Batch͔Βͷenqueue • Workerͷϓϩηε • ॲཧ༰ •
ॲཧ͕͗͢Δ͜ͱͰγεςϜʹෛՙ App App Queue RDBMS Worker Worker Worker Worker -
CRMπʔϧͷࣄྫ • ৴ͷ৴ϝσΟΞͷબʹΑܾͬͯ·Δ •RDBMSͷॻ͖ࠐΈ(ΞϓϦPM) •Mail •RDBMSͷॻ͖ࠐΈ(ΞϓϦ௨) •Push৴ • ॲཧ͕ҰఆͰͳ͍ɾ৴ຖʹมԽ •
৴ʹ͔͔Δ࣌ؒΛ͘͢ΔͨΊWorkerͷΛखಈͰௐ • ௐ࿙ΕʹΑͬͯఆ֎ͷෛՙɾো CRM Queue RDBMS Worker Worker Worker Worker Mail Push ߴ -
CruiseControl • ʮΛ੍ޚ͢ΔαʔϏεʯΛߏங • Worker͕ॲཧΛ։࢝͢ΔલʹCruiseControlʹ ͍߹ΘͤΔ • ॲཧ͕͍߹wait͕ೖΔ • Worker͕ेʹ͋Εॲཧ͕Ұఆʹ
• CRMπʔϧͷߴͳ৴ͱαʔϏεͷ҆ఆՔಇΛ࣮ݱ CRM App Queue Worker Worker Worker Cruise Control RDBMS Mail Push -
CruiseControl = NGINX • ngx_http_limit_req_module Λར༻ • pathͱheaderʹΑͬͯΛ੍ޚ limit_req_zone $http_x_limit_req
zone=r10:50m rate=10r/s; limit_req_zone $http_x_limit_req zone=r50:50m rate=50r/s; limit_req_zone $http_x_limit_req zone=r100:50m rate=100r/s; server { listen 8080; root /path/to/root; location /r10 { limit_req zone=r10 burst=4294967296; } location /r50 { limit_req zone=r50 burst=4294967296; } location /r100 { limit_req zone=r100 burst=4294967296; } } % curl -H 'X-Limit-Req: push-msg' cruisecontrol:8080/r100