SRE大全 メルカリ編 後半 hbstudy#75
SREେશ: ϝϧΧϦฤ [ޙ]2017/08/17 hbstudy#75Masahiro Nagano @kazeburo
View Slide
AGENDA• ޙ• PHP ΞϓϦέʔγϣϯͷ࠷దԽࣄྫ• ηΩϡϦςΟͷऔΓΈ(ύεϫʔυϦετ߈ܸࣄྫ)• ϝϧΧϦSREͷࠓޙ• SREͷׂͷݱࡏͱະདྷɺϚΠΫϩαʔϏε
PHP ΞϓϦέʔγϣϯͷ࠷దԽࣄྫ
PHP7.1Խ͠·ͨ͠
CPU Usage1)1Y 1)1YCPU༻!!
Response time(95percentile)‒PHP 7.1.x ‒PHP 5.6.xResponse Time 20-30%վળ!!!
PHP 7.1Խ• 5.6ܥ͔ΒͷΞοϓάϨʔυ• CIͰͷςετɺΫϥΠΞϯτͷࣗಈςετɺQAʹΑΔखಈςετΛܦͯશҠߦ• ͦͷଞҠߦͷޮՌ• CIʹ͔͔Δ࣌ؒॖ
ΊͰͨ͠ΊͰͨ͠ Ͱͳͯ͘..
PHP7.1ԽҎ֎ͷPHP ΞϓϦέʔγϣϯͷ࠷దԽࣄྫ
JSONಡΈࠐΈ• ػೳͷ֦ॆʹΑΓɺࠃࡍԽରԠͷmessage֨ೲϑΝΠϧ͕ेKBʹΒΉ• messageϑΝΠϧ΄ͱΜͲͷϦΫΤετͰಡ·ΕΔͨΊෛՙ͕૿େ
JSONಡΈࠐΈ• 1ͷαʔόͰར༻͞Εͨmessage keyΛϩάʹهɺΑ͘ར༻͞ΕΔmessageΛಛఆ• grep | sort | uniq -c | sort -n• Α͘ར༻͞ΕΔmessage͚ͩΛूΊͨ essential.json Λ࡞• ϕϯνϚʔΫ All: 9msec => Essential: 1msec ҎԼ
JSONಡΈࠐΈ: ߴԽmsecͷվળ!!
େྔͷྫ֎Ϋϥε• app_exception.php ͱ͍͏ϑΝΠϧʹ330ݸͷྫ֎Ϋϥε͕ఆٛ• ػೳ͕૿͑Εྫ֎Ϋϥε૿͑Δ• ຖϦΫΤετͰಡΈࠐ·ΕΔ
େྔͷྫ֎Ϋϥε• αʔό্Ͱى͖͍ͯΔ࣮ࡍʹى͖͍ͯΔྫ֎Λूܭ• ൃੜ݅ͷଟ͍ͷΛ app_exception_base.php ʹΓग़͢• ϕϯνϚʔΫ 10msec => 1msec• try-catchͰϩάʹͰͳ͍ͷ͋ͬͨ
େྔͷྫ֎Ϋϥε: ରࡦޙmsecͷվળ!!!
PHP(Opcache) ಈ࡞ΠϝʔδmemoryDestroy!!startendDBAPIshared memphpphpphpphpphpphpphpphpphpResponseRequestOPCODECopy࣮ߦ࣮ߦ࣮ߦίϯύΠϧ݁ՌͰͳ͘ɺOPCODEΛcache͓ͯ͠Γɺhashclassߏங = ϦΫΤετΛ͑ͯcache͢Δͷ͍͠
PHPͷಈ࡞֎෦ͱͷ௨৴PHP෦ͷॲཧhashclassͷߏஙͳͲΛؚΉPHPͷಈ࡞͕͍࣌ؒ֎෦ͷ͍߹Θͤͱ͋Θͤͯνϡʔχϯά͢ΔϙΠϯτ
ϘτϧωοΫΛ୳͢ํ๏• NewRelic• σʔλϕʔεɺmemcachedɺ֎෦APIͷݺͼग़͠ճͱ࣌ؒɻN+1ɺpreload࿙Εͷൃݟ• PHP෦ͷτϨʔεͰ͖Δ• strace• System Call ͷτϨʔε• PHP෦Ͱى͖͍ͯΔ͜ͱτϨʔεͰ͖ͳ͍• System CallͱιʔείʔυΛݟൺͯϘτϧωοΫΛ୳͢ => Ͳ͏ͬͯʁ
strace$ sudo strace -tt -s 256 -p $(pgrep -n -f httpd) |& tee strace.txt15:29:31.122566 read(20, “GET /...)15:29:31.122844 stat(...15:29:31.123914 getcwd("/", 4095) = 215:29:31.123936 chdir(“/var/www/vhosts/app/webroot") = 015:29:31.123966 setitimer(ITIMER_PROF, {it_interval={0, 0}, it_value={60, 0}}, NULL) = 015:29:31.123987 fcntl(18, F_SETLK, {type=F_RDLCK, whence=SEEK_SET, start=1, len=1}) = 015:29:31.125310 open(“/var/www/vhosts/app/foo/bar.json”) = 0## Ctrl-C$ less strace.txtPHP ෦ͷಈ࡞traceͰ͖ͳ͍ιʔείʔυͱরΒ͠߹ΘͤΔʹώϯτͱͳΔsystem call͕΄͍͠ => OpcacheͷઃఆΛΈΔ
php.ini in Productionopcache.enable = 1opcache.revalidate_freq = anyopcache.validate_timestamps = 0࠷େੑೳopcache.enable = 1opcache.revalidate_freq = 1opcache.validate_timestamps = 1ϝϧΧϦӡ༻தϑΝΠϧͷߋ৽νΣοΫ͠ͳ͍ϑΝΠϧͷߋ৽νΣοΫ͢Δ1ඵ͝ͱʹνΣοΫnginx dynamic upstreamΛ͍ɺbalancer͔Β֎ͯ͠ɺrsync => 1ඵsleep ͯ͠balancerʹ͢
php.ini for straceopcache.enable = 1opcache.revalidate_freq = 0opcache.validate_timestamps = 1TUSBDF࣮ߦ༻ຖϦΫΤετͰνΣοΫ͢Δ18:21:01.253488 stat("/var/www/current/app/webroot/index.php", {st_mode=S_IFREG|0644, st_size=356, ...}) = 018:21:01.253572 stat("/var/www/current/dietcake/dietcake.php", {st_mode=S_IFREG|0644, st_size=1013, ...}) = 018:21:01.253697 stat("/var/www/current/dietcake/core/exception.php", {st_mode=S_IFREG|0644, st_size=46, ...}) = 018:21:01.253775 stat("/var/www/current/dietcake/core/inflector.php", {st_mode=S_IFREG|0644, st_size=289, ...}) = 0....18:21:01.254005 stat("/var/www/current/dietcake/core/dispatcher.php", {st_mode=S_IFREG|0644, st_size=1300, ...}) = 018:21:01.254044 stat(“/var/www/current/app/aaaaaaa.php", {st_mode=S_IFREG|0644, st_size=5457, ...}) = 018:21:01.254135 stat("/var/www/current/app/app_xxx.php", {st_mode=S_IFREG|0644, st_size=10423, ...}) = 018:21:01.254618 stat("/var/www/current/app/app_exception.php", {st_mode=S_IFREG|0644, st_size=45680, ...}) = 018:21:01.257390 stat("/var/www/current/app/app_yyy.php", {st_mode=S_IFREG|0644, st_size=1133, ...}) = 0
App࠷దԽͷϞνϕʔγϣϯͱλΠϛϯά• Appαʔό͕૿͑Δͱ• ނো͕૿͑ɺσϓϩΠ࣌ؒ৳ͼΔ• AppαʔόΛ૿ͯ͘͠ͳΔͷ͛Δ͕ɺϨεϙϯε͕มΘΒͳ͍• ϝϧΧϦͰAppαʔόͷCPU༻͕ఆظతʹ50%Λ͑࢝ΊͨΒରࡦΛߦ͏• αʔόͷՃ or ΞϓϦέʔγϣϯͷίʔυͷ࠷దԽ• قઅతཁҼͰΞΫηε͕૿͑ΔͷͰɺ࣌ؒͷ͋Δͱ͖ʹ४උɾ࠷దԽͷωλΛ͓ͬͯ͘• αʔόαΠυΤϯδχΞͱऔΓΉ͜ͱ͕Ͱ͖Εͳ͓ྑ͍• SREݚमͰऔΓΉͳͲ
ηΩϡϦςΟͷऔΓΈύεϫʔυϦετ߈ܸͷࣄྫ
ύεϫʔυϦετ߈ܸ• ͓٬͞·ͷΞΧϯτʹରͯ͠ɺϥϯμϜͳύεϫʔυ͘͠ผͰ࿙Ӯͨ͠ύεϫʔυจࣈྻΛ͍ϩάΠϯΛࢼߦ• ͞·͟·ͳنͷ߈ܸ͕ߦΘΕ͍ͯΔ• ·ͣؾ͘͜ͱ͕Ͱ͖ΔΈͷߏங͕ॏཁ
ύεϫʔυϦετ߈ܸͷݕ• ϩάΠϯࣦഊΛAPIͷϩάͱͯ͢͠• ΞΧϯτͳ͠/ύεϫʔυҧ͍• ϩάΛnorikraͰूܭɺmackerelͰՄࢹԽ ͱࢹ• ϝϧΧϦ͕TVʹऔΓ্͛ΒΕΔͱΞϥʔτ͕དྷΔ͜ͱ
ύεϫʔυϦετ߈ܸ͔Βͷޚ• ൺֱత୯७ͳ߈ܸΞϓϦέʔγϣϯͰࣗಈతʹޚ• ಉҰͷϝʔϧΞυϨεʹΑΔϩάΠϯࢼߦ• ಉҰͷIPΞυϨεʹΑΔϩάΠϯࢼߦ• ߈ܸͱஅ͞Εͨ߹֘IPɺ֘ΞΧϯτΛҰఆظؒڋ൱• ͓٬͞·ʹύεϫʔυͷϦηοτଅ͢
େنͳύεϫʔυϦετ߈ܸ• ੈքத͔Β߈ܸ͕ߦΘΕΔ• 2016ʹ࣮ࡍʹى͖ͨ߈ܸͷΞΫηεݩͷࠃ• 1IPʹ͖ͭɺճͷϩάΠϯࢼߦͰ͋ΓɺࣗಈͰ͛ͳ͍ͦͷଞ18%Armenia2%Azerbaijan2%Bahrain2%Georgia2% Japan2%Russian2% Indonesia3%Nepal3%Pakistan5%Thailand5%Taiwan6%Viet Nam6%Brazil10%India30%
େنͳύεϫʔυϦετ߈ܸͷඋ͑• ߈ܸ͞Ε͍͢Web൛ͰCAPTCHAΛಋೖ• Client Reputation ͷར༻• GeoIP• Anonymous proxy detection• IP Reputation
Client reputationhttp://www.cyren.com/security-center/ip-reputation-checkhttps://www.ip2location.com/demo
߈ܸݩIP Reputation-PX.JEEMF)JHI4ׂ͙͜ͱ͕Ͱ͖ͨՄೳੑ͕͋Δ*༷ʑͳใϦιʔεΛΈ߹ΘͤͯαʔϏεͷ҆શੑΛߴΊ͍ͯ·͢* ߈ܸͷ͋ͬͨλΠϛϯάͷreputationͰͳ͍ͷͰଟগζϨ͕͋Γ·͢
ϝϧΧϦSREͷࠓޙSREͷׂͷݱࡏͱະདྷɺMicroservice
Mercari SRE (࠶ܝ)• ͍ͭͰշద͔ͭ҆શʹར༻Ͱ͖Δʮ৴པੑͷߴ͍ʯαʔϏεͷ࣮ݱ• ʮ৽نαʔϏεͷ։ൃҎ֎ͷΤϯδχΞϦϯάશ෦Δʯ• ʮ։ൃʯͱʮӡ༻ʯͷ• Productionڥͷ࡞ۀશൠΛSRE͕ड͚࣋ͭ
Mercari SREͷҐஔ͚ͮ։ൃνʔϜSREBOTInfrastructureσʔλϕʔε࡞ۀ/ௐࠪґཔεΩʔϚ/ΞʔΩςΫνϟ૬ஊߏஙʗӡ༻σʔλϕʔε࡞ۀௐࠪߏஙɾར༻σϓϩΠLogಗ໊DBར༻ར༻ө/ࢀরௐࠪͷڠྗɺमਖ਼ґཔ
SREͷ՝• ૿͑ΔιϑτΣΞΤϯδχΞɺαʔϏε/ػೳʹରͯ͠SREͷਓͷ૿Ճ͕͍͔ͭͳ͍• ProductionڥͱιϑτΣΞΤϯδχΞͷҙࣝͷဃ
SREͷ՝લ લ ݱࡏ4& 43&
MicroserviceԽ• ։ൃऀʹ Ownership• ։ൃ৫ͷεέʔϥϏϦςΟͱॊೈੑΛߴΊɺαʔϏεΛΑΓ͍͛ͯ͘• ʮϝϧΧϦνϟϯωϧʯͳͲͷ৽͍͠ػೳ• AtteΧϧͳͲͷ࿈ܞΞϓϦ• Ҭ͝ͱͷಠࣗαʔϏε => US/UKͰਐΜͰ͍Δ
Microservice in US©2011 Amazon Web Services LLC or its affiliates. All rights reserved.User Users Client Multimedia Corporatedata centerTraditionalserverMobile ClientInternet AWS ManagementConsoleIAM Add-on Example:IAM Add-onTasks (HIT) TaskMechanical TurkNon-Service Specificbackend for frontendsGKECore/طଘAPIprotobufJSON over HTTPsGRPCService A Service BSpannerGKE GKEDeployج൫SaaSPubSub
MicroserviceԽʹΉ͚ͯ• Requirements(ཁ݅)ͷࡦఆ• Microserviceج൫ͷߏங• Deploy, Log, Database, Monitoring...• ຊ൪ͷΞʔΩςΫνϟͱͯ͠ͷ࠾༻• લਐͭͭ͠՝Λݟ͚ͭɺͻͱͭͻͱͭղܾ͍ͯ͘͠
ࠓޙͷMercari SRE։ൃνʔϜSREBOTInfrastructureσʔλϕʔε࡞ۀ/ௐࠪґཔεΩʔϚ/ΞʔΩςΫνϟ૬ஊߏஙʗӡ༻σʔλϕʔε࡞ۀௐࠪߏஙɾར༻σϓϩΠLogಗ໊DBར༻ར༻ө/ࢀরௐࠪͷڠྗमਖ਼ґཔMicroserviceج൫MircoserviceInfrastructureσϓϩΠߏஙࣗಈԽ͞Εͨӡ༻
ࠓޙͷMercari SRE։ൃνʔϜSREBOTInfrastructureσʔλϕʔε࡞ۀ/ௐࠪґཔεΩʔϚ/ΞʔΩςΫνϟ૬ஊߏஙʗӡ༻σʔλϕʔε࡞ۀௐࠪߏஙɾར༻σϓϩΠLogಗ໊DBར༻ར༻ө/ࢀরௐࠪͷڠྗमਖ਼ґཔMircoserviceInfrastructureσϓϩΠߏஙࣗಈԽ͞Εͨӡ༻Microserviceج൫ϚΠΫϩαʔϏεԽ͕ਐΉ͜ͱͰ͚ͩ͜͜ʹͳ͍ͬͯ͘ͷ͔ʁ
൱ͱߟ͍͑ͯ·͢
ࠓޙͷMercari SRE։ൃνʔϜSREBOTInfrastructureσʔλϕʔε࡞ۀ/ௐࠪґཔεΩʔϚ/ΞʔΩςΫνϟ૬ஊߏஙʗӡ༻σʔλϕʔε࡞ۀௐࠪߏஙɾར༻σϓϩΠLogಗ໊DBར༻ར༻ө/ࢀরௐࠪͷڠྗमਖ਼ґཔMicroserviceج൫/RequirementMircoserviceInfrastructureσϓϩΠߏஙࣗಈԽ͞Εͨӡ༻ੵۃతʹؔΘ͍ͬͯ͘
ࠓޙͷMercari SRE• Microserviceͷج൫ߏஙɾӡ༻• SREͷϊϋΛͬͯαʔϏεʹੵۃతؔ༩• εέʔϥϏϦςΟɺՄ༻ੑͷվળ• ࣗಈԽͷਪਐɺΦϖϨʔγϣϯͷվળ• ηΩϡϦςΟͷ࣮ࢪ• Globalͳnetwork/infrastructureͷޮӡ༻
SRE More!!!https://twitter.com/kazeburo/status/890131903529054210
Mercari SRE More!!!Microservice, Automation, Performance, ScalabilityDatabase, Network, Distributed System, OS,Cloud, Hardware, SecuritySWE + ༷ʑͳࣝɾܦݧΛੜ͔ͯ͠ʮ৽ͨͳՁΛੜΈग़͢ੈքతϚʔέοτΛΔʯ৴པੑΛ࣋ͬͯࢧ͑Δશํ໘ͰΑΖ͓͘͠ئ͍͠·͢!!
Ҏ্
kazeburo
grimoh
dach
.png){kind=icon}
pharma_x_tech
line_developers
nagiss
antimon2
yamahiro
kawajiritakayuki
culvert
moriyuya
k_koheyi
samanthasiow
holman
eileencodes
orderedlist
afnizarnur
palkan
jacobian
62gerente
mthomps
geeforr
chriscoyier
reverentgeek