SREେશ: ϝϧΧϦฤ [ޙ൒] 2017/08/17 hbstudy#75 Masahiro Nagano @kazeburo

AGENDA • ޙ൒ • PHP ΞϓϦέʔγϣϯͷ࠷దԽࣄྫ • ηΩϡϦςΟͷऔΓ૊Έ(ύεϫʔυϦετ߈ܸࣄྫ) • ϝϧΧϦSREͷࠓޙ • SREͷ໾ׂͷݱࡏͱະདྷɺϚΠΫϩαʔϏε

PHP ΞϓϦέʔγϣϯͷ࠷దԽࣄྫ

PHP7.1Խ͠·ͨ͠

CPU Usage 1)1Y 1)1Y CPU࢖༻཰൒෼!!

Response time(95percentile) ‒PHP 7.1.x ‒PHP 5.6.x Response Time 20-30%վળ!!!

PHP 7.1Խ • 5.6ܥ͔ΒͷΞοϓάϨʔυ • CIͰͷςετɺΫϥΠΞϯτͷࣗಈςετɺQAʹΑΔखಈςετΛܦͯ׬શҠ ߦ • ͦͷଞҠߦͷޮՌ • CIʹ͔͔Δ࣌ؒ΋୹ॖ

ΊͰͨ͠ΊͰͨ͠
 Ͱ͸ͳͯ͘..

PHP7.1ԽҎ֎ͷ PHP ΞϓϦέʔγϣϯͷ࠷దԽࣄྫ

JSONಡΈࠐΈ • ػೳͷ֦ॆʹΑΓɺࠃࡍԽରԠͷmessage֨ೲϑΝΠϧ͕਺ेKBʹ๲ΒΉ • messageϑΝΠϧ͸΄ͱΜͲͷϦΫΤετͰಡ·ΕΔͨΊෛՙ͕૿େ

JSONಡΈࠐΈ • 1୆ͷαʔόͰར༻͞Εͨmessage keyΛϩάʹه࿥ɺΑ͘ར༻͞ΕΔ messageΛಛఆ • grep | sort | uniq -c | sort -n • Α͘ར༻͞ΕΔmessage͚ͩΛूΊͨ essential.json Λ࡞੒ • ϕϯνϚʔΫ All: 9msec => Essential: 1msec ҎԼ

JSONಡΈࠐΈ: ߴ଎Խ ਺msecͷվળ!!

େྔͷྫ֎Ϋϥε • app_exception.php ͱ͍͏ϑΝΠϧʹ330ݸͷྫ֎Ϋϥε͕ఆٛ • ػೳ͕૿͑Ε͹ྫ֎Ϋϥε΋૿͑Δ • ຖϦΫΤετͰಡΈࠐ·ΕΔ

େྔͷྫ֎Ϋϥε • αʔό্Ͱى͖͍ͯΔ࣮ࡍʹى͖͍ͯΔྫ֎Λूܭ • ൃੜ݅਺ͷଟ͍΋ͷΛ app_exception_base.php ʹ੾Γग़͢ • ϕϯνϚʔΫ 10msec => 1msec • try-catchͰϩάʹͰͳ͍΋ͷ΋͋ͬͨ

େྔͷྫ֎Ϋϥε: ରࡦޙ ਺msecͷվળ!!!

PHP(Opcache) ಈ࡞Πϝʔδ memory Destroy!! start end DB API shared mem php php php php php php php php php Response Request OPCODE Copy ࣮ߦ ࣮ߦ ࣮ߦ ίϯύΠϧ݁ՌͰ͸ͳ͘ɺOPCODEΛcache͓ͯ͠Γɺhash΍class͸౎౓ߏங = ௿଎ ϦΫΤετΛ௒͑ͯcache͢Δͷ͸೉͍͠

PHPͷಈ࡞ ֎෦ͱͷ௨৴ PHP಺෦ͷॲཧ hash΍classͷߏஙͳͲΛؚΉPHPͷಈ࡞͕࣌ؒ௕͍ ֎෦΁ͷ໰͍߹Θͤͱ͋Θͤͯνϡʔχϯά͢ΔϙΠϯτ

ϘτϧωοΫΛ୳͢ํ๏ • NewRelic • σʔλϕʔεɺmemcachedɺ֎෦APIͷݺͼग़͠ճ਺ͱ࣌ؒɻN+1΍ɺpreload࿙Εͷൃݟ • PHP಺෦ͷτϨʔε΋Ͱ͖Δ • strace • System Call ͷτϨʔε • PHP಺෦Ͱى͖͍ͯΔ͜ͱ͸τϨʔεͰ͖ͳ͍ • System CallͱιʔείʔυΛݟൺ΂ͯϘτϧωοΫΛ୳͢ => Ͳ͏΍ͬͯʁ

strace $ sudo strace -tt -s 256 -p $(pgrep -n -f httpd) |& tee strace.txt 15:29:31.122566 read(20, “GET /...) 15:29:31.122844 stat( ... 15:29:31.123914 getcwd("/", 4095) = 2 15:29:31.123936 chdir(“/var/www/vhosts/app/webroot") = 0 15:29:31.123966 setitimer(ITIMER_PROF, {it_interval={0, 0}, it_value={60, 0}}, NULL) = 0 15:29:31.123987 fcntl(18, F_SETLK, {type=F_RDLCK, whence=SEEK_SET, start=1, len=1}) = 0 15:29:31.125310 open(“/var/www/vhosts/app/foo/bar.json”) = 0 ## Ctrl-C $ less strace.txt PHP ಺෦ͷಈ࡞͸traceͰ͖ͳ͍ ιʔείʔυͱরΒ͠߹ΘͤΔʹ͸ώϯτͱͳΔsystem call͕΄͍͠ => OpcacheͷઃఆΛΈΔ

php.ini in Production opcache.enable = 1 opcache.revalidate_freq = any opcache.validate_timestamps = 0 ࠷େੑೳ opcache.enable = 1 opcache.revalidate_freq = 1 opcache.validate_timestamps = 1 ϝϧΧϦӡ༻த ϑΝΠϧͷߋ৽͸νΣοΫ͠ͳ͍ ϑΝΠϧͷߋ৽͸νΣοΫ͢Δ 1ඵ͝ͱʹνΣοΫ nginx dynamic upstreamΛ࢖͍ɺbalancer͔Β֎ͯ͠ɺ rsync => 1ඵsleep ͯ͠balancerʹ໭͢

php.ini for strace opcache.enable = 1 opcache.revalidate_freq = 0 opcache.validate_timestamps = 1 TUSBDF࣮ߦ༻ ຖϦΫΤετͰνΣοΫ͢Δ 18:21:01.253488 stat("/var/www/current/app/webroot/index.php", {st_mode=S_IFREG|0644, st_size=356, ...}) = 0 18:21:01.253572 stat("/var/www/current/dietcake/dietcake.php", {st_mode=S_IFREG|0644, st_size=1013, ...}) = 0 18:21:01.253697 stat("/var/www/current/dietcake/core/exception.php", {st_mode=S_IFREG|0644, st_size=46, ...}) = 0 18:21:01.253775 stat("/var/www/current/dietcake/core/inflector.php", {st_mode=S_IFREG|0644, st_size=289, ...}) = 0 .... 18:21:01.254005 stat("/var/www/current/dietcake/core/dispatcher.php", {st_mode=S_IFREG|0644, st_size=1300, ...}) = 0 18:21:01.254044 stat(“/var/www/current/app/aaaaaaa.php", {st_mode=S_IFREG|0644, st_size=5457, ...}) = 0 18:21:01.254135 stat("/var/www/current/app/app_xxx.php", {st_mode=S_IFREG|0644, st_size=10423, ...}) = 0 18:21:01.254618 stat("/var/www/current/app/app_exception.php", {st_mode=S_IFREG|0644, st_size=45680, ...}) = 0 18:21:01.257390 stat("/var/www/current/app/app_yyy.php", {st_mode=S_IFREG|0644, st_size=1133, ...}) = 0

App࠷దԽͷϞνϕʔγϣϯͱλΠϛϯά • Appαʔό͕૿͑Δͱ • ނো཰͕૿͑ɺσϓϩΠ࣌ؒ΋৳ͼΔ • AppαʔόΛ૿΍ͯ͠΋஗͘ͳΔͷ͸๷͛Δ͕ɺϨεϙϯε଎౓͕มΘΒͳ͍ • ϝϧΧϦͰ͸AppαʔόͷCPU࢖༻཰͕ఆظతʹ50%Λ௒͑࢝ΊͨΒରࡦΛߦ͏ • αʔόͷ௥Ճ or ΞϓϦέʔγϣϯͷίʔυͷ࠷దԽ • قઅతཁҼͰΞΫηε͕૿͑ΔͷͰɺ࣌ؒͷ͋Δͱ͖ʹ४උɾ࠷దԽͷωλΛ΋͓ͬͯ͘ • αʔόαΠυΤϯδχΞͱऔΓ૊Ή͜ͱ͕Ͱ͖Ε͹ͳ͓ྑ͍ • SREݚमͰऔΓ૊ΉͳͲ

No content

ηΩϡϦςΟͷऔΓ૊Έ ύεϫʔυϦετ߈ܸͷࣄྫ

ύεϫʔυϦετ߈ܸ • ͓٬͞·ͷΞΧ΢ϯτʹରͯ͠ɺϥϯμϜͳύεϫʔυ΋͘͠͸ผͰ࿙Ӯ ͨ͠ύεϫʔυจࣈྻΛ࢖͍ϩάΠϯΛࢼߦ • ͞·͟·ͳن໛ͷ߈ܸ͕ߦΘΕ͍ͯΔ • ·ͣ͸ؾ෇͘͜ͱ͕Ͱ͖Δ࢓૊Έͷߏங͕ॏཁ

ύεϫʔυϦετ߈ܸͷݕ஌ • ϩάΠϯࣦഊΛAPIͷϩάͱͯ͠࢒͢ • ΞΧ΢ϯτͳ͠/ύεϫʔυҧ͍ • ϩάΛnorikraͰूܭɺmackerelͰՄࢹԽ
 ͱ؂ࢹ • ϝϧΧϦ͕TVʹऔΓ্͛ΒΕΔͱΞϥʔτ͕དྷΔ͜ͱ΋

ύεϫʔυϦετ߈ܸ͔Βͷ๷ޚ • ൺֱత୯७ͳ߈ܸ͸ΞϓϦέʔγϣϯ಺Ͱࣗಈతʹ๷ޚ • ಉҰͷϝʔϧΞυϨεʹΑΔϩάΠϯࢼߦ • ಉҰͷIPΞυϨεʹΑΔϩάΠϯࢼߦ • ߈ܸͱ൑அ͞Εͨ৔߹͸֘౰IPɺ֘౰ΞΧ΢ϯτΛҰఆظؒڋ൱ • ͓٬͞·ʹύεϫʔυͷϦηοτଅ͢

େن໛ͳύεϫʔυϦετ߈ܸ • ੈքத͔Β߈ܸ͕ߦΘΕΔ • 2016೥ʹ࣮ࡍʹى͖ͨ߈ܸͷΞΫη εݩͷࠃ • 1IPʹ͖ͭɺ਺ճͷϩάΠϯࢼߦͰ ͋ΓɺࣗಈͰ͸๷͛ͳ͍ ͦͷଞ 18% Armenia 2% Azerbaijan 2% Bahrain 2% Georgia 2% Japan 2% Russian 2% Indonesia 3% Nepal 3% Pakistan 5% Thailand 5% Taiwan 6% Viet Nam 6% Brazil 10% India 30%

େن໛ͳύεϫʔυϦετ߈ܸ΁ͷඋ͑ • ߈ܸ͞Ε΍͍͢Web൛Ͱ͸CAPTCHAΛಋೖ • Client Reputation ͷར༻ • GeoIP • Anonymous proxy detection • IP Reputation

Client reputation http://www.cyren.com/security-center/ip-reputation-check https://www.ip2location.com/demo

߈ܸݩIP Reputation -PX .JEEMF )JHI 4ׂ͸๷͙͜ͱ͕Ͱ͖ͨՄೳੑ͕͋Δ* ༷ʑͳ৘ใϦιʔεΛ૊Έ߹ΘͤͯαʔϏεͷ҆શੑΛߴΊ͍ͯ·͢ * ߈ܸͷ͋ͬͨλΠϛϯάͷreputationͰ͸ͳ͍ͷͰଟগζϨ͕͋Γ·͢

ϝϧΧϦSREͷࠓޙ SREͷ໾ׂͷݱࡏͱະདྷɺMicroservice

Mercari SRE (࠶ܝ) • ͍ͭͰ΋շద͔ͭ҆શʹར༻Ͱ͖Δʮ৴པੑͷߴ͍ʯαʔϏεͷ࣮ݱ • ʮ৽نαʔϏεͷ։ൃҎ֎ͷΤϯδχΞϦϯά͸શ෦΍Δʯ • ʮ։ൃʯͱʮӡ༻ʯͷ෼཭ • Production؀ڥͷ࡞ۀશൠΛSRE͕ड͚࣋ͭ

Mercari SREͷҐஔ͚ͮ ։ൃνʔϜ SRE BOT Infrastructure σʔλϕʔε࡞ۀ/ௐࠪґཔ εΩʔϚ/ΞʔΩςΫνϟ૬ஊ ߏஙʗӡ༻ σʔλϕʔε࡞ۀ ௐࠪ ߏஙɾར༻ σϓϩΠ Log ಗ໊DB ར༻ ར༻ ൓ө/ࢀর ໰୊ௐࠪͷڠྗɺमਖ਼ґཔ

SREͷ՝୊ • ૿͑Διϑτ΢ΣΞΤϯδχΞɺαʔϏε/ػೳʹରͯ͠SREͷਓ਺ͷ૿Ճ ͕௥͍͔ͭͳ͍ • Production؀ڥͱιϑτ΢ΣΞΤϯδχΞͷҙࣝͷဃ཭

SREͷ՝୊ ೥લ ൒೥લ ݱࡏ 4& 43&

MicroserviceԽ • ։ൃऀʹ Ownership • ։ൃ૊৫ͷεέʔϥϏϦςΟͱॊೈੑΛߴΊɺαʔϏεΛΑΓ޿͍͛ͯ͘ • ʮϝϧΧϦνϟϯωϧʯͳͲͷ৽͍͠ػೳ • Atte΍Χ΢ϧͳͲͷ࿈ܞΞϓϦ • ஍Ҭ͝ͱͷಠࣗαʔϏε => US/UKͰ͸ਐΜͰ͍Δ

Microservice in US ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Tasks (HIT) Task Mechanical Turk Non-Service Specific backend for frontends GKE Core/طଘAPI protobuf JSON over HTTPs GRPC Service A Service B Spanner GKE GKE Deployج൫ SaaS PubSub

MicroserviceԽʹΉ͚ͯ • Requirements(ཁ݅)ͷࡦఆ • Microserviceج൫ͷߏங • Deploy, Log, Database, Monitoring... • ຊ൪ͷΞʔΩςΫνϟͱͯ͠ͷ࠾༻ • લਐͭͭ͠՝୊Λݟ͚ͭɺͻͱͭͻͱͭղܾ͍ͯ͘͠

ࠓޙͷMercari SRE ։ൃνʔϜ SRE BOT Infrastructure σʔλϕʔε࡞ۀ/ௐࠪґཔ εΩʔϚ/ΞʔΩςΫνϟ૬ஊ ߏஙʗӡ༻ σʔλϕʔε࡞ۀ ௐࠪ ߏஙɾར༻ σϓϩΠ Log ಗ໊DB ར༻ ར༻ ൓ө/ࢀর ໰୊ௐࠪͷڠྗ मਖ਼ґཔ Microserviceج൫ Mircoservice Infrastructure σϓϩΠ ߏங ࣗಈԽ͞Εͨӡ༻

ࠓޙͷMercari SRE ։ൃνʔϜ SRE BOT Infrastructure σʔλϕʔε࡞ۀ/ௐࠪґཔ εΩʔϚ/ΞʔΩςΫνϟ૬ஊ ߏஙʗӡ༻ σʔλϕʔε࡞ۀ ௐࠪ ߏஙɾར༻ σϓϩΠ Log ಗ໊DB ར༻ ར༻ ൓ө/ࢀর ໰୊ௐࠪͷڠྗ मਖ਼ґཔ Mircoservice Infrastructure σϓϩΠ ߏங ࣗಈԽ͞Εͨӡ༻ Microserviceج൫ ϚΠΫϩαʔϏεԽ͕ਐΉ͜ͱͰ ͚ͩ͜͜ʹͳ͍ͬͯ͘ͷ͔ʁ

൱ ͱߟ͍͑ͯ·͢

ࠓޙͷMercari SRE ։ൃνʔϜ SRE BOT Infrastructure σʔλϕʔε࡞ۀ/ௐࠪґཔ εΩʔϚ/ΞʔΩςΫνϟ૬ஊ ߏஙʗӡ༻ σʔλϕʔε࡞ۀ ௐࠪ ߏஙɾར༻ σϓϩΠ Log ಗ໊DB ར༻ ར༻ ൓ө/ࢀর ໰୊ௐࠪͷڠྗ मਖ਼ґཔ Microserviceج൫/Requirement Mircoservice Infrastructure σϓϩΠ ߏங ࣗಈԽ͞Εͨӡ༻ ੵۃతʹؔΘ͍ͬͯ͘

ࠓޙͷMercari SRE • Microserviceͷج൫ߏஙɾӡ༻ • SREͷϊ΢ϋ΢Λ΋ͬͯαʔϏεʹੵۃతؔ༩ • εέʔϥϏϦςΟɺՄ༻ੑͷվળ • ࣗಈԽͷਪਐɺΦϖϨʔγϣϯͷվળ • ηΩϡϦςΟͷ࣮ࢪ • Globalͳnetwork/infrastructureͷޮ཰ӡ༻

SRE More!!! https://twitter.com/kazeburo/status/890131903529054210

Mercari SRE More!!! Microservice, Automation, Performance, Scalability Database, Network, Distributed System, OS, Cloud, Hardware, Security SWE + ༷ʑͳ஌ࣝɾܦݧΛੜ͔ͯ͠ ʮ৽ͨͳՁ஋ΛੜΈग़͢ੈքతϚʔέοτΛ૑Δʯ ৴པੑΛ࣋ͬͯࢧ͑Δ શํ໘ͰΑΖ͓͘͠ئ͍͠·͢!!

Ҏ্