Upgrade to Pro — share decks privately, control downloads, hide ads and more …

SRE大全 メルカリ編 後半 #hbstudy 75 / SRE Taizen Mercari 2 hbstudy#75

kazeburo
August 02, 2017

SRE大全 メルカリ編 後半 #hbstudy 75 / SRE Taizen Mercari 2 hbstudy#75

SRE大全 メルカリ編 後半 hbstudy#75

kazeburo

August 02, 2017
Tweet

More Decks by kazeburo

Other Decks in Technology

Transcript

  1. JSONಡΈࠐΈ • 1୆ͷαʔόͰར༻͞Εͨmessage keyΛϩάʹه࿥ɺΑ͘ར༻͞ΕΔ messageΛಛఆ • grep | sort |

    uniq -c | sort -n • Α͘ར༻͞ΕΔmessage͚ͩΛूΊͨ essential.json Λ࡞੒ • ϕϯνϚʔΫ All: 9msec => Essential: 1msec ҎԼ
  2. PHP(Opcache) ಈ࡞Πϝʔδ memory Destroy!! start end DB API shared mem

    php php php php php php php php php Response Request OPCODE Copy ࣮ߦ ࣮ߦ ࣮ߦ ίϯύΠϧ݁ՌͰ͸ͳ͘ɺOPCODEΛcache͓ͯ͠Γɺhash΍class͸౎౓ߏங = ௿଎ ϦΫΤετΛ௒͑ͯcache͢Δͷ͸೉͍͠
  3. ϘτϧωοΫΛ୳͢ํ๏ • NewRelic • σʔλϕʔεɺmemcachedɺ֎෦APIͷݺͼग़͠ճ਺ͱ࣌ؒɻN+1΍ɺpreload࿙Εͷൃݟ • PHP಺෦ͷτϨʔε΋Ͱ͖Δ • strace •

    System Call ͷτϨʔε • PHP಺෦Ͱى͖͍ͯΔ͜ͱ͸τϨʔεͰ͖ͳ͍ • System CallͱιʔείʔυΛݟൺ΂ͯϘτϧωοΫΛ୳͢ => Ͳ͏΍ͬͯʁ
  4. strace $ sudo strace -tt -s 256 -p $(pgrep -n

    -f httpd) |& tee strace.txt 15:29:31.122566 read(20, “GET /...) 15:29:31.122844 stat( ... 15:29:31.123914 getcwd("/", 4095) = 2 15:29:31.123936 chdir(“/var/www/vhosts/app/webroot") = 0 15:29:31.123966 setitimer(ITIMER_PROF, {it_interval={0, 0}, it_value={60, 0}}, NULL) = 0 15:29:31.123987 fcntl(18, F_SETLK, {type=F_RDLCK, whence=SEEK_SET, start=1, len=1}) = 0 15:29:31.125310 open(“/var/www/vhosts/app/foo/bar.json”) = 0 ## Ctrl-C $ less strace.txt PHP ಺෦ͷಈ࡞͸traceͰ͖ͳ͍ ιʔείʔυͱরΒ͠߹ΘͤΔʹ͸ώϯτͱͳΔsystem call͕΄͍͠ => OpcacheͷઃఆΛΈΔ
  5. php.ini in Production opcache.enable = 1 opcache.revalidate_freq = any opcache.validate_timestamps

    = 0 ࠷େੑೳ opcache.enable = 1 opcache.revalidate_freq = 1 opcache.validate_timestamps = 1 ϝϧΧϦӡ༻த ϑΝΠϧͷߋ৽͸νΣοΫ͠ͳ͍ ϑΝΠϧͷߋ৽͸νΣοΫ͢Δ 1ඵ͝ͱʹνΣοΫ nginx dynamic upstreamΛ࢖͍ɺbalancer͔Β֎ͯ͠ɺ rsync => 1ඵsleep ͯ͠balancerʹ໭͢
  6. php.ini for strace opcache.enable = 1 opcache.revalidate_freq = 0 opcache.validate_timestamps

    = 1 TUSBDF࣮ߦ༻ ຖϦΫΤετͰνΣοΫ͢Δ 18:21:01.253488 stat("/var/www/current/app/webroot/index.php", {st_mode=S_IFREG|0644, st_size=356, ...}) = 0 18:21:01.253572 stat("/var/www/current/dietcake/dietcake.php", {st_mode=S_IFREG|0644, st_size=1013, ...}) = 0 18:21:01.253697 stat("/var/www/current/dietcake/core/exception.php", {st_mode=S_IFREG|0644, st_size=46, ...}) = 0 18:21:01.253775 stat("/var/www/current/dietcake/core/inflector.php", {st_mode=S_IFREG|0644, st_size=289, ...}) = 0 .... 18:21:01.254005 stat("/var/www/current/dietcake/core/dispatcher.php", {st_mode=S_IFREG|0644, st_size=1300, ...}) = 0 18:21:01.254044 stat(“/var/www/current/app/aaaaaaa.php", {st_mode=S_IFREG|0644, st_size=5457, ...}) = 0 18:21:01.254135 stat("/var/www/current/app/app_xxx.php", {st_mode=S_IFREG|0644, st_size=10423, ...}) = 0 18:21:01.254618 stat("/var/www/current/app/app_exception.php", {st_mode=S_IFREG|0644, st_size=45680, ...}) = 0 18:21:01.257390 stat("/var/www/current/app/app_yyy.php", {st_mode=S_IFREG|0644, st_size=1133, ...}) = 0
  7. App࠷దԽͷϞνϕʔγϣϯͱλΠϛϯά • Appαʔό͕૿͑Δͱ • ނো཰͕૿͑ɺσϓϩΠ࣌ؒ΋৳ͼΔ • AppαʔόΛ૿΍ͯ͠΋஗͘ͳΔͷ͸๷͛Δ͕ɺϨεϙϯε଎౓͕มΘΒͳ͍ • ϝϧΧϦͰ͸AppαʔόͷCPU࢖༻཰͕ఆظతʹ50%Λ௒͑࢝ΊͨΒରࡦΛߦ͏ •

    αʔόͷ௥Ճ or ΞϓϦέʔγϣϯͷίʔυͷ࠷దԽ • قઅతཁҼͰΞΫηε͕૿͑ΔͷͰɺ࣌ؒͷ͋Δͱ͖ʹ४උɾ࠷దԽͷωλΛ΋͓ͬͯ͘ • αʔόαΠυΤϯδχΞͱऔΓ૊Ή͜ͱ͕Ͱ͖Ε͹ͳ͓ྑ͍ • SREݚमͰऔΓ૊ΉͳͲ
  8. େن໛ͳύεϫʔυϦετ߈ܸ • ੈքத͔Β߈ܸ͕ߦΘΕΔ • 2016೥ʹ࣮ࡍʹى͖ͨ߈ܸͷΞΫη εݩͷࠃ • 1IPʹ͖ͭɺ਺ճͷϩάΠϯࢼߦͰ ͋ΓɺࣗಈͰ͸๷͛ͳ͍ ͦͷଞ

    18% Armenia 2% Azerbaijan 2% Bahrain 2% Georgia 2% Japan 2% Russian 2% Indonesia 3% Nepal 3% Pakistan 5% Thailand 5% Taiwan 6% Viet Nam 6% Brazil 10% India 30%
  9. Mercari SREͷҐஔ͚ͮ ։ൃνʔϜ SRE BOT Infrastructure σʔλϕʔε࡞ۀ/ௐࠪґཔ εΩʔϚ/ΞʔΩςΫνϟ૬ஊ ߏஙʗӡ༻ σʔλϕʔε࡞ۀ

    ௐࠪ ߏஙɾར༻ σϓϩΠ Log ಗ໊DB ར༻ ར༻ ൓ө/ࢀর ໰୊ௐࠪͷڠྗɺमਖ਼ґཔ
  10. Microservice in US ©2011 Amazon Web Services LLC or its

    affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Tasks (HIT) Task Mechanical Turk Non-Service Specific backend for frontends GKE Core/طଘAPI protobuf JSON over HTTPs GRPC Service A Service B Spanner GKE GKE Deployج൫ SaaS PubSub
  11. MicroserviceԽʹΉ͚ͯ • Requirements(ཁ݅)ͷࡦఆ • Microserviceج൫ͷߏங • Deploy, Log, Database, Monitoring...

    • ຊ൪ͷΞʔΩςΫνϟͱͯ͠ͷ࠾༻ • લਐͭͭ͠՝୊Λݟ͚ͭɺͻͱͭͻͱͭղܾ͍ͯ͘͠
  12. ࠓޙͷMercari SRE ։ൃνʔϜ SRE BOT Infrastructure σʔλϕʔε࡞ۀ/ௐࠪґཔ εΩʔϚ/ΞʔΩςΫνϟ૬ஊ ߏஙʗӡ༻ σʔλϕʔε࡞ۀ

    ௐࠪ ߏஙɾར༻ σϓϩΠ Log ಗ໊DB ར༻ ར༻ ൓ө/ࢀর ໰୊ௐࠪͷڠྗ मਖ਼ґཔ Microserviceج൫ Mircoservice Infrastructure σϓϩΠ ߏங ࣗಈԽ͞Εͨӡ༻
  13. ࠓޙͷMercari SRE ։ൃνʔϜ SRE BOT Infrastructure σʔλϕʔε࡞ۀ/ௐࠪґཔ εΩʔϚ/ΞʔΩςΫνϟ૬ஊ ߏஙʗӡ༻ σʔλϕʔε࡞ۀ

    ௐࠪ ߏஙɾར༻ σϓϩΠ Log ಗ໊DB ར༻ ར༻ ൓ө/ࢀর ໰୊ௐࠪͷڠྗ मਖ਼ґཔ Mircoservice Infrastructure σϓϩΠ ߏங ࣗಈԽ͞Εͨӡ༻ Microserviceج൫ ϚΠΫϩαʔϏεԽ͕ਐΉ͜ͱͰ ͚ͩ͜͜ʹͳ͍ͬͯ͘ͷ͔ʁ
  14. ࠓޙͷMercari SRE ։ൃνʔϜ SRE BOT Infrastructure σʔλϕʔε࡞ۀ/ௐࠪґཔ εΩʔϚ/ΞʔΩςΫνϟ૬ஊ ߏஙʗӡ༻ σʔλϕʔε࡞ۀ

    ௐࠪ ߏஙɾར༻ σϓϩΠ Log ಗ໊DB ར༻ ར༻ ൓ө/ࢀর ໰୊ௐࠪͷڠྗ मਖ਼ґཔ Microserviceج൫/Requirement Mircoservice Infrastructure σϓϩΠ ߏங ࣗಈԽ͞Εͨӡ༻ ੵۃతʹؔΘ͍ͬͯ͘
  15. Mercari SRE More!!! Microservice, Automation, Performance, Scalability Database, Network, Distributed

    System, OS, Cloud, Hardware, Security SWE + ༷ʑͳ஌ࣝɾܦݧΛੜ͔ͯ͠ ʮ৽ͨͳՁ஋ΛੜΈग़͢ੈքతϚʔέοτΛ૑Δʯ ৴པੑΛ࣋ͬͯࢧ͑Δ શํ໘ͰΑΖ͓͘͠ئ͍͠·͢!!