SRE大全 メルカリ編 hbstudy#75
SREେશ: ϝϧΧϦฤ [લ]2017/08/17 hbstudy#75Masahiro Nagano @kazeburo
View Slide
AGENDA• લ• ࣗݾհɺϝϧΧϦ͕SREΛ࠾༻ͨ͠ཧ༝• ϝϧΧϦSREνʔϜհɺOnCallɺϝϧΧϦʹ͍ͭͯ• ઌͷCDNมߋͰͷΠϯγσϯτʹ͍ͭͯ
AGENDA• ޙ• PHP ΞϓϦέʔγϣϯͷ࠷దԽࣄྫ• ηΩϡϦςΟͷऔΓΈ(ύεϫʔυϦετ߈ܸࣄྫ)• ϝϧΧϦSREͷࠓޙ• SREͷׂͷݱࡏͱະདྷɺϚΠΫϩαʔϏε
Me• Masahiro Nagano / խ• @kazeburo• Mercari, Inc Principal Engineer Site Reliability Engineering (SRE) Team• BASE, Inc Technical Advisor
Me• ~ 2006: ژͰελʔτΞοϓࢀՃ• ։ൃΛ͠ͳ͕ΒΠϯϑϥͷ໘ΛݟΔɻDC࡞ۀͬͨ• ΞϓϦέʔγϣϯͷνϡʔχϯάɺۭ͍ͨϦιʔεͰ৽ػೳͷՃͱ͍͏αΠΫϧ• mod_perlɺSquidʹΑΔReverse Proxy• 2006 ~: mixi• ʮΞϓϦέʔγϣϯӡ༻νʔϜʯ / DCʹߦ͔ͳ͘ͳͬͨ• େنͳը૾৴/memcached/Q4M
Me• 2010 ~: livedoor (NHN Japan => LINE)• livedoorLINEϑΝϛϦʔͷαʔϏεΛԣஅͯ͠ΠϯϑϥύϑΥʔϚϯεͷվળ• livedoor blog ͷMySQLνϡʔχϯά• GrowthForecast/HRForecast/Plack Optimization/MHA• 2015/02: mercari
࠷ۙͷ׆ಈ• ొஃ• AWS Dev Day Tokyo 2017• YAPC::Fukuoka 2017, YAPC::Hokkaido 2016• هࣄ• WEB+DB PRESS Vol.88, Vol.92-97 ࿈ࡌ• ܦSYSTEMS 2017 7݄߸, ITPro
WEB+DB PRESS Vol.1008%#13&44߸㊗8%#13&447PMͷग़൛ֶੜ࣌ͷ࠷ॳʹखʹऔͬͨ࣌ʹ7PMʹهࣄΛॻ͘ͳΜͯ͜ͱશ͘૾͍ͯ͠·ͤΜͰͨ͠ɻͱͯޫӫͰ͢
ϝϧΧϦ͕SREΛ࠾༻ͨ͠ཧ༝
ࣗশ `!ΠϯϑϥΤϯδχΞ`• (ݸਓతʹ) ΠϯϑϥΤϯδχΞͷٙ• 2006ҎདྷDC࡞ۀ͍ͯ͠ͳ͍ɻDC࡞ۀΛߦ͏νʔϜଞʹ͍Δ• σʔληϯλʔνʔϜ͕༻ҙͨ͠αʔόͷೳྗΛҾ͖ग़͠ɺΞϓϦέʔγϣϯΤϯδχΞ͕࡞ͨ͠ίʔυΛ࠷ߴͷܗͰಈ͔͢ͷ͕ࣗΒ(νʔϜ)ͷׂ• αʔϏεͷՄ༻ੑϋʔυΣΞͷνʔϜͰͳ͘ɺιϑτΣΞΛѻ͏νʔϜͷ
ΦϖϨʔγϣϯΤϯδχΞ• 2010ग़൛ʮWeb Operationsʯ• ʮΣϒΦϖϨʔγϣϯٕܳͰ͋ΓՊֶͰͳ͍ʯ• ܧଓతσϓϩΠɺDevOpsɺࣗಈԽɺࢹͳͲΦϖϨʔγϣϯʹؔ͢ΔΤοηΠ• ͨͩ͠ɺΦϖϨʔγϣϯ=ӡ༻ΛϧʔνϯϫʔΫͱଊ͑Δਓଟ͍
SREͱͷग़ձ͍• 2012/7 ༑ਓͱͷIRCͰͷձ͔Β• ΠϯϑϥͱαʔϏεͷՔಇɺ҆ఆੑΛ୲͢ΔνʔϜ͕SRE• https://research.googleblog.com/2012/07/site-reliability-engineers-solving-most.html ͜ͷهࣄ͕ެ։͞Εͨࠒ• twitter ͷbioൃදεϥΠυʹʮSite ReliabilityʯΛՃͯ͠ҙࣝ• https://www.slideshare.net/kazeburo/yapc2102mysql/2 (2012/9)• 2015/11 ϝϧΧϦʹͯνʔϜ໊ͱͯ͠࠾༻
ϝϧΧϦͰSREΛ࠾༻ͨ͠ཧ༝• ϝϧΧϦΛ͓٬͞·ʹͬͯ͘Β͏ʹʮ͍ͭͰշదʹ҆શʹ͑Δʯ৴པੑ͕ॏཁ• Πϯϑϥ=ϞϊɺΦϖϨʔγϣϯ=ࣄͰͳ͘ɺ৴པੑʹΛ࣋ͭ͜ͱΛ໌Β͔ʹ͢Δ• ւ֎Ͱ௨͡Δ໊শ• JP/US/UKͰͷల։ɻάϩʔόϧͰͷ࠾༻ৗʹҙࣝ• ઌਐతͳऔΓΈͱͯ͠
ϝϧΧϦSREʹ͍ͭͯ
Mercari SRE• ͍ͭͰշద͔ͭ҆શʹར༻Ͱ͖Δʮ৴པੑͷߴ͍ʯαʔϏεͷ࣮ݱ• ʮ৽نαʔϏεͷ։ൃҎ֎ͷΤϯδχΞϦϯάશ෦Δʯ• ݱࡏϝϯόʔʮ9ਓʯ• શһ౦ژۈ• ࣾπʔϧͷӡ༻վળɺAIܥͷۀʹܞΘΔϝϯόʔ͍Δ
Mercari SRE ͷۀൣғOperations Software Eng.ج൫ߏஙOnCall (োରԠ)AutomationεέʔϥϏϦςΟɾՄ༻ੑվળDBAɺϛυϧΣΞߏஙΞϓϦέʔγϣϯͷઃܭϨϏϡʔϩάऩूɾੳج൫ͷߏஙɺӡ༻αʔόϓϩϏδϣχϯάɾσϓϩΠɺϚΠΫϩαʔϏεج൫ͷඋηΩϡϦςΟʗෆਖ਼ར༻ݕग़
SRE൪/OnCall• ΈΜͳେ͖োରԠ• ΞϥʔτରԠ൪ͱి൪• 4໊ͰOnCallͷScheduling• (2໊τϨʔχϯάத)
SRE൪/OnCall• ༵0͔࣌Β༵24࣌·Ͱ1िؒͰަ• Ξϥʔτͷड͚औΓͱҰ࣌ରԠ• ฏνʔϜϝϯόʔ͕ग़ࣾ͢Δ·Ͱࣗػ• 9͔࣌Βࣗػ͠ɺUS͔ΒͷґཔͳͲʹରԠ• UK͔ΒͷରԠͰؒ࡞ۀ͋Δ• ٳ15-20ҎʹରԠ։࢝Ͱ͖Δ͜ͱ͕·͘͠ɺߦಈʹ੍ݶ͋Δ• ਂٳͷରԠͳͲɺՈͷڠྗඞཁ
൪/OnCall Λࢧ͑Δٕज़• ࢹ• Mackerel, slacklog• ௨/Scheduling• Slack, PagerDuty, Twilio
mackerelWorker BatchApp AppMySQL cronmackerel-agentfluent-plugin-mackerelmkrՄࢹԽ/ᮢͷઃఆMetricsऩू௨
[ઢ] ࠷ۙ࡞ͬͨmackerel-plugin#!/usr/bin/perluse HTTP::Date;my $NUM_LOG_WATCH = 1000;my $CHECK_RANGE = 300; #5minmy $exceptions = 0;my $now = time;open( my $messages_tail, "-|", "tail","-$NUM_LOG_WATCH","/var/log/messages") or die $!;while (<$messages_tail>) {if ( $_ !~ m!Machine Check Exception! ) {next;}if ( my ($time) = ($_ =~ m!^(\w{3}\s+\d{1,2}\s\d{2}:\d{2}:\d{2})\s!) ) {$time = str2time($time);if ( $now - $time < $CHECK_RANGE ) {$exceptions++;}}}if ( $exceptions > 0 ) {print "CRITICAL: Machine Check Exception Found in this 5 minutes\n";exit 2;}print "OK: No Machine Check Exception found\n";exit 0;% dmesg | tail sbridge: HANDLING MCE MEMORY ERRORCPU 0: Machine Check Exception: 0 Bank 8:cc0427c000010090TSC 0 ADDR 37805ac0 MISC 45048ce86 PROCESSOR0:406f1 TIME 1495654896 SOCKET 0 APIC 0[Hardware Error]: Machine check events loggedEDAC MC1: CE row 0, channel 0, label"CPU_SrcID#0_Ha#0_Channel#0_DIMM": 4255 Unknownerror(s): memory read on FATAL area OVERFLOW:cpu=0 Err=0001:0090 (ch=0), addr = 0x37805ac0=> socket=0, ha=1, Channel=0(mask=1), rank=0DIFDLNBDIJOFFYDFQUJPOT ϝϞϦʔΤϥʔΛݟ͚ͭΔ
[ઢ] ࠷ۙ࡞ͬͨmackerel-plugin#!/bin/shset -eif [ ! -f /opt/MegaRAID/MegaCli/MegaCli64 ]; thenexitfiif ( /opt/MegaRAID/MegaCli/MegaCli64 -PDList -aALL|grep 'Firmware state'|grep -v "Online, Spun Up" > /dev/null 2>&1 );then/opt/MegaRAID/MegaCli/MegaCli64 -PDList -aALL|grep 'Firmware state'exit 2fi/opt/MegaRAID/MegaCli/MegaCli64 -PDList -aALL|grep 'Firmware state'exit 0DIFDLSBJEEJTL.FHB$MJΛ͔ͭͬͯ3"*%ͷঢ়ଶΛࢹ
slacklog ίϚϯυ$ slacklog -t alert-information --notify -- perl -e 'die "TEST!"'!LB[VIP͞ΜͷDSPOMPHΛࢀߟʹ࡞CBUDICBDLVQͷࣦഊΛݕslackboard௨ͷूIUUQTHJUIVCDPNDVCJDEBJZBTMBDLCPBSE
slackͰͷΞϥʔτͷ• ؾ͘ͷ͕͍͠• ௨ԻͷΧελϚΠζͰ͖ͳ͍• 1ճ͔͠௨Ͱ͖ͳ͍ɻྲྀΕ͕ͪ• Push͜ͳ͍߹• ετϨε⤴ / ӡ༻ʹͳΓ͕ͪ
PagerDuty4DIFEVMFϩʔςʔγϣϯཧ &TDBMBUJPOϧʔϧ
PagerDuty• ༷ʑͳखஈͰ௨Λߦ͏͜ͱ͕Ͱ͖Δ• mail• SMS• App• ి• 12ʹҰిΛೖΕΔϧʔϧͰӡ༻• Appͷ௨͕ศར
Emergency Call☎US/UKؚΊɺؾܰʹͬͯΒ͑ΔΑ͏ࣾࠂ ࠷ۙͰSREνʔϜͰΤεΧϨʔγϣϯతͰར༻
ϝϧΧϦʹ͍ͭͯ
Mercari• ࠃ࠷େڃͷϑϦϚΞϓϦ• 3Ͱ؆୯ʹग़• ҆৺҆શͳܾࡁ
US/UK ͷల։JP2016/08US AppStore3ҐUS UK2017/03/15ϦϦʔε
Mercari KPIμϯϩʔυGMV(૯औҾֹ)7500ສDL(JP+US)݄ؒ100ԯԁҎ্ग़ 1100ສҎ্
ຊ࠷େͷϑϦϚΞϓϦ1,2001ؒͷग़(peek࣌ؒଳ)Ҏ্
ग़͔Β͙͢ʹചΕΔ24࣌ؒҎചΕͨͷ50%͕24࣌ؒҎʹऔҾཱ
γεςϜ͔ΒΈͨϝϧΧϦ©2011 Amazon Web Services LLC or its affiliates. All rights reserved.Client Multimedia Corporatedata centerTraditionalserverMobile ClientIAM Add-on Example:IAM Add-onence)Assignment/TaskRequesterWorkersग़!DB Search5-දࣔݕࡧө©2011 Amazon Web Services LLC or its affiliates. All rights reserved.User Users Client Multimedia Corporatedata centerTraditionalserverMobile ClientInternet AWS ManagementConsoleIAM Add-on Example:IAM Add-onAmazon Mechanical TurkOn-Demand WorkforceHuman IntelligenceTasks (HIT)Assignment/TaskRequesterWorkersAmazonMechanical TurkNon-Service Specific©2011 Amazon Web Services LLC or its affiliates. All rights reserved.User Users Client Multimedia Corporatedata centerTraditionalserverMobile ClientInternet AWS ManagementConsoleIAM Add-on Example:IAM Add-onAmazon Mechanical TurkOn-Demand WorkforceHuman IntelligenceTasks (HIT)Assignment/TaskRequesterWorkersAmazonMechanical TurkNon-Service Specific©2011 Amazon Web Services LLC or its affiliates. All rights reserved.User Users Client Multimedia Corporatedata centerTraditionalserverMobile ClientInternet AWS ManagementConsoleIAM Add-on Example:IAM Add-onAmazon Mechanical TurkOn-Demand WorkforceHuman IntelligenceTasks (HIT)Assignment/TaskRequesterWorkersAmazonMechanical TurkNon-Service Specific©2011 Amazon Web Services LLC or its affiliates. All rights reserved.User Users Client Multimedia Corporatedata centerTraditionalserverMobile ClientInternet AWS ManagementConsoleIAM Add-on Example:IAM Add-onAmazon Mechanical TurkOn-Demand WorkforceHuman IntelligenceTasks (HIT)Assignment/TaskRequesterWorkersAmazonMechanical TurkNon-Service Specific©2011 Amazon Web Services LLC or its affiliates. All rights reserved.User Users Client Multimedia Corporatedata centerTraditionalserverMobile ClientInternet AWS ManagementConsoleIAM Add-on Example:IAM Add-onAmazon Mechanical TurkOn-Demand WorkforceHuman IntelligenceTasks (HIT)Assignment/TaskRequesterWorkersAmazonMechanical TurkNon-Service Specific©2011 Amazon Web Services LLC or its affiliates. All rights reserved.User Users Client Multimedia Corporatedata centerTraditionalserverMobile ClientInternet AWS ManagementConsoleIAM Add-on Example:IAM Add-onAmazon Mechanical TurkOn-Demand WorkforceHuman IntelligenceTasks (HIT)Assignment/TaskRequesterWorkersAmazonMechanical TurkNon-Service Specific©2011 Amazon Web Services LLC or its affiliates. All rights reserved.User Users Client Multimedia Corporatedata centerTraditionalserverMobile ClientInternet AWS ManagementConsoleIAM Add-on Example:IAM Add-onAmazon Mechanical TurkOn-Demand WorkforceHuman IntelligenceTasks (HIT)Assignment/TaskRequesterWorkersAmazonMechanical TurkNon-Service Specific©2011 Amazon Web Services LLC or its affiliates. All rights reserved.User Users Client Multimedia Corporatedata centerTraditionalserverMobile ClientInternet AWS ManagementConsoleIAM Add-on Example:IAM Add-onAmazon Mechanical TurkOn-Demand WorkforceHuman IntelligenceTasks (HIT)Assignment/TaskRequesterWorkersAmazonMechanical TurkNon-Service Specific©2011 Amazon Web Services LLC or its affiliates. All rights reserved.User Users Client Multimedia Corporatedata centerTraditionalserverMobile ClientInternet AWS ManagementConsoleIAM Add-on Example:IAM Add-onAmazon Mechanical TurkOn-Demand WorkforceHuman IntelligenceTasks (HIT)Assignment/TaskRequesterWorkersAmazonMechanical TurkNon-Service Specific©2011 Amazon Web Services LLC or its affiliates. All rights reserved.User Users Client Multimedia Corporatedata centerTraditionalserverMobile ClientInternet AWS ManagementConsoleIAM Add-on Example:IAM Add-onAmazon Mechanical TurkOn-Demand WorkforceHuman IntelligenceTasks (HIT)Assignment/TaskRequesterWorkersAmazonMechanical TurkNon-Service Specific©2011 Amazon Web Services LLC or its affiliates. All rights reserved.User Users Client Multimedia Corporatedata centerTraditionalserverMobile ClientInternet AWS ManagementConsoleIAM Add-on Example:IAM Add-onAmazon Mechanical TurkOn-Demand WorkforceHuman IntelligenceTasks (HIT)Assignment/TaskRequesterWorkersAmazonMechanical TurkNon-Service SpecificେྔͷϦΫΤετ©2011 Amazon Web Services LLC or its affiliates. All rights reserved.User Users Client Multimedia Corpdata cMobile ClientInternet AWS ManagementConsoleIAM Add-on Example:IAM Add-onHuman IntelligenceTasks (HIT)Assignment/TaskRequesterWorkersAmazonMechanical TurkNon-Service SpecificϦΫΤετԠDB Searchߪೖ!ඵʙ30ඵ ඵʙߴʹେྔͷτϥϯβΫγϣϯΛѻ͏ը૾ ܾࡁAI
ΠϯϑϥετϥΫνϟੴङDCઐ༻αʔόJPCloudUSCloudUK
ΠϯϑϥετϥΫνϟJP US UKDNS: Amazon Route53CDN: Akamai, Fastly, ImageFluxStorage: Amazon S3Analysis: Google BigQueryܾࡁ/ྲྀαʔϏε ܾࡁ/ྲྀαʔϏε ܾࡁ/ྲྀαʔϏε
ΞʔΩςΫνϟ• ࡾ+ΞϧϑΝͳΞʔΩςΫνϟ• Reverse Proxy(nginx) Application(Apache+mod_php) Database(MySQL) Cache(memcached) Search(Solr)• ଟ͘Λʮઐ༻αʔόʯʹͯߏ• εέʔϧΞτͱεέʔϧΞοϓΛಉ࣌ʹߦ͏Diagonal Scale• ओʹ24ίΞ~56ίΞ·ͰͷαʔόΛར༻• Databaseʹ ioMemory NVMe Λࡌͨ͠αʔόΛ࠾༻nginx nginx nginx©2011 Amazon Web Services LLC or its affiliates. All rights reserved.Users Client Multimedia Corporatedata centerTraditionalserverMobile ClientWS ManagementConsoleIAM Add-on Example:IAM Add-onHuman IntelligenceTasks (HIT)Assignment/TaskRequesterWorkersicDNS-RRApp App AppApp App AppMySQL MySQLmemcachedmemcachedutil utilcloud cloudJPSolrSolr
CDNΓସ͑ʹ͏Web൛ϝϧΧϦʹ͓͚Δݸਓใྲྀग़ Πϯγσϯτʹ͍ͭͯ
ଟ͘ͷօ༷ʹ͝৺ɺ͝໎Λֻ͓͚͍ͨ͠·ͨ͠ਂ͓͘ͼਃ͋͛͠·͢
͜͜Ͱൃੜͨ͠ࣄͱݪҼٴͼ ݱঢ়ͷରࡦʹ͍ͭͯհ͍ͨ͠·͢ɻ
ൃੜͨ͠ࣄ• ϝϧΧϦWeb൛ͷΞΫηε্ͱɺηΩϡϦςΟ্ͷͨΊɺCDNͷΓସ͑Λߦ͍·ͨ͠ɻͦͷࡍʹΓସ͑ઌͷCDNʹ͓͚Δಈ࡞ʹ͍ͭͯɺΓସ͑Λ୲ͨ͠ΤϯδχΞͷѲ͕ෆ͓ͯ͠Γɺ͓٬͞·ͷϨεϙϯε͕ผͷ͓٬͞·ʹҙਤͤͣදࣔ͞Εɺ݁Ռͱͯ͠ݸਓΛಛఆͰ͖ΔใΛؚΉ༰͕ຊਓҎ֎ʹӾཡ͞ΕΔঢ়ଶͱͳΓ·ͨ͠
Timeline• 6/22• 9:41ɹ CDNͷΓସ͑Λ࣮ࢪʢൃੜʣ• 14:41ɹΧελϚʔαϙʔτʹ͓ͯ٬͞·͔Βͷ͍߹ΘͤΛ֬ೝ͠ɺࣾใࠂ• 15:05ɹCDNͷΓସ͑Λதࢭ͠ɺैདྷͷCDN͢• 15:16ɹWeb൛ͷϝϧΧϦΛϝϯςφϯεϞʔυΓସ͑• 15:38ɹΓସ͑ઌCDNͷઃఆΛdeactivate͠ɺΞΫηεΛःஅ• 15:47ɹWeb൛ͷϝϧΧϦϝϯςφϯεϞʔυΛऴྃ• 17:55 ίʔϙϨʔταΠτʹ͓ΒͤΛܝࡌ• 20:45 Tech blogʹͯৄࡉެ։
Timeline• 6/28• Γସ͑ઌCDNͷઃఆΛ࠶Activate• CacheΛແޮԽ͢ΔઃఆΛߦ͍ɺΓସ͑ઌCDNͷΤϯδχΞʹϨϏϡʔΛґཔ• nginxͷઃఆΛߋ৽͠ɺcacheʹؔ͢ΔϔομΛมߋ• ࣾͰݕূ• 6/29• Γସ͑ઌCDNͷΤϯδχΞػͷͱɺΓସ͑Λ࣮ࢪ
Γସ͑ઌCDNʹ͓͚Δcacheͷಈ࡞• CacheΛແޮԽ͢ΔͨΊʹ “Cache-Control: private” ͘͠ "Set-Cookie"͕ඞཁ• ”Cache-Control: no-cache” “no-store” ແࢹ͞ΕΔ• Expiresϔομར༻͞ΕΔ͕ɺͷղऍʹࣦഊ͋Δ͍աڈͷ߹ “0ඵ” ͱͯ͠ѻΘΕΔ• ʮ0ඵͷΩϟογϡ͕ଘࡏ͢Δʯ• (্هઃఆʹΑΓΧελϚΠζ͕Մೳ)
0ඵͷcache• CDN͔ΒΦϦδϯͷϦΫΤετͷॲཧதʹɺಉ͡URLʹରͯ͠ϦΫΤετ͕ൃੜ͢Δͱɺ࠷ॳͷϨεϙϯεΛͬͯɺ2ͭҎ߱ͷϦΫΤετʹಉ͡Ϩεϙϯε͕ฦ͞ΕΔ• ੩తίϯςϯπͰඇৗʹ༗ޮͰ͋Δ• ͜ͷ༷ʹؔ͢ΔѲ͕Ͱ͖ͯͳ͔ͬͨ ©2011 Amazon Web Services LLC or its affiliates. All rights reserved.User Users Client MultimediaMobile ClientInternet AWS ManagementConsoleIAM Add-on Example:IAM Add-onHuman IntelligenceTasks (HIT)Assignment/TaskRequesterWorkersAmazonMechanical TurkNon-Service Specific©2011 Amazon Web Services LLC or its affiliates. All rUser Users Client Mobile ClientInternet AWS ManagementConsoleIAM Add-on Example:IAM Add-onHuman IntelligenceTasks (HIT)Assignment/TaskWorkersAmazonMechanical TurkNon-Service SpecificoriginCDN(1)(2)(3)(4)(5) (5)
࠶ସͷࡍʹߦͬͨରࡦ• CDNͷઃఆΛมߋ͠ɺURLɺΦϦδϯͷϔομʹؔΘΒͣΩϟογϡΛ͠ͳ͍ɺ·ͨ0ඵͷΩϟογϡͷͨΊʹϨεϙϯεΛͨͳ͍Α͏ʹ͠ɺCDNఏڙࣾͷΤϯδχΞͷϨϏϡʔΛड͚ͨ• CDNͷϩάΛϦΞϧλΠϜʹS3ʹૹ৴͠ɺlambdaʹΑΓϩάΛղੳɺmackerelͰՄࢹԽɻcache͕ΘΕ͍ͯͳ͍͔Λࢹ• nginx/ApacheʹͯΩϟογϡ͞ΕͮΒ͘͢ΔϔομΛՃ• ϔομCDNͷมߋΛࢹ͢ΔscriptΛ࡞͠ɺslacklogܦ༝Ͱఆظ࣮ߦ
cache aware nginx configuration• ExpiresϔομΘͳ͍• ݹ͍ϒϥβ͚ʹPragmaϔομͰରԠ• Cache-ControlҎ֎ʹΩϟογϡΛආ͚ΔͨΊ͚ͩͷ Set-Cookieૹ৴more_clear_headers 'Expires';more_set_headers "Cache-Control: private, no-cache, no-store, must-revalidate" "Pragma: no-cache";add_header Set-Cookie "merCtx=\"\"; HttpOnly" always;OHJOYDPOG
ࠓޙ՝• css/js/fontͷassetsͷΩϟογϡ• ݱঢ়ɺશʹΩϟογϡແޮ• CDNͷઃఆͷܧଓతΠϯςΫάϨʔγϣϯ(CI)• Web൛Ҏ֎ͰͷCDNͷར༻• DDoSੑɺηΩϡϦςΟରࡦ
લऴྃ
kazeburo
culvert
nrryuya
oracle4engineer
pf_businessdivision
egmc
lmi
irotoris
nasa_desu
antimon2
kashira
coconala_engineer
bengo4com
roundedbygravity
jrom
aki_iinuma
schacon
mojombo
sachag
kneath
cherdarchuk
brianwarren
vanstee
davidbonilla
phodgson