速さは機能！Eightを支えるパフォーマンス・チューニング！ / Performance is a Feature！

■イベント
Observability Meetup #2
https://newrelic.co.jp/events/meetup/observability-meetup-2

■登壇概要
タイトル：
速さは機能！Eightを支えるパフォーマンス・チューニング！

登壇者：
Eight事業部 Engineering Manager 藤井洋太郎

▼Sansan Builders Box
https://buildersbox.corp-sansan.com/

Transcript

1. ଎͞͸ػೳʂ
   ࠃ಺࠷େڃͷϏδωε4/4Λࢧ͑ΔύϑΥʔϚϯενϡʔχϯάʂ Sansanגࣜձࣾ
   &JHIUࣄۀ෦
   1MBUGPSN
   6OJU
   -FBEFS
   
   &OHJOFFSJOH
   .BOBHFS
   ౻Ҫ
   ༸ଠ࿠ 
   0CTFSWBCJMJUZ
   .FFUVQ
   
   !4MBDL
   +BQBO
2. - ౻Ҫ༸ଠ࿠ (Fujii Yotaro) - ɹɹɹɹɹ (ݸਓ޲໊͚ࢗΞϓϦ) - 2014೥4݄ʹ৽ଔͱͯ͠ೖࣾ - Platform Unit(ج൫νʔϜ) - Unit Leader ݉ Engineering Manager - ΞʔΩςΫνϟ࡮৽΍ηΩϡϦςΟରԠɺ ύϑΥʔϚϯεվળ౳Λ୲౰ ࣗݾ঺հ
3. - Eight ͷ঺հ - New Relic ͷಋೖ؀ڥ - APM Λ࢖ͬͨύϑΥʔϚϯεվળͷࣄྫ঺հ - վળʹΑΔޮՌ / ൓Ԡ - ·ͱΊ ຊ೔ͷΞδΣϯμ
4. Eight ͷ঺հ
5. EightΛ͝ར༻தͷํ ✋
6. ໊ࢗͰͭͳ͕ΔɺϏδωεͷͨΊͷSNS
7. ໊ࢗͰͭͳ͕ΔɺϏδωεͷͨΊͷSNS ໊ࢗ؅ཧ ਓࣄ৘ใΛ௨஌ ؔ࿈͢Δاۀͷχϡʔε Ϗδωεϝοηʔδ
8. ΞΫςΟϒϢʔβ͸250ສਓ Ϣʔβ਺ 250ສ+ ೥ؒ ਺ԯຕ
9. EightͰاۀ಺ͷਓ຺Λڞ༗
10. ࠾༻αʔϏε΋ϦϦʔε
11. Eight ͷߏ੒
12. Ϋϥ΢υ͸AWS ϚωʔδυαʔϏεΛϑϧ׆༻֤͠छػೳΛఏڙ  
     =.'4)> =.*, =.'4 KPI 7+'4 7+0< />! />! '481   !  ! % 35)>  "  35/> " />  ;( $ %  *, 26 #81 
    9&:- =.'49&:- Recommendation Feed XPSLF S
13. ։ൃ؀ڥ(ݴޠ/ϑϨʔϜϫʔΫ/αʔϏε)
14. Eightʹ͓͚Δ New Relic ͷར༻ঢ়گ
15. EightͰओʹར༻͍ͯ͠Δ New Relic αʔϏε ΞϓϦέʔγϣϯੑೳͷৗற؂ࢹ APIར༻ঢ়گͷ֬ೝ / چAPIΛఀࢭ͢Δࡍʹ΋ར༻ ΞϓϦέʔγϣϯΤϥʔऩू ύϑΥʔϚϯεվળ APM(Application Performance Monitoring) ϑϩϯτΤϯυͷ ϨϯμϦϯάύϑΥʔϚϯε JSΤϥʔͷऩू Browser ΞϓϦͷCPU/ϝϞϦ࢖༻ྔ ୺຤/ར༻ࠃຖͷ௨৴؀ڥ Τϥʔ෼ੳ Mobile ֤छRelicαʔϏεͱ࿈ܞ ੑೳྼԽݕ஌ & Slack௨஌ Alerts ࠓ೔ͷ࿩ͷϝΠϯ
16. - ઃఆϑΝΠϧͰͷϑϥάมߋ͚ͩͰɺ؆୯ʹNew Relic AgentΛಋೖͰ͖Δɻ Ϛγϯʹ༨ܭͳϛυϧ΢ΣΞΛೖΕΔඞཁ͕ͳ͍ͱ͜Ζ΋ັྗ - ແྉ൛Ͱ΋(ࠓͷॴ)୆਺੍ݶແ͘ಋೖͰ͖Δʂ New Relic APMͷಛ௃ Eight Ͱͷಋೖྫ Staging؀ڥ Production؀ڥ ϓϥϯ New Relic Lite(ແঈ൛) New Relic Pro(༗ঈ൛) ద༻୆਺ શ୆ Availability Zoneຖʹ1୆ͣͭ σʔλอ࣋ظؒ 1೔ 90೔ؒ ػೳ ΞϓϦέʔγϣϯશମͷঢ়گ/ ֓ཁ͸े෼೺ѲՄೳ ֤τϥϯβΫγϣϯ΍SQL౳ͷৄࡉ Λ೺Ѳ͢ΔͨΊʹ͸ඞਢ
17. - Agent Λ༗ޮԽ͢ΔͱɺCPUར༻཰͕ 5ʙ10% ΄Ͳ্ঢ͢Δɻ - ͢΂ͯͷαʔόʔʹ͓͍ͯҰ཯༗ޮԽͯ͠͠·͏ͱɺ ύϑΥʔϚϯεྼԽ΍අ༻ରޮՌͷ໘Ͱ௼Γ߹Θͳ͘ͳΔͷͰɺόϥϯε͕ॏཁ New Relic APM Λར༻͢Δ্Ͱͷ஫ҙ఺ NewRelic Agent Λ༗ޮԽͨ͠ϓϩηε Agent ແޮͳϓϩηε Rails Appͷ CPU࢖༻཰ྫ
18. Ϧʔδϣϯ͕ҟͳΔͱɺωοτϫʔΫύϑΥʔϚϯεʹ΋͕ࠩग़Δ Availability Zone 1 Availability Zone 2 Aurora Elasticsearch Service ElastiCache CloudSearch ಉҰϦʔδϣϯ΍ɺ͍ۙڑ཭ʹ͋ΔαʔϏεͷ΄͏͕౰વύϑΥʔϚϯε͕Α͘ͳΔ App App ͍ۙ ԕ͍
19. Eightͷ ຊ൪؀ڥͰ͸ Multi-AZ ຖʹ1୆ͣͭAgentΛ༗ޮԽ Availability Zone 2 Availability Zone 1 Load Balancer ʮϦΫΤετ਺ʯ × ʮϨεϙϯελΠϜʯͰΦʔτεέʔϦϯά͞ΕΔ
20. DynamoDBΛར༻͠ɺAgent༗ޮԽ͢ΔΠϯελϯεΛ੍ݶ Availability Zone 1 Auto Scaling Group Name Availability Zone Instance ID Auto Scaling Group1 Availability Zone 1 instance-1 Scaling Group1 Availability Zone 2 instance-4 … Availability Zone 2 Auto Scaling Group1 instance-1 instance-2 instance-3 instance-4 Auto Scaling Group1 Availability Zone 2 instance-4 ΠϯελϯεՔಇ࣌ʹ ʮಉҰεέʔϦϯάάϧʔϓ × AZʯͷItemΛ֬ೝ ແ͚Ε͹ɺઃఆϑΝΠϧ Λॻ͖׵্͑ͨͰΞϓϦ έʔγϣϯՔಇ Insert(Lock) Πϯελϯεୀ໾࣌ʹࣗ਎ͷInstance IDʹඥͮ͘ItemΛ࡟আ͢Δ͜ͱͰɺ্ཱ͕࣍ͪΔΠϯελϯεͰ༗ޮԽ
21. - Ϧʔδϣϯ͝ͱͰҟৗݕ஌͕Մೳ - αʔϏεશମͷੑೳධՁΛਖ਼͘͠ߦ͑Δɻ - New Relic Agent ͷՔಇඅ༻ʹؔͯ͠΋࠷దԽͰ͖ɺ ࣋ଓՄೳͳঢ়ଶͰར༻͍͚ͯ͠Δɻ ৗʹҟͳΔϦʔδϣϯͰ্खʹ Agent Λ༗ޮԽ͢Δ͜ͱͰɺ ແཧແ͘ɺ࠷େݶ׆༻͢Δ͜ͱ͕େࣄ
22. ΍ͬͱɺຊ୊
23. New Relic APMΛ࢖ͬͨ “γϯϓϧ” Ͱ “ޮՌత” ͳ ύϑΥʔϚϯεɾνϡʔχϯά
24. ύϑΥʔϚϯεɾνϡʔχϯάʹର͢Δҹ৅ ΞʔΩςΫνϟมߋ ΞϓϦέʔγϣϯίʔυ վળ վળޮՌ େ খ ޻਺ େ খ
25. ΄Μͱʹʁ
26. - ޮՌ͕খ͍͞ͱࢥΘΕ͕͕ͪͩɺ࣮ࡍ͸ ΞϓϦέʔγϣϯίʔυͷվળ͚ͩͰ΋େ͖ͳޮՌ վળͷϑΝʔετϑΣʔζͰ͸අ༻ରޮՌ͸͔ͳΓେ͖͍ ޮՌ ޮՌ͕ݱΕΔ·Ͱ ΞʔΩςΫνϟ࡮৽ ίʔυʹΑΔվળ ӽ͑ΒΕͳ͍น
27. ΞϓϦέʔγϣϯίʔυվળ͚ͩͰͰ͖Δ ޮՌతͳύϑΥʔϚϯενϡʔχϯάํ๏
28. վળ࣠ΛҎԼͷ2఺ʹϑΥʔΧε͢Δ ᶃॲཧྔ͕ଟ͍ػೳΛத৺ʹର৅ΛߜΔ ᶄॲཧྔʹؔΘΒͣɺ஗͍ػೳʹߜΔ ͋ͨΓલͷ͜ͱա͗Δ͕ɺϑΥʔΧεϙΠϯτΛߜΔ͜ͱͰޮՌ͕࠷େԽ͢Δ
29. ͪͳΈʹ Eight Ͱ͸ɺ 1ϲ݄ ͰαʔϏεશମͷॲཧΛ ໿20% ߴ଎Խ
30. ᶃαʔόʔॲཧྔ͕ଟ͍ػೳͷվળ
31. - 1݅1݅ͷॲཧ͸ͱͯ΋଎͍͕ɺߴස౓ʹॲཧ͍ͯ͠Δέʔε͸ҙ֎ͱ͋Δɻ - ৽ணνΣοΫͷΑ͏ͳόοΫάϥ΢ϯυͰͷϙʔϦϯάॲཧ - ΩϟογϡΛט·͍ͤͯΔऔಘॲཧͳͲ - ஗ͯͨ͘͘͞Μ࢖ΘΕ͍ͯΔػೳ͸౰વɺߴස౓ʹར༻͞ΕΔ଎͍ػೳ΋ ʮ͞Βʹվળʯ͢Δ͜ͱͰվળޮՌΛ࠷େԽ͠΍͍͢ɻ ॲཧྔ͕ଟ͍ = ॲཧ͕஗͍ɺͱ͸ݶΒͳ͍ ྫ͑͹ɺ20%Λ઎Ί͍ͯΔػೳ1ͭΛ2ഒߴ଎ԽͰ͖Ε͹ɺ αʔόʔશମͷॲཧྔΛ10%࡟ݮͰ͖Δɻ
32. New Relic Λ࢖͑͹؆୯ʹՄࢹԽͰ͖Δ ػೳ(API)͝ͱͷॲཧྔ͸ Transactions ͷ Most time consuming Ͱ
33. - ੵΈॏͶͱͯ͠ͷɺϢʔβମݧ޲্ Ϣʔβ͔Β௚઀࢖ΘΕΔػೳΛߴ଎Խ͢Δ͜ͱ͸ɺϢʔβମݧͷ޲্ʹ௚݁ɻ ·ͨόοΫάϥϯυͰଟ͘ୟ͔ΕΔػೳΛߴ଎Խ͢Δ͜ͱͰ΋ɺΑΓϦΞϧλ ΠϜੑΛอͬͨϑΟʔυόοΫ΍ɺψϧαΫͱ͍ͬͨ໨ʹݟ͑ͳ͍ମݧ޲্ʹ ΋ͭͳ͕Δɻ - αʔϏεӡ༻ίετ࡟ݮ Webॲཧྔ͕গͳ͘ͳΕ͹଴ͪड͚Δαʔόʔͷ୆਺ΛݮΒͤΔɻ 20%࢖༻͍ͯ͠ΔAPI͕10%ʹͳΕ͹ɺ୯७ܭࢉͰαʔόʔӡ༻ίετ΋1ׂ࡟ ݮͰ͖Δ ظ଴͞ΕΔޮՌ
34. ᶄॲཧྔʹؔΘΒͣɺ஗͍ػೳʹߜΔ
35. - ஗͍ػೳ͸ຊ൪؀ڥͷΈͰݦࡏԽ͢Δ͜ͱ͕ଟ͍ (։ൃ؀ڥͰ΋஗͔ͬͨΒؾ෇͍ͯ௚͍ͯ͠·͢ΑͶ..?Έͳ͞Μʁ) - σʔλྔ΍ɺར༻ස౓ɺϢʔβݻ༗ͷಛఆͷ؀ڥʹΑͬͯύ ϑΥʔϚϯεྼԽ͍ͯ͠Δ͜ͱ͕΄ͱΜͲɻ - ࣄલʹؾ෇͚Δέʔε΋͋Δ͕ɺશͯΛ໢ཏͰ͖ΔΘ͚Ͱ͸ͳ͸͍ ʮ஗͍ʯػೳ͸౰વ଎͍ͨ͘͠ɻͰ΋ҙ֎ͱݕ஌ͣ͠Β͍
36. New Relic ͷ ฏۉ/MaxϨεϙϯελΠϜ Ͱநग़ ஗͍ػೳͷநग़͸ Transactions ͷ Slowest average response time Ͱ
37. Show all transactions tableͰɺશػೳΛҰཡͰදࣔ & ιʔτ ฏۉ(ms)Ͱιʔτ Max(ms)Ͱιʔτ
38. - ϔϏʔϢʔβ޲͚ͷମݧ޲্Λ͠΍͍͢ - ਺ઍຕʙ1ສຕ໊ࢗΛ͍࣋ͬͯΔϢʔβ΋ଟ͘ଘࡏ - σʔλͷಉظॲཧɺݕࡧॲཧɺιʔτॲཧͳͲ - Α͘࢖͍ͬͯͩͬͯ͘͞ΔϢʔβ͞Μ΄Ͳɺ஗͍ମݧʹͳͬͯ͠·͏ɻͱ͍͏ ͜ͱ͸ආ͚͍ͨɻ - ؅ཧऀػೳͳͲɺීஈ࢖͍Ͱ͸ؾ͖ͮʹ͍͘ػೳ΋վળͰ͖Δ - ϨεϙϯελΠϜ͚ͩʹϑΥʔΧε͢Δ͜ͱͰɺ ར༻཰͸ߴ͘ͳ͍͕ॏཁͳػೳʹ΋ϑΥʔΧεΛ౰ͯΒΕΔɻ ظ଴͞ΕΔޮՌ
39. ্هͷ؍఺Ͱର৅ΛϦετΞοϓ͠ɺ վળର৅͕ܾ·ͬͨΒ
40. - ஗͍ݪҼͱͳΔݪҼ͸༷ʑ - ॲཧ಺༰͕ଟ͍ - DBʹର͢ΔΫΤϦ͕஗͍ - ֎෦αʔϏεར༻ʹ͓͚Δͱͷ௨৴͕஗͍ - etc.. - ͕ͩɺΞϓϦέʔγϣϯίʔυͷมߋ͚ͩͰվળͰ͖Δέʔε͸ݶΒΕΔɻ ϘτϧωοΫΛ୳͢
41. ϘτϧωοΫΛ୳͢ - ͍ΘΏΔʮN+1(1+N)໰୊ʯͱͳ͍ͬͯͳ͍͔ʁ - SQL΍αʔϏεؔ࿈ܥͳͲͰͷ໰͍߹Θ͕ͤଟ͗͢Δ - ίωΫγϣϯϓʔϧ౳͕ద౰Ͱͳ͘ɺ௨৴ίετ͕େ͖͍౳ - ্هʹର͢Δվળ͸ൺֱత༰қͰ͔ͭվળޮՌ͕େ͖͍ ܁Γฦ͠ߦ͍ͬͯΔॲཧ͸ͳ͍͔ʁ
42. ҰͭҰͭͷ Transaction ͷৄࡉΛΈ͍ͯ͘
43. τϥϯβΫγϣϯৄࡉ Transaction(API)ຖʹ͋Δఔ౓ͷཻ౓Ͱ֤ॲཧͷฏۉίʔϧ਺͕·ͱ·͍ͬͯΔ DynamoDB΁ͷϦΫΤετ͕17ʙ18ճʂʂ ΩϟογϡετΞ΁ͷΞΫηε͕30ճʂʂ
44. - 1݅1݅ͷॲཧ͸ૣͯ͘΋ɺ਺ेʙ਺ඦճ࣮ߦ͢Ε͹૬౰ͳίετ ͜ΕΒͷ࣮ߦΛόονॲཧͯ͠1ճʹ͚ۙͮΕ͹ xxx ms ʹͳΔɺͱ͍ͬͨ໨ ඪઃఆ͕͠΍͍͢ɻ - ·ͨɺϧʔϓΛݮΒ͢͜ͱͰɺNewRelic্ͷ֤Segment(ॲཧ)ʹ ෼ྨ͞Εͳ͍ࡉ෦ͷॲཧ΋ߴ଎Խ͞ΕΔɻ - ίϯςΩετεΠονϯάɺϝιουίʔϧ࣌ͷ୳ࡧ͸ Controller΍Ruby શମͷॲཧʹ·ͱΊ͋͛ΒΕ͍ͯΔɻ ϧʔϓ͍ͯ͠Δॲཧ͸վળ͠΍͍͢ & ޮՌ͕େ͖͍
45. ֤ετϨʔδʹର͢ΔΞϓϩʔνྫ 45
46. - RDBΛѻ͏ͱ͖͕ɺ࠷΋ N+1 ͕ൃੜ͠΍͍͢ ςʔϒϧઃܭͰஸೡʹਖ਼نԽ͢Ε͹͢Δ΄ͲɺN+1໰୊Λҙࣝ͢Δඞཁ͕͋Δ RDB ᶃϦετऔಘ ᶄৄࡉऔಘ ɹ× Ϧετऔಘ෼ ộ employees ςʔϒϧ business_cards ςʔϒϧ ྫ) ࣾһͱڞ༗͍ͯ͠Δ໊ࢗΛ ͢΂ͯऔಘ͍ͨ͠ ࣾһ͕100ਓ͍Δ৔߹ɺ101ճͷSQL͕ൃߦ͞Εͯ͠·͏..!!!
47. - ϑϨʔϜϫʔΫ͕ఏڙ͍ͯ͠Δ ORM Λͦͷ··ҙࣝͤͣʹ࢖͏ͱɺN+1ʹͳͬͯ͠·͏ɻ - جຊతʹ Eager Loading ͷ࢓૊Έ͕༻ҙ͞Ε͍ͯΔ͸ͣͳͷͰɺNewRelicͷtransaction৘ ใΛࢀߟʹ͠ͳ͕Βɺνϡʔχϯά͍ͯ͘͠ RDB Employee.includes(:business_cards).where(company_name: ‘Sansanגࣜձࣾ’) RailsͷActiveRecordͰ͋Ε͹ɺ ্هͷΑ͏ʹɺҰׅઌಡΈ͢ΔࢦఆΛ͢Δ͜ͱͰɺ ൃߦ͢ΔSQLΛ1ճ͋Δ͍͸ 2(1+1)ճ·ͰݮΒ͢͜ͱ͕Ͱ͖Δ✨ ࢀߟ) Rails Guide : ؔ࿈෇͚ΛҰׅಡΈࠐΈ͢Δ
48. - ΩϟογϡΞΫηεͷվળ͸ҙ֎ͱ໡఺ʹͳΔ͜ͱ͕ଟ͍ ΩϟογϡΛಋೖ͢Δ͜ͱࣗମ͕ߴ଎Խʹͭͳ͕ΔͷͰɺෳ਺ճΞΫηε͍ͯ͠Δ͜ͱΛݟಀ͕ͪ͠ɻ ΩϟογϡετΞ(Memcached/ Redis) Redis Memcached Ϛελʔσʔλ - Memcached/Redis ڞʹҰׅऔಘ/ߋ৽༻ΠϯλʔϑΣʔε͕ఏڙ͞Ε͍ͯΔ - Memcached : Multi-GET / Multi-SET - Redis : MGET / MSET - ·ͨίωΫγϣϯϓʔϦϯάΛར༻͠ɺ઀ଓ࣌ͷΦʔόϔουΛഉআ͢Δ޻෉΋ॏཁ
49. - ֤ݴޠɾϑϨʔϜϫʔΫ΍ΫϥΠΞϯτϥΠϒϥϦʹ͸ Multi ॲཧ͢Δ I/F ͸جຊతʹ༻ҙ͞Ε͍ͯΔɻ(͸ͣ) - Rails Ͱ͸ *_multi ϝιουΛར༻͢Δ͜ͱͰҰׅऔಘ/ߋ৽͕Ͱ͖Δɻ ΩϟογϡετΞ(Memcached/ Redis) Rails.cache ॲཧ಺༰ Rails.cache.read_multi ҰׅಡΈࠐΈ Rails.cache.write_multi Ұׅॻ͖ࠐΈ Rails.cache.fetch_multi ҰׅಡΈࠐΈɺΩϟογϡ͕ͳ͚Ε͹ॻ͖ࠐΈ
50. ݕࡧΤϯδϯ/Elasticsearch Elasticsearch Service Aurora - ໊ࢗͷશจݕࡧͰར༻͍ͯ͠Δ Elasticsearch ʹରͯ͠΋ෳ਺ճߋ৽…. ɾESʹ͸ɺҰׅͰૢ࡞ՄೳͳBulk API ͕ఏڙ͞Ε͍ͯΔ ɾRubyͰ͸ elastic/elasticsearch-ruby gem ౳͔Β ɹBulk APIΛ؆୯ʹར༻Մೳʂ (Indexing͸ಉظͰ͸ͳ͘ɺόοΫάϥ΢ϯυॲཧʹճ͢ͷ͕Ұ൪)
51. - DynamoDB ౳Λσʔλιʔεͱͨ͠ॲཧͰɺݱঢ়ͷઃܭͰ͸Ұׅॲཧʹஔ͖ ׵͑ΒΕͳ͍έʔε΋͋Δɻ(ػೳͱͯ͠ఏڙ͞Ε͍ͯͳ͍έʔε) DynamoDB - ΞϓϦέʔγϣϯଆͰϦΫΤετΛฒྻԽ͢Δɻ - ΩϟογϡΛڬΉߏ੒ʹมߋ͢ΔϓϥΫςΟε΋ޮՌత ͦͷଞ / DynamoDB, etc. query DynamoDB fetch multi-get cache update
52. վળͷޮՌɾ൓Ԡ 52
53. ʮॲཧྔ͕ଟ͍ػೳʯͱʮ஗͍ػೳʯͷվળྫ ໊ࢗࡱӨ/௥Ճ଎౓ 1.5ഒ➚ Feedऔಘ଎౓ 1.5 - 2ഒ➚ ໊ࢗ/Ϣʔβݕࡧ଎౓ 2ഒ➚ اۀΞΧ΢ϯτ޲͚ Ϣʔβݕࡧ 10+ഒ➚
54. վΊͯ) αʔϏεશମͰΈΔͱ 20% Ҏ্ߴ଎Խ
55. - EightͷWebαʔόʔ܈͸ʮϦΫΤετ਺ʯͱʮฏۉϨεϙϯελΠϜʯʹ ΑͬͯΦʔτεέʔϦϯά͍ͤͯ͞ΔͨΊɺ ϨεϙϯελΠϜ͕޲্ͨ͜͠ͱͰɺαʔόʔίετ΋10ʙ20%ఔ౓࡟ݮ αʔϏεશମͷߴ଎Խ = ӡ༻ίετͷ࡟ݮ Auto Scaling Group response time request count ×
56. ސ٬͔ΒͷFeedback 57
57. اۀ޲͚ػೳͷ୲౰ऀ༷ΑΓ ʮUI͕଎͍ɺϨεϙϯε͕଎͍ɺઈົʯ ฐࣾProdcuct Manager ฐࣾӦۀ୲౰
58. - ಈ͍͍ͯΔ͔Βେৎ෉ɺͰ͸࠷ߴͷαʔϏεʹ͸ͳΒͳ͍ɻ - զʑͷΑ͏ͳ SRE(Site Reliability Engineering) Λ୲౰͢ΔνʔϜͦ͜ɺ ʮ߈ΊʯͷύϑΥʔϚϯεվળΛ͍ͯ͘͠΂͖ɻ - දʹ͸Θ͔Γ΍͘͢ݱΕͳ͍Α͏ͳɺʮ଎͞ʯʮ҆ఆੑʯΛ௥ٻ͢Δ͜ͱ ͰαʔϏεϨϕϧΛҰஈɺೋஈҾ্͖͛Δ͜ͱ͕Ͱ͖Δɻ ଎͍ɺ͸ɺ఻ΘΔ ଎͞ɺ͸ͦΕ͚ͩͰػೳͨΓಘΔ
59. - ଟػೳͳͰศརͳ New Relicʂ ͦͷதͰ΋ ॲཧྔɺॲཧ࣌ؒ ͷ2࣠ʹϑΥʔΧεͯ͠վળ͢Δ͚ͩͰ ΋େ͖ͳޮՌΛಘΔ͜ͱ͕Ͱ͖Δɻ - ύϑΥʔϚϯεྼԽ͚ͩͰͳ͘ɺطଘͷػೳͷվળʹ޲͖߹͏͜ͱͰɺ ʮϢʔβମݧͷ޲্ʯʮίετͷ࡟ݮʯʹͭͳ͕͍ͬͯ͘ɻ ·ͱΊ ଎͞͸ػೳʂ
60. ͓ΘΓ 61 ຊ೔ͷ಺༰͸ฐࣾϒϩάͰ΋؆୯ʹ঺հ͍ͯ͠·͢! https://buildersbox.corp-sansan.com/entry/2019/09/11/110000