delightworks-tech-night-1

Transcript

1. ϞϯελʔετϥΠΫͷ؂ࢹπʔϧࠓੲ෺ޠ ʙ্רʙ ։ൃຊ෦ SREάϧʔϓ খ஑஌༟ גࣜձࣾϛΫγΟ
2. ࣗݾ঺հ 2
3. ࣗݾ঺հ ‣ࢯ໊ ‣খ஑஌༟ ‣ܦྺ ‣2008೥ϛΫγΟʹೖࣾ ‣SNSʮmixiʯͳͲͰΠϯϑϥɺγεςϜӡ༻ ‣ͦͷޙɺϞϯελʔετϥΠΫΛ͸͡Ίͱͨ͠ήʔϜΞϓϦͷӡ༻ ‣Ϟϯετ֤छαΠτɺࣾ಺Ͱར༻͢ΔαʔϏεӡ༻πʔϧɺ෺ཧΠ ϯϑϥ؀ڥͷӡ༻౳ʑɺ෯޿͘ैࣄ ‣։ൃຊ෦ SRE άϧʔϓॴଐ 3
4. ϞϯελʔετϥΠΫ 4
5. ϞϯελʔετϥΠΫ ࣗ෼ͷϞϯελʔΛҾͬுͬͯ஄͖ɺఢͷϞϯελʔʹ౰ͯͯ౗͍ͯ͘͠ͱ͍͏ɺεϚʔτϑΥϯͷಛੑΛ׆༻ͨ͠ɺ ୭Ͱ΋؆୯ʹָ͠ΊΔΞΫγϣϯRPGͰ͢ɻήʔϜ͸λʔϯ੍Λͱ͓ͬͯΓɺ Ұॹʹ͍Δ༑ͩͪͱ࠷େ4ਓ·Ͱಉ࣌ʹ༡΂ΔڠྗϓϨΠʢϚϧνϓϨΠʣ͕ಛ௕Ͱ͢ɻ 2013೥ͷ10݄ͷఏڙ։͔࢝Βݱࡏ※·Ͱͷੈքྦྷܭར༻ऀ਺4,900ສਓΛಥഁ※ 2018೥12݄࣌఺ ʮੈքྦྷܭར༻ऀ਺ 4,900ສਓΛಥഁͨ͠εϚϗΞϓϦʯ 5
6. ͓͠ͳ͕͖ 6 ‣Ϟϯετͷαʔόߏ੒ ‣؂ࢹγεςϜͷߏ੒ʹ͍ͭͯ ‣ࢮ׆ɾϝτϦοΫε؂ࢹ ‣ΞϥʔτରԠ ‣·ͱΊ
7. αʔόߏ੒ 7
8. αʔόߏ੒(γεςϜ) Unicorn memcached MariaDB Redis Fluetnd resque worker LoadBalancer 8
9. αʔόߏ੒(Πϯϑϥ) 9 ‣Քಇαʔόʔ਺ ‣໿1,000୆ ‣ϚϧνΫϥ΢υߏ੒ ‣ΦϯϓϨαʔόʔͱΫϥ΢υͷซ༻ ‣ࣗࣾDC 2ڌ఺ ‣ύϒϦοΫΫϥ΢υΛෳ਺
10. αʔόߏ੒ DataCenter 1 •DB •memcached DataCenter 2 •DB •memcached 10 application Cloud1 application Cloud3 application Cloud4 application Cloud2
11. αʔόߏ੒ 11 ‣Application ‣13,000 ʙ 26,000 core ‣DB ‣෺ཧϚγϯ150୆ 1ηοτ ‣ͦΕͧΕͷDCʹ഑ஔ
12. ؂ࢹγεςϜߏ੒ 12
13. ؂ࢹγεςϜ 13 ‣ࢮ׆؂ࢹ ‣ Nagios ‣ ϝτϦοΫε؂ࢹ ‣ CloudForecast ‣ Kibana + elasticsearch ‣ Grafana + InfluxDB
14. ؂ࢹγεςϜ 14 ‣ͳͥ͜ΕΒΛબΜͩͷ͔ ‣ΦϯϓϨɺΫϥ΢υͰڞ௨Ͱ࢖͑Δ ‣SNS(mixi.jp)͔Βͷࢿ࢈
15. ࢮ׆؂ࢹ 15
16. ࢮ׆؂ࢹ 16 ‣ࠔͬͨ͜ͱ ‣ର৅αʔό͕୯७ʹଟ͍ ‣؂ࢹαʔό͕ࣗ෼Λ؂ࢹͰ͖ͳ͍ ‣αʔό͕ࢮΜͩͷ͔Ϋϥ΢υͱͷ઀ ଓ͕੾Εͨͷ͔໰୊
17. ࢮ׆؂ࢹ 17 ‣֤ڌ఺ʹNagios Λߏங ‣ͦΕͧΕʹ૬ޓ؂ࢹ
18. ࢮ׆؂ࢹ 18 DC 1 DC 2 Cloud 1
19. ࢮ׆؂ࢹ 19 ‣ࠔͬͨ͜ͱ ‣Nagios ͷઃఆϑΝΠϧ(cfg)͕൥ࡶ ‣ෳ਺ͷ؂ࢹαʔόͷߋ৽͕໘౗
20. ࢮ׆؂ࢹ 20 ‣Nagios ‣ ؂ࢹઃఆͷcfg ϑΝΠϧΛYAML͔Βੜ੒ ‣ ֤ڌ఺ͷߋ৽ΛҰׅͰߦ͏πʔϧΛ಺੡ ‣ cfg ϑΝΠϧߋ৽ ‣ syntax check && nagios restart
21. ࢮ׆؂ࢹ 21
22. ࢮ׆؂ࢹ 22 DC 1 DC 2 Cloud 1
23. ࢮ׆؂ࢹ 23 ‣ࠔͬͨ͜ͱ ‣؂ࢹ߲໨ΛΧελϚΠζ͍ͨ͠
24. ࢮ׆؂ࢹ 24 ‣SNMP ͷextend ػೳΛར༻ ‣Net-SNMPͷ֦ுػೳ ‣೚ҙͷίϚϯυ݁ՌΛSNMPͰฦ͢
25. ࢮ׆؂ࢹ 25 ‣check plugin ΋͍͔ͭࣾ͘಺Ͱࣗ࡞ ‣ྫ: ‣αʔόͷuptime ΛνΣοΫ ‣filesystem ͷreadonly ΛνΣοΫ
26. ࢮ׆؂ࢹ 26 ‣ࠔͬͨ͜ͱ ‣ ෳ਺୆ͷ؂ࢹαʔό؅ཧ͸൥ࡶ ‣ ؂ࢹ͢ΔͨΊʹ༷ʑͳιϑτ΢ΣΞ͕ඞཁ ‣ libmysqlclient, snmp… ౳
27. ࢮ׆؂ࢹ 27 ‣ ͦ͜Ͱ ‣؂ࢹγεςϜͷ࡮৽ ‣ৄ͘͠͸ޙฤʹͯ
28. ϝτϦοΫε؂ࢹ 28
29. ࢮ׆؂ࢹ 29 ࠷ॳ
30. ϝτϦοΫε؂ࢹ 30 ‣CloudForecast ‣https://github.com/kazeburo/ cloudforecast ‣monitor pluginΛࣗ࡞
31. ࢮ׆؂ࢹ 31 ࣍ʹ
32. ϝτϦοΫε؂ࢹ 32 ‣Kibana + Elasticsearch ‣application αʔόͷlog ౳Λ஝ੵ ‣ूܭ΍ݕࡧͳͲ ‣1/100 ʹ samplingͰऩू
33. ϝτϦοΫε؂ࢹ Application Elasticsearch + Kibana Fluetnd 33 αϯϓϦϯά΍σʔλՃ޻
34. ࢮ׆؂ࢹ 34 ࣍ʹ
35. ϝτϦοΫε؂ࢹ 35 ‣grafana + InfluxDB ‣ ֤छσʔλΛूܭͯ͠InfluxDB ʹ஝ੵ ‣ ͦΕΒΛgrafana ͰՄࢹԽ ‣ μογϡϘʔυΛ࡞੒ ‣ Ξϥʔτ΋ઃఆ
36. ϝτϦοΫ؂ࢹ 36 ‣ͳͥ͜ΕΒΛબΜͩͷ͔ ‣ΦϯϓϨɺΫϥ΢υͰڞ௨Ͱ࢖͑Δ ‣༷ʑͳϝτϦοΫεΛ௥ՃͰ͖Δ
37. ؂ࢹΞϥʔτ 37
38. ؂ࢹΞϥʔτ 38 ‣؂ࢹͰҟৗ͕͋ͬͨ৔߹ͷΞϥʔτ ‣ϞϯετͰ͸PagerDuty Λར༻ ‣On-Call౰൪੍ ‣αʔό։ൃ & SRE Ͱϩʔςʔγϣϯ
39. ؂ࢹΞϥʔτ 39 ‣PagerDuty ‣ https://www.pagerduty.com/ ‣ ֤छ؂ࢹγεςϜͱ࿈ܞͯ͠௨஌ΛૹΕΔ ‣ ॊೈͳΤεΧϨʔγϣϯϧʔϧ͕૊ΊΔ
40. ؂ࢹΞϥʔτ 40 ‣On-Call ౰൪ ‣γεςϜͷো֐ʹඋ͑ΔͨΊͷ଴ػ ‣ೋਓҰ૊ͷ౰൪੍ ‣Ξϥʔτൃੜ࣌ʹ͸15෼ͰରԠ։࢝
41. ؂ࢹΞϥʔτ 41 Ϛωʔδϟɾࣄۀ੹೚ऀ ։ൃɾSRE ຊ೔ͷରԠ౰൪ ো֐ൃੜʂ
42. ·ͱΊ 42
43. ·ͱΊ 43 ‣ϞϯετͷΠϯϑϥͷ؂ࢹʹ͍ͭͯ ‣ࢮ׆؂ࢹ ‣ϝτϦοΫε؂ࢹ ‣ΞϥʔτରԠ
44. Thank you!