delightworks-tech-night-1

Transcript

1. ϞϯελʔετϥΠΫͷ؂ࢹπʔϧࠓੲ෺ޠ ʙ্רʙ ։ൃຊ෦ SREάϧʔϓ খ஑஌༟ גࣜձࣾϛΫγΟ

2. ࣗݾ঺հ 2

3. ࣗݾ঺հ ‣ࢯ໊ ‣খ஑஌༟ ‣ܦྺ ‣2008೥ϛΫγΟʹೖࣾ ‣SNSʮmixiʯͳͲͰΠϯϑϥɺγεςϜӡ༻ ‣ͦͷޙɺϞϯελʔετϥΠΫΛ͸͡Ίͱͨ͠ήʔϜΞϓϦͷӡ༻ ‣Ϟϯετ֤छαΠτɺࣾ಺Ͱར༻͢ΔαʔϏεӡ༻πʔϧɺ෺ཧΠ ϯϑϥ؀ڥͷӡ༻౳ʑɺ෯޿͘ैࣄ ‣։ൃຊ෦

   SRE άϧʔϓॴଐ 3

4. ϞϯελʔετϥΠΫ 4

5. ϞϯελʔετϥΠΫ ࣗ෼ͷϞϯελʔΛҾͬுͬͯ஄͖ɺఢͷϞϯελʔʹ౰ͯͯ౗͍ͯ͘͠ͱ͍͏ɺεϚʔτϑΥϯͷಛੑΛ׆༻ͨ͠ɺ ୭Ͱ΋؆୯ʹָ͠ΊΔΞΫγϣϯRPGͰ͢ɻήʔϜ͸λʔϯ੍Λͱ͓ͬͯΓɺ Ұॹʹ͍Δ༑ͩͪͱ࠷େ4ਓ·Ͱಉ࣌ʹ༡΂ΔڠྗϓϨΠʢϚϧνϓϨΠʣ͕ಛ௕Ͱ͢ɻ 2013೥ͷ10݄ͷఏڙ։͔࢝Βݱࡏ※·Ͱͷੈքྦྷܭར༻ऀ਺4,900ສਓΛಥഁ※ 2018೥12݄࣌఺ ʮੈքྦྷܭར༻ऀ਺ 4,900ສਓΛಥഁͨ͠εϚϗΞϓϦʯ 5

6. ͓͠ͳ͕͖ 6 ‣Ϟϯετͷαʔόߏ੒ ‣؂ࢹγεςϜͷߏ੒ʹ͍ͭͯ ‣ࢮ׆ɾϝτϦοΫε؂ࢹ ‣ΞϥʔτରԠ ‣·ͱΊ

7. αʔόߏ੒ 7

8. αʔόߏ੒(γεςϜ) Unicorn memcached MariaDB Redis Fluetnd resque worker LoadBalancer 8

9. αʔόߏ੒(Πϯϑϥ) 9 ‣Քಇαʔόʔ਺ ‣໿1,000୆ ‣ϚϧνΫϥ΢υߏ੒ ‣ΦϯϓϨαʔόʔͱΫϥ΢υͷซ༻ ‣ࣗࣾDC 2ڌ఺ ‣ύϒϦοΫΫϥ΢υΛෳ਺

10. αʔόߏ੒ DataCenter 1 •DB •memcached DataCenter 2 •DB •memcached 10

    application Cloud1 application Cloud3 application Cloud4 application Cloud2

11. αʔόߏ੒ 11 ‣Application ‣13,000 ʙ 26,000 core ‣DB ‣෺ཧϚγϯ150୆ 1ηοτ

    ‣ͦΕͧΕͷDCʹ഑ஔ

12. ؂ࢹγεςϜߏ੒ 12

13. ؂ࢹγεςϜ 13 ‣ࢮ׆؂ࢹ ‣ Nagios ‣ ϝτϦοΫε؂ࢹ ‣ CloudForecast ‣

    Kibana + elasticsearch ‣ Grafana + InfluxDB

14. ؂ࢹγεςϜ 14 ‣ͳͥ͜ΕΒΛબΜͩͷ͔ ‣ΦϯϓϨɺΫϥ΢υͰڞ௨Ͱ࢖͑Δ ‣SNS(mixi.jp)͔Βͷࢿ࢈

15. ࢮ׆؂ࢹ 15

16. ࢮ׆؂ࢹ 16 ‣ࠔͬͨ͜ͱ ‣ର৅αʔό͕୯७ʹଟ͍ ‣؂ࢹαʔό͕ࣗ෼Λ؂ࢹͰ͖ͳ͍ ‣αʔό͕ࢮΜͩͷ͔Ϋϥ΢υͱͷ઀ ଓ͕੾Εͨͷ͔໰୊

17. ࢮ׆؂ࢹ 17 ‣֤ڌ఺ʹNagios Λߏங ‣ͦΕͧΕʹ૬ޓ؂ࢹ

18. ࢮ׆؂ࢹ 18 DC 1 DC 2 Cloud 1

19. ࢮ׆؂ࢹ 19 ‣ࠔͬͨ͜ͱ ‣Nagios ͷઃఆϑΝΠϧ(cfg)͕൥ࡶ ‣ෳ਺ͷ؂ࢹαʔόͷߋ৽͕໘౗

20. ࢮ׆؂ࢹ 20 ‣Nagios ‣ ؂ࢹઃఆͷcfg ϑΝΠϧΛYAML͔Βੜ੒ ‣ ֤ڌ఺ͷߋ৽ΛҰׅͰߦ͏πʔϧΛ಺੡ ‣ cfg

    ϑΝΠϧߋ৽ ‣ syntax check && nagios restart

21. ࢮ׆؂ࢹ 21

22. ࢮ׆؂ࢹ 22 DC 1 DC 2 Cloud 1

23. ࢮ׆؂ࢹ 23 ‣ࠔͬͨ͜ͱ ‣؂ࢹ߲໨ΛΧελϚΠζ͍ͨ͠

24. ࢮ׆؂ࢹ 24 ‣SNMP ͷextend ػೳΛར༻ ‣Net-SNMPͷ֦ுػೳ ‣೚ҙͷίϚϯυ݁ՌΛSNMPͰฦ͢

25. ࢮ׆؂ࢹ 25 ‣check plugin ΋͍͔ͭࣾ͘಺Ͱࣗ࡞ ‣ྫ: ‣αʔόͷuptime ΛνΣοΫ ‣filesystem ͷreadonly

    ΛνΣοΫ

26. ࢮ׆؂ࢹ 26 ‣ࠔͬͨ͜ͱ ‣ ෳ਺୆ͷ؂ࢹαʔό؅ཧ͸൥ࡶ ‣ ؂ࢹ͢ΔͨΊʹ༷ʑͳιϑτ΢ΣΞ͕ඞཁ ‣ libmysqlclient, snmp…

    ౳

27. ࢮ׆؂ࢹ 27 ‣ ͦ͜Ͱ ‣؂ࢹγεςϜͷ࡮৽ ‣ৄ͘͠͸ޙฤʹͯ

28. ϝτϦοΫε؂ࢹ 28

29. ࢮ׆؂ࢹ 29 ࠷ॳ

30. ϝτϦοΫε؂ࢹ 30 ‣CloudForecast ‣https://github.com/kazeburo/ cloudforecast ‣monitor pluginΛࣗ࡞

31. ࢮ׆؂ࢹ 31 ࣍ʹ

32. ϝτϦοΫε؂ࢹ 32 ‣Kibana + Elasticsearch ‣application αʔόͷlog ౳Λ஝ੵ ‣ूܭ΍ݕࡧͳͲ ‣1/100

    ʹ samplingͰऩू

33. ϝτϦοΫε؂ࢹ Application Elasticsearch + Kibana Fluetnd 33 αϯϓϦϯά΍σʔλՃ޻

34. ࢮ׆؂ࢹ 34 ࣍ʹ

35. ϝτϦοΫε؂ࢹ 35 ‣grafana + InfluxDB ‣ ֤छσʔλΛूܭͯ͠InfluxDB ʹ஝ੵ ‣ ͦΕΒΛgrafana

    ͰՄࢹԽ ‣ μογϡϘʔυΛ࡞੒ ‣ Ξϥʔτ΋ઃఆ

36. ϝτϦοΫ؂ࢹ 36 ‣ͳͥ͜ΕΒΛબΜͩͷ͔ ‣ΦϯϓϨɺΫϥ΢υͰڞ௨Ͱ࢖͑Δ ‣༷ʑͳϝτϦοΫεΛ௥ՃͰ͖Δ

37. ؂ࢹΞϥʔτ 37

38. ؂ࢹΞϥʔτ 38 ‣؂ࢹͰҟৗ͕͋ͬͨ৔߹ͷΞϥʔτ ‣ϞϯετͰ͸PagerDuty Λར༻ ‣On-Call౰൪੍ ‣αʔό։ൃ & SRE Ͱϩʔςʔγϣϯ

39. ؂ࢹΞϥʔτ 39 ‣PagerDuty ‣ https://www.pagerduty.com/ ‣ ֤छ؂ࢹγεςϜͱ࿈ܞͯ͠௨஌ΛૹΕΔ ‣ ॊೈͳΤεΧϨʔγϣϯϧʔϧ͕૊ΊΔ

40. ؂ࢹΞϥʔτ 40 ‣On-Call ౰൪ ‣γεςϜͷো֐ʹඋ͑ΔͨΊͷ଴ػ ‣ೋਓҰ૊ͷ౰൪੍ ‣Ξϥʔτൃੜ࣌ʹ͸15෼ͰରԠ։࢝

41. ؂ࢹΞϥʔτ 41 Ϛωʔδϟɾࣄۀ੹೚ऀ ։ൃɾSRE ຊ೔ͷରԠ౰൪ ো֐ൃੜʂ

42. ·ͱΊ 42

43. ·ͱΊ 43 ‣ϞϯετͷΠϯϑϥͷ؂ࢹʹ͍ͭͯ ‣ࢮ׆؂ࢹ ‣ϝτϦοΫε؂ࢹ ‣ΞϥʔτରԠ

44. Thank you!