delightworks-tech-night-1

 delightworks-tech-night-1

B276b9088550bc522536ab9d471aeebe?s=128

Tomohiro Koike

March 07, 2019
Tweet

Transcript

  1. ϞϯελʔετϥΠΫͷ؂ࢹπʔϧࠓੲ෺ޠ ʙ্רʙ ։ൃຊ෦ SREάϧʔϓ খ஑஌༟ גࣜձࣾϛΫγΟ

  2. ࣗݾ঺հ 2

  3. ࣗݾ঺հ ‣ࢯ໊ ‣খ஑஌༟ ‣ܦྺ ‣2008೥ϛΫγΟʹೖࣾ ‣SNSʮmixiʯͳͲͰΠϯϑϥɺγεςϜӡ༻ ‣ͦͷޙɺϞϯελʔετϥΠΫΛ͸͡Ίͱͨ͠ήʔϜΞϓϦͷӡ༻ ‣Ϟϯετ֤छαΠτɺࣾ಺Ͱར༻͢ΔαʔϏεӡ༻πʔϧɺ෺ཧΠ ϯϑϥ؀ڥͷӡ༻౳ʑɺ෯޿͘ैࣄ ‣։ൃຊ෦

    SRE άϧʔϓॴଐ 3
  4. ϞϯελʔετϥΠΫ 4

  5. ϞϯελʔετϥΠΫ ࣗ෼ͷϞϯελʔΛҾͬுͬͯ஄͖ɺఢͷϞϯελʔʹ౰ͯͯ౗͍ͯ͘͠ͱ͍͏ɺεϚʔτϑΥϯͷಛੑΛ׆༻ͨ͠ɺ ୭Ͱ΋؆୯ʹָ͠ΊΔΞΫγϣϯRPGͰ͢ɻήʔϜ͸λʔϯ੍Λͱ͓ͬͯΓɺ Ұॹʹ͍Δ༑ͩͪͱ࠷େ4ਓ·Ͱಉ࣌ʹ༡΂ΔڠྗϓϨΠʢϚϧνϓϨΠʣ͕ಛ௕Ͱ͢ɻ 2013೥ͷ10݄ͷఏڙ։͔࢝Βݱࡏ※·Ͱͷੈքྦྷܭར༻ऀ਺4,900ສਓΛಥഁ※ 2018೥12݄࣌఺ ʮੈքྦྷܭར༻ऀ਺ 4,900ສਓΛಥഁͨ͠εϚϗΞϓϦʯ 5

  6. ͓͠ͳ͕͖ 6 ‣Ϟϯετͷαʔόߏ੒ ‣؂ࢹγεςϜͷߏ੒ʹ͍ͭͯ ‣ࢮ׆ɾϝτϦοΫε؂ࢹ ‣ΞϥʔτରԠ ‣·ͱΊ

  7. αʔόߏ੒ 7

  8. αʔόߏ੒(γεςϜ) Unicorn memcached MariaDB Redis Fluetnd resque worker LoadBalancer 8

  9. αʔόߏ੒(Πϯϑϥ) 9 ‣Քಇαʔόʔ਺ ‣໿1,000୆ ‣ϚϧνΫϥ΢υߏ੒ ‣ΦϯϓϨαʔόʔͱΫϥ΢υͷซ༻ ‣ࣗࣾDC 2ڌ఺ ‣ύϒϦοΫΫϥ΢υΛෳ਺

  10. αʔόߏ੒ DataCenter 1 •DB •memcached DataCenter 2 •DB •memcached 10

    application Cloud1 application Cloud3 application Cloud4 application Cloud2
  11. αʔόߏ੒ 11 ‣Application ‣13,000 ʙ 26,000 core ‣DB ‣෺ཧϚγϯ150୆ 1ηοτ

    ‣ͦΕͧΕͷDCʹ഑ஔ
  12. ؂ࢹγεςϜߏ੒ 12

  13. ؂ࢹγεςϜ 13 ‣ࢮ׆؂ࢹ ‣ Nagios ‣ ϝτϦοΫε؂ࢹ ‣ CloudForecast ‣

    Kibana + elasticsearch ‣ Grafana + InfluxDB
  14. ؂ࢹγεςϜ 14 ‣ͳͥ͜ΕΒΛબΜͩͷ͔ ‣ΦϯϓϨɺΫϥ΢υͰڞ௨Ͱ࢖͑Δ ‣SNS(mixi.jp)͔Βͷࢿ࢈

  15. ࢮ׆؂ࢹ 15

  16. ࢮ׆؂ࢹ 16 ‣ࠔͬͨ͜ͱ ‣ର৅αʔό͕୯७ʹଟ͍ ‣؂ࢹαʔό͕ࣗ෼Λ؂ࢹͰ͖ͳ͍ ‣αʔό͕ࢮΜͩͷ͔Ϋϥ΢υͱͷ઀ ଓ͕੾Εͨͷ͔໰୊

  17. ࢮ׆؂ࢹ 17 ‣֤ڌ఺ʹNagios Λߏங ‣ͦΕͧΕʹ૬ޓ؂ࢹ

  18. ࢮ׆؂ࢹ 18 DC 1 DC 2 Cloud 1

  19. ࢮ׆؂ࢹ 19 ‣ࠔͬͨ͜ͱ ‣Nagios ͷઃఆϑΝΠϧ(cfg)͕൥ࡶ ‣ෳ਺ͷ؂ࢹαʔόͷߋ৽͕໘౗

  20. ࢮ׆؂ࢹ 20 ‣Nagios ‣ ؂ࢹઃఆͷcfg ϑΝΠϧΛYAML͔Βੜ੒ ‣ ֤ڌ఺ͷߋ৽ΛҰׅͰߦ͏πʔϧΛ಺੡ ‣ cfg

    ϑΝΠϧߋ৽ ‣ syntax check && nagios restart
  21. ࢮ׆؂ࢹ 21

  22. ࢮ׆؂ࢹ 22 DC 1 DC 2 Cloud 1

  23. ࢮ׆؂ࢹ 23 ‣ࠔͬͨ͜ͱ ‣؂ࢹ߲໨ΛΧελϚΠζ͍ͨ͠

  24. ࢮ׆؂ࢹ 24 ‣SNMP ͷextend ػೳΛར༻ ‣Net-SNMPͷ֦ுػೳ ‣೚ҙͷίϚϯυ݁ՌΛSNMPͰฦ͢

  25. ࢮ׆؂ࢹ 25 ‣check plugin ΋͍͔ͭࣾ͘಺Ͱࣗ࡞ ‣ྫ: ‣αʔόͷuptime ΛνΣοΫ ‣filesystem ͷreadonly

    ΛνΣοΫ
  26. ࢮ׆؂ࢹ 26 ‣ࠔͬͨ͜ͱ ‣ ෳ਺୆ͷ؂ࢹαʔό؅ཧ͸൥ࡶ ‣ ؂ࢹ͢ΔͨΊʹ༷ʑͳιϑτ΢ΣΞ͕ඞཁ ‣ libmysqlclient, snmp…

  27. ࢮ׆؂ࢹ 27 ‣ ͦ͜Ͱ ‣؂ࢹγεςϜͷ࡮৽ ‣ৄ͘͠͸ޙฤʹͯ

  28. ϝτϦοΫε؂ࢹ 28

  29. ࢮ׆؂ࢹ 29 ࠷ॳ

  30. ϝτϦοΫε؂ࢹ 30 ‣CloudForecast ‣https://github.com/kazeburo/ cloudforecast ‣monitor pluginΛࣗ࡞

  31. ࢮ׆؂ࢹ 31 ࣍ʹ

  32. ϝτϦοΫε؂ࢹ 32 ‣Kibana + Elasticsearch ‣application αʔόͷlog ౳Λ஝ੵ ‣ूܭ΍ݕࡧͳͲ ‣1/100

    ʹ samplingͰऩू
  33. ϝτϦοΫε؂ࢹ Application Elasticsearch + Kibana Fluetnd 33 αϯϓϦϯά΍σʔλՃ޻

  34. ࢮ׆؂ࢹ 34 ࣍ʹ

  35. ϝτϦοΫε؂ࢹ 35 ‣grafana + InfluxDB ‣ ֤छσʔλΛूܭͯ͠InfluxDB ʹ஝ੵ ‣ ͦΕΒΛgrafana

    ͰՄࢹԽ ‣ μογϡϘʔυΛ࡞੒ ‣ Ξϥʔτ΋ઃఆ
  36. ϝτϦοΫ؂ࢹ 36 ‣ͳͥ͜ΕΒΛબΜͩͷ͔ ‣ΦϯϓϨɺΫϥ΢υͰڞ௨Ͱ࢖͑Δ ‣༷ʑͳϝτϦοΫεΛ௥ՃͰ͖Δ

  37. ؂ࢹΞϥʔτ 37

  38. ؂ࢹΞϥʔτ 38 ‣؂ࢹͰҟৗ͕͋ͬͨ৔߹ͷΞϥʔτ ‣ϞϯετͰ͸PagerDuty Λར༻ ‣On-Call౰൪੍ ‣αʔό։ൃ & SRE Ͱϩʔςʔγϣϯ

  39. ؂ࢹΞϥʔτ 39 ‣PagerDuty ‣ https://www.pagerduty.com/ ‣ ֤छ؂ࢹγεςϜͱ࿈ܞͯ͠௨஌ΛૹΕΔ ‣ ॊೈͳΤεΧϨʔγϣϯϧʔϧ͕૊ΊΔ

  40. ؂ࢹΞϥʔτ 40 ‣On-Call ౰൪ ‣γεςϜͷো֐ʹඋ͑ΔͨΊͷ଴ػ ‣ೋਓҰ૊ͷ౰൪੍ ‣Ξϥʔτൃੜ࣌ʹ͸15෼ͰରԠ։࢝

  41. ؂ࢹΞϥʔτ 41 Ϛωʔδϟɾࣄۀ੹೚ऀ ։ൃɾSRE ຊ೔ͷରԠ౰൪ ো֐ൃੜʂ

  42. ·ͱΊ 42

  43. ·ͱΊ 43 ‣ϞϯετͷΠϯϑϥͷ؂ࢹʹ͍ͭͯ ‣ࢮ׆؂ࢹ ‣ϝτϦοΫε؂ࢹ ‣ΞϥʔτରԠ

  44. Thank you!