How to monitor your Symfony application

How to monitor your Symfony application

8665aad8f35b1710df79e9aef52d6daa?s=128

Alexandre Salomé

September 16, 2016
Tweet

Transcript

  1. 2.

    About me • The “Poney” guy • Architect on the

    back-office software, Auchan Retail France • 7 years of experience on Symfony • Developer since my childhood
  2. 4.

    Summary • A little theory • Overview of existing solutions

    • What to monitor • Alerting • Our solution at Auchan Retail France
  3. 9.

    Metrics: Examples • System metrics ◦ Load average ◦ RAM

    usage ◦ Disk I/O • Service metrics ◦ Number of SQL queries ◦ Cache hits and misses • Application metrics ◦ Number of registrations ◦ Page generation duration
  4. 10.

    Metrics: Aggregation • Important when querying • Different aggregations ◦

    Sum ◦ Average ◦ Max ◦ Min ◦ 90th percentile • Can be used to reduce storage size ◦ Every minute from now to 1 week ago ◦ Every 15 minutes from 1 week ago to 1 month ago ◦ Every hour from 1 month ago to 6 months ago ◦ Every day from 6 month ago to …
  5. 11.

    Metrics: Deviation Some metrics are pushed as growing numbers: •

    MySQL query count • Network transfer To get the rate, you need to compute the deviation :
  6. 13.

    Events An event is a message from an application, your

    system, or service. It’s in a text format: 2016-09-06 20:47:13 - Alexandre is preparing slides for the conference
  7. 14.

    Events: examples • Linux logs • Apache or Nginx logs

    • Symfony logs • MySQL logs • Slow query logs
  8. 15.

    Events: field extraction Parse messages with a regex : 2016-01-14

    12:34:32 boston-01: User “alice” connected to the application from IP 12.34.56.78 Get a data table: Date 2016-01-14 Time 12:34:32 Server boston-01 Event type login Username alice IP 12.34.56.78
  9. 16.

    Events: field extraction • Logstash provides a lot of built-in

    regular expressions : Example: parsing of Apache/Nginx logs grok { match => { "message" => "%{COMBINEDAPACHELOG}" } }
  10. 17.

    Metrics and Events > Comparison Metrics are good at: •

    Time series data • Consolidation over time • Mathematics • Storage size Numbers Events are good at: • Storing any message in any format • Extracting fields from messages for indexation and queries Text
  11. 19.

    Metrics storages • ++ Graphite : aggregation • + InfluxDB

    : clustering • OpenTSDB : scalable • Promotheus
  12. 20.

    Metrics from your system and services A good solution: collectd

    • Plugins for system and services metrics : CPU usage, RAM, load average, network, MySQL, Apache, AMQP, Carbon, CPU Temperature, Filesystem, Disk, IRQ, NFS, PostgreSQL, Syslog, MongoDB, Redis, File count, … • You can add custom metrics by using the Exec plugin Sends all metrics to your storage
  13. 21.

    Metrics from your system and services A complete solution: Zabbix

    • Agents to collect metrics • Web UI to get realtime alerting • Alert by Mail/SMS/Anything • Complete metrics extraction ◦ System metrics ◦ Service metrics ◦ Remote calls
  14. 22.

    Metrics buffering with StatsD • A Node.JS application to buffer

    your metrics flow • Lot of available backends • Manage different metric types ◦ Counters (+1, +3, +2) ◦ Sampling ( ◦ Gauges (200, +3, -2) • A very simple UDP protocol • Flush metrics every X seconds • Optimize performance
  15. 23.

    Metrics from your Symfony application • <3 m6web/statsd-bundle <3 •

    algatux/influxdb-bundle • https://packagist.org/search/?q=<your-solution>-bundle
  16. 24.

    Events: the so-famous ELK • ElasticSearch is the storage •

    LogStash is the log processing tool • Kibana is the dashboard
  17. 25.

    See also Events storage: • Graylog • Fluentd Awesome Sysadmin

    : https://github.com/kahun/awesome-sysadmin
  18. 29.

    A good fail • Metrics are events • 1 hour

    = 10 MB • 1 day = 200 MB • 1 week = 1.5 GB
  19. 31.

    Anything that changes can be measured • Measure anything and

    everything • 3 levels: ◦ System: the Debian/Archlinux/whatever system you are using ◦ Services: Apache, MySQL, Docker, Nginx, Redis, … ◦ Applications: your Symfony application How to Measure Anything: Finding the Value of Intangibles in Business By Douglas W. Hubbard
  20. 32.

    System Metrics • Load average • RAM • Free disk

    • IOWait • Network usage • Inodes Events • System logs
  21. 33.

    Services Metrics • MySQL ◦ Query count ◦ Cache hit/miss

    • Apache ◦ Query count ◦ Busy/idle workers • HAProxy • Redis • …. Events • Apache|Nginx access logs • Apache|Nginx error logs • MySQL logs • ElasticSearch logs • ...
  22. 34.

    Applications Metrics • Memory/duration per route • Feature usage •

    Custom metrics ◦ Registration ◦ Checkout process Events • Symfony logs • Custom logs ◦ Registration GeoIP ◦ Checkout details ◦ Feature details
  23. 38.

    Application metrics • Use application events ◦ Don’t couple your

    application code to your monitoring • M6Web/StatsdBundle provides a smart way to achieve this: m6_statsd: clients: default: Events: forum.read: increment : mysite.forum.read
  24. 39.

    Application events • Use Symfony monolog channels to route your

    messages and create powerful dashboard • Example: the deprecated channel for deprecation message
  25. 43.

    What to measure • System and service performance ◦ Load

    average ◦ Free disk • System and service errors ◦ Syslog errors ◦ HTTP codes >= 500 • User behavior ◦ Feature usage ◦ Registration count ◦ Page views
  26. 44.
  27. 45.

    A little note on alerting • It’s nice to measure,

    it’s better to be alerted • Define rules and get notified when a rule is violated • Don’t put thresolds at 95% : if your filesystem is filled at 95%, your system is probably already suffering ◦ Prefer 60% • Handling the problem before it happens avoids recovering over a crash • The alerting rules can be complex ◦ On work hours, send a mail to the team ◦ Otherwise, send an SMS to the IT manager phone ◦ If the IT manager is on holidays, send to his backup
  28. 50.

    Splunk • ELK + Cash effect = Splunk • The

    whole company can use it • On-the-fly field extraction ◦ Beautiful interface to configure them • Powerful expression language: ◦ index=apache sourcetype=frontend | timechart count BY host ◦ index=apache sourcetype=frontend host=auchan.fr | stats avg(response_time) BY path • Powerful graph constructor • Data models → Pivot tables for business
  29. 51.

    Conclusion • Track everything that changes • Instrumentalize your application

    • Track your critical business features • Create decisional dashboards • Alert at 60%, not at 95% • If you have (lot of) money, take Splunk
  30. 54.

    Photos credits • Andrew Malone - Measuring - https://flic.kr/p/aqhCH8 •

    Sebastian Schulze - SymfonyLive 2010 - https://flic.kr/p/7Ef7vx • KimManleyOrt - At the Math Grad House - https://flic.kr/p/m2UBWH • Usehung - Chemistry - https://flic.kr/p/4uT7Er • Cybjorg - Gauges - https://flic.kr/p/5r3LuJ • Shan Ambrose - alert - https://flic.kr/p/cAk4KC • Nicolas Buffler - Projet 365 - 209/365 - https://flic.kr/p/mkHfLF • Derek Bridges - Questions - https://flic.kr/p/5DeuzB • Poneys - Internet