Monitoring & Profiling at ResearchGate

Transcript

1. Monitoring and Profiling at ResearchGate @BastianHofmann
3. http://www.alexa.com/
4. 11 million users
5. 193 countries
6. ~1800 request/s
7. lots of data
8. >100 million publications
9. ~ 140 components
10. ~ 400 repositories
11. haproxy node memcache postgresql mongodb solr infinispan hbase mongodb solr community services
12. + async events, stream and batch processing
13. We need to know whats going on
14. Monitoring
15. Logging
16. Profiling
17. Real-Time
18. Many roads
20. Logging
21. Error Logs
22. Access Logs
23. Enrich with additional information
24. We turn Notices, Warnings, Errors into Exceptions
25. set_error_handler( function($errno, $msg, $file, $line) { switch ($errno) { case E_RECOVERABLE_ERROR: case E_USER_ERROR: throw new \ErrorException($msg, null, $errno, $file, $line); case E_WARNING: case E_USER_WARNING: case E_CORE_WARNING: case E_COMPILE_WARNING: throw new WarningException($msg, null, $errno, $file, $line); case E_NOTICE: case E_USER_NOTICE: throw new NoticeException($msg, null, $errno, $file, $line); case E_STRICT: throw new StrictException($msg, null, $errno, $file, $line); } return true; }); $a = []; try { $b = $a['doesNotExist']; } catch (NoticeException $e) { }
26. Log as JSON
27. Correlation with logs from other services
28. web server http service http service http service http service create unique trace_id for request user request trace_id trace_id trace_id trace_id log log log log log
30. Central log management
31. Aggregated in a central place
32. Easily full-text searchable
33. Always Log to file
34. Logstash + Elasticsearch + Kibana
35. Logstash elasticsearch webserver webserver webserver AMQP log log log logstash logstash logstash
36. Kibana http://www.elasticsearch.org/overview/kibana/
38. Monitoring
39. Graphite http://graphite.wikidot.com/
41. StatsD https://github.com/etsy/statsd/
42. webserver webserver webserver statsd statsd statsd graphite
43. What do we measure?
44. Business Metrics
45. Technical Metrics
46. Error rates
47. Database Queries
48. HTTP calls
49. Cache access
50. Rendering time
51. …
52. Response Times
53. Easiest approach: Get it out of the access log
54. 192.168.56.1 - - [09/Jul/2012:19:18:19 +0200] "GET /rg_trunk/webroot/c/af10c/ images/template/rg_logo_default.png HTTP/ 1.1" 200 882 "http://devm/rg_trunk/webroot/ directory/publications/"
55. Or measure it in your PHP process
56. Is this actually what we want to measure?
57. http://www.stevesouders.com/blog/2012/02/10/the-performance-golden-rule/
61. Navigation Timing API https://developer.mozilla.org/en-US/docs/ Navigation_timing
63. Profiling
64. XHProf with tideways.io Extension https://github.com/tideways/php-profiler-extension
65. XHGUI https://github.com/perftools/xhgui
70. Active in production for a subset of requests
71. Symfony Debug Toolbar
74. Extend it http://symfony.com/doc/current/cookbook/profiler/ data_collector.html
80. Active in production for a subset of requests
81. Alerts
83. Dashboards
88. http://twitter.com/BastianHofmann http://profiles.google.com/bashofmann http://lanyrd.com/people/BastianHofmann http://speakerdeck.com/u/bastianhofmann https://github.com/bashofmann https://www.researchgate.net/profile/Bastian_Hofmann/ mail@bastianhofmann.de