Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
Monitoring & Profiling at ResearchGate
Search
Sponsored
·
Your Podcast. Everywhere. Effortlessly.
Share. Educate. Inspire. Entertain. You do you. We'll handle the rest.
→
Bastian Hofmann
November 30, 2016
Programming
280
2
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
Monitoring & Profiling at ResearchGate
Bastian Hofmann
November 30, 2016
Transcript
88 slides
Monitoring and Profiling at ResearchGate @BastianHofmann
None
http://www.alexa.com/
11 million users
193 countries
~1800 request/s
lots of data
>100 million publications
~ 140 components
~ 400 repositories
haproxy node memcache postgresql mongodb solr infinispan hbase mongodb solr
community services
+ async events, stream and batch processing
We need to know whats going on
Monitoring
Logging
Profiling
Real-Time
Many roads
None
Logging
Error Logs
Access Logs
Enrich with additional information
We turn Notices, Warnings, Errors into Exceptions
set_error_handler( function($errno, $msg, $file, $line) { switch ($errno) { case
E_RECOVERABLE_ERROR: case E_USER_ERROR: throw new \ErrorException($msg, null, $errno, $file, $line); case E_WARNING: case E_USER_WARNING: case E_CORE_WARNING: case E_COMPILE_WARNING: throw new WarningException($msg, null, $errno, $file, $line); case E_NOTICE: case E_USER_NOTICE: throw new NoticeException($msg, null, $errno, $file, $line); case E_STRICT: throw new StrictException($msg, null, $errno, $file, $line); } return true; }); $a = []; try { $b = $a['doesNotExist']; } catch (NoticeException $e) { }
Log as JSON
Correlation with logs from other services
web server http service http service http service http service
create unique trace_id for request user request trace_id trace_id trace_id trace_id log log log log log
None
Central log management
Aggregated in a central place
Easily full-text searchable
Always Log to file
Logstash + Elasticsearch + Kibana
Logstash elasticsearch webserver webserver webserver AMQP log log log logstash
logstash logstash
Kibana http://www.elasticsearch.org/overview/kibana/
None
Monitoring
Graphite http://graphite.wikidot.com/
None
StatsD https://github.com/etsy/statsd/
webserver webserver webserver statsd statsd statsd graphite
What do we measure?
Business Metrics
Technical Metrics
Error rates
Database Queries
HTTP calls
Cache access
Rendering time
…
Response Times
Easiest approach: Get it out of the access log
192.168.56.1 - - [09/Jul/2012:19:18:19 +0200] "GET /rg_trunk/webroot/c/af10c/ images/template/rg_logo_default.png HTTP/ 1.1"
200 882 "http://devm/rg_trunk/webroot/ directory/publications/"
Or measure it in your PHP process
Is this actually what we want to measure?
http://www.stevesouders.com/blog/2012/02/10/the-performance-golden-rule/
None
None
None
Navigation Timing API https://developer.mozilla.org/en-US/docs/ Navigation_timing
None
Profiling
XHProf with tideways.io Extension https://github.com/tideways/php-profiler-extension
XHGUI https://github.com/perftools/xhgui
None
None
None
None
Active in production for a subset of requests
Symfony Debug Toolbar
None
None
Extend it http://symfony.com/doc/current/cookbook/profiler/ data_collector.html
None
None
None
None
None
Active in production for a subset of requests
Alerts
None
Dashboards
None
None
None
None
http://twitter.com/BastianHofmann http://profiles.google.com/bashofmann http://lanyrd.com/people/BastianHofmann http://speakerdeck.com/u/bastianhofmann https://github.com/bashofmann https://www.researchgate.net/profile/Bastian_Hofmann/
[email protected]
More Decks by Bastian Hofmann
See All by Bastian Hofmann
Monitoring in Kubernetes with Prometheus and Grafana
bastianhofmann
0
390
Creating a fast Kubernetes Development Workflow
bastianhofmann
0
160
Highly available cross-region deployments with Kubernetes
bastianhofmann
1
190
From source to Kubernetes in 30 minutes
bastianhofmann
0
210
Introduction to Kubernetes
bastianhofmann
1
140
CI/CD with Kubernetes
bastianhofmann
0
260
Creating a fast Kubernetes Development Workflow
bastianhofmann
1
300
Deploying your first Micro-Service application to Kubernetes
bastianhofmann
2
220
Creating a fast Kubernetes Development Workflow
bastianhofmann
0
290
Other Decks in Programming
See All in Programming
Claude Codeのおかげでやりたいことがやれるようになったんだよォー!
sms_tech
0
190
クラウドエージェントは、 何がそんなに便利なのか / what makes cloud agents so useful
rkaga
5
980
[DroidKnights 2026] Workflow AX: From Auto-Documented Policies to E2E Testing
raeminkang
0
610
プロジェクトコードをキレイキレイしよう!暗黙的な参照を読み解く静的解析
ostk0069
0
220
protojson における文字化けの扱い
kazuki1023
0
460
RaspberryPi 5で軽くはじめるk3sとtailscaleを用いたホームサーバー構築 (1)
nasum
0
430
Debian JP Projectのサーバー構成
uwabami
1
170
SODA - FACT BOOK(EN)
sodainc
1
280
AI時代のFlutter品質設計 — アプリ品質を守るQuality Layers
omatty198
0
390
正解の無い問題を 「できるだけ上手く」解く ヒューリスティックで感性を磨こう。〜現実課題をプログラミングで解きあうコミュニティについて〜
allegrogiken
0
110
Enforcing Your Architecture with Unit Tests in the AI Era (Swift Connection X)
steliosf
PRO
1
190
AppFunctionsの内部実装を紐解く
hiroaki404
0
120
Featured
See All Featured
First, design no harm
axbom
PRO
2
1.4k
Typedesign – Prime Four
hannesfritz
42
3.2k
jQuery: Nuts, Bolts and Bling
dougneiner
66
8.7k
Producing Creativity
orderedlist
PRO
348
41k
Claude Code どこまでも/ Claude Code Everywhere
nwiizo
68
61k
Darren the Foodie - Storyboard
khoart
PRO
5
4.1k
Ten Tips & Tricks for a 🌱 transition
stuffmc
1
290
How Software Deployment tools have changed in the past 20 years
geshan
3
35k
Optimizing for Happiness
mojombo
378
72k
Have SEOs Ruined the Internet? - User Awareness of SEO in 2025
akashhashmi
0
560
A Soul's Torment
seathinner
9
3.9k
Learning to Love Humans: Emotional Interface Design
aarron
275
41k