Application performance management with
PacketBeat, Elasticsearch and Kibana
Tudor Golubenco (@tudor_g)
Slide 2
Slide 2 text
What is PacketBeat?
¯\_(ツ)_/¯
Slide 3
Slide 3 text
What is PacketBeat
• “Open Source Application Monitoring”
• “Monitoring & Troubleshooting for Distributed Applications”
• “Distributed Wireshark with a lot more analytics features”
• “Application Performance Management”
Slide 4
Slide 4 text
How it works?
ಠ_ಠ
Slide 5
Slide 5 text
How it works
• Captures the wire traffic
• Follows TCP streams, decodes HTTP, MySQL, PgSQL, REDIS,
Thrift-RPC
• Looks for requests, waits for the matching response
• Records response time, URLs, response codes, etc
Slide 6
Slide 6 text
Show me!
(￣^￣)
Slide 7
Slide 7 text
No content
Slide 8
Slide 8 text
What do we do with the data?
¯\(°_o)/¯
Slide 9
Slide 9 text
The traditional way
• Decide what metrics you need (requests per second for each
server, response time percentiles, etc.)
• Write code to extract these metrics, store them in a DB
• Store the transactions in a DB
• Drilling down is difficult
• Features like “Top 10 method with errors” are difficult to implement
Slide 10
Slide 10 text
PacketBeat + ELK
Slide 11
Slide 11 text
Why ELK?
• Already proven to scale and perform for logs
• Clear and simple flow for the data
• You don’t have to pre-create the metrics
• Ad-hoc troubleshooting and analytics by using Kibana
• Drilling down to the problematic transactions is trivial
• Top N features are trivial
• Slicing by different dimensions is easy
Slide 12
Slide 12 text
Show me!
(￣^￣)
Slide 13
Slide 13 text
No content
Slide 14
Slide 14 text
No content
Slide 15
Slide 15 text
No content
Slide 16
Slide 16 text
No content
Slide 17
Slide 17 text
Pros of wire data
• Captures a lot of things that other approaches miss
• No changes to the code or to the monitored application
• Minimal knowledge about the monitored app is required
• No latency overhead
• When using tap points, zero CPU/memory overhead on the app
servers
Slide 18
Slide 18 text
Cons of wire data
• There can be, like, tons of data
• Compared to log processing, larger CPU requirements
• Privacy concerns
• Doesn’t work for encrypted protocols
• Doesn’t work for “in-house” protocols
Slide 19
Slide 19 text
Next steps
( ͡ ° ͜ ʖ ͡ °)
Slide 20
Slide 20 text
More protocols
• Available:
• HTTP
• MySQL
• PostgreSQL
• REDIS
• Thrift-RPC
• Soon (tm):
• DNS
• Memcache
• MongoDB, RethinkDB
• Oracle, MSSQL
• XMLRPC / JSONRPC
• Your suggestions?
Slide 21
Slide 21 text
Sampling
• Wire data can be huge
• Troubleshooting convenience vs hardware requirements
• Sample by:
• protocol (e.g. store all MySQL requests, sample REDIS 1/10)
• method (e.g. store all PUTs requests, sample GETs 1/10)
• status code (e.g. store all errors, sample successes)
• response time (e.g. store all slow transactions)
Slide 22
Slide 22 text
String obfuscation
• Replace: select * from users where username=“Tudor” and id=3
• With: select * from users where username=S8 and id=N3
• Makes TopN charts better
• “The Mature Optimisation Handbook” - Carlos Bueno
Slide 23
Slide 23 text
Bonito
• Our own UI
• Similar to Kibana, but focused more on app performance
• Will be a Kibana 4 plugin
Slide 24
Slide 24 text
No content
Slide 25
Slide 25 text
Deploying PacketBeat
(´ ▽｀).。ｏὑ
Slide 26
Slide 26 text
Deploying
• Getting started guide
• packetbeat-deploy
• ansible roles for Packetbeat, Elasticsearch, Logstash, Redis,
Kibana
• supports multiple ES nodes or all-in-one server
• ansible-playbook -i hosts site.yml
Slide 27
Slide 27 text
Thanks!
(ƅ˜ƅ)/
Slide 28
Slide 28 text
Keep in touch
• Twitter: @packetbeat or @tudor_g
• www: packetbeat.com
• github.com/packetbeat/packetbeat