Instrumenting Application Stack in Dynamically Scaling Environment

DMG212 AWS re:Invent 2013

Transcript

1. Instrumenting your Application Stack in a Dynamically Scaling Environment Mike Fiedler, Datadog November 15, 2013
2. The Past • Server uptime • Intertwined applications • SPOFs
3. The Present • SOA • Scale or Fail • Data-driven decisions
4. Modern application stack User LB Web Data
5. Failure is inevitable
6. Design to tolerate failure
7. Things that produce metrics business/service Services • applicatio n • OS • instances API-based • S3, S*S • RDS, RedShift • etc Users • RUM
8. The Desire
9. CPU comparison (hypervisor)
10. CPU comparison (Agent)
11. My Stack • HAProxy • Apache HTTP • Bottle.py – Custom python webapp • MongoDB • EC2 • Chef • Datadog • Users
12. Tracking Database operations
13. Moving up the stack…
14. Even higher…
15. Using metadata for more flavor
16. Replace an instance, keep the metrics
17. Enter…
18. Aggregating numbers
19. Comparing numbers
20. My App from bottle import route import pymongo import json db = pymongo.Connection('mongodb://localhost').test @route('/insert/:name') def insert(name): doc = {'name': name} db.words.update(doc, {"$inc":{"count": 1}}, upsert=True) return json.dumps(doc, default=default)
21. Add 2 lines and you win! from bottle import route import pymongo import json from statsd import statsd db = pymongo.Connection('mongodb://localhost').test @route('/insert/:name') @statsd.increment('fullstack.insert') def insert(name): doc = {'name': name} db.words.update(doc, {"$inc":{"count": 1}}, upsert=True) return json.dumps(doc, default=default)
22. Profit!
23. Correlation
24. Don’t be caught unaware
26. Questions Time!
27. We are sincerely eager to hear your feedback on this presentation and on re:Invent. Please fill out an evaluation form when you have a chance.