Beyond grep – EuroPython Edition

Beyond grep – EuroPython Edition

Knowing that your application is up and running is great. However in order to make informed decisions about the future, you also need to know in what state your application currently is and how its state is developing over time.

This talk combines two topics that are usually discussed separately. However I do believe that they have a lot of overlap and ultimately a similar goal: giving you vital insights about your system in production.

We'll have a look at their commonalities, differences, popular tools, and how to apply everything in your own systems while avoiding some common pitfalls.

174e7b0ff60963f821d0b9a4f1a3ef52?s=128

Hynek Schlawack

July 21, 2015
Tweet

Transcript

  1. 4.
  2. 5.
  3. 6.
  4. 7.
  5. 8.
  6. 9.
  7. 10.
  8. 17.
  9. 18.
  10. 19.
  11. 20.
  12. 21.
  13. 22.
  14. 23.
  15. 24.
  16. 25.
  17. 26.
  18. 36.
  19. 37.

    Raven-Python Integrations: • logging • Django • WSGI • …9

    others Transports: • gevent • aiohttp • Twisted • …8 others
  20. 38.

    Vanilla from raven import Client client = Client("https://yoursentry") try: 1

    / 0 except ZeroDivisionError: client.captureException()
  21. 39.

    Vanilla from raven import Client client = Client("https://yoursentry") try: 1

    / 0 except ZeroDivisionError: client.captureException()
  22. 40.

    Vanilla from raven import Client client = Client("https://yoursentry") try: 1

    / 0 except ZeroDivisionError: client.captureException()
  23. 43.
  24. 45.
  25. 46.
  26. 47.
  27. 53.

    System Metrics vs App Metrics • load • network traffic

    • I/O • … • counters • timers
  28. 54.

    System Metrics vs App Metrics • load • network traffic

    • I/O • … • counters • timers • gauges
  29. 55.

    System Metrics vs App Metrics • load • network traffic

    • I/O • … • counters • timers • gauges • …
  30. 58.
  31. 61.

    Math • # reqs / s? • worst 0.01% ⟨req

    time⟩? • don’t try this alone!
  32. 66.
  33. 69.
  34. 77.

    Approaches 1. external aggregation: StatsD, Riemann + no state, simple

    – no direct introspection 2. aggregate in-app, deliver to DB
  35. 78.

    Approaches 1. external aggregation: StatsD, Riemann + no state, simple

    – no direct introspection 2. aggregate in-app, deliver to DB + in-app dashboard, useful in dev
  36. 79.

    Approaches 1. external aggregation: StatsD, Riemann + no state, simple

    – no direct introspection 2. aggregate in-app, deliver to DB + in-app dashboard, useful in dev – state w/i app
  37. 80.
  38. 85.

    Scales from greplin import scales from greplin.scales.meter import MeterStat STATS

    = scales.collection( "/Resource", MeterStat("reqs"), scales.PmfStat("request_time") )
  39. 86.

    Scales from greplin import scales from greplin.scales.meter import MeterStat STATS

    = scales.collection( "/Resource", MeterStat("reqs"), scales.PmfStat("request_time") )
  40. 87.
  41. 93.

    Dashboard Scales … "request_time": { "count": 567315293, "99percentile": 0.10978688716888428, "75percentile":

    0.013181567192077637, "min": 0.0002448558807373047, "max": 30.134822130203247, "98percentile": 0.08934824466705339, "95percentile": 0.027234303951263434, "median": 0.009176492691040039, "999percentile": 0.14235656142234793, "stddev": 0.01676855570363413, "mean": 0.013247184020535955 }, …
  42. 97.
  43. 98.
  44. 99.
  45. 100.
  46. 101.
  47. 103.
  48. 104.
  49. 105.
  50. 109.
  51. 110.
  52. 111.
  53. 112.
  54. 113.
  55. 114.
  56. 117.
  57. 122.

    Original Logger BoundLogger Processor 1 Processor n Return Value Return

    Value bind values log.bind(key=value) Context log events log.info(event, another_key=another_value) + structlog
  58. 127.

    >>> from structlog import get_logger >>> log = get_logger() >>>

    log = log.bind(user='anonymous', some_key=23) >>> log = log.bind(user='hynek', another_key=42) >>> log.info('user.logged_in', happy=True) some_key=23 user='hynek' another_key=42 happy=True event='user.logged_in'
  59. 128.

    >>> from structlog import get_logger >>> log = get_logger() >>>

    log = log.bind(user='anonymous', some_key=23) >>> log = log.bind(user='hynek', another_key=42) >>> log.info('user.logged_in', happy=True) some_key=23 user='hynek' another_key=42 happy=True event='user.logged_in'
  60. 129.

    >>> from structlog import get_logger >>> log = get_logger() >>>

    log = log.bind(user='anonymous', some_key=23) >>> log = log.bind(user='hynek', another_key=42) >>> log.info('user.logged_in', happy=True) some_key=23 user='hynek' another_key=42 happy=True event='user.logged_in'
  61. 130.

    >>> from structlog import get_logger >>> log = get_logger() >>>

    log = log.bind(user='anonymous', some_key=23) >>> log = log.bind(user='hynek', another_key=42) >>> log.info('user.logged_in', happy=True) some_key=23 user='hynek' another_key=42 happy=True event='user.logged_in'
  62. 131.

    >>> from structlog import get_logger >>> log = get_logger() >>>

    log = log.bind(user='anonymous', some_key=23) >>> log = log.bind(user='hynek', another_key=42) >>> log.info('user.logged_in', happy=True) some_key=23 user='hynek' another_key=42 happy=True event='user.logged_in'
  63. 135.

    def request_extractor(_, __, event): req = event.pop("request", None) if req

    is not None: event.update({ "client_addr": req.client_addr, "user_id": req.authenticated_userid, }) return event
  64. 136.

    def request_extractor(_, __, event): req = event.pop("request", None) if req

    is not None: event.update({ "client_addr": req.client_addr, "user_id": req.authenticated_userid, }) return event
  65. 137.

    def request_extractor(_, __, event): req = event.pop("request", None) if req

    is not None: event.update({ "client_addr": req.client_addr, "user_id": req.authenticated_userid, }) return event
  66. 138.

    def request_extractor(_, __, event): req = event.pop("request", None) if req

    is not None: event.update({ "client_addr": req.client_addr, "user_id": req.authenticated_userid, }) return event
  67. 142.
  68. 145.

    Capture • into files • to syslog / a queue

    • pipe into a logging agent
  69. 146.
  70. 150.
  71. 151.

    {"event": "user.login", "user": "guido"} log = log.bind(user="guido") log.info("user.login") structlog logstash-forwarder

    logstash stdout logging /var/log/app/current runit’s svlogd (adds TAI64 timestamp)
  72. 152.

    {"event": "user.login", "user": "guido"} log = log.bind(user="guido") log.info("user.login") structlog logstash-forwarder

    logstash 1010001101 Elasticsearch stdout logging /var/log/app/current runit’s svlogd (adds TAI64 timestamp)
  73. 160.
  74. 165.
  75. 182.
  76. 185.
  77. 186.
  78. 187.
  79. 188.
  80. 189.