Lies, Damn Lies, and Metrics (Strange Loop 2016)

4c3ed917e59156a36212d48155831482?s=47 André Arko
September 17, 2016

Lies, Damn Lies, and Metrics (Strange Loop 2016)

Metrics are great, and measuring things can provide tremendously useful insights. But there's a problem: metrics lie to you. Metrics just report the numbers that were measured. Analyzing those numbers is up to us, and that analysis can go wrong in so, so many ways. Learn how to arm yourself against human intuition, interpreter pauses, routing, instrumentation lag, and other issues. Don't get so caught up in instrumenting that you lose sight of why metrics exist! Make sure your metrics are telling you actionable information, instead of just accurate numbers.

4c3ed917e59156a36212d48155831482?s=128

André Arko

September 17, 2016
Tweet

Transcript

  1. 3.
  2. 5.
  3. 13.

    “Normal” 5 -5 -4 -3 -2 -1 0 1 2

    3 4 0 0.1 0.2 0.3 0.4
  4. 14.

    “Normal” 5 -5 -4 -3 -2 -1 0 1 2

    3 4 0 0.1 0.2 0.3 0.4
  5. 15.

    Real Life 5 -5 -4 -3 -2 -1 0 1

    2 3 4 0 0.1 0.2 0.3 0.4
  6. 19.

    The problem with averages: If you put one hand in

    a bucket of ice and the other in a bucket of hot coals, on average, you’re comfortable. Erik Michaels-Ober @sferik
  7. 21.

    10 0 1 2 3 4 5 6 7 8

    9 250 0 50 100 150 200
  8. 23.

    10 0 1 2 3 4 5 6 7 8

    9 250 0 50 100 150 200
  9. 25.

    10 0 1 2 3 4 5 6 7 8

    9 250 0 50 100 150 200
  10. 27.

    10 0 1 2 3 4 5 6 7 8

    9 1000 0 250 500 750
  11. 29.
  12. 31.
  13. 35.
  14. 36.

    Average of X Average of Y Variance of X Variance

    of Y Correlation of X and Y Linear regression All four data sets have the same
  15. 40.
  16. 42.
  17. 47.
  18. 51.