Instrumenting The Rest Of The Company: Hunting For Metrics

@esigler Instrumenting The Rest Of The Company: Hunting For Useful
Metrics Eric Sigler, Head of DevOps, PagerDuty

@esigler Alternatively: ”Lies, Damn Lies, and Hacky Scripts"

@esigler

@esigler Engineer Eng Engineer Eng? Manager Mgr Manager

@esigler (No stock photos harmed in the making of this
talk.)

@esigler "We have problem $foo, so we're going to do
$bar."

@esigler "What data did you use to understand $foo? And
how will we know if $bar improved anything?”

@esigler “We can’t really measure  either $foo and/or $bar.”

@esigler “Without data, you're just another person with an opinion.”
- W. Edwards Deming

@esigler

@esigler (Turns out other managers do this too.)

@esigler

@esigler "We have a problem with people not knowing what
the chatbot does, so we're going to write better documentation."

@esigler

@esigler ?

@esigler

@esigler “If only there was some way we could track
events, and show them over time.”

@esigler

@esigler Outcome: Writing a smarter help function in the chat
bot. (And simplifying some commands).

@esigler

@esigler Takeaway: Reuse existing tools when it makes sense.

@esigler

@esigler "We have slow tests in CI, so we're going
to complain a lot about it.”

@esigler “Define slow.”

@esigler Local != CI

@esigler

@esigler “Tests take forever to start.”

@esigler

@esigler ?

@esigler

@esigler Outcome: More workers. (And, knowing how many to budget
for.)

@esigler Takeaway: Look for ways to reverse engineer existing metrics.

@esigler

@esigler "We have to ship code faster, so we're going
to reorganize."

@esigler

@esigler But it doesn’t show where the bottlenecks are.

@esigler Pipe GitHub metrics into  &

@esigler

@esigler Then start making changes.

@esigler

@esigler Outcome: Productivity success! (With massive organizational change to enable
it.)

@esigler Takeaway: Look for proxy metrics

@esigler Potpourri: Data collection (chat, email, calendars) Cross-validation of metrics
(“Sniff test”) Cognitive biases around metrics Plotting against organization events

@esigler Takeaways: Useful metrics are everywhere You aren’t alone in
digging for metrics Existing tools can be repurposed Look to reverse engineer your way to a metric Look for proxy metrics (but choose wisely)

@esigler Thank you!

@esigler Image credits:    https://commons.wikimedia.org/wiki/File:Staff_meeting.jpg https://blogs-images.forbes.com/kellyallan/files/2015/06/Deming-in-Tuxedo-DEM-1078-Dr.-Deming2-1940x1130.jpg (Wherever I grabbed that
screenshot from Pulp Fiction, my apologies I am a terrible person for not capturing the URL)

Instrumenting The Rest Of The Company: Hunting For Metrics

Instrumenting The Rest Of The Company: Hunting For Metrics

More Decks by Eric Sigler

Other Decks in Technology

Featured

Transcript