Upgrade to Pro — share decks privately, control downloads, hide ads and more …

SLO Creation and You: Or, How We Learned to Stop Worrying and Love the Queue Length

SLO Creation and You: Or, How We Learned to Stop Worrying and Love the Queue Length

Similar material to DevOps Days, but updated for ExploreTech 2018


Lisa Seelye

October 24, 2018


  1. Lisa Seelye @thedoh SLO Creation and You Or: How We

    Learned to Stop Worrying and Love the Queue Length 1
  2. Lisa Seelye @thedoh About Me 2 By @burst at https://www.pexels.com/photo/architecture-bridge-building-business-374754/

  3. Lisa Seelye @thedoh About Our Story 3 Empty Theater Seats

    by @jamie-fernandez-201894 at https://www.pexels.com/photo/empty-theater-seats-758976/ / CC0 / cropped
  4. Lisa Seelye @thedoh Our Cast of Characters 4 • Service

    Level Objectives (SLOs) ◦ Service Level Agreement (SLAs) as the evil twin • RabbitMQ • Service Ownership • New Service Owners (aka Developers and Product Owners)
  5. Lisa Seelye @thedoh Act I: We Made an SLO 5

  6. Lisa Seelye @thedoh Creating The SLO 6 Group Hand Fist

    Bump by @rawpixel.com at https://www.pexels.com/photo/group-hand-fist-bump-1068523/
  7. Lisa Seelye @thedoh 99.95% Monthly Availability 7 So… What do

    you think?
  8. Lisa Seelye @thedoh Conspicuously Absent Objectives • Queue length •

    Queue consumer count 8
  9. Lisa Seelye @thedoh The Reaction “We think you should have

    queue length as an SLO.” 9
  10. Lisa Seelye @thedoh Act II: All About Queue Length 10

  11. Lisa Seelye @thedoh Causes of long queues - Capacity 11

    Rush Hour by msvg at https://www.flickr.com/photos/msvg/4476789745/ CC-BY 2.0 / cropped
  12. Lisa Seelye @thedoh Causes of long queues - Usage Spike

    12 Image from NewRelic.com for FreshBooks (used with permission)
  13. Lisa Seelye @thedoh Causes of long queues - Buggy Deploy

    13 Bugs by searleb at https://www.flickr.com/photos/searleb/3122477836/ CC-BY 2.0 / cropped
  14. Lisa Seelye @thedoh Causes of long queues - Normal Growth

    14 Landscape Photography of Pavement Road by @jc-estrada-341132 at https://www.pexels.com/photo/landscape-photography-of-pavement-road-1046606/ CC0 / cropped
  15. Lisa Seelye @thedoh Causes of long queues - RabbitMQ Did

    It! 15 Images from FreshBooks
  16. Lisa Seelye @thedoh The Value of Queue Length • We

    might not have enough capacity • We might be a problem with the queue services • Is RabbitMQ well-provisioned? 16
  17. Lisa Seelye @thedoh Act III: Asking Questions 17

  18. Lisa Seelye @thedoh Key Goals • Quicker on-call alert handling

    • Easier capacity planning • Sets service expectations • Hold ourselves accountable 18
  19. Lisa Seelye @thedoh Ask Direct Questions • Specific questions have

    specific answers • Usually service specific • “Why is that important” 19
  20. Lisa Seelye @thedoh What To Ask? - Is My Service

    Healthy? 20 No more use... by smithser at https://www.flickr.com/photos/smithser/3434266313 CC-BY 2.0 / cropped
  21. Lisa Seelye @thedoh What To Ask? - Service Misuse? 21

    DSC_1607 by justinbaeder at https://www.flickr.com/photos/justinbaeder/5317820857 CC-BY-2.0 / cropped
  22. Lisa Seelye @thedoh 22 White Pocket Watch With Gold-colored Frame

    on Brown Wooden Board by @iseeghoststoo at https://www.pexels.com/photo/white-pocket-watch-with-gold-colored-frame-on-brown-wooden-board-1010513/ CC0 / cropped What To Ask? - Customer Wait Time?
  23. Lisa Seelye @thedoh 23 The big queue at an ATM

    in Masalli, Azerbaijan by Ds02006 at https://commons.wikimedia.org/wiki/File:ATM_Masalli.jpg / Public Domain / cropped What To Ask? - Consumer Throughput?
  24. Lisa Seelye @thedoh 24 King's Highway 12 - Ontario by

    dougtone at https://www.flickr.com/photos/dougtone/9190014238/ / CC-SA 2.0 / cropped What To Ask? - Violating Queue Limits?
  25. Lisa Seelye @thedoh Wait, Did You Just Suggest Queue Length??

  26. Lisa Seelye @thedoh We Have Our Questions … Now What?

    26 Stealthy Cosmo by pargon at https://www.flickr.com/photos/pargon/2381366401 / CC-BY 2.0 / cropped
  27. Lisa Seelye @thedoh Epilogue 27

  28. Lisa Seelye @thedoh Looking Back • Unfamiliar with RabbitMQ instrumentation

    • Correlated queue length with problem • Alert fatigue :( 28
  29. Lisa Seelye @thedoh Key Lessons 29 • Old ways weren’t

    working • Be open to a discussion with people
  30. Lisa Seelye @thedoh One Last Thing... 30 • SLOs aren’t

    just for software - Think Customer Support
  31. Lisa Seelye @thedoh The End 31 SLOs sound cool? Learn

    More in Google’s SRE Book (Ch. 4) https://landing.google.com/sre/book.html Special thank you to FreshBooks For granting permission to give this talk.