Upgrade to Pro — share decks privately, control downloads, hide ads and more …

The UnNamed Talk

The UnNamed Talk

@highgroove tech talk - 04/06/12

6385d06011a9633cd45cab0662ae9eb8?s=128

John Vincent

April 06, 2012
Tweet

Transcript

  1. The Unnamed Talk @highgroove Tech Talk 04/06/2012

  2. Alternate Titles • “Operations Eye for the Dev Person” •

    “Operations 101” • “Everything you want to know about Ops but were afraid to ask” • “Foreign Exchange for make benefit of glorious Nation of Operations” • “Care and Feeding of your Ops team” • “Ops is from Venus, Development is from Mars”
  3. But first a word from our “sponsors” • Hosted or

    On-premise cloud management • Abstracts all the Things • Governance • You should ask me about it
  4. Who the hell am I? • @lusis every-effing-where • Atlanta

    Native • DevOps-y kind of guy • Does Ruby (most of the time) • Does Python (other times) • Does Java (well, once like 2 days ago) • #monitoringsucks • Hate Driven Development
  5. Let's talk about Ops

  6. Ops Tradition

  7. Ops Tradition You shall not pass! (lol)

  8. @shitallspawsays “If you care at all about these things, you're

    doing Ops” • Metrics collection • PaaS/IaaS evaluation/investigation • Automation (auto-build, auto-recovery) • Fault tolerance • Availability • Monitoring • Performance • Capex and Opex forecasting • Outage response
  9. @shitallspawsays “If you care at all about these things, you're

    doing Ops” • Metrics collection • PaaS/IaaS evaluation/investigation • Automation (auto-build, auto-recovery) • Fault tolerance • Availability • Monitoring • Performance • Capex and Opex forecasting • Outage response
  10. What is “The Metric” “an analytical measurement intended to quantify

    the state of a system” - W. I. Pedia
  11. Does this look familiar? Started GET "/" for 127.0.0.1 at

    2012-03-10 14:28:14 +0100 Processing by HomeController#index as HTML Rendered text template within layouts/application (0.0ms) Rendered layouts/_assets.html.erb (2.0ms) Rendered layouts/_top.html.erb (2.6ms) Rendered layouts/_about.html.erb (0.3ms) Rendered layouts/_google_analytics.html.erb (0.4ms) Completed 200 OK in 79ms (Views: 78.8ms | ActiveRecord: 0.0ms)
  12. Does this look familiar? Started GET "/" for 127.0.0.1 at

    2012-03-10 14:28:14 +0100 Processing by HomeController#index as HTML Rendered text template within layouts/application (0.0ms) Rendered layouts/_assets.html.erb (2.0ms) Rendered layouts/_top.html.erb (2.6ms) Rendered layouts/_about.html.erb (0.3ms) Rendered layouts/_google_analytics.html.erb (0.4ms) Completed 200 OK in 79ms (Views: 78.8ms | ActiveRecord: 0.0ms)
  13. So that's great for humans but • what if you

    want to do this?
  14. Well you could.... • grep ms | awk | sed

    | grep | sort | cry • Performant? • Realistic? • Scalable? • Do you seriously want to do this?
  15. What about Business Metrics? • Failed Logins • Shopping Cart

    Abandon Rates • New Account creations • <insert shit the business cares about here>
  16. What about Business Metrics? • Failed Logins • Shopping Cart

    Abandon Rates • New Account creations • <insert shit the business cares about here> AND DID ANY OF IT GET BETTER OR WORSE WITH A NEW CODE RELEASE?
  17. Let's talk about Ops

  18. Baking operations into your code (easy) • Visibility – Implement

    a “ping” page • Low-impact • Exercise the full stack (call DB, perform some safe action, Easy to Parse, Easy to Check) • Please don't expect folks to robot a login session
  19. Baking operations into your code (awesome) • Visibility – Push

    Metrics • StatsD + Graphite (DIY) • Librato, New Relic (aaS) • Read this - http://codeascraft.etsy.com/2011/02/15/measure-an • Watch This - http://pivotallabs.com/talks/139-metrics-metrics-e • Try This - https://github.com/johnewart/ruby-metrics
  20. Let's talk about Logs

  21. Remember This? Started GET "/" for 127.0.0.1 at 2012-03-10 14:28:14

    +0100 Processing by HomeController#index as HTML Rendered text template within layouts/application (0.0ms) Rendered layouts/_assets.html.erb (2.0ms) Rendered layouts/_top.html.erb (2.6ms) Rendered layouts/_about.html.erb (0.3ms) Rendered layouts/_google_analytics.html.erb (0.4ms) Completed 200 OK in 79ms (Views: 78.8ms | ActiveRecord: 0.0ms)
  22. Logstash • Started by @jordansissel and @fetep ~2004 • Inputs

    → Filters → Outputs • Jruby (and increasingly better YARV support) • UNIX pipe with Radioactive Super Powers
  23. Works like this

  24. More like this

  25. Can give you this

  26. Config language Not Ruby. It'll be okay.

  27. Basic Config

  28. Slightly Advanced Config

  29. Filters all up in your shit

  30. Remember This? Started GET "/" for 127.0.0.1 at 2012-03-10 14:28:14

    +0100 Processing by HomeController#index as HTML Rendered text template within layouts/application (0.0ms) Rendered layouts/_assets.html.erb (2.0ms) Rendered layouts/_top.html.erb (2.6ms) Rendered layouts/_about.html.erb (0.3ms) Rendered layouts/_google_analytics.html.erb (0.4ms) Completed 200 OK in 79ms (Views: 78.8ms | ActiveRecord: 0.0ms)
  31. Lookie here

  32. Annoying to parse though • Try this: https://github.com/mattmatt/lograge

  33. Remember This? Started GET "/" for 127.0.0.1 at 2012-03-10 14:28:14

    +0100 Processing by HomeController#index as HTML Rendered text template within layouts/application (0.0ms) Rendered layouts/_assets.html.erb (2.0ms) Rendered layouts/_top.html.erb (2.6ms) Rendered layouts/_about.html.erb (0.3ms) Rendered layouts/_google_analytics.html.erb (0.4ms) Completed 200 OK in 79ms (Views: 78.8ms | ActiveRecord: 0.0ms)
  34. Lookie here GET /jobs/833552.json format=json action=jobs#show status=200 duration=58.33 view=40.43 db=15.26

    • yes that's a single line! • No more multiline • No more awkward parsing
  35. Remember this?

  36. Lookie here

  37. Logs need to be readable by humans AND machines •

    Single lines are better • JSON is readable by humans and Machines • Every language has JSON support
  38. Baking operations into your code • Take little effort •

    Has great value • Makes everyone happy • Don't forget the non-functional requirements (The “ilities”) • Stop inventing Timestamp formats (ISO8601 is a thing) • Turn Kittens into PopTarts
  39. Ops AMA time! • Packaging? Artifacts? - FPM • Monitoring?

    - Nagios, Sensu, Riemann • Logging? - Loggly, Papertrail, Logstash (woot), Graylog2 • Alerting? - Pagerduty
  40. Thank you so much!