The UnNamed Talk

Transcript

1. The Unnamed Talk @highgroove Tech Talk 04/06/2012

2. Alternate Titles • “Operations Eye for the Dev Person” •

   “Operations 101” • “Everything you want to know about Ops but were afraid to ask” • “Foreign Exchange for make benefit of glorious Nation of Operations” • “Care and Feeding of your Ops team” • “Ops is from Venus, Development is from Mars”

3. But first a word from our “sponsors” • Hosted or

   On-premise cloud management • Abstracts all the Things • Governance • You should ask me about it

4. Who the hell am I? • @lusis every-effing-where • Atlanta

   Native • DevOps-y kind of guy • Does Ruby (most of the time) • Does Python (other times) • Does Java (well, once like 2 days ago) • #monitoringsucks • Hate Driven Development

5. Let's talk about Ops

6. Ops Tradition

7. Ops Tradition You shall not pass! (lol)

8. @shitallspawsays “If you care at all about these things, you're

   doing Ops” • Metrics collection • PaaS/IaaS evaluation/investigation • Automation (auto-build, auto-recovery) • Fault tolerance • Availability • Monitoring • Performance • Capex and Opex forecasting • Outage response

9. @shitallspawsays “If you care at all about these things, you're

   doing Ops” • Metrics collection • PaaS/IaaS evaluation/investigation • Automation (auto-build, auto-recovery) • Fault tolerance • Availability • Monitoring • Performance • Capex and Opex forecasting • Outage response

10. What is “The Metric” “an analytical measurement intended to quantify

    the state of a system” - W. I. Pedia

11. Does this look familiar? Started GET "/" for 127.0.0.1 at

    2012-03-10 14:28:14 +0100 Processing by HomeController#index as HTML Rendered text template within layouts/application (0.0ms) Rendered layouts/_assets.html.erb (2.0ms) Rendered layouts/_top.html.erb (2.6ms) Rendered layouts/_about.html.erb (0.3ms) Rendered layouts/_google_analytics.html.erb (0.4ms) Completed 200 OK in 79ms (Views: 78.8ms | ActiveRecord: 0.0ms)

12. Does this look familiar? Started GET "/" for 127.0.0.1 at

    2012-03-10 14:28:14 +0100 Processing by HomeController#index as HTML Rendered text template within layouts/application (0.0ms) Rendered layouts/_assets.html.erb (2.0ms) Rendered layouts/_top.html.erb (2.6ms) Rendered layouts/_about.html.erb (0.3ms) Rendered layouts/_google_analytics.html.erb (0.4ms) Completed 200 OK in 79ms (Views: 78.8ms | ActiveRecord: 0.0ms)

13. So that's great for humans but • what if you

    want to do this?

14. Well you could.... • grep ms | awk | sed

    | grep | sort | cry • Performant? • Realistic? • Scalable? • Do you seriously want to do this?

15. What about Business Metrics? • Failed Logins • Shopping Cart

    Abandon Rates • New Account creations • <insert shit the business cares about here>

16. What about Business Metrics? • Failed Logins • Shopping Cart

    Abandon Rates • New Account creations • <insert shit the business cares about here> AND DID ANY OF IT GET BETTER OR WORSE WITH A NEW CODE RELEASE?

17. Let's talk about Ops

18. Baking operations into your code (easy) • Visibility – Implement

    a “ping” page • Low-impact • Exercise the full stack (call DB, perform some safe action, Easy to Parse, Easy to Check) • Please don't expect folks to robot a login session

19. Baking operations into your code (awesome) • Visibility – Push

    Metrics • StatsD + Graphite (DIY) • Librato, New Relic (aaS) • Read this - http://codeascraft.etsy.com/2011/02/15/measure-an • Watch This - http://pivotallabs.com/talks/139-metrics-metrics-e • Try This - https://github.com/johnewart/ruby-metrics

20. Let's talk about Logs

21. Remember This? Started GET "/" for 127.0.0.1 at 2012-03-10 14:28:14

    +0100 Processing by HomeController#index as HTML Rendered text template within layouts/application (0.0ms) Rendered layouts/_assets.html.erb (2.0ms) Rendered layouts/_top.html.erb (2.6ms) Rendered layouts/_about.html.erb (0.3ms) Rendered layouts/_google_analytics.html.erb (0.4ms) Completed 200 OK in 79ms (Views: 78.8ms | ActiveRecord: 0.0ms)

22. Logstash • Started by @jordansissel and @fetep ~2004 • Inputs

    → Filters → Outputs • Jruby (and increasingly better YARV support) • UNIX pipe with Radioactive Super Powers

23. Works like this

24. More like this

25. Can give you this

26. Config language Not Ruby. It'll be okay.

27. Basic Config

28. Slightly Advanced Config

29. Filters all up in your shit

30. Remember This? Started GET "/" for 127.0.0.1 at 2012-03-10 14:28:14

    +0100 Processing by HomeController#index as HTML Rendered text template within layouts/application (0.0ms) Rendered layouts/_assets.html.erb (2.0ms) Rendered layouts/_top.html.erb (2.6ms) Rendered layouts/_about.html.erb (0.3ms) Rendered layouts/_google_analytics.html.erb (0.4ms) Completed 200 OK in 79ms (Views: 78.8ms | ActiveRecord: 0.0ms)

31. Lookie here

32. Annoying to parse though • Try this: https://github.com/mattmatt/lograge

33. Remember This? Started GET "/" for 127.0.0.1 at 2012-03-10 14:28:14

    +0100 Processing by HomeController#index as HTML Rendered text template within layouts/application (0.0ms) Rendered layouts/_assets.html.erb (2.0ms) Rendered layouts/_top.html.erb (2.6ms) Rendered layouts/_about.html.erb (0.3ms) Rendered layouts/_google_analytics.html.erb (0.4ms) Completed 200 OK in 79ms (Views: 78.8ms | ActiveRecord: 0.0ms)

34. Lookie here GET /jobs/833552.json format=json action=jobs#show status=200 duration=58.33 view=40.43 db=15.26

    • yes that's a single line! • No more multiline • No more awkward parsing

35. Remember this?

36. Lookie here

37. Logs need to be readable by humans AND machines •

    Single lines are better • JSON is readable by humans and Machines • Every language has JSON support

38. Baking operations into your code • Take little effort •

    Has great value • Makes everyone happy • Don't forget the non-functional requirements (The “ilities”) • Stop inventing Timestamp formats (ISO8601 is a thing) • Turn Kittens into PopTarts

39. Ops AMA time! • Packaging? Artifacts? - FPM • Monitoring?

    - Nagios, Sensu, Riemann • Logging? - Loggly, Papertrail, Logstash (woot), Graylog2 • Alerting? - Pagerduty

40. Thank you so much!