$30 off During Our Annual Pro Sale. View Details »

The UnNamed Talk

The UnNamed Talk

@highgroove tech talk - 04/06/12

John Vincent

April 06, 2012
Tweet

More Decks by John Vincent

Other Decks in Technology

Transcript

  1. The Unnamed Talk
    @highgroove Tech Talk
    04/06/2012

    View Slide

  2. Alternate Titles

    “Operations Eye for the Dev Person”

    “Operations 101”

    “Everything you want to know about Ops but
    were afraid to ask”

    “Foreign Exchange for make benefit of
    glorious Nation of Operations”

    “Care and Feeding of your Ops team”

    “Ops is from Venus, Development is from
    Mars”

    View Slide

  3. But first a word from our “sponsors”

    Hosted or On-premise cloud management

    Abstracts all the Things

    Governance

    You should ask me about it

    View Slide

  4. Who the hell am I?

    @lusis every-effing-where

    Atlanta Native

    DevOps-y kind of guy

    Does Ruby (most of the time)

    Does Python (other times)

    Does Java (well, once like 2 days ago)

    #monitoringsucks

    Hate Driven Development

    View Slide

  5. Let's talk about Ops

    View Slide

  6. Ops Tradition

    View Slide

  7. Ops Tradition
    You shall not pass! (lol)

    View Slide

  8. @shitallspawsays
    “If you care at all about these things, you're doing Ops”

    Metrics collection

    PaaS/IaaS evaluation/investigation

    Automation (auto-build, auto-recovery)

    Fault tolerance

    Availability

    Monitoring

    Performance

    Capex and Opex forecasting

    Outage response

    View Slide

  9. @shitallspawsays
    “If you care at all about these things, you're doing Ops”

    Metrics collection

    PaaS/IaaS evaluation/investigation

    Automation (auto-build, auto-recovery)

    Fault tolerance

    Availability

    Monitoring

    Performance

    Capex and Opex forecasting

    Outage response

    View Slide

  10. What is “The Metric”
    “an analytical measurement intended to quantify
    the state of a system” - W. I. Pedia

    View Slide

  11. Does this look familiar?
    Started GET "/" for 127.0.0.1 at 2012-03-10 14:28:14 +0100
    Processing by HomeController#index as HTML
    Rendered text template within layouts/application (0.0ms)
    Rendered layouts/_assets.html.erb (2.0ms)
    Rendered layouts/_top.html.erb (2.6ms)
    Rendered layouts/_about.html.erb (0.3ms)
    Rendered layouts/_google_analytics.html.erb (0.4ms)
    Completed 200 OK in 79ms (Views: 78.8ms | ActiveRecord:
    0.0ms)

    View Slide

  12. Does this look familiar?
    Started GET "/" for 127.0.0.1 at 2012-03-10 14:28:14 +0100
    Processing by HomeController#index as HTML
    Rendered text template within layouts/application (0.0ms)
    Rendered layouts/_assets.html.erb (2.0ms)
    Rendered layouts/_top.html.erb (2.6ms)
    Rendered layouts/_about.html.erb (0.3ms)
    Rendered layouts/_google_analytics.html.erb (0.4ms)
    Completed 200 OK in 79ms (Views: 78.8ms | ActiveRecord:
    0.0ms)

    View Slide

  13. So that's great for humans but

    what if you want to do this?

    View Slide

  14. Well you could....

    grep ms | awk | sed | grep | sort | cry

    Performant?

    Realistic?

    Scalable?

    Do you seriously want to do this?

    View Slide

  15. What about Business Metrics?

    Failed Logins

    Shopping Cart Abandon Rates

    New Account creations


    View Slide

  16. What about Business Metrics?

    Failed Logins

    Shopping Cart Abandon Rates

    New Account creations


    AND DID ANY OF IT GET BETTER OR
    WORSE WITH A NEW CODE RELEASE?

    View Slide

  17. Let's talk about Ops

    View Slide

  18. Baking operations into your code
    (easy)

    Visibility – Implement a “ping” page

    Low-impact

    Exercise the full stack (call DB, perform
    some safe action, Easy to Parse, Easy to
    Check)

    Please don't expect folks to robot a login
    session

    View Slide

  19. Baking operations into your code
    (awesome)

    Visibility – Push Metrics

    StatsD + Graphite (DIY)

    Librato, New Relic (aaS)

    Read this -
    http://codeascraft.etsy.com/2011/02/15/measure-an

    Watch This -
    http://pivotallabs.com/talks/139-metrics-metrics-e

    Try This -
    https://github.com/johnewart/ruby-metrics

    View Slide

  20. Let's talk about Logs

    View Slide

  21. Remember This?
    Started GET "/" for 127.0.0.1 at 2012-03-10 14:28:14 +0100
    Processing by HomeController#index as HTML
    Rendered text template within layouts/application (0.0ms)
    Rendered layouts/_assets.html.erb (2.0ms)
    Rendered layouts/_top.html.erb (2.6ms)
    Rendered layouts/_about.html.erb (0.3ms)
    Rendered layouts/_google_analytics.html.erb (0.4ms)
    Completed 200 OK in 79ms (Views: 78.8ms | ActiveRecord:
    0.0ms)

    View Slide

  22. Logstash

    Started by @jordansissel and @fetep ~2004

    Inputs → Filters → Outputs

    Jruby (and increasingly better YARV support)

    UNIX pipe with Radioactive Super Powers

    View Slide

  23. Works like this

    View Slide

  24. More like this

    View Slide

  25. Can give you this

    View Slide

  26. Config language
    Not Ruby. It'll be okay.

    View Slide

  27. Basic Config

    View Slide

  28. Slightly Advanced Config

    View Slide

  29. Filters all up in your shit

    View Slide

  30. Remember This?
    Started GET "/" for 127.0.0.1 at 2012-03-10 14:28:14 +0100
    Processing by HomeController#index as HTML
    Rendered text template within layouts/application (0.0ms)
    Rendered layouts/_assets.html.erb (2.0ms)
    Rendered layouts/_top.html.erb (2.6ms)
    Rendered layouts/_about.html.erb (0.3ms)
    Rendered layouts/_google_analytics.html.erb (0.4ms)
    Completed 200 OK in 79ms (Views: 78.8ms | ActiveRecord:
    0.0ms)

    View Slide

  31. Lookie here

    View Slide

  32. Annoying to parse though

    Try this:
    https://github.com/mattmatt/lograge

    View Slide

  33. Remember This?
    Started GET "/" for 127.0.0.1 at 2012-03-10 14:28:14 +0100
    Processing by HomeController#index as HTML
    Rendered text template within layouts/application (0.0ms)
    Rendered layouts/_assets.html.erb (2.0ms)
    Rendered layouts/_top.html.erb (2.6ms)
    Rendered layouts/_about.html.erb (0.3ms)
    Rendered layouts/_google_analytics.html.erb (0.4ms)
    Completed 200 OK in 79ms (Views: 78.8ms | ActiveRecord:
    0.0ms)

    View Slide

  34. Lookie here
    GET /jobs/833552.json format=json
    action=jobs#show status=200
    duration=58.33 view=40.43 db=15.26

    yes that's a single line!

    No more multiline

    No more awkward parsing

    View Slide

  35. Remember this?

    View Slide

  36. Lookie here

    View Slide

  37. Logs need to be readable by humans
    AND machines

    Single lines are better

    JSON is readable by humans and Machines

    Every language has JSON support

    View Slide

  38. Baking operations into your code

    Take little effort

    Has great value

    Makes everyone happy

    Don't forget the non-functional
    requirements (The “ilities”)

    Stop inventing Timestamp formats (ISO8601 is
    a thing)

    Turn Kittens into PopTarts

    View Slide

  39. Ops AMA time!

    Packaging? Artifacts? - FPM

    Monitoring? - Nagios, Sensu, Riemann

    Logging? - Loggly, Papertrail, Logstash
    (woot), Graylog2

    Alerting? - Pagerduty

    View Slide

  40. Thank you so much!

    View Slide