The UnNamed Talk

@highgroove tech talk - 04/06/12

Transcript

1. The Unnamed Talk
   @highgroove Tech Talk
   04/06/2012
   

   View Slide

2. Alternate Titles
   ●
   “Operations Eye for the Dev Person”
   ●
   “Operations 101”
   ●
   “Everything you want to know about Ops but
   were afraid to ask”
   ●
   “Foreign Exchange for make benefit of
   glorious Nation of Operations”
   ●
   “Care and Feeding of your Ops team”
   ●
   “Ops is from Venus, Development is from
   Mars”
   

   View Slide

3. But first a word from our “sponsors”
   ●
   Hosted or On-premise cloud management
   ●
   Abstracts all the Things
   ●
   Governance
   ●
   You should ask me about it
   

   View Slide

4. Who the hell am I?
   ●
   @lusis every-effing-where
   ●
   Atlanta Native
   ●
   DevOps-y kind of guy
   ●
   Does Ruby (most of the time)
   ●
   Does Python (other times)
   ●
   Does Java (well, once like 2 days ago)
   ●
   #monitoringsucks
   ●
   Hate Driven Development
   

   View Slide

5. Let's talk about Ops
   

   View Slide

6. Ops Tradition
   

   View Slide

7. Ops Tradition
   You shall not pass! (lol)
   

   View Slide

8. @shitallspawsays
   “If you care at all about these things, you're doing Ops”
   ●
   Metrics collection
   ●
   PaaS/IaaS evaluation/investigation
   ●
   Automation (auto-build, auto-recovery)
   ●
   Fault tolerance
   ●
   Availability
   ●
   Monitoring
   ●
   Performance
   ●
   Capex and Opex forecasting
   ●
   Outage response
   

   View Slide

9. @shitallspawsays
   “If you care at all about these things, you're doing Ops”
   ●
   Metrics collection
   ●
   PaaS/IaaS evaluation/investigation
   ●
   Automation (auto-build, auto-recovery)
   ●
   Fault tolerance
   ●
   Availability
   ●
   Monitoring
   ●
   Performance
   ●
   Capex and Opex forecasting
   ●
   Outage response
   

   View Slide

10. What is “The Metric”
    “an analytical measurement intended to quantify
    the state of a system” - W. I. Pedia
    

    View Slide

11. Does this look familiar?
    Started GET "/" for 127.0.0.1 at 2012-03-10 14:28:14 +0100
    Processing by HomeController#index as HTML
    Rendered text template within layouts/application (0.0ms)
    Rendered layouts/_assets.html.erb (2.0ms)
    Rendered layouts/_top.html.erb (2.6ms)
    Rendered layouts/_about.html.erb (0.3ms)
    Rendered layouts/_google_analytics.html.erb (0.4ms)
    Completed 200 OK in 79ms (Views: 78.8ms | ActiveRecord:
    0.0ms)
    

    View Slide

12. Does this look familiar?
    Started GET "/" for 127.0.0.1 at 2012-03-10 14:28:14 +0100
    Processing by HomeController#index as HTML
    Rendered text template within layouts/application (0.0ms)
    Rendered layouts/_assets.html.erb (2.0ms)
    Rendered layouts/_top.html.erb (2.6ms)
    Rendered layouts/_about.html.erb (0.3ms)
    Rendered layouts/_google_analytics.html.erb (0.4ms)
    Completed 200 OK in 79ms (Views: 78.8ms | ActiveRecord:
    0.0ms)
    

    View Slide

13. So that's great for humans but
    ●
    what if you want to do this?
    

    View Slide

14. Well you could....
    ●
    grep ms | awk | sed | grep | sort | cry
    ●
    Performant?
    ●
    Realistic?
    ●
    Scalable?
    ●
    Do you seriously want to do this?
    

    View Slide

15. What about Business Metrics?
    ●
    Failed Logins
    ●
    Shopping Cart Abandon Rates
    ●
    New Account creations
    ●
    
    

    View Slide

16. What about Business Metrics?
    ●
    Failed Logins
    ●
    Shopping Cart Abandon Rates
    ●
    New Account creations
    ●
    
    AND DID ANY OF IT GET BETTER OR
    WORSE WITH A NEW CODE RELEASE?
    

    View Slide

17. Let's talk about Ops
    

    View Slide

18. Baking operations into your code
    (easy)
    ●
    Visibility – Implement a “ping” page
    ●
    Low-impact
    ●
    Exercise the full stack (call DB, perform
    some safe action, Easy to Parse, Easy to
    Check)
    ●
    Please don't expect folks to robot a login
    session
    

    View Slide

19. Baking operations into your code
    (awesome)
    ●
    Visibility – Push Metrics
    ●
    StatsD + Graphite (DIY)
    ●
    Librato, New Relic (aaS)
    ●
    Read this -
    http://codeascraft.etsy.com/2011/02/15/measure-an
    ●
    Watch This -
    http://pivotallabs.com/talks/139-metrics-metrics-e
    ●
    Try This -
    https://github.com/johnewart/ruby-metrics
    

    View Slide

20. Let's talk about Logs
    

    View Slide

21. Remember This?
    Started GET "/" for 127.0.0.1 at 2012-03-10 14:28:14 +0100
    Processing by HomeController#index as HTML
    Rendered text template within layouts/application (0.0ms)
    Rendered layouts/_assets.html.erb (2.0ms)
    Rendered layouts/_top.html.erb (2.6ms)
    Rendered layouts/_about.html.erb (0.3ms)
    Rendered layouts/_google_analytics.html.erb (0.4ms)
    Completed 200 OK in 79ms (Views: 78.8ms | ActiveRecord:
    0.0ms)
    

    View Slide

22. Logstash
    ●
    Started by @jordansissel and @fetep ~2004
    ●
    Inputs → Filters → Outputs
    ●
    Jruby (and increasingly better YARV support)
    ●
    UNIX pipe with Radioactive Super Powers
    

    View Slide

23. Works like this
    

    View Slide

24. More like this
    

    View Slide

25. Can give you this
    

    View Slide

26. Config language
    Not Ruby. It'll be okay.
    

    View Slide

27. Basic Config
    

    View Slide

28. Slightly Advanced Config
    

    View Slide

29. Filters all up in your shit
    

    View Slide

30. Remember This?
    Started GET "/" for 127.0.0.1 at 2012-03-10 14:28:14 +0100
    Processing by HomeController#index as HTML
    Rendered text template within layouts/application (0.0ms)
    Rendered layouts/_assets.html.erb (2.0ms)
    Rendered layouts/_top.html.erb (2.6ms)
    Rendered layouts/_about.html.erb (0.3ms)
    Rendered layouts/_google_analytics.html.erb (0.4ms)
    Completed 200 OK in 79ms (Views: 78.8ms | ActiveRecord:
    0.0ms)
    

    View Slide

31. Lookie here
    

    View Slide

32. Annoying to parse though
    ●
    Try this:
    https://github.com/mattmatt/lograge
    

    View Slide

33. Remember This?
    Started GET "/" for 127.0.0.1 at 2012-03-10 14:28:14 +0100
    Processing by HomeController#index as HTML
    Rendered text template within layouts/application (0.0ms)
    Rendered layouts/_assets.html.erb (2.0ms)
    Rendered layouts/_top.html.erb (2.6ms)
    Rendered layouts/_about.html.erb (0.3ms)
    Rendered layouts/_google_analytics.html.erb (0.4ms)
    Completed 200 OK in 79ms (Views: 78.8ms | ActiveRecord:
    0.0ms)
    

    View Slide

34. Lookie here
    GET /jobs/833552.json format=json
    action=jobs#show status=200
    duration=58.33 view=40.43 db=15.26
    ●
    yes that's a single line!
    ●
    No more multiline
    ●
    No more awkward parsing
    

    View Slide

35. Remember this?
    

    View Slide

36. Lookie here
    

    View Slide

37. Logs need to be readable by humans
    AND machines
    ●
    Single lines are better
    ●
    JSON is readable by humans and Machines
    ●
    Every language has JSON support
    

    View Slide

38. Baking operations into your code
    ●
    Take little effort
    ●
    Has great value
    ●
    Makes everyone happy
    ●
    Don't forget the non-functional
    requirements (The “ilities”)
    ●
    Stop inventing Timestamp formats (ISO8601 is
    a thing)
    ●
    Turn Kittens into PopTarts
    

    View Slide

39. Ops AMA time!
    ●
    Packaging? Artifacts? - FPM
    ●
    Monitoring? - Nagios, Sensu, Riemann
    ●
    Logging? - Loggly, Papertrail, Logstash
    (woot), Graylog2
    ●
    Alerting? - Pagerduty
    

    View Slide

40. Thank you so much!
    

    View Slide