Berlin 2013 - Session - Mark McGranaghan

Transcript

1. Fewer Better Systems Monitorama EU 2013 Mark McGranaghan
2. @mmcgrana
3. Fewer Better Systems
4. Unix
5. everything is a file
6. /var/db /usr/lib /dev/tcp /usr/bin /etc
7. /dev/tcp
8. problem problem problem
9. everything is a ...
10. failover
11. primary secondary
12. primary secondary
13. primary secondary
14. primary secondary?
15. https://twitter.com/b6n/status/161899319459463168
16. the best systems are used constantly
17. Fewer Better Systems
18. everything is a ...
19. the best systems are used constantly
20. logs / events alert criteria / metrics integration testing / QoS monitoring errors / results
21. logs / events
22. logs: stream of unstructured information events: stream of structured information
23. logs 64.242.88.10 - [19/Sep/2013:10:27:39] "GET /users/7 HTTP/1.1" 200 [notice] SQL (0.5ms) SELECT users Completed in 64ms (View: 52, DB: 10) | 200 OK [/users/7]
24. invent ways to encode data in text...
25. data "data" | data <data> - data [data] (data)
26. meanwhile...
27. Apache log parsers / analyzers Postgres log parses / analyzers Redis log parsers / analyzers Heroku log parsers / analyzers ...
28. everything is a ...
29. events { :time "2013-09-19 10:27:39" :action "users.get" :user_id 7 :method "GET" :path "/users/7" :ip "64.242.88.10" ... }
30. 64.242.88.10 - [19/Sep/2013:10:27:39] "GET /users/7 HTTP/1.1" 200
31. events { :time "2013-09-19 10:27:39" :action "users.get" :user_id 7 :method "GET" :path "/users/7" :ip "64.242.88.10" ... }
32. encode data as data, uniformly
33. analyze with general tools
34. open source
35. http://fluentd.org
36. { :time "2013-09-19 10:27:39", :tag "web.request", :record { :ip "64.242.88.10", :path "/users/7", ... } }
37. Web apps ---+ +--> file | | +--> ---+ /var/log ------> Fluentd ------> mail +--> ---+ | | Apache ---- +--> Fluentd http://fluentd.org
38. problem problem problem
39. something happened at some time: event events as data, not text general-purpose event processing applicable to all information
40. everything is a ...
41. alert criteria / metrics
42. alert criteria: measure, alert if out of bounds metrics: measure, store for analysis
43. measure measure alert store
44. measure measure alert store steady-state
45. measure measure alert store alert!
46. measure measure alert store steady-state
47. measure alert store
48. production
50. every alert has time series alter time series come from metrics stack alert source data stored all the time
51. the best systems are used constantly
52. integration testing / QoS monitoring
53. https://plus.google.com/112678702228711889851/posts/eVeouesvaVX
54. integration testing: is good for production? QoS monitoring: is it good in production?
55. integration testing run through common user flows, assert no errors, ensure performance adequate
56. quality of service (QoS) monitoring users running through flows asserting no/minimal errors, ensuring performance adequate
57. integration prod staging user load QoS monitoring
58. Integration prod staging user load QoS monitoring
59. staging prod user load load gen QoS monitoring QoS monitoring load gen
60. invest in load generation/replay invest in granular QoS monitoring applicable to all environments, all the time
61. the best systems are used constantly
62. errors / results
63. raise(“it’s tricky”)
64. errors: something happened, it was bad results: something happened, it was OK
65. begin res = call_fn(arg) # handle result rescue => err # handle error end
68. exceptions are only exceptional at small scale “1 in a billion” @ 100k op/s ≃ 10 times a day
69. begin res = call_fn(arg) # handle result rescue => err # handle error end
70. open source
71. http://golang.org
72. http://golang.org res, err := RunOp(arg) if err != nil { // handle error } // handle result
73. begin res = run_op(arg) # handle result rescue => err # handle error end
74. locality? in general: not local in space - service-level errors etc not local in time - defined post hoc!
75. what even is an error? you don’t know at dev-time when it’s just a result... emit event for later analysis
76. treat “exceptions” / results symmetrically to the greatest extent possible expect to define errors at analysis-time, not just dev-time or run-time, based on results
77. everything is a ...
78. logs / events / metrics alert criteria / metrics integration testing / QoS monitoring errors / results
79. a challenge
80. everything is a ...
81. the best systems are used constantly
82. Fewer Better Systems