WCEU 2020 - Everything is broken: an introduction to testing, logging, monitoring and metrics

‘You cannot make an omelette without breaking a few eggs.’ Things are inevitably going to break. And that’s stressful. If your triaging process involves manually seeking through log files or you only find out something is broken after a customer calls you, then you’re probably not fixing problems efficiently.

In this lightning talk Niels will discuss how to prevent, proactively detect and quickly react to issues. He will provide a short overview of tools that help with testing and preventing issues. The talk will cover logging frameworks and platforms that will actively alert you and help find out what’s going wrong. Finally, Niels will advise on adding metrics to your site in order to prevent future issues.

Transcript

1. Everything is broken An introduction to logging, monitoring and metrics.
2. My name is Niels de Blaauw. I’m lead developer at Level Level, a full-service WordPress Agency in the Netherlands. @nielsdeblaauw
3. Everything is broken.
4. Industry Average: "about 15 - 50 errors per 1000 lines of delivered code." (Steve McConnell 2004)
6. WordPress currently tracks 3749 known bugs (defects).
7. You’re not just using WordPress Core. Webserver WordPress Core Plugins Mail Payments Themes
9. The bad news: You can’t prevent mistakes from happening. The good news: You can prepare for mistakes happening.
10. The good news: You can prepare for mistakes happening.
11. 4 milestones: 1. Log exceptions. 2. Catch undefined behaviour early. 3. Define behaviour for exceptions. 4. Monitor the rest. 1. Log 2. Catch early 3. Define behaviour 4. Monitor Follow along!
12. 4 milestones: 1. Log exceptions. 2. Catch undefined behaviour early. 3. Define behaviour for exceptions. 4. Monitor the rest. 1. Log 2. Catch early 3. Define behaviour 4. Monitor Follow along!
13. Log event Message Context Level Output 1. Log 2. Catch early 3. Define behaviour 4. Monitor
14. Log event Message Context Level Output 1. Log 2. Catch early 3. Define behaviour 4. Monitor What is happening?
15. Log event Message Context Level Output 1. Log 2. Catch early 3. Define behaviour 4. Monitor When? Who? Where?
16. Log event Message Context Level Output 1. Log 2. Catch early 3. Define behaviour 4. Monitor Informational? Warning? Critical?
17. Log event Message Context Level Output 1. Log 2. Catch early 3. Define behaviour 4. Monitor Files Slack Text message
18. <?php require_once( 'vendor/autoload.php' ); // Use statements skipped for clarity. // Create a log channel $log = new Logger( 'MyFirstLogger' ); $log->pushHandler( new SlackWebhookHandler( 'https://slack.com/SECRET' ) ); 1. Log 2. Catch early 3. Define behaviour 4. Monitor
19. <?php require_once( 'vendor/autoload.php' ); // Use statements skipped for clarity. // Create a log channel $log = new Logger( 'MyFirstLogger' ); $log->pushHandler( new SlackWebhookHandler( 'https://slack.com/SECRET' ) ); // Add URL, referrer, etc... $log->pushProcessor( new WebProcessor() ); 1. Log 2. Catch early 3. Define behaviour 4. Monitor
20. <?php require_once( 'vendor/autoload.php' ); // Use statements skipped for clarity. // Create a log channel $log = new Logger( 'MyFirstLogger' ); $log->pushHandler( new SlackWebhookHandler( 'https://slack.com/SECRET' ) ); // Add URL, referrer, etc... $log->pushProcessor( new WebProcessor() ); // Catch unhandled exceptions ErrorHandler::register( $logger ); 1. Log 2. Catch early 3. Define behaviour 4. Monitor
21. <?php require_once( 'vendor/autoload.php' ); // Use statements skipped for clarity. // Create a log channel $log = new Logger( 'MyFirstLogger' ); $log->pushHandler( new SlackWebhookHandler( 'https://slack.com/SECRET' ) ); // Add URL, referrer, etc... $log->pushProcessor( new WebProcessor() ); // Catch unhandled exceptions ErrorHandler::register( $logger ); UndefinedFunction(); ⚠ 1. Log 2. Catch early 3. Define behaviour 4. Monitor
22. 4 milestones: 1. Log exceptions. 2. Catch undefined behaviour early. 3. Define behaviour for exceptions. 4. Monitor the rest. 1. Log 2. Catch early 3. Define behaviour 4. Monitor
23. 1. Coding standards 2. Integration testing 3. Static analysis tool 1. Log 2. Catch early 3. Define behaviour 4. Monitor
24. 1. Coding standards register_post_type( 'author', array() ); // Bad. Reserved slug name. 1. Log 2. Catch early 3. Define behaviour 4. Monitor
25. 2. Integration testing 1. Log 2. Catch early 3. Define behaviour 4. Monitor
26. 1. Log 2. Catch early 3. Define behaviour 4. Monitor € 15.00 product 2. Integration testing
27. 2. Integration testing 1. Log 2. Catch early 3. Define behaviour 4. Monitor € 15.00 product Order 4x
28. 2. Integration testing 1. Log 2. Catch early 3. Define behaviour 4. Monitor € 15.00 product Order 4x ✔ € 60.00
29. 2. Integration testing 1. Log 2. Catch early 3. Define behaviour 4. Monitor € 15.00 product Order 4x Add 50% discount ⚠ € 30.00
30. 3. Static analysis tools 1. Log 2. Catch early 3. Define behaviour 4. Monitor register_post_type()
31. 3. Static analysis tools 1. Log 2. Catch early 3. Define behaviour 4. Monitor register_post_type() get_rest_controller() \WP_Post_Type ✔
32. 3. Static analysis tools 1. Log 2. Catch early 3. Define behaviour 4. Monitor register_post_type() get_rest_controller() \WP_Post_Type \WP_Error ⚠
33. 4 milestones: 1. Log exceptions. 2. Catch undefined behaviour early. 3. Define behaviour for exceptions. 4. Monitor the rest. 1. Log 2. Catch early 3. Define behaviour 4. Monitor
34. Failure: The XML document is well formed but the document is not valid (10002) 1. Log 2. Catch early 3. Define behaviour 4. Monitor
35. Two questions matter 1. What is the situation? 2. What can we do to resolve the situation? 1. Log 2. Catch early 3. Define behaviour 4. Monitor
36. Logging provides asynchronous communication between creator and the maintainer. 1. Log 2. Catch early 3. Define behaviour 4. Monitor
37. 1. Log 2. Catch early 3. Define behaviour 4. Monitor register_post_type() get_rest_controller() \WP_Post_Type \WP_Error ✔
38. 1. Log 2. Catch early 3. Define behaviour 4. Monitor register_post_type() get_rest_controller() \WP_Post_Type \WP_Error ✔ Log event Message, context, level.
39. 4 milestones: 1. Log exceptions. 2. Catch undefined behaviour early. 3. Define behaviour for exceptions. 4. Monitor the rest. 1. Log 2. Catch early 3. Define behaviour 4. Monitor
40. 1. Log 2. Catch early 3. Define behaviour 4. Monitor Access logs Error logs Application data System data
41. 1. Log 2. Catch early 3. Define behaviour 4. Monitor Access logs Error logs Application data System data Data store
42. 1. Log 2. Catch early 3. Define behaviour 4. Monitor Access logs Error logs Application data System data Data store Dashboards Alerts
43. 1. Log 2. Catch early 3. Define behaviour 4. Monitor Stats.d $statsd->increment( ‘shop.orders’ );
44. 1. Log 2. Catch early 3. Define behaviour 4. Monitor Stats.d Graphite Store time series data
45. 1. Log 2. Catch early 3. Define behaviour 4. Monitor Stats.d Graphite Grafana
46. To recap: You can’t prevent mistakes from happening. You can prepare for mistakes happening. 4 milestones: 1. Log exceptions. 2. Catch undefined behaviour early. 3. Define behaviour for exceptions. 4. Monitor the rest.
47. Let’s keep in touch! Download slides and resources: on twitter.com/nielsdeblaauw @nielsdeblaauw