Things Your Application Does While You're Not Looking (Zendcon 2015)

Transcript

1. Things Your Application Does While You’re Not Looking Josh Butts

   ZendCon 2015

2. About Me • VP of Engineering
 at Offers.com • Austin

   PHP Organizer • github.com/jimbojsb • @jimbojsb 2

3. About Offers.com • We help people save money • Launched

   in 2009 • 100k lines of PHP across multiple apps • Millions of Uniques / Month 3

4. Agenda • What is application health? • How can we

   collect data to determine if our application is healthy • How can we make this data actionable? 4

5. What is application health? • Depends on who you ask

   • Combination of performance and quality – Uptime – Response time – Error rate 5

6. Uptime • Set realistic expectations - no one is up

   100% of the time • How many 9’s can you tolerate? • Measure uptime monthly • Planned maintenance counts! 6

7. Up isn’t good enough, but it’s a start • Ping

   monitoring is an absolute minimum • ICMP ping is not good enough • Need to at least check the status code • Should really check for a content snippet • You should outsource this – Pingdom – UptimeRobot 7

8. Error Rate • Number of requests that generate an E_WARNING

   or above / total requests • Uncaught exceptions: E_Fatal • What’s acceptable? – 0% is not realistic – 1% is a good place to start – 0.5% is what we use 8

9. Why Error Rate is Hard • PHP error handlers are

   terrible • You really need an extension • There are a few third party tools that do this, but they aren’t cheap 9

10. Silent Killers • Does a caught exception count as towards

    your error rate? • How many times do you drop the exception? • Would you even know if your password reset page was throwing a 500 error? • Even the best testing can’t fix stupid users 10

11. APPLICATION LOGS 11

12. Application Logs • Logs are your best source for debugging

    production errors • Log facts • Speak to your future self • Use a service or tool to aggregate logs 12

13. Log Highlights • Be wordy, but avoid pointless words •

    Take advantage of log levels • Take advantage of different application environments • Keep your logs to “one-liners” 13

14. Log Levels • DEBUG • INFO • NOTICE • WARNING

    • ERROR • CRITICAL • ALERT • EMERGENCY 14

15. DEBUG • Most detailed and verbose level • Database queries

    • “per-item” information in a loop • Probably turn this off in production 15

16. INFO • This is the “default” for most things •

    General events – user logins – application state changes – material domain object modifications 16

17. NOTICE • Like INFO but slightly more important • You

    might actually care about these • Transactions with values that are normal but higher or lower than expected • Might review these weekly 17

18. WARN • Undesired behavior that isn’t necessarily wrong • Calling

    deprecated APIs • Unexpected null result sets 18

19. ERROR • Runtime logic errors • Unexpected invalid arguments •

    Caught exceptions • Doesn’t require immediate attention • Look at these daily 19

20. CRITICAL • First level where you should consider real-time notifications

    • Unable to connect to a 3rd party service • Connection timeouts • High latency 20

21. ALERT • Application is partially down or non-functional • Failed

    to connect to a critical internal resource • This should send SMS messages, wake people up • Recommend a time threshold 21

22. EMERGENCY • Everything has gone to hell • Hardware failures

    • Wake everyone up, keep calling until someone acknowledges • Rare to see this, because logging has probably also failed 22

23. PHP Logging Software • Monolog • Pretty much everyone uses

    this one • Log4PHP • Pretty much no one uses this one • The one that comes with your favorite framework 23

24. Basic Monolog Setup 24 <?php $loggerName = 'myapp'; $logger =

    new \Monolog\Logger($loggerName);

25. Useful Monolog Setup 25 <?php $loggerName = 'myapp'; $logger =

    new \Monolog\Logger($loggerName); $file = __DIR__ . '/app.log'; touch($file); chmod($file, 0666); $logger->pushHandler(new Monolog\Handler\StreamHandler($file));

26. SAPI-aware Monolog 26 $sapi = php_sapi_name(); $loggerName = php_sapi_name() ==

    'cli' ? "myapp-cli" : "myapp-web"; $logger = new \Monolog\Logger($loggerName); if ($sapi == 'cli') { $logger->pushHandler(new \Monolog\Handler\StreamHandler("php:// stdout")); } else { // file setup here, touch, chmod, etc $logger->pushHandler(new Monolog\Handler\StreamHandler($file)); }

27. Add extra info to your logs 27 $logger->pushProcessor(function($record) { $record["extra"]

    = array(gethostname()); return $record; });

28. Logging to a Service 28 $handler = new Monolog\Handler\SyslogUdpHandler('data.logentries.com', 12345);

    $handler->setFormatter(new \Monolog\Formatter\LineFormatter()); $logger->pushHandler($handler);

29. Environment-Aware Log Levels 29 if (APPLICATION_ENV == 'production') { $udpHandler

    = new Monolog\Handler \SyslogUdpHandler('data.logentries.com', 12345, \Monolog\Logger::INFO); $udpHandler->setFormatter(new \Monolog\Formatter\LineFormatter()); $emailHandler = new Monolog\Handler\SwiftMailerHandler($swiftMailer, \Monolog\Logger::ALERT); $logger->pushHandler($udpHandler); $logger->pushHandler($emailHandler); }

30. STATS 30

31. Metrics Collection • Everyone likes graphs • Data visualizations help

    you spot outliers in real-time • Create a dashboard that displays them 31

32. Example Baseline metrics • PHP execution time • PHP memory

    usage • Number of database queries per request • Job queue length • Time to process jobs • Emails sent 32

33. Application Metrics • User logins / failed logins • Password

    resets • Page views for key pages • Deployments • Caught exceptions • Overall page views 33

34. Statsd / Graphite • Statsd is a node.js tool that

    collects stats from your application • Graphite is a visualization tool that lets you access information from Statsd in graph form 34

35. Graphite UI 35

36. Graphite Example 36

37. Types of Statsd Metrics • Counters • Timers • Gauges

    • Sets 37

38. Examples of Counters • Count every request • Count every

    transactional email sent • Count every job from your job queue by type • Count every caught exception 38

39. Examples of Timers • Time your index.php at the top

    and bottom • Time your crontabs, especially overnight ones • You can even submit timers for multi-page events (conversion funnels, etc) 39

40. Metric Naming • . delimited names • Think of it

    like namespaces • Use a top-level namespace per- app (client-side) 40

41. Send PHP data to Statsd 41 $connection = new Domnikl\Statsd\Connection\UdpSocket('localhost');

    $client = new Domnikl\Statsd\Client($connection, 'myapp');

42. Time your “Page Render” 42 <?php $connection = new Domnikl\Statsd\Connection\UdpSocket('localhost');

    $client = new Domnikl\Statsd\Client($connection, ‘orca'); $client->startTiming(‘render_time'); $application = new Application(); $response = $application->dispatch(); echo $response; $client->endTiming('render_time');

43. Page Render Example 43

44. Count Your Pageviews 44 $connection = new Domnikl\Statsd\Connection\UdpSocket('localhost'); $client =

    new Domnikl\Statsd\Client($connection, 'orca'); $client->startTiming('render_time'); $application = new Application(); $response = $application->dispatch(); echo $response; $client->increment('pageviews'); $client->endTiming('render_time');

45. Pageview Count Example 45

46. Job Queue Example 46 class Worker { protected $statsd; public

    function run($job) { try { $this->processJob($job); $this->statsd->increment("worker.success"); } catch (\Exception $e) { $this->buryJob($job); $this->statsd->increment("worker.buried"); } } }

47. Job Queue Example 47

48. Logs vs Stats • Why not both? • Logs are

    searchable • Stats are graph-able, visual • Make sure you can correlate logs and stats 48

49. Make it Actionable • You have to actually look at

    this stuff • Identify problems with stats • Investigate problems with logs • Revisit your data collection when you encounter anything serious • Get tools to help you 49

50. Got Budget? 50

51. QUESTIONS Anyone have 51

52. JOIND.IN/15514 I’d love your feedback: 52