Monitoring and Logging on Doorbell.io

Transcript

1. Monitoring(and(Logging(on( Doorbell.io !@manavo
2. What%is%Doorbell.io
3. The$Stack LAMP%with: • PHP$5.5,$using$Laravel$4 • Memcache • Beanstalkd$(message$queue) • Mashape$(3rd$party$APIs$like$senDment$analysis) • Mandrill/Mailgun$(sending/receiving$emails) • Twilio$(sending$SMS)
4. Logging&and&Monitoring&101 • Knowing(when(the(site(is(unreachable • Logging(uncaught(PHP(and(JS(errors • Finding(breaking(changes(from(libraries • Making(sure(the(Queue(processor(will(always(be( running
5. Logging&and&Monitoring&101 • Knowing(when(the(site(is(unreachable Pingdom • Logging(uncaught(PHP(and(JS(errors • Finding(breaking(changes(from(libraries Sentry,.New.Relic • Making(sure(the(Queue(processor(will(always(be( running Supervisor
6. Case%study%#1:%Queues
7. Queues%in%Laravel Queue::push( 'SendEmail@send', array('thread_id' => 2) ); OR Queue::push(function($job) use ($id) { Account::delete($id); $job->delete(); });
8. Queues%in%Doorbell.io%.%New%feedback 1. Queue&closure 2. Closure&loops&through&email&addresses&to&no7fy 3. Send&email 4. Delete&job
9. Problem(#1:(Dele-ng(the(job(at(the(end // Done with the job, delete it $job->delete(); If#an#error#occurs,#the#job#never#gets#deleted. There%was%no%maximum%number%of%tries. So#it#runs#again,#and#again,#and#again,#and#again,# and....
10. Problem(#1(solu-on Quick&fix: 1. Manually)delete)job)from)Queue 2. Stop)using)the)Queue)for)this)specific)problem Long%term*fix: Figure'out'why'the'Queue'processor'failed
11. Problem(#2 A"good"message:
12. Problem(#2 A"bad"message: This%means%Greek%in%the%serialized%data...
13. Which%leads%to... Blocked(Queue Saved&by Because'CPU'usage'was'100%
14. Moral&of&the&story Moved&the&$job->delete()&call&to&the&top&of&the& func0on,&so&it&won't&keep&retrying&forever Make%sure%you%fix%the%root%cause%of%the%problem,%or% you'll%have%the%same%thing%happening%again
15. Case%study%#2:%External%services
16. Case%study%#2:%External%services • Easily(add(func.onality(to(applica.on((sen.ment( analysis,(etc) • They(break,(and(it's(out(of(your(control
17. Examples • Sen%ment(analysis(failed(occasionally • Incoming(webhooks(from(Mailgun(failed • FogBugz(makes(a(change(to(authen%ca%on
18. Updated'Stack • Pingdom • New,Relic • Sentry • Supervisor • Papertrailapp.com • Custom0dashboard
19. Papertrail
20. Logging&in&PHP Good$library$to$use:$Monolog h4ps:/ /github.com/Seldaek/monolog
21. Custom'Dashboard Basic,'but'enough'to'see'threads'with'no'sen3ment.
22. Moral&of&the&story Logs%hosted%on%a%server%don't%allow%you%to%be%reac5ve
23. Case%study%#3:%Your%own%bugs,%and% mysteries
24. Example(#1:(Innocent(looking(commit Saves&making&a&query&to&the&sen3ment&analysis&API
25. Reality(of(commit Hard%coded%every%new%feedback%thread%to%be% posi5ve,%without%analyzing... Which%leads%to:
26. Example(#2:(Stuck(queue,(New(Relic(doesn't( report(CPU(100% For$reasons$s)ll$unknown$to$me,$the$queue$got$stuck,$ and$didn't$know$it. This%means: • Not%running%sen-ment%analysis • No%email%no-fica-ons%sent%for%new%feedback% threads%or%replies
27. Even%worse... People&reported&on&Doorbell&itself&the&bug,&and&I&got& no&no6fica6on...
28. Report'#1
29. Report'#2
30. Solu%on'#1:'Charts Replies'per'day
31. Solu%on'#1:'Charts Sen$ment'analysis'of'new'threads'per'day'now
32. Solu%on'#1:'Charts Sen$ment'analysis'of'new'threads'per'day'before'fix
33. Solu%on'#2:'Extend'Custom'Dashboard
34. Solu%on'#3:'Email'report'cron Cron%running%every%10%minutes,%le3ng%me%know%if%the% queue%is%not%empty.
35. Conclusion • It$is$necessary • Good$monitoring/logging$is$possible$on$a$budget • Reac8ve$not$proac8ve • Peace$of$mind
36. Ques%ons?