Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Monitoring and Logging on Doorbell.io
Search
Philip Manavopoulos
June 11, 2014
Programming
2
500
Monitoring and Logging on Doorbell.io
Philip Manavopoulos
June 11, 2014
Tweet
Share
Other Decks in Programming
See All in Programming
童醫院敏捷轉型的實踐經驗
cclai999
0
200
なぜ「共通化」を考え、失敗を繰り返すのか
rinchoku
1
580
AIコーディング道場勉強会#2 君(エンジニア)たちはどう生きるか
misakiotb
1
250
データの民主化を支える、透明性のあるデータ利活用への挑戦 2025-06-25 Database Engineering Meetup#7
y_ken
0
320
Google Agent Development Kit でLINE Botを作ってみた
ymd65536
2
200
技術同人誌をMCP Serverにしてみた
74th
1
440
datadog dash 2025 LLM observability for reliability and stability
ivry_presentationmaterials
0
180
ニーリーにおけるプロダクトエンジニア
nealle
0
610
LINEヤフー データグループ紹介
lycorp_recruit_jp
0
1.2k
Team topologies and the microservice architecture: a synergistic relationship
cer
PRO
0
1.1k
Enterprise Web App. Development (2): Version Control Tool Training Ver. 5.1
knakagawa
1
120
Webの外へ飛び出せ NativePHPが切り拓くPHPの未来
takuyakatsusa
2
440
Featured
See All Featured
Understanding Cognitive Biases in Performance Measurement
bluesmoon
29
1.8k
Making the Leap to Tech Lead
cromwellryan
134
9.4k
Unsuck your backbone
ammeep
671
58k
Designing Dashboards & Data Visualisations in Web Apps
destraynor
231
53k
Into the Great Unknown - MozCon
thekraken
39
1.9k
Code Reviewing Like a Champion
maltzj
524
40k
Why You Should Never Use an ORM
jnunemaker
PRO
58
9.4k
The Pragmatic Product Professional
lauravandoore
35
6.7k
Building Adaptive Systems
keathley
43
2.6k
Typedesign – Prime Four
hannesfritz
42
2.7k
Mobile First: as difficult as doing things right
swwweet
223
9.7k
BBQ
matthewcrist
89
9.7k
Transcript
Monitoring(and(Logging(on( Doorbell.io !@manavo
What%is%Doorbell.io
The$Stack LAMP%with: • PHP$5.5,$using$Laravel$4 • Memcache • Beanstalkd$(message$queue) • Mashape$(3rd$party$APIs$like$senDment$analysis)
• Mandrill/Mailgun$(sending/receiving$emails) • Twilio$(sending$SMS)
Logging&and&Monitoring&101 • Knowing(when(the(site(is(unreachable • Logging(uncaught(PHP(and(JS(errors • Finding(breaking(changes(from(libraries • Making(sure(the(Queue(processor(will(always(be( running
Logging&and&Monitoring&101 • Knowing(when(the(site(is(unreachable Pingdom • Logging(uncaught(PHP(and(JS(errors • Finding(breaking(changes(from(libraries Sentry,.New.Relic •
Making(sure(the(Queue(processor(will(always(be( running Supervisor
Case%study%#1:%Queues
Queues%in%Laravel Queue::push( 'SendEmail@send', array('thread_id' => 2) ); OR Queue::push(function($job) use
($id) { Account::delete($id); $job->delete(); });
Queues%in%Doorbell.io%.%New%feedback 1. Queue&closure 2. Closure&loops&through&email&addresses&to&no7fy 3. Send&email 4. Delete&job
Problem(#1:(Dele-ng(the(job(at(the(end // Done with the job, delete it $job->delete(); If#an#error#occurs,#the#job#never#gets#deleted.
There%was%no%maximum%number%of%tries. So#it#runs#again,#and#again,#and#again,#and#again,# and....
Problem(#1(solu-on Quick&fix: 1. Manually)delete)job)from)Queue 2. Stop)using)the)Queue)for)this)specific)problem Long%term*fix: Figure'out'why'the'Queue'processor'failed
Problem(#2 A"good"message:
Problem(#2 A"bad"message: This%means%Greek%in%the%serialized%data...
Which%leads%to... Blocked(Queue Saved&by Because'CPU'usage'was'100%
Moral&of&the&story Moved&the&$job->delete()&call&to&the&top&of&the& func0on,&so&it&won't&keep&retrying&forever Make%sure%you%fix%the%root%cause%of%the%problem,%or% you'll%have%the%same%thing%happening%again
Case%study%#2:%External%services
Case%study%#2:%External%services • Easily(add(func.onality(to(applica.on((sen.ment( analysis,(etc) • They(break,(and(it's(out(of(your(control
Examples • Sen%ment(analysis(failed(occasionally • Incoming(webhooks(from(Mailgun(failed • FogBugz(makes(a(change(to(authen%ca%on
Updated'Stack • Pingdom • New,Relic • Sentry • Supervisor •
Papertrailapp.com • Custom0dashboard
Papertrail
Logging&in&PHP Good$library$to$use:$Monolog h4ps:/ /github.com/Seldaek/monolog
Custom'Dashboard Basic,'but'enough'to'see'threads'with'no'sen3ment.
Moral&of&the&story Logs%hosted%on%a%server%don't%allow%you%to%be%reac5ve
Case%study%#3:%Your%own%bugs,%and% mysteries
Example(#1:(Innocent(looking(commit Saves&making&a&query&to&the&sen3ment&analysis&API
Reality(of(commit Hard%coded%every%new%feedback%thread%to%be% posi5ve,%without%analyzing... Which%leads%to:
Example(#2:(Stuck(queue,(New(Relic(doesn't( report(CPU(100% For$reasons$s)ll$unknown$to$me,$the$queue$got$stuck,$ and$didn't$know$it. This%means: • Not%running%sen-ment%analysis • No%email%no-fica-ons%sent%for%new%feedback% threads%or%replies
Even%worse... People&reported&on&Doorbell&itself&the&bug,&and&I&got& no&no6fica6on...
Report'#1
Report'#2
Solu%on'#1:'Charts Replies'per'day
Solu%on'#1:'Charts Sen$ment'analysis'of'new'threads'per'day'now
Solu%on'#1:'Charts Sen$ment'analysis'of'new'threads'per'day'before'fix
Solu%on'#2:'Extend'Custom'Dashboard
Solu%on'#3:'Email'report'cron Cron%running%every%10%minutes,%le3ng%me%know%if%the% queue%is%not%empty.
Conclusion • It$is$necessary • Good$monitoring/logging$is$possible$on$a$budget • Reac8ve$not$proac8ve • Peace$of$mind
Ques%ons?