Building a Firehose with NodeJS

August 2011 Forward First Tuesday - building a firehose in nodeJS

Transcript

1. Firehose   Building    a  real-­‐1me  data  stream  with  NodeJS     Luca  Grulla@Forward   London,  2/8/2011    
2. Introduc1on   Or  what  we  wanted  to  build  
3. We  like  to  have  a  (near)  real-­‐1me  view   of  the  data  stream  
4.     Resilient    
5.     Scalable  
6.     Accurate   (but  not  necessarily  precise)  
8. Verse  and  Chorus   Or  how  we  built  it  
9. Stack   •  NodeJS  (0.4.9)   •  CoffeeScript   •  ZeroMQ  
10. Architecture  
11. Interlude   Or  joys  and  pi4alls  of  building  a  tail   library  for  node  
12. Living  the  asynchronous  dream    
13. Living  the  asynchronous  dream     •  Error  manifests  itself  only  at  high  throughput   (7MB/secs)   •  The  few  implementa1ons  you  can  find  around   are  wrong  (including  Log.io  –  we  are  planning   to  contribute  back)  
14. Breathing  in  an  asynchronous    way  
15. Learnings   •  You  need  to  think  asynchronous  to  write   asynchronous  code   •  Dispatch  internal  events  if  you  need  to  keep   control  of  the  flow   •  “Internal  queues”  to  scale(up  to  a  certain   limit)  
16. Performance   •  Beyond  5MB  we  start  to  fill  the  internal  queue   •  We  are  limited  by  IO,  not  from  CPU/memory   •  Consistent  throughput  
17. Solo   Or  a  short  rant  on  declara<ve  vs   impera<ve  style  
18. Event-­‐driven  approach   •  Event-­‐driven  is  implicitly  declara1ve   – You  tell  to  the  world  that  something  just   happened  and  do  not  care  of  what  next   •  Fire  events  instead  of  passing  around  states   – It’s  cleaner   – It’s  simpler  to  maintain   – It’s  idioma1c  
20.     Ques1ons  ?  
21. References   •  npm  install  tail   •  hhps://github.com/forward/node-­‐tail