Rise of the Real-time Stack

B45193666b6922c140244eb153cc5f3c?s=47 Druid
May 06, 2014

Rise of the Real-time Stack

Data Collective and Metamarkets hosted leading technologists in the open source world for drinks, food and conversation about the rise of real-time processing systems, the current limitations of the data engineering space, the value of open source solutions, and where real-time technology is headed.

B45193666b6922c140244eb153cc5f3c?s=128

Druid

May 06, 2014
Tweet

Transcript

  1. present     The  Rise  of  the  Real-­‐Time  Stack:  

    An  Evening  with  Open  Source  Technologists     Storm            Druid            Ka-a            Spark                     @dcvc   @metamarkets   @foundersden     &  
  2. Opening  Remarks:       Mike  Driscoll  &  Fangjin  Yang

      Metamarkets  
  3. ENGINEERING  AT  MMX:  A  BRIEF  HISTORY   Event Streams 2013

    Insight Hadoop
  4. ENGINEERING  AT  MMX:  A  BRIEF  HISTORY   Event Streams 2013

    Insight Hadoop Hadoop Druid
  5. ENGINEERING  AT  MMX:  A  BRIEF  HISTORY   2013 Insight Event

    Streams Hadoop Hadoop Druid Kafka Storm MR (Spark/ Hadoop) Druid
  6. Next:       Andy  Feng,    Yahoo!   STORM

       
  7. Apache  Storm:   Distributed  RealIme  CompuIng   Andy  Feng  (afeng@apache.org)

      DisInguished  Architect,  Yahoo   CommiPer,  Apache  Storm  
  8. Apache  Storm   •  “Hadoop  for  RealIme”   –  a

     distributed,  fault-­‐tolerant,  and  high-­‐performance  realIme   computaIon  system  that  provides  strong  guarantees  on  the   processing  of  data   •  Emerging  standard   –  IniIally  developed  and  deployed  at  BackType  in  2011   –  Open  sourced  via  github.com  in  September  2011   –  Apache  incubaIon  in  September  2013   •  Apache  Storm  0.9.1-­‐incuba6ng  release  coming   –  7  commiPers,  50  contributors,  12,000  readers  
  9. Companies  Using  Storm   and  many  others  (hPps://github.com/nathanmarz/storm/wiki/Powered-­‐By)  

  10. •  100  Topologies,  400  nodes   &  growing   – 

    Ad  budget  management   –  Ad  preprocessing   –  Content  preprocessing   –  Social  signal  processing   –  User  interest  understanding   –  Yahoo  Finance   –  Yahoo  Weather   –  E-­‐Commerce   –  System  monitoring   –  …   Storm  Use  Cases  @  Yahoo  
  11. Storm  Concept   •  Data   –  Stream   • 

    Unbounded  sequence  of  tuples   –  Tuple   •  Ordered  list  of  elements     •  ApplicaIon   –  Topology   •  DAG  of  spouts  &  bolts   –  Spout   •  Source  of  Stream   –  Bolt   •  Processes  input  streams     •  Produces  new  streams    
  12. Parallelism  &  RouIng   N  tasks  for  1  spout/bolt  

    RouMng   •  Shuffle  grouping   –  Tuples  are  randomly  distributed   •  Fields  grouping   –  The  stream  is  parIIoned  by  the  specified   fields   •  All  grouping   –  The  stream  is  replicated  to  ALL  tasks     •  Global  grouping   –  The  stream  goes  to  a  SINGLE  task   •  Local  or  shuffle  grouping   –  Local  tasks  within  the  current  process   tasks  preferred,  otherwise  apply  shuffle   grouping   •  Direct  grouping   –  producer  selects  receiving  task   •  Customized  rouMng    
  13. Reliability  Guarantees     Tuple  tree     3  opMons

     of  guarantees     13 •  BEST  EFFORT   –  STORM  will  not  track  tuple  tree   •  AT  LEAST  ONCE   –  A  spout  tuple  is  not  fully  processed   unIl  all  tuples  in  the  tree  have  been   completed   –  Storm  tracks  tuple  trees  for  you  in  an   extremely  efficient  way   •  EXACT  ONCE   –  A  small  batch  of  tuples  are  processed   at  a  Ime   –  Each  batch  completely  succeeds  or   completely  fails   X  
  14. •  High-­‐level  operaIons   –  Filter,  Join,  Group,  AggregaIon,  

    FuncIon   •  Exactly-­‐once  guarantee   –  Stream  =  Small  batches  of  tuples   –  State  updates  are  ordered  among   batches     •  Stateful  processing   –  Ex.,  In-­‐Memory,  Memcached,   Hbase     Trident:  Higher  AbstracIon  
  15. Storm  Topology  on  Cluster   Worker   Nimbus   Zoo

      keepers   Supervisor   Worker   Worker   2 Zoo   keepers   Supervisor   Supervisor   Worker   Supervisor   1 3 4 5 6 7
  16. Storm:  Player  in  Big-­‐Data  Ecosystem  

  17. AddiIonal  Info   •  Documents   –  hPps://github.com/nathanmarz/storm/wiki   – 

    hPp://storm-­‐project.net/   •  Code   –  hPps://github.com/apache/incubator-­‐storm   •  Mailing  Lists   –  Storm  users  …  user@storm.incubator.apache.org   –  Storm  developers  …  dev@storm.incubator.apache.org  
  18. Backup  

  19. Storm  Enhanced  for  MulI-­‐tenancy  @  Yahoo   hPps://github.com/yahoo/incubator-­‐storm/tree/security  

  20. Next:       Eric  TscheTer,  Creator  of  Druid  

      DRUID      
  21. DRUID:     IMMEDIATE,  SLICE-­‐N-­‐DICE     AGGREGATION  ENGINE  

      ERIC TSCHETTER
  22. “ ” “ ” REQUIREMENTS   REQUIREMENTS  

  23. None
  24. INGESTION  

  25. None
  26. SLICE-­‐N-­‐DICE  

  27. None
  28. AVAILABLE  

  29. • Ingestion •  Make it queryable in real-time • Slice-N-Dice •  Arbitrary

    boolean filters • Available •  Downtime is evil REQUIREMENTS  
  30. “ ” HOW?  

  31. * , '

  32. “SUM  OF  PARTS”  ARCHITECTURE  

  33. WHERE  DOES  IT  FIT?   Storm Kafka Druid Spark

  34. BUZZWORD  SOUP   •Column-oriented   • Only scan what you need

    •Distributed   • Parallelize through horizontal scaling • Fault-tolerance through replication •Real-time   • Make data queryable immediately upon ingestion
  35. •Bitmap Indexes   • Apply Boolean filters without looking at data

    •Compressed   • Dictionary encoding • LZF •Fault-tolerant   • Hot Replication BUZZWORD  SOUP  -­‐  PART  2   • Rebalancing
  36. •Approximation   • HyperLogLog • Approximate TopN •Caching   • Perfect cache invalidation

    • Per-segment cache   •Rolling Deploys BUZZWORD  SOUP  -­‐  PART  3   • No downtime software deploys
  37. CLUSTER  SIZES   •Cluster 1   •  > 3 trillion

    events (impressions, bids, etc.) •  > 110 billion rows •  80 machines •  90% query latency < 1s, 95% < 2s •Cluster 2   • real-time 150k events/s (7B events/day)
  38. DRUID  IS  OPEN   URL:  hPp://www.druid.io     LICENSE:  GPL

     v2     Get  Involved:     github.com/metamx/druid     druid-­‐development@googlegroups.com     hTps://groups.google.com/d/forum/druid-­‐ development     #druid-­‐dev  on  irc.freenode.net    
  39. Next:       Jun  Rao,  LinkedIn   KAFKA  

     
  40. Building  a  Real-­‐Time  Data  Pipeline:   Apache  Kapa  at  Linkedin

      Rise  of  the  Real-­‐Time  Stack     Jun  Rao   Feb  2014  
  41. We  have  a  lot  of  data.   We  want  to

     leverage  this  data  to  build  products.     Data pipeline
  42. Network  update  stream  

  43. System  and  applicaIon  metrics/ logging  

  44. Point-­‐to-­‐point  pipelines   Oracle Oracle Oracle User Tracking Hadoop Log

    Search Monitoring Data Warehous e Social Graph Rec. Engine Search Email Voldemort Voldemort Voldemort Espresso Espresso Espresso Logs Operational Metrics Production Services ... Security
  45. Oracle Oracle Oracle User Tracking Hadoop Log Search Monitorin g

    Data Warehous e Social Graph Rec Engine & Life Search Email Voldemort Voldemort Voldemort Espresso Espresso Espresso Logs Operational Metrics Production Services ... Security Data Pipeline Central  data  pipeline  
  46. Apache  Kapa:  a  messaging  system   Topic 1 Topic 2

    Producers Producers Consumer Consumer Topic partitions Brokers
  47. Features  and  technologies   •  High  throughput   –  Simple

     storage,  batch  api,  zero-­‐copy  transfer   •  Distributed   –  Built-­‐in  cluster  management,  parallel  coordinated   consumpIon   •  Fault  tolerant   –  Auto  data  replicaIon,  auto  failover  on  producer/ consumer    
  48. Usage  at  LinkedIn   •  16  brokers  in  each  cluster

      •  28  billion  messages/day   •  Peak  rates   –  Writes:  460,000  messages/second   –  Reads:  2,300,000  messages/second   •  ~  700  topics   •  Every  producIon  service  is  a  producer   •  ~50  live  services  consuming  user-­‐acIvity  data   •  Many  ad  hoc  consumers   •  3k  connecIons/broker  
  49. Next:       Matei  Zaharia,    Databricks   SPARK

       
  50. Panel   Next:      

  51. Q&A     Next:      

  52. Thanks  For  Coming!     THE  END