Upgrade to Pro — share decks privately, control downloads, hide ads and more …

View, Act, and React: Shaping Business Activity...

View, Act, and React: Shaping Business Activity with Analytics, BigData Queries, and Complex Event Processing

Srinath Perera

November 06, 2013

Transcript

19 slides
  1. View,  Act,  and  React:  Shaping   Business  Ac6vity  with  Analy6cs,

      BigData  Queries,  and  Complex   Event  Processing       Srinath  Perera   Director,  Research   WSO2  
  2. Start   •  1942,  Asimov  wrote  a  book   called

     FoundaAon,  in  which     the  character  Hari  Seldon  use   mathemaAcal  models  to   predict  the  future  of   civilizaAon  and  then  to  save   it.     •  Paul  Krugman,(  the  Nobel   Laureate  in  Economics),  said   his  interest  in  economic   begin  with  foundaAon.     •  We  are  entering  that  Era  of   our  history  where  Mr.   Asimov  might  have  a  point.   Image  cedit,  CC  licence,  hRp:// ansem315.deviantart.com/art/Asimov-­‐ FoundaAon-­‐395188263    
  3. Consider  a  Day  in  your  Life   •  What  is

     the  best  road  to  take?   •  Would  there  be  any  bad   weather?   •  What  is  the  best  way  to  invest   the  money?   •  Should  I  take  that  loan?   •  Is  there  a  way  to  do  this  faster?   •  What  others  did  in  similar   cases?   •  Which  product  should  I  buy?      
  4. Why  it  is  hard?   •  System  build  of  many

      computers  (1000  nodes  to   store  1PB  with  1TB  each)     •  That  handles  lots  of  data  (10Gb   network  =>  83  days  to  copy   1PB)   •  Running  complex  logic  (models   can  be  complex  as  the  system)     •  This  pushes  us  to  the  fronAer   of  Distributed  Systems  and   Databases     hRp://www.flickr.com/photos/mariachily/5250487136,    Licensed  CC  
  5. Event  Streams   •  We  view  the  world  as  event

      streams     Event  stream  is  series  of  events   over  Ame     •  We  use  SQL  like  languages  (Hive/   CEP)  to  process  event  streams  and   create  new  event  streams   { 'name':'PlayStream', 'version':'1.0.0', 'payloadData':[ 'name':'sid', 'ts':'BIGINT', 'x':'DOUBLE', ... ] } Each  stream    has  a  name   Each  event  has   a-ributes,  that  has   types   Select from PlayStream[x>2500 and .. ] İnsert into NearGoalStream
  6. Demo  Usecase  (DEBS  2013)   •  Football  game,  players  and

     ball   has  sensors  (DESB  Challenge   2013)   •  sid,  ts,  x,y,z,  v,a   •  Use  cases:  Running  analysis,   Ball  Possession  and  Shots  on   Goal,  Heatmap  of  AcAvity   •  Siddhi  did  100K+  on  each   usecase   •  For  this  talk,  we  will  look  at  user   acAvity  by  region  of  the  field.    
  7. Data  CollecAon   •  Can  receive   events  via  SOAP,

      HTTP,  JMS,  ..   •  WSO2  Events  is   highly  opAmized   version  (400K   events  TPS)   •  Default  Agents   and  you  can   write  custom   agents.     Agent agent = new Agent(agentConfiguration); publisher = new AsyncDataPublisher( "tcp://localhost:7612", .. ); StreamDefinition definition = new StreamDefinition(STREAM_NAME, VERSION); definition.addPayloadData("sid", STRING); ... publisher.addStreamDefinition(definitio n); ... Event event = new Event(); event.setPayloadData(eventData); publisher.publish(STREAM_NAME, VERSION, event);
  8. BAM  Hive  Query   Find  how  much  Ame  spent  in

     each  cell.     CREATE EXTERNAL TABLE IF NOT EXISTS PlayStream … select sid, ceiling((y+33000)*7/10000 + x/10000) as cell, count(sid) from PlayStream GROUP BY sid, ceiling((y+33000)*7/10000 + x/10000);
  9. CEP  Query   define partition sidPrt by PlayStream.sid, LocBySecStream.sid from

    PlayStream#window.timeBatch(1sec) select sid, avg(x) as xMean, avg(y) as yMean, avg(z) as zMean insert into LocBySecStream partition by sidPrt from every e1 = LocBySecStream -> e2 = LocBySecStream [e1.yMean + 10000 > yMean or yMean + 10000 > e1.yMean] within 2sec select e1.sid insert into LongAdvStream partition by sidPrt ;   Calculate  the   mean  locaAon  of   each  player  every   second     Detect  more   than  10m  run    

More Decks by Srinath Perera

Other Decks in Technology