View, Act, and React: Shaping Business Activity with Analytics, BigData Queries, and Complex Event Processing

Transcript

1. View,  Act,  and  React:  Shaping   Business  Ac6vity  with  Analy6cs,   BigData  Queries,  and  Complex   Event  Processing       Srinath  Perera   Director,  Research   WSO2  
2. Start   •  1942,  Asimov  wrote  a  book   called  FoundaAon,  in  which     the  character  Hari  Seldon  use   mathemaAcal  models  to   predict  the  future  of   civilizaAon  and  then  to  save   it.     •  Paul  Krugman,(  the  Nobel   Laureate  in  Economics),  said   his  interest  in  economic   begin  with  foundaAon.     •  We  are  entering  that  Era  of   our  history  where  Mr.   Asimov  might  have  a  point.   Image  cedit,  CC  licence,  hRp:// ansem315.deviantart.com/art/Asimov-­‐ FoundaAon-­‐395188263    
3. Consider  a  Day  in  your  Life   •  What  is  the  best  road  to  take?   •  Would  there  be  any  bad   weather?   •  What  is  the  best  way  to  invest   the  money?   •  Should  I  take  that  loan?   •  Is  there  a  way  to  do  this  faster?   •  What  others  did  in  similar   cases?   •  Which  product  should  I  buy?      
4. Bigdata  Landscape  
5. Big  Data  Architecture  
6. Why  it  is  hard?   •  System  build  of  many   computers  (1000  nodes  to   store  1PB  with  1TB  each)     •  That  handles  lots  of  data  (10Gb   network  =>  83  days  to  copy   1PB)   •  Running  complex  logic  (models   can  be  complex  as  the  system)     •  This  pushes  us  to  the  fronAer   of  Distributed  Systems  and   Databases     hRp://www.flickr.com/photos/mariachily/5250487136,    Licensed  CC  
7. Big  Data  Architecture  with  WSO2  
8. Event  Streams   •  We  view  the  world  as  event   streams     Event  stream  is  series  of  events   over  Ame     •  We  use  SQL  like  languages  (Hive/   CEP)  to  process  event  streams  and   create  new  event  streams   { 'name':'PlayStream', 'version':'1.0.0', 'payloadData':[ 'name':'sid', 'ts':'BIGINT', 'x':'DOUBLE', ... ] } Each  stream    has  a  name   Each  event  has   a-ributes,  that  has   types   Select from PlayStream[x>2500 and .. ] İnsert into NearGoalStream
9. Demo  Usecase  (DEBS  2013)   •  Football  game,  players  and  ball   has  sensors  (DESB  Challenge   2013)   •  sid,  ts,  x,y,z,  v,a   •  Use  cases:  Running  analysis,   Ball  Possession  and  Shots  on   Goal,  Heatmap  of  AcAvity   •  Siddhi  did  100K+  on  each   usecase   •  For  this  talk,  we  will  look  at  user   acAvity  by  region  of  the  field.    
10. Demo  High-­‐level  Architecture  
11. Data  CollecAon   •  Can  receive   events  via  SOAP,   HTTP,  JMS,  ..   •  WSO2  Events  is   highly  opAmized   version  (400K   events  TPS)   •  Default  Agents   and  you  can   write  custom   agents.     Agent agent = new Agent(agentConfiguration); publisher = new AsyncDataPublisher( "tcp://localhost:7612", .. ); StreamDefinition definition = new StreamDefinition(STREAM_NAME, VERSION); definition.addPayloadData("sid", STRING); ... publisher.addStreamDefinition(definitio n); ... Event event = new Event(); event.setPayloadData(eventData); publisher.publish(STREAM_NAME, VERSION, event);
12. Business  AcAvity  Monitor  
13. BAM  Hive  Query   Find  how  much  Ame  spent  in  each  cell.     CREATE EXTERNAL TABLE IF NOT EXISTS PlayStream … select sid, ceiling((y+33000)*7/10000 + x/10000) as cell, count(sid) from PlayStream GROUP BY sid, ceiling((y+33000)*7/10000 + x/10000);
14. Complex  Event  Processor  
15. CEP  Query   define partition sidPrt by PlayStream.sid, LocBySecStream.sid from PlayStream#window.timeBatch(1sec) select sid, avg(x) as xMean, avg(y) as yMean, avg(z) as zMean insert into LocBySecStream partition by sidPrt from every e1 = LocBySecStream -> e2 = LocBySecStream [e1.yMean + 10000 > yMean or yMean + 10000 > e1.yMean] within 2sec select e1.sid insert into LongAdvStream partition by sidPrt ;   Calculate  the   mean  locaAon  of   each  player  every   second     Detect  more   than  10m  run    
16. Run  Demo  
17. VisualizaAon  
18. Conclusion  
19. Thank  You