PRINCIPLES AND PATTERNS FOR STREAMING DATA ANALYSIS Voxxed Days Ticino Galder Zamarreño Arrizabalaga
 @galderz
 20th October 2018 

@GALDERZ #INFINISPAN #VDT18 2 Since 2006 ENGINEER @galderz Community Lead and Core Developer INFINISPAN CO-FOUNDER (2008) OTIS PAIR PROGRAMMING BUDDY 

@GALDERZ #INFINISPAN #VDT18 3 DATA IS OVERWHELMING US Delays can have a big impact EXPONENTIAL DATA GROWTH YEAR ON YEAR Smartphones, IOT devices, trillions of internet connected devices... REAL-TIME STREAMING DATA PROCESSING IS CHALLENGING 

@GALDERZ #INFINISPAN #VDT18 4 HIGH LEVEL ARCHITECTURE 

@GALDERZ #INFINISPAN #VDT18 5 COLLECTION TIER 

@GALDERZ #INFINISPAN #VDT18 6 COMMON INTERACTION PATTERNS 

@GALDERZ #INFINISPAN #VDT18 7 REQUEST/RESPONSE PATTERN connection request client server response 

@GALDERZ #INFINISPAN #VDT18 8 REQUEST/RESPONSE PATTERN connection request client server response NON-BLOCKING ASYNCHRONOUS BLOCKING SYNCHRONOUS 

@GALDERZ #INFINISPAN #VDT18 9 REQUEST/ACKNOWLEDGMENT PATTERN connection request client server ack 

@GALDERZ #INFINISPAN #VDT18 10 PUBLISH / SUBSCRIBE PATTERN producer consumer broker topic A topic B msg msg subscribe subscribe 

@GALDERZ #INFINISPAN #VDT18 11 ONE-WAY PATTERN connection request client server 

@GALDERZ #INFINISPAN #VDT18 12 STREAM PATTERN connection request client server response response … 

@GALDERZ #INFINISPAN #VDT18 13 MESSAGE QUEUE TIER Decoupling collection and analysis tier 

@GALDERZ #INFINISPAN #VDT18 14 e.g. analysis tier being more processing-intensive or analysis tier consuming messages in batches Fast collection tier combined with slow analysis tier WHY DECOUPLE? 

@GALDERZ #INFINISPAN #VDT18 15 DURABLE MESSAGING Disaster recovery Offline consumption Fault tolerance 

@GALDERZ #INFINISPAN #VDT18 16 DELIVERY SEMANTICS At-least-once : messages not lost but might be repeated Exactly-once : messages not lost and consumed only once At-most-once : messages might get lost 

@GALDERZ #INFINISPAN #VDT18 17 BULLSHIT! 

@GALDERZ #INFINISPAN #VDT18 18 • Guaranteed delivery vs guaranteed processing of message • What if a subscriber consumes the message and then it crashes? • Guaranteed delivery and processing requires application awareness and collaboration • So subscriber can IDEMPOTENTLY process a message and know to which point it's processed it • At this point you're capable of doing at least once • Also requires consumer to acknowledge processing to publisher EXACTLY-ONCE MISLEADING OR LIE! 

@GALDERZ #INFINISPAN #VDT18 19 ANALYSIS TIER 

@GALDERZ #INFINISPAN #VDT18 20 IN-FLIGHT ANALYSIS Traditional RDMS : data at rest and query for answers Streaming : data moved through the query Data always in motion from message queue tier 

@GALDERZ #INFINISPAN #VDT18 21 CONTINUOUS QUERY New data that matches query pushed to client Use cases : tracking behaviour, traffic/safety, fraud analytics... Query constantly evaluated 

@GALDERZ #INFINISPAN #VDT18 22 SLIDING WINDOW e.g. traffic information in my area for last hour Combines queries with time constraints 

@GALDERZ #INFINISPAN #VDT18 23 DATA ACCESS TIER 

@GALDERZ #INFINISPAN #VDT18 24 PROTOCOLS TO SEND DATA TO CLIENTS Protocol Message frequency Communication direction Message latency Efficiency Fault tolerance / Reliability Webhooks Low Uni-directional (server to client) Average Low None HTTP Long Polling Average Bi-directional Average Average None Server-sent events High Uni-directional Low High None by default. Can be implemented. WebSocket s High Bi-directional Low High None by default. Can be implemented. 

@GALDERZ #INFINISPAN #VDT18 25 APPLIED ARCHITECTURE 

@GALDERZ #INFINISPAN #VDT18 26 Platform-as-a-Service (PaaS) Platform for developing and running applications Public or private and multi-language OpenShift is a Kubernetes distro with extras THE PLATFORM 

@GALDERZ #INFINISPAN #VDT18 27 APPLIED ARCHITECTURE 

@GALDERZ #INFINISPAN #VDT18 28 Vert.x is a toolkit for building reactive apps On JVM, event-driven and non-blocking RxJava integrates with Vert.x Great at event transform and coordination Works best with many source of events (modern apps!) THE GLUE 

@GALDERZ #INFINISPAN #VDT18 29 APPLIED ARCHITECTURE 

@GALDERZ #INFINISPAN #VDT18 30 INFINISPAN - IN-MEMORY KEY/VALUE STORE 

@GALDERZ #INFINISPAN #VDT18 31 THE DATA transport.opendata.ch + sbb.ch { "x":"8290840" ,"y":"47483629" ,"name":"IR 1978" ,"poly":[ {"x":"8290840","y":"47483629",...} , {"x":"8290193","y":"47483647"...,"msec":"2000" , ...] } 

@GALDERZ #INFINISPAN #VDT18 32 COMPONENT ARCHITECTURE datagrid infinispan pod infinispan pod infinispan pod datagrid-hotrod service /eventbus/delayed-trains delayed trains /eventbus/delayed-positions app main http vert.x verticle pod station boards vert.x verticle train positions vert.x verticle delayed positions 

@GALDERZ #INFINISPAN #VDT18 33 DEMO TIME! 

THANK YOU! github.com/infinispan-demos/streaming-data-kubernetes infinispan.org redhat.com/en/technologies/jboss-middleware/data-grid openshift.com | vertx.io