Streaming Analytics - Comparison of Open Source Frameworks and Products

Slide 1

Slide 1 text

Fast Data and Streaming Analytics in the Era of Hadoop, R and Apache Spark Kai Wähner [email protected] @KaiWaehner www.kai-waehner.de LinkedIn / Xing  Please connect!

Slide 2

Slide 2 text

Key Messages – Streaming Analytics processes Data while it is in Motion! – Automation and Proactive Human Interaction are BOTH needed! – Time to Market is the Key Requirement for most Use Cases!

Slide 3

Slide 3 text

Agenda – Real World Use Cases – Introduction to Stream Processing – Market Overview – Relation to other Big Data Components

Slide 4

Slide 4 text

Agenda – Real World Use Cases – Introduction to Stream Processing – Market Overview – Relation to other Big Data Components

Slide 5

Slide 5 text

© Copyright 2015 TIBCO Software Inc. Find and Act on “Critical Business Moments” “Business Moments” occur in Every Facet of Enterprise Operations, they drive competitive differentiation, customer satisfaction and business success! Optimize Pricing Identify fraud Make cross- sell offers Restock inventory Reroute trucks Deliver proactive customer service Predict equipment failure & fix proactively Anticipate and handle disruptions

Slide 6

Slide 6 text

Operational Intelligence in Action © Copyright 2000-2015 TIBCO Software Inc. Actions by Operations Human decisions in real time informed by up to date information The Challenge: Empower operations staff to see and seize key business moments 6 Automated action based on models of history combined with live context and business rules The Challenge: Create, understand, and deploy algorithms & rules that automate key business reactions Machine-to-Machine Automation

Slide 7

Slide 7 text

Slide 8

Slide 8 text

Slide 9

Slide 9 text

Data Monitoring • Motor temperature • Motor vibration • Current • Intake pressure • Intake temperature  Flow Electrical power cable Pump Intake Protector ESP motor Pump monitoring unit Electric Submersible Pumps (ESP) Predictive Analytics (Fault Management)

Slide 10

Slide 10 text

Voltage Temperature Vibration Device history Temporal analytic: “If vibration spike is followed by temp spike then voltage spike [within 12 minutes] then flag high severity alert.” Predictive Analytics (Fault Management)

Slide 11

Slide 11 text

Live Surveillance of Equipment © Copyright 2000-2014 TIBCO Software Inc. Continuous, live geospatial display of pump health and predictive signal breeches Alerts based on predictive signals Compare live readings and signals to historical average and means Continuous, live visualization of stats per 100’s of wells

Slide 12

Slide 12 text

Slide 13

Slide 13 text

IoT for High Tech Manufacturing Yield Optimization © Copyright 2000-2014 TIBCO Software Inc. • Before: Solar Panel Manufacturer with No Unified View of Manufacturing Process – Multiple manufacturing facilities, multiple processes – no way to compare production to yield expectations • Negative Consequences: Sub-Optimal Production – Operations are sub-optimal: high tolerance leads to better yield but less output; tight tolerance means high throughput but lower yield • Business Outcome: Higher Yield and More Runs – Process Manufacturing can run tighter tolerances and adjust them mid-run, predicting yield and adjusting to changing variables – Systems proactively re-route high-value customers around affected network areas in real-time • How We Do It: The TIBCO Fast Data Platform – IoT, Spotfire, StreamBase, and TERR for predictive modeling, high-speed network by TIBCO “For every 1% increase in shipped product, we make $11MM in profit. The demand is there, we just need to fulfill it.” - Head of Quality, Solar Panel Manufacturer

Slide 14

Slide 14 text

Slide 15

Slide 15 text

High Tech Manufacturing Yield Optimization © Copyright 2000-2014 TIBCO Software Inc. Continuously computed real-time analytics on streams by StreamBase (thresholds, min / max, average) Analysis, alerts and triggers are based on streaming analytics

Slide 16

Slide 16 text

High Tech Manufacturing Yield Optimization © Copyright 2000-2014 TIBCO Software Inc. Manufacturing operations staff drill down on any machine, any time, to inspect and fix problems before they impact yield

Slide 17

Slide 17 text

Slide 18

Slide 18 text

Slide 19

Slide 19 text

Slide 20

Slide 20 text

Slide 21

Slide 21 text

Slide 22

Slide 22 text

Slide 23

Slide 23 text

Slide 24

Slide 24 text

Slide 25

Slide 25 text

Slide 26

Slide 26 text

Slide 27

Slide 27 text

Slide 28

Slide 28 text

Challenges of the 21st Century Retailer • Retailing and Retail Challenges are changing • Consumers expect better and integrated customer experience across all channels – Rapid adoption of mobile is a major driver – Customers want an integrated service across physical and digital channels… Simultaneously – Customer experience is becoming one of the main differentiators • Real-Time, one-on-one marketing can: – Improve a retailer’s relevance with the customer – Increase customer wallet-share • Key to being able to achieve this is: – Identifying and knowing your customer, in depth in real-time – Understanding the opportunity their past behavior reveals – Understanding your inventory (availability, velocity, pipeline)

Slide 29

Slide 29 text

29 © Copyright 2000-2014 TIBCO Software Inc. All Customers are different… Treat them that way… Capture – Engage – Expand - Monetize Patterns – Real time MORE PERSONAL MORE CONTEXT social CRM POS mobile web e-mails

Slide 30

Slide 30 text

Slide 31

Slide 31 text

New Real-Time Fraud Detection Based on Deep Historical Insight Real-time fraud action can be taken based on historical insight – system not “whiplashed” by real-time events Streaming Analytics for Gift Card Fraud Protection

Slide 32

Slide 32 text

32 © Copyright 2000-2015 TIBCO Software Inc. Internet of Things Hybrid Stores Smart Tags Smart Shelves Smart Warehouse Faster Delivery Buy Online Pickup at Store Same Day Delivery Omni Channel 2.0 Store Fulfillment Social Media Predictive Shopping National Retailer Loyalty 2018

Slide 33

Slide 33 text

Slide 34

Slide 34 text

34 © Copyright 2000-2014 TIBCO Software Inc. Real Time Close Loop Model Develop model Deploy into Stream Processing flow Act Automatically monitor real-time transactions Automatically trigger action Analyze Analyze data via Data Discovery Uncover patterns, trends, correlations

Slide 35

Slide 35 text

Agenda – Real World Use Cases – Introduction to Stream Processing – Market Overview – Relation to other Big Data Components

Slide 36

Slide 36 text

Traditional Data Processing: Challenges • Introduces too much “decision latency” into the business. • Responses are delivered “after-the- fact”. • Maximum value of the identified situation is lost. – Cross-sell / up-sell opportunities are lost, impending equipment failure is missed, business processes are slow to respond and lack timely context. • Decisions are made on old and stale data. © Copyright 2000-2015 TIBCO Software Inc. Store Analyze Act

Slide 37

Slide 37 text

The New Era: Fast Data Processing • Events are analyzed and processed in real-time as they arrive. • Decisions are timely, contextual, and based on fresh data. • Decision latency is eliminated, resulting in:  Superior Customer Experience  Operational Excellence  Instant Awareness and Timely Decisions © Copyright 2000-2015 TIBCO Software Inc. Act Analyze Store

Slide 38

Slide 38 text

Slide 39

Slide 39 text

39 Act while data is in motion! Time Business Value Business Event Data Ready for Analysis Analysis Completed Decision Made $$$$ $$$ $$ $ Action Taken Stream Processing speeds action and increases business value by seizing opportunities while they matter

Slide 40

Slide 40 text

Operational Analytics Operations Live UI SENSOR DATA TRANSACTIONS MESSAGE BUS MACHINE DATA SOCIAL DATA Streaming Analytics Action Aggregate Rules Stream Processing Analytics Correlate Live Datamart Continuous query processing Alerts Manual action, escalation HISTORICAL ANALYSIS MS Excel SAS Data Scientists Cleansed Data History Data Discovery R Enterprise Service Bus ERP MDM DB WMS SOA BIG DATA Data Warehouse, Hadoop Internal Data Integration Bus API Event Server Streaming Analytics Reference Architecture Spark

Slide 41

Slide 41 text

Agenda – Real World Use Cases – Introduction to Stream Processing – Market Overview – Relation to other Big Data Components

Slide 42

Slide 42 text

Slide 43

Slide 43 text

Slide 44

Slide 44 text

Concepts (Continuous Queries, Sliding Windows) Patterns (Counting, Sequencing, Tracking, Trends) Build everything by yourself!  45 What Streaming Alternative do you need? Time to Market Streaming Frameworks Streaming Products Slow Fast Streaming Concepts © Copyright 2000-2015 TIBCO Software Inc.

Slide 45

Slide 45 text

Slide 46

Slide 46 text

Slide 47

Slide 47 text

Library (Java, .NET, Python) Query Language (often similar to SQL) Scalability (horizontal and vertical, fail over) Connectivity (technologies, markets, products) Operators (Filter, Sort, Aggregate) 48 What Streaming Alternative do you need? Time to Market Streaming Frameworks Streaming Products Slow Fast Streaming Concepts © Copyright 2000-2015 TIBCO Software Inc.

Slide 48

Slide 48 text

Slide 49

Slide 49 text

Slide 50

Slide 50 text

Slide 51

Slide 51 text

Slide 52

Slide 52 text

53 Amazon Kinesis – The Cloud ... © Copyright 2000-2015 TIBCO Software Inc. … is easy to setup and scale! But you do not have full control  • Any data that is older than 24 hours is automatically deleted • Every Kinesis application consists of just one procedure, so you can’t use Kinesis to perform complex stream processing unless you connect multiple applications • Kinesis can only support a maximum size of 50KB for each data item http://diamondstream.com/amazon-kinesis-big-real-time-data-processing-solution/ (blog post from 2014, might be outdated, but shows that you do not have full control over a cloud service)

Slide 53

Slide 53 text

54 Apache Spark © Copyright 2000-2015 TIBCO Software Inc. General Data-processing Framework  However, focus is especially on Analytics (these days) http://fortune.com/2015/09/09/cloudera-spark-mapreduce/

Slide 54

Slide 54 text

55 Apache Spark – Focus on Analytics © Copyright 2000-2015 TIBCO Software Inc. http://aptuz.com/blog/is-apache-spark-going-to-replace-hadoop/ http://fortune.com/2015/09/09/cloudera-spark-mapreduce/ http://www.ebaytechblog.com/2014/05/28/using-spark-to-ignite-data-analytics/ http://www.forbes.com/sites/paulmiller/2015/06/15/ibm-backs-apache-spark-for-big-data-analytics/ “[IBM’s initiatives] include: • deepening the integration between Apache Spark and existing IBM products like the Watson Health Cloud; • open sourcing IBM’s existing SystemML machine learning technology;

Slide 55

Slide 55 text

56 Spark Streaming © Copyright 2000-2015 TIBCO Software Inc. Spark Streaming • is no real streaming solution • uses micro-batches • cannot process data in real-time (i.e. no ultra-low latency) • allows easy combination with other Spark components (SQL, Machine Learning, etc.)

Slide 56

Slide 56 text

Slide 57

Slide 57 text

Slide 58

Slide 58 text

Visual IDE (Dev, Test, Debug) Simulation (Feed Testing, Test Generation) Live UI (monitoring, proactive interaction) Maturity (24h support, consulting) Integration (ootb integration: ESB, MDM, etc.) Library (Java, .NET, Python) Query Language (often similar to SQL) Scalability (horizontal and vertical, fail over) Connectivity (technologies, markets, products) Operators (Filter, Sort, Aggregate) What Streaming Alternative do you need? Time to Market Streaming Frameworks Streaming Products Slow Fast Streaming Concepts

Slide 59

Slide 59 text

Slide 60

Slide 60 text

Slide 61

Slide 61 text

TIBCO StreamBase • Performance: Latency, Throughput, Scalability – Multi-threaded and clustered server from version 1 – High throughput: Millions of messages, 100,000s of quotes, 10,000s of orders – Low-latency: microsecond latency for algo trading, pre-trade risk, market data • Take Advantage of High Performance Hardware – Multicore (12, 24, 32 core) large memory (10s of gigabytes) – 64-bit Linux, Windows, Solaris deployment – Hardware acceleration (GPU, Solace, Tervela) • Enterprise Deployment – High availability and fault tolerance – Distributed state management for large data sets – Management and monitoring tools – Security and entitlements Integration – Continuous deployment and QA Process Support StreamSQL compiler and static optimizer In process, in thread adapter architecture Visual parallelism and scaling ActiveSpaces integration for distributed shared state Data parallelism and dispatch StreamBase Server Innovations “The StreamBase engine is for real. We couldn’t break it, and believe me, I tried” SVP Development, Top 5 Broker Dealer

Slide 62

Slide 62 text

StreamBase: The Power of Visual Programming © Copyright 2000-2015 TIBCO Software Inc. 1) Get ideas into market in days or weeks, not months or years 2) Unlock the power of IT and data scientists working together

Slide 63

Slide 63 text

64 © Copyright 2000-2013 TIBCO Software Inc. Code Anyone Can Read Limit Gift Card Activation Amounts at One Location Aggregate Capture card activations per location Sales too high! Log to any database No Fraud Sales too high?

Slide 64

Slide 64 text

Visual Debugger Feed Simulation Unit Testing “StreamBase’s modeling tools are easy to use and will enable the exchange to quickly react to the ever changing needs of our customers.” Steve Goldman, Director of Enterprise Architecture StreamBase Development Studio

Slide 65

Slide 65 text

Live Datamart Continuous Query Processor Alerts BusinessEvents FTL EMS ActiveSpaces Live Datamart BusinessWorks Social Media Data Market Data Sensor Data Historical Data ActiveSpaces Datagrid Enterprise data Market Data IoT Mobile Social LiveView Desktop Command & Control ACTION Continuous Query

Slide 66

Slide 66 text

67 Dynamic aggregation Live visualization Ad-hoc continuous query Alerts Action LiveView Desktop

Slide 67

Slide 67 text

Live Datamart Clients and APIs • Rich Desktop Client – Drag&Drop, no coding • Rich Web Client – Drag&Drop, no coding • HTML5 and Javascript API – D3, jQuery, ExtJS, Google Charts, Bing, AngularJS • .NET API – For custom .NET development • Java API – For custom Java GUI development • Combination – Rich Client + HTML5 Extensions

Slide 68

Slide 68 text

Predictive Sensor Analytics Live Demo (Stream Processing)

Slide 69

Slide 69 text

Slide 70

Slide 70 text

71 What do you need (out-of-the-box)? © Copyright 2000-2015 TIBCO Software Inc. • A stream processing programming language for streaming analytics • Visual development and debugging instead of coding • Out-of-the-box connectivity to streaming and historical data sources • Performance (real-time vs. micro-batches) • Automated monitoring and alerts • Live UI for proactive human interaction • Maturity and proven deployments • Fault tolerance • Commercial support • Professional services and training

Slide 71

Slide 71 text

Slide 72

Slide 72 text

Example: Apache Storm + TIBCO Live Datamart External Data Snapshot Results Continuous Query Processor Query TIBCO Live Datamart Continuous Alerting Active Tables Active Tables Continuous Updates Clients Message Bus Public Data Customer Data StreamBase Bolt StreamBase Spout Operational Data StreamBase Bolt and Spout connect Apache Storm to StreamBase to provide real-time analytics on operational data

Slide 73

Slide 73 text

Agenda – Real World Use Cases – Introduction to Stream Processing – Market Overview – Relation to other Big Data Components

Slide 74

Slide 74 text

Slide 75

Slide 75 text

76 © Copyright 2000-2014 TIBCO Software Inc. Real Time Close Loop Model Develop model Deploy into Stream Processing flow Act Automatically monitor real-time transactions Automatically trigger action Analyze Analyze data via Data Discovery Uncover patterns, trends, correlations

Slide 76

Slide 76 text

Real Time Close Loop: Understand – Anticipate – Act Big Data  store everything in Hadoop, DWH, NoSQL, etc.  even without structure  even if you do not need it today http://blogs.teradata.com/international/tag/hadoop/

Slide 77

Slide 77 text

Real Time Close Loop: Understand – Anticipate – Act Data Discovery + Statistics + Machine Learning to find insights and patterns in historical data

Slide 78

Slide 78 text

Real Time Close Loop: Understand – Anticipate – Act Streaming Analytics to operationalize insights and patterns in real time Stream Processing Hadoop Open Source R TERR SAS MATLAB In- database analytics Spark

Slide 79

Slide 79 text

R with Revolution Analytics (now Microsoft) © Copyright 2000-2015 TIBCO Software Inc. Open Source GPL License http://www.revolutionanalytics.com/webinars/introducing-revolution-r-open-enhanced-open-source-r-distribution-revolution-analytics

Slide 80

Slide 80 text

R with TIBCO Runtime for R (TERR) TIBCO TERR delivers production-grade R analytics to enterprises  Flexibility & analytic power of R language  Time-to-market agility  Enterprise-grade platform • A TIBCO licensed & supported product • Not GPL, not a repackaging of the Open source R engine • Deployment in TIBCO products and 3rd party applications (e.g. Hadoop) http://spotfire.tibco.com/discover-spotfire/what-does-spotfire-do/predictive-analytics/tibco-enterprise-runtime-for-r-terr

Slide 81

Slide 81 text

Slide 82

Slide 82 text

Spark MLlib © Copyright 2000-2015 TIBCO Software Inc. MLlib is Spark’s machine learning (ML) library. Its goal is to make practical machine learning scalable and easy. It consists of common learning algorithms and utilities, including classification, regression, clustering, collaborative filtering, dimensionality reduction, as well as lower-level optimization primitives and higher-level pipeline APIs. You can even combine Mllib module with R language

Slide 83

Slide 83 text

Predictive Sensor Analytics Live Demo (Data Discovery, Statistics)

Slide 84

Slide 84 text

Slide 85

Slide 85 text

Case Study: Streaming Analytics for Betting • Situation: Today, 80% of Betting is Done After the Game Starts • It’s not your father’s bookie anymore! • Problem: How to Analyze Big Betting Data? • Thousands of concurrent games, constantly adjusting odds, dozens of betting networks – firms must correlate millions of events a day to find the best betting opportunities in real-time • Solution: TIBCO for Fast Data Architecture • TXOdds uses TIBCO to correlate, aggregate, and analyze large volumes of streaming betting data in real-time and publish innovative predictive betting analytics to their customers • Result: TXOdds First to Market with Innovative Zero Latency Betting Analytics • Innovative real-time analytics help players who can process electronic data in real-time the edge “With StreamBase, in two months we had our first betting analytics feed live, and we continually deploy new ideas and evolve our old ones.” - Alex Kozlenkov, VP of technology, TXOdds

Slide 86

Slide 86 text

87 “WHEN 5 KEY BOOKIES RAISE THE SAME ODDS IN A 5-SECOND WINDOW, BET LESS” ? ? ? ? ? ? ? ? ?

Slide 87

Slide 87 text

88 “WHEN THE REAL-TIME ODDS ARE 5% GREATER THAN THE HISTORICAL SPREAD, INCREASE MY BET” ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?

Slide 88

Slide 88 text

Reference Architecture: Streaming Betting Analytics Event Processing MONITOR REAL-TIME ANALYTICS AGGREGATE HISTORICAL COMPARISON Predictive odds analytics Zero Latency Betting Analytics GLOBAL, DISTRIBUTED INFRASTRUCTURE Historical odds deviations B U S BETTING LINES SCORES NEWS HADOOP Context: Historical Betting Data, Odds, Outcomes B U S CACHE CACHE CACHE Real-Time Analytics CORRELATE StreamBase LiveView SOCIAL

Slide 89

Slide 89 text

Twitter (#TomBradyBrokenLeg) Twitter (#Boston) Brady’s Stats Actionable Insights Real-Time Social Media Analytics Twitter (#NFL) Something relevant happening? Every minute counts! Change Odds (automated or manually triggered): • Stop live-betting for the currently running game? • How many interceptions will the Quarterback throw? • Will the Patriots win the Super Bowl? • …

Slide 90

Slide 90 text

91 Real-Time Social Sentiment Analysis

Slide 91

Slide 91 text

Did you get the Key Message?

Slide 92

Slide 92 text

– Streaming Analytics processes Data while it is in Motion! – Automation and Proactive Human Interaction are BOTH needed! – Time to Market is the Key Requirement for most Use Cases! Key Messages

Slide 93

Slide 93 text

Questions? Kai Wähner [email protected] @KaiWaehner www.kai-waehner.de LinkedIn / Xing  Please connect!