Alternatives for Stream Processing - Apache Apex, Flink, Spark Streaming, StreamBase, Apama, Striim, SQLStream, et al

Slide 1

Slide 1 text

Kai Wähner Technology Evangelist kontakt@kai-waehner.de LinkedIn @KaiWaehner www.kai-waehner.de Big Data Spain @ Madrid (November 2016) Comparison of Streaming Analytics Frameworks

Slide 2

Slide 2 text

© Copyright 2000-2016 TIBCO Software Inc. Key Take-Aways • Streaming Analytics processes Data while it is in Motion! • Automation and Proactive Human Interaction are BOTH needed! • Streaming Analytics is Complementary to Hadoop and Machine Learning!

Slide 3

Slide 3 text

Slide 4

Slide 4 text

Slide 5

Slide 5 text

Slide 6

Slide 6 text

Slide 7

Slide 7 text

Slide 8

Slide 8 text

Data Monitoring • Motor temperature • Motor vibration • Current • Intake pressure • Intake temperature Ø Flow Electrical power cable Pump Intake Protector ESP motor Pump monitoring unit Electric Submersible Pumps (ESP) Predictive Analytics - Fault Management

Slide 9

Slide 9 text

Voltage Temperature Vibration Device history Temporal analytic: “If vibration spike is followed by temp spike then voltage spike [within 4 hours] then flag high severity alert.” Predictive Analytics - Fault Management

Slide 10

Slide 10 text

© Copyright 2000-2016 TIBCO Software Inc. Live Surveillance of Equipment Continuous, live geospatial display of pump health and predictive signal breeches Alerts based on predictive signals Compare live readings and signals to historical average and means Continuous, live visualization of stats per 100’s of wells

Slide 11

Slide 11 text

Slide 12

Slide 12 text

Slide 13

Slide 13 text

Slide 14

Slide 14 text

© Copyright 2000-2016 TIBCO Software Inc. All Customers are different… Treat them that way… 14 Capture – Engage – Expand - Monetize Patterns – Real time MORE PERSONAL MORE CONTEXT social CRM POS mobile web e-mails

Slide 15

Slide 15 text

Slide 16

Slide 16 text

© Copyright 2000-2013 TIBCO Software Inc. ““For every 1% increase in shipped product, we make $11MM in profit. The demand is there, we just need to fulfill it.“ - Head of Quality, Solar Panel Manufacturer

Slide 17

Slide 17 text

Scenario: Predictive Scrapping of Parts in an Assembly Line Goal: Scrap parts as early as possible automatically to reduce costs in a manufacturing process. Question: When to scrap a part in Station 1 instead of doing re-work or sending it to Station 2? Station 1 Station 2 Cost Before 9€ 7€ 13€ Total Cost 29€ (or more) Scrap? Scrap?

Slide 18

Slide 18 text

Slide 19

Slide 19 text

Slide 20

Slide 20 text

Slide 21

Slide 21 text

Slide 22

Slide 22 text

© Copyright 2000-2016 TIBCO Software Inc. Traditional Data Processing: ”Request – Response” • Data is collected from a variety of sources, and placed in a persistent store. – Relational database. – NoSQL store. – Hadoop environment. • Analytical processes are executed against the stored data to detect opportunities or threats. • Actions are identified, delivered, and executed across various business channels. Store Analyze Act

Slide 23

Slide 23 text

© Copyright 2000-2016 TIBCO Software Inc. Traditional Data Processing: Challenges Store Analyze Act • Introduces too much “decision latency” into the business. • Responses are delivered “after-the- fact”. • Maximum value of the identified situation is lost. – Cross-sell / up-sell opportunities are lost, impending equipment failure is missed, business processes are slow to respond and lack timely context. • Decisions are made on old and stale data.

Slide 24

Slide 24 text

Slide 25

Slide 25 text

© Copyright 2000-2016 TIBCO Software Inc. Event Value Decreases Over Time Value Time • Events are often most valuable “close to” the point of collection. • As time passes, events tend to lose their value. • The ability to proactively identify “threats” or “opportunities” will typically decrease. • Real-time capability is needed to maximize event value.

Slide 26

Slide 26 text

Slide 27

Slide 27 text

© Copyright 2000-2016 TIBCO Software Inc. The New Era: Streaming Analytics • Events are analyzed and processed in real-time as they arrive. • Decisions are timely, contextual, and based on fresh data. • Decision latency is eliminated, resulting in: ü Superior Customer Experience ü Operational Excellence ü Instant Awareness and Timely Decisions Act & Monitor Analyze Store

Slide 28

Slide 28 text

© Copyright 2000-2016 TIBCO Software Inc. Streaming Analytics: What Is A “Stream”? Clickstream Sensors Social Data Logs • Consists of pieces of data typically generated due to a change of state. • One or more identifiers • Timestamp & payload • Immutable • Typically unbounded; there is no end to the data. • Batch dataset: “bounded”. • Can be raw or derived.

Slide 29

Slide 29 text

© Copyright 2000-2016 TIBCO Software Inc. Streaming Analytics Processing Pipeline APIs Adapters / Channels Integration Messaging Stream Ingest Transformation Aggregation Enrichment Filtering Stream Preprocessing Process Management Analytics (Real Time) Applications & APIs Analytics / DW Reporting Stream Outcomes • Contextual Rules • Windowing • Patterns • Deep ML • Analytics • … Stream Analytics & Processing Index / Search Normalization

Slide 30

Slide 30 text

© Copyright 2000-2016 TIBCO Software Inc. Streaming Analytics Processing Pipeline Separation of concerns to easily adjust one part in response to changing business requirements without the need for rewriting other parts!

Slide 31

Slide 31 text

© Copyright 2000-2016 TIBCO Software Inc. Streaming Analytics: Ingest APIs Adapters / Channels Integration Messaging Stream Ingest • Stream data may come from a number sources, either at the edge, in the data center, or via the cloud. • Need to handle a variety of data formats and protocols, all at global scale. • Pay attention to “event time” vs. “processing time” !! • Event Time: Time the event was created. • Processing Time: Time the event was received or processed. • Event time is typically more relevant, and will lead to more predictable results. • Eliminate time skew associated with clock synchronization, system outages, processing latency, network issues, etc.

Slide 32

Slide 32 text

© Copyright 2000-2016 TIBCO Software Inc. Streaming Analytics: Preprocessing Transformation Aggregation Enrichment Filtering Stream Preprocessing Normalization • Stream data often needs to be manipulated before it is processed by downstream components. • Normalization • Transformation • May filter unwanted events close to the source to eliminate “noise”. • Events may also be enriched with additional context to provide additional data for further processing. • Customer details, equipment details, location information, etc. • Data may be stored in a high-speed cache or other store for rapid access.

Slide 33

Slide 33 text

© Copyright 2000-2016 TIBCO Software Inc. Streaming Analytics: Processing Batch • Transform • Deep ML • Analytics • Data Lake • … Stream Analytics & Processing Real-Time • RT Analytics • Contextual Rules • Windowing • Patterns • … • Streams may be immediately pushed to a data lake. • May be raw or preprocessed. • Used for subsequent analysis as part of an immutable data layer. • Typically processed in batch in this part of the architecture. • In parallel, streams may be processed in real-time against a number of constructs. • Real-time analytics. • Graph analysis / Geo Analysis • Rules. • Results from the real-time processing may be fed into the batch component. • The results of batch processing may also be pushed into the real- time layer.

Slide 34

Slide 34 text

Slide 35

Slide 35 text

Slide 36

Slide 36 text

Slide 37

Slide 37 text

Slide 38

Slide 38 text

© Copyright 2000-2016 TIBCO Software Inc. Automation and Augmented Intelligence for Humans Actions by Operations Human decisions in real time informed by up to date information 38 Automated action based on models of history combined with live context and business rules Machine-to-Machine Automation

Slide 39

Slide 39 text

Big Data Reference Architecture Augmented Intelligence Operations SENSOR DATA TRANSACTIONS MESSAGE BUS MACHINE DATA SOCIAL DATA Streaming Analytics Action Aggregate Rules Stream Processing Analytics Correlate Continuous query processing Alerts Manual action, escalation Data Discovery Python R Data Scientists Cleansed Data History Visual Analytics Spark Integration ERP MDM DB WMS SOA / Microservices BIG DATA Data Warehouse, Hadoop Internal Data Integration Bus API Event Server H2O.ai Live UI

Slide 40

Slide 40 text

Slide 41

Slide 41 text

© Copyright 2000-2016 TIBCO Software Inc. Streaming Analytics Market Growing Significantly “Everything Flows: The value of stream processing and streaming integration” (September 2016) http://hortonworks.com/info/value-streaming-integration/

Slide 42

Slide 42 text

Slide 43

Slide 43 text

© Copyright 2000-2016 TIBCO Software Inc. Alternatives for Stream Processing Concepts (Continuous Queries, Sliding Windows) Patterns (Counting, Sequencing, Tracking, Trends) Build everything by yourself! L Time to Market Streaming Frameworks Streaming Products Slow Fast Streaming Concepts

Slide 44

Slide 44 text

Slide 45

Slide 45 text

© Copyright 2000-2016 TIBCO Software Inc. Alternatives for Stream Processing Library (Java, .NET, Python) Query Language (often similar to SQL) Scalability (horizontal and vertical, fail over) Connectivity (technologies, markets, products) Operators (Filter, Sort, Aggregate) Time to Market Streaming Frameworks Streaming Products Slow Fast Streaming Concepts Different frameworks (ingest, preprocess, analytics) combined!

Slide 46

Slide 46 text

Slide 47

Slide 47 text

© Copyright 2000-2016 TIBCO Software Inc. Example for an Open Source Streaming Pipeline http://hortonworks.com/hadoop-tutorial/realtime-event-processing-nifi-kafka-storm “Realtime Event Processing in Hadoop with Apache NiFi, Kafka and Storm”

Slide 48

Slide 48 text

Dataflow Streaming Pipeline (Ingest, Preprocess) Augmented Intelligence Operations SENSOR DATA TRANSACTIONS MESSAGE BUS MACHINE DATA SOCIAL DATA Streaming Analytics Action Aggregate Rules Stream Processing Analytics Correlate Continuous query processing Alerts Manual action, escalation Data Discovery Python R Data Scientists Cleansed Data History Visual Analytics Spark Integration ERP MDM DB WMS SOA / Microservices BIG DATA Data Warehouse, Hadoop Internal Data Integration Bus API Event Server H2O.ai Live UI

Slide 49

Slide 49 text

Slide 50

Slide 50 text

Streaming Analytics Augmented Intelligence Operations SENSOR DATA TRANSACTIONS MESSAGE BUS MACHINE DATA SOCIAL DATA Streaming Analytics Action Aggregate Rules Stream Processing Analytics Correlate Continuous query processing Alerts Manual action, escalation Data Discovery Python R Data Scientists Cleansed Data History Visual Analytics Spark Integration ERP MDM DB WMS SOA / Microservices BIG DATA Data Warehouse, Hadoop Internal Data Integration Bus API Event Server H2O.ai Live UI

Slide 51

Slide 51 text

Slide 52

Slide 52 text

Slide 53

Slide 53 text

Slide 54

Slide 54 text

Slide 55

Slide 55 text

Slide 56

Slide 56 text

Slide 57

Slide 57 text

© Copyright 2000-2016 TIBCO Software Inc. AWS Kinesis – Public Cloud Trade-Off … is easy to setup and scale. But you do not have full control! L • Any data that is older than 24 hours is automatically deleted • Every Kinesis application consists of just one procedure, so you can’t use Kinesis to perform complex stream processing unless you connect multiple applications • Kinesis can only support a maximum size of 50KB for each data item http://diamondstream.com/amazon-kinesis-big-real-time-data-processing-solution/ (blog post from 2014, might be outdated, but shows that you do not have full control over a cloud service)

Slide 58

Slide 58 text

Slide 59

Slide 59 text

© Copyright 2000-2016 TIBCO Software Inc. Apache Spark – Focus on Analytics http://aptuz.com/blog/is-apache-spark-going-to-replace-hadoop/ http://fortune.com/2015/09/09/cloudera-spark-mapreduce/ http://www.ebaytechblog.com/2014/05/28/using-spark-to-ignite-data-analytics/ http://www.forbes.com/sites/paulmiller/2015/06/15/ibm-backs-apache-spark-for-big-data-analytics/ “[IBM’s initiatives] include: • deepening the integration between Apache Spark and existing IBM products like the Watson Health Cloud; • open sourcing IBM’s existing SystemML machine learning technology;

Slide 60

Slide 60 text

© Copyright 2000-2016 TIBCO Software Inc. Spark Streaming Spark Streaming • is no real streaming solution • uses micro-batches • cannot process data in real-time (i.e. no ultra-low latency) • allows easy combination with other Spark components (SQL, Machine Learning, etc.)

Slide 61

Slide 61 text

Slide 62

Slide 62 text

Slide 63

Slide 63 text

Slide 64

Slide 64 text

Slide 65

Slide 65 text

Slide 66

Slide 66 text

Alternatives for Stream Processing Library (Java, .NET, Python) Query Language (often similar to SQL) Scalability (horizontal and vertical, fail over) Connectivity (technologies, markets, products) Operators (Filter, Sort, Aggregate) Time to Market Streaming Frameworks Streaming Products Slow Fast Streaming Concepts Single Tool (Complete Processing Pipeline) Visual IDE (Dev, Test, Debug) Simulation (Feed Testing, Test Generation) Live UI (monitoring, proactive interaction) Maturity (24/7 support, consulting) Integration (out-of-the-box: ESB, MDM, Analytics, etc.)

Slide 67

Slide 67 text

Slide 68

Slide 68 text

Dataflow Streaming Pipeline + Streaming Analytics Augmented Intelligence Operations SENSOR DATA TRANSACTIONS MESSAGE BUS MACHINE DATA SOCIAL DATA Streaming Analytics Action Aggregate Rules Stream Processing Analytics Correlate Continuous query processing Alerts Manual action, escalation Data Discovery Python R Data Scientists Cleansed Data History Visual Analytics Spark Integration ERP MDM DB WMS SOA / Microservices BIG DATA Data Warehouse, Hadoop Internal Data Integration Bus API Event Server H2O.ai Live UI

Slide 69

Slide 69 text

Slide 70

Slide 70 text

© Copyright 2000-2016 TIBCO Software Inc. TIBCO StreamBase • Performance: Latency, Throughput, Scalability • Multi-threaded and clustered server from version 1 • High throughput: Millions of messages, 100,000s of quotes, 10,000s of orders • Low-latency: microsecond latency for algo trading, pre-trade risk, market data • Take Advantage of High Performance Hardware • Multicore (12, 24, 32 core) large memory (10s of gigabytes) • 64-bit Linux, Windows, Solaris deployment • Hardware acceleration (GPU, Solace, Tervela) • Enterprise Deployment • High availability and fault tolerance • Distributed state management for large data sets • Management and monitoring tools • Security and entitlements Integration • Continuous deployment and QA Process Support StreamSQL compiler and static optimizer In process, in thread adapter architecture Visual parallelism and scaling In-Memory Data Grid integration for distributed shared state Data parallelism and dispatch StreamBase Server Innovations

Slide 71

Slide 71 text

Slide 72

Slide 72 text

Visual Debugger Feed Simulation Unit Testing StreamBase Development Studio TIBCO StreamBase - Visual Programming

Slide 73

Slide 73 text

Live UI for Augmented Intelligence Augmented Intelligence Operations SENSOR DATA TRANSACTIONS MESSAGE BUS MACHINE DATA SOCIAL DATA Streaming Analytics Action Aggregate Rules Stream Processing Analytics Correlate Continuous query processing Alerts Manual action, escalation Data Discovery Python R Data Scientists Cleansed Data History Visual Analytics Spark Integration ERP MDM DB WMS SOA / Microservices BIG DATA Data Warehouse, Hadoop Internal Data Integration Bus API Event Server H2O.ai Live UI

Slide 74

Slide 74 text

© Copyright 2000-2016 TIBCO Software Inc. Live User Interface Live UI Continuous Query Processor Alerts CEP MQTT JMS In-Memory Data Grid Integration Social Media Data Market Data Sensor Data Historical Data In-Memory Data Grid Enterprise data Market Data IoT Mobile Social Browser / App Command & Control ACTION Continuous Query

Slide 75

Slide 75 text

Slide 76

Slide 76 text

© Copyright 2000-2016 TIBCO Software Inc. Live UI - Products Characteristics to Check • Alternative clients (rich client, browser, mobile app) • Maturity for enterprise use cases • Performance and scalability • “Big data native” deployment (YARN, Mesos) • Monitoring and proactive actions • Streaming engine under the hood (not just visualization layer) • New Ad-hoc queries by the business user (without the help of IT department) • Various visual components • Extendibility (graphical designer vs. coding) … or build your own solution using Websockets, Angular JS, etc.

Slide 77

Slide 77 text

Slide 78

Slide 78 text

© Copyright 2000-2016 TIBCO Software Inc. Customer Example: Apache Storm + TIBCO Live Datamart External Data Snapshot Results Continuous Query Processor Query TIBCO Live Datamart Continuous Alerting Active Tables Active Tables Continuous Updates Clients Message Bus Public Data Customer Data StreamBase Bolt StreamBase Spout Operational Data StreamBase Bolt and Spout connect Apache Storm to StreamBase to provide real-time analytics on operational data

Slide 79

Slide 79 text

Slide 80

Slide 80 text

Slide 81

Slide 81 text

Slide 82

Slide 82 text

Data Discovery via Visual Analytics, Big Data and Machine Learning Augmented Intelligence Operations SENSOR DATA TRANSACTIONS MESSAGE BUS MACHINE DATA SOCIAL DATA Streaming Analytics Action Aggregate Rules Stream Processing Analytics Correlate Continuous query processing Alerts Manual action, escalation Data Discovery Python R Data Scientists Cleansed Data History Visual Analytics Spark Integration ERP MDM DB WMS SOA / Microservices BIG DATA Data Warehouse, Hadoop Internal Data Integration Bus API Event Server H2O.ai Live UI

Slide 83

Slide 83 text

Find Insights and Patterns in Historical Data Visual Analytics + Machine Learning

Slide 84

Slide 84 text

Apply Insights and Analytic Models to Proactive Actions Streaming Analytics H20.ai Open Source R TERR Spark ML MATLAB SAS PMML

Slide 85

Slide 85 text

Slide 86

Slide 86 text

Case Study: Streaming Analytics for Betting • Situation: Today, 80% of Betting is Done After the Game Starts • It’s not your father’s bookie anymore! • Problem: How to Analyze Big Betting Data? • Thousands of concurrent games, constantly adjusting odds, dozens of betting networks – firms must correlate millions of events a day to find the best betting opportunities in real-time • Solution: TIBCO for Fast Data Architecture • TXOdds uses TIBCO to correlate, aggregate, and analyze large volumes of streaming betting data in real-time and publish innovative predictive betting analytics to their customers • Result: TXOdds First to Market with Innovative Zero Latency Betting Analytics • Innovative real-time analytics help players who can process electronic data in real-time the edge “With StreamBase, in two months we had our first betting analytics feed live, and we continually deploy new ideas and evolve our old ones.” - Alex Kozlenkov, VP of technology, TXOdds

Slide 87

Slide 87 text

© Copyright 2000-2016 TIBCO Software Inc. Big Data Architecture for Streaming Betting Analytics Event Processing MONITOR REAL-TIME ANALYTICS AGGREGATE HISTORICAL COMPARISON Predictive odds analytics Zero Latency Betting Analytics GLOBAL, DISTRIBUTED INFRASTRUCTURE Historical odds deviations B U S BETTING LINES SCORES NEWS HADOOP Context: Historical Betting Data, Odds, Outcomes B U S CACHE CACHE CACHE Real-Time Analytics CORRELATE Live Datamart SOCIAL

Slide 88

Slide 88 text

Real-Time Social Media Analytics Twitter (#TomBradyBrokenLeg) Twitter (#Boston) Brady’s Stats Actionable Insights Twitter (#NFL) Something relevant happening? Every second counts! Change Odds (automated or manually triggered): Stop live-betting for the current running game? • Who will win the game? • How many interceptions will the Quarterback throw? • Will the Patriots win the Super Bowl? • …

Slide 89

Slide 89 text

Slide 90

Slide 90 text

Slide 91

Slide 91 text

Slide 92

Slide 92 text

Big Data Architecture for Predictive Maintenance Operational Analytics Operations Live UI CSV Batch JSON Real Time XML Real Time Streaming Analytics Action Aggregate Rules Analytics Correlate Live Datamart Continuous query processing Alerts Manual action, escalation HISTORICAL ANALYSIS Data Scientists Flume HDFS Spotfire R / TERR HDFS Hadoop (Cloudera) StreamBase TIBCO Fast Data Platform H2O Oracle RDBMS Avro Parquet … PMML Internal Data

Slide 93

Slide 93 text

Slide 94

Slide 94 text

Slide 95

Slide 95 text

Monitor Patterns à TIBCO Live Datamart Augmented Intelligence (“Monitor the manufacturing process and change rules in real time!”) Live Dartmart Desktop Client

Slide 96

Slide 96 text

Monitor Patterns à TIBCO Live Datamart Augmented Intelligence (“Monitor the manufacturing process and change rules in real time!”) Live Dartmart Web API

Slide 97

Slide 97 text

TIBCO Spotfire + StreamBase + Live Datamart + H2O.ai Live Demo Live Demo

Slide 98

Slide 98 text

Slide 99

Slide 99 text

Questions? Please contact me! Kai Wähner Technology Evangelist kontakt@kai-waehner.de @KaiWaehner www.kai-waehner.de LinkedIn