How to Build Streaming Data Applications: Evaluating the Top Contenders 1

page
HOW TO BUILD STREAMING DATA
APPLICATIONS: EVALUATING THE TOP
CONTENDERS
Akmal B. Chaudhri
about.me/akmalchaudhri

View Slide

page
© 2015 VoltDB PROPRIETARY page
INTRODUCTION
2

View Slide

page
VOLTDB OVERVIEW
Mike Stonebraker
Founded in 2009 by database luminary
FAST
World Record Cloud Benchmark:
YCSB (Yahoo Cloud Serving Benchmark)
- 2.4m million tps (transactions per second)
Other Stonebraker Companies
Customers
3
Technology
•  In-Memory (but data is durable to disk)
•  Scale-Out shared-nothing architecture
•  Reliability and fault tolerance
•  SQL + Java with ACID
•  Hadoop and data warehouse integration
•  Open source and commercially licensed (24X7)
© 2015 VoltDB PROPRIETARY

View Slide

page
VOLTDB BENCHMARK ON AMAZON VIRTUAL AND
IBM SOFTLAYER BARE-METAL SERVERS
•  Yahoo Cloud Serving Benchmark (YCSB) is
a popular industry-standard benchmark for
cloud databases
•  AWS – virtualized servers
•  SoftLayer - bare-metal servers
•  Workload “B” - 95% reads with 5% updates.
•  Results: Best in class cloud performance
(run in the cloud)!
•  AWS - 285k tps for 3 nodes scaling linearly to
724k tps for a 12 node cluster
•  IBM SoftLayer - 1.02 million tps for 3 nodes
scaling linearly to 2.4 million tps for a 12 node
cluster
SoftLayer
AWS
SoftLayer: Update and Read Latency
Latency (ms)
Throughput (ops/sec)

View Slide

page
PREDICTION
5
All businesses will compete
on their ability to make
decisions “in the moment”
using Fast Data.

View Slide

page
FAST DATA SOURCES AND DRIVERS
Mobile
IoT
Social
Sensors
Logs
Data is doubling every two years
•  26 billion connected devices by
2020 (Gartner 2014)
•  37% of most data will be
processed at the edge in
milliseconds (Cisco IoT Study 12/11/14)
Mobile
IoT
6

View Slide

page
Mobile
Billing and rights management, subscriber marketing, etc.
IoT, Energy, Sensor
Smart grid/meters, asset tracking & management
Personalized Targeting
Ad optimization, audience segmenting
Capital Markets
Risk, market data management, customer mgt
Infrastructure
Data pipeline, system performance, streaming ETL
EVERY COMPANY HAS FAST DATA PROBLEMS
UK Smart
Meter
7
VoltDB Customers

View Slide

page
FAST DATA IS A COMPETITIVE ADVANTAGE TODAY!
Instant insight
Instant action
Instant awareness
8
* VoltDB customers
“Event triggered, real-time
recommendations based on
customer behavior have 10-15
times the response rates than
mass marketing”
“We get competitive advantage
by analyzing device and user
data to create an interactive
and personalized consumer
experience across all devices.”
“Real time contextual offers
increase offer uptake rates by
75% and data revenues by
15%.”
*
*

View Slide

page
TRADITIONAL RDBMS
•  Heavy Overhead
•  1000s of concurrent versions
•  Contention for locked records
•  Contention for latching on lock table
•  Index bottlenecks
•  Disk I/O bottlenecks
•  Architecture limits scaling
© 2015 VoltDB PROPRIETARY 9

View Slide

page
ARCHITECTURE IS IMPORTANT
Fast data requires
a different
architecture.

View Slide

page
BIG DATA + FAST DATA
11

View Slide

page
Collect' Explore'
(Data'Science)'
Analyze'
Act'
(Discoveries/'
Op:miza:ons)'
Big data
ecosystem has
several
components

View Slide

page
DATA ARCHITECTURE FOR FAST + BIG DATA
Enterprise Apps
ETL
CRM ERP Etc.
Data Lake
(HDFS, etc.)
BIG DATA
SQL on
Hadoop
Map
Reduce
Exploratory
Analytics
BI
Reporting
Fast Operational
Database
FAST DATA
Export
Ingest /
Interactive
Real-time
Analytics
Fast Serve
Analytics
Decisioning
13

View Slide

page
Calculations Serving of Results
Real Time, Per Event, Interactive
VOLTDB AND FAST DATA PIPELINE
14

View Slide

page
IN THE BIG CORNER
Systems facilitating exploration and analytics of large collections.
15
Example Technologies
Columnar OLAP warehouses
Hadoop Ecosystem
•  MapReduce
•  Hive, Pig
•  SQL.next: Impala, Drill, Shark
Example Applications
•  User segmentation & pre-scoring
•  Seasonal trending
•  Recommendation matrices
•  Building search indexes
•  Data Science: statistical clustering,
machine learning

View Slide

page
IN THE FAST CORNER
Systems facilitating real time ingest, analytics and decisions against
incoming streams of events.
16
Example Technologies
•  Streaming frameworks (e.g. Spark)
•  Fast OLAP (e.g. HANA)
•  Fast OLTP (e.g. VoltDB)
Example Applications
•  Micro-personalization
•  Recommendation serving
•  Alerting/alarming
•  Operational monitoring
•  Data enrichment (ETL elimination)
•  High throughput authorization
•  Ex: API quota enforcement

View Slide

page
TYPICAL FAST DATA QUESTIONS
17
Hadoop&
Volume'
SQL&/&OLAP&
Data'Science'
Fast&
Velocity'
•  Is the fast layer streaming?
•  It is often more like fast OLTP
•  How do the pieces communicate?
•  OLAP analytics from Big -> Fast
•  New events from Fast -> Big
•  Where do “analytics” belong?
•  Analytics per-event: with Fast
•  Analytics across history: with Big
•  Are streaming frameworks equivalent?
•  Traditional SQL CEP (Esper, Streambase)
•  Tuple DAGs (Storm)
•  Window processors on Hadoop (Spark)
&

View Slide

page
HOW TO SOLVE IT*
18
*"With"admiring"credit"to"G."Polya"
Considering'Data' Considering'Processing'
What&are&the&types&of&
data&to&be&managed&in&
fast&data&applica>ons?&
How&does&data&ﬂow&
through&fast&data&
applica>ons?&
What&are&the&
calcula>ons&&&analy>cs&
that&are&necessary?&

View Slide

page
Data Temporality
Incoming events Click stream, tick stream, sensors,
metrics
Real-Time
Analytic Results
Event metadata Device version, location, user
profiles, point-of-interest data
OLAP Analytics Used in
Real-Time Decisions
Responses/side effects
Examples
Event Stream
Persistent
(Queryable)
Persistent
(Look-Ups)
Outgoing
events
Persistent
(Look-Ups)
Event Stream
Event Stream
Counters, streaming aggregates,
Time-series rollups
Scoring models, seasonal usage,
demographic trends
Policy enforcement decisions,
personalization recommendations
Enriched, filtered, correlated
transform of input feed

View Slide

page
SOURCES OF STATE
1.  Analytics outputs must be query-able.
2.  “Lookup tables” to create groupings for analytics
and to supply enrichment data.
3.  Session managements: grouping, filtering and
aggregating create intermediate state.
20

View Slide

through&fast&data&
applica>ons?&
What&are&the&

View Slide

page
DATA FLOWS
Real-time Analytics
•  Streaming summaries for operations
•  KPI measurement
•  Analytics for apps
22
Real-Time Analytics

View Slide

page
DATA FLOWS
23
Fast Request/Response (and side effects)
•  Mobile Authorization
•  Campaign Evaluation
•  Quota Enforcement
•  Micro-Personalization
•  Recommendation Serving
Request/
Response

View Slide

page
DATA FLOWS
Data Pipelines
•  Data enrichment
•  Sessionization and re-assembly of incoming events.
•  Correlation (by time, location, identity)
•  Filtering
24
Pipeline
Data Lake

View Slide

through&fast&data&
applica>ons?&
What&are&the&

View Slide

Continuous Query
Transactional Event
Evaluation
Transformation

View Slide

page
FAST DATA STACK
Applications, Message Queues, Data Sources
Ingest
Analyze Decide
•  Counters
•  Aggregations
•  Time series
•  Statistics
•  Store results
•  Query and
recombine
•  Fast serving
•  Per-event policy evaluations
•  Responses (synchronous):
authorization, personalization
•  Side-effects (asynchronous): alerts,
alarms
Export & Pipeline

View Slide

Ingest
Analyze Decide
Counters
Aggregations
Time series
Statistics
Store results
Query and
recombine
Fast serving
Per-event policy evaluations
Responses (synchronous)
Side-effects (asynchronous)
Export & Pipeline
APACHE-ISH TECHNOLOGY STACK
Kafka / RabbitMQ
Storm, Flume, Sqoop
Storm +
Serving Layer
Spark +
Serving Layer
Cassandra,
HBase
Hadoop, Message queues

View Slide

Ingest
Analyze Decide
Counters
Aggregations
Time series
Statistics
Store results
Query and
recombine
Fast serving
Export & Pipeline
VOLTDB TECHNOLOGY STACK
Kafka / RabbitMQ
VoltDB
SQL, Java for
Analytics
Transactions /
ACID
Hadoop, Message queues

View Slide

OLTP
(Transactions First)
Streaming
Event Processors
OLAP
(Columnar Analytics)

View Slide

Ingest
Analyze Decide
Counters
Aggregations
Time series
Statistics
Store results
Query and
recombine
Fast serving
Export & Pipeline
STREAM TECHNOLOGY STACK

View Slide

Ingest
Analyze Decide
Counters
Aggregations
Time series
Statistics
Store results
Query and
recombine
Fast serving
Export & Pipeline
OLAP TECHNOLOGY STACK

View Slide

page
Applications
&
Streams
Logs, Sensors,
Meter Readings,
IoT, Location
Real-Time
Applications
Message Queue
Ingest
Kafka Loader
CSV loaders
C++, C#, PHP, Python
Java (and others)
Export
CSV Data
Thrift Messages
JDBC
HTTP
Local File
Extensible Connectors
SQL
Views
Java
Analyze
ACID
Txns
State
Decide
Downstream
Pipeline
Hadoop
Data Warehouse
Message Queue
STREAMING DATA PIPELINE

View Slide

page
FAST DATA PATTERNS
34

View Slide

page
THREE FAST DATA APPLICATION PATTERNS
•  Real-Time Analytics
•  Real-time analytics for operations
•  Real-time KPI measurement
•  Real-time analytics for apps
•  Data Pipelines
•  Streaming data enrichment
•  Sessionization / re-assembly
•  Correlation (by time, by location, by id)
•  Filtering
•  Pre-aggregation
35
•  Fast Request/Response
•  Mobile Authorization
•  Campaign Authorization
•  Fast API Quota Enforcement
•  Micro-Personalization
•  Recommendation Serving

View Slide

page
VOLTDB: REAL-TIME ANALYTICS
36
VoltDB
Metadata
(Dimension table)
Session state
(Fact table) •  Operational analytics and
monitoring
•  RT analytics enabling user-
facing applications
•  KPI for internal BI/Dashboards
•  In-memory MPP SQL over
ODBC/JDBC
•  Cheap + correct materialized
views for streaming
aggregations
SQL, Views
Ingest

View Slide

page
VOLTDB: DATA PIPELINES WITH EXPORT
37
VoltDB
Metadata
(Dimension table)
Session state
(Fact table)
•  Filtering (ex: only RFID /
iBeacon readings that show
change from previous
location).
•  Sessionization
•  Common version re-writing
•  Data enrichment
•  MPP streaming Export
•  Row data, Thrift messages, CSV
•  OLAP, HDFS and message
queues
Export

View Slide

page
VOLTDB: REQUEST/RESPONSE DECISIONS
38
•  Authorization
•  RT balance checks, quota
enforcement
•  Personalization and
Recommendation Serving
•  Combine pre-score with
immediate context
•  Fully ACID transaction model.
•  Thousands to Millions per
second
•  At less than 5ms latencies
Metadata&
(Dimension&table)&
Session&state&
(Fact&table)&
ACID&Transac>ons&

View Slide

page
VOLTDB V5.0
39

View Slide

page
VOLTDB V5.0 – ACCELERATING FAST DATA
APPLICATION DEVELOPMENT
•  Hadoop/Big Data Ecosystem Integrations
•  Fast Data Pipeline Sample Applications
•  Ease of Database Development (traditional API)
•  VoltDB Management Center (VMC)
•  Updated Hortonworks HDP Certification
40

View Slide

page
FAST DATA INTEGRATIONS - IMPORTERS
•  Kafka Loader
•  Subscribe to a Kafka topic and insert each message into a VoltDB
Table
•  JDBC Loader
•  Load a JDBC result set into a VoltDB Table
•  Vertica Udx
•  User-defined function to load Vertica result sets into a VoltDB
Table
•  Apache Hive and Apache Pig
•  Hadoop OutputFormat to load Hive and Pig result sets into VoltDB

View Slide

page
FAST DATA INTEGRATIONS - EXPORTERS
•  HDFS Export
•  Hadoop export via WebHDFS and HttpFS
•  HTTP Export
•  Delivery and Alerting via HTTP post/get
•  Kafka Export, RabbitMQ Export
•  Message queue delivery
•  Export format configurable
•  Avro, CSV, TSV, more coming…

View Slide

page
FAST DATA PIPELINE SAMPLE APPLICATION
•  Streaming Data, Real-time Analytics
•  Export to Hadoop
•  Export to OLAP (Vertica, others)
•  Place historical decision making intelligence into VoltDB
•  Closed Loop, via Hive, Pig OutputFormat or Vertica Udx
•  Download: https://github.com/VoltDB/app-fastdata
•  And see our blog posts:
http://voltdb.com/blog/fast-data-look-voltdb-sample-app

View Slide

page
LAMBDA ARCHITECTURE SAMPLE APPLICATION
•  Type of application: Real-time analytics
•  Demonstrates how to simplify the “Speed
Layer”
•  Using VoltDB, developers can replace both the
streaming and the operational data store portions of
the speed layer.
•  Less code, greatly reduced complexity
•  Improving the Lambda Architecture
•  Perform real-time analytics AND react, per event, to
the incoming data stream
•  Try it yourself: http://voltdb.com/community/applications
HOW MANY UNIQUE
USERS INTERACTED WITH
MY APP TODAY?

View Slide

page
VOLTDB MANAGEMENT CENTER (VMC)
A browser-based management tool for monitoring, examining, and querying a running VoltDB database

View Slide

page
UPDATED HORTONWORKS CERTIFICATION

View Slide

page
CUSTOMER CASE STUDIES
47

View Slide

Million meters under management,
saving millions in efficiency, reduced waste
VOLTDB DELIVERS SUPERIOR CUSTOMER VALUE
Customers Business Value
Internet Service
Provider
Discover 100% of DoS attacks, and
improved response time by 97%
Communications
Service Provider
Improved infrastructure utilization
by 150%
Online Game Analytics
Increased free-to-pay conversion rate
by 30%
Mobile Network Management
Saves $0.5 million/customer installation;
unlimited scale in the cloud
Mobile Ad Service
Provider
OpEx – 93% reduction in servers (100 to 7)
Saved millions in ad budget overages
48
Smart Meter, Energy
Management

View Slide


View Slide

page
TRY V5.0 TODAY FOR FREE
•  VoltDB Enterprise Edition
•  Production-ready
•  Fully durable, highly available
•  Commercial license, fully supported
•  http://voltdb.com/download/software
•  Sample apps (in a Docker container)
•  http://voltdb.com/community/demo
•  VoltDB Community Edition – open source
•  http://github.com/voltdb
VoltDB runs over 6 BILLION transactions/day in production!

View Slide

Capability Spark,Streaming Storm TIBCO,Streambase IBM,Streams Google,Dataflow Amazon,Kinesis VoltDB
Focus Micro&Batching&for&Hadoop
Infrastructure&for&data&
capture Complex&Event&Processing
Stream&processing&and&
analytics&without&queries
Next&gen&MapReduce&in&the&
cloud
Infrastructure&for&data&
capture
Stream&processing,&analytics&with&
queries,&and&realCtime&decision&
making
Programming&Model Java,&Scala Clojure,&Java,&Ruby,&Python SQL
Proprietary&C&Stream&
Processing&Language&(SPL) Java Java
Java,&Relational,&SQL,&ACIDC
compliant
Latency&(milliseconds) >&&1,000&milliseconds milliseconds 1&millisecond 1&millisecond >&&2,000&milliseconds 35C100&milliseconds 1&milllisecond
Data&Capture/Ingestion Batch
! ! ! ! ! !
Stateful,Operation X X X X X X
!
Ad,hoc,queries
Interactive,SQL X X X X X X
!
Analytics,w/o,Queries
! with&add&on&DDLs
! ! ! ! !
Analytics,with,queries,and,perKevent,
decision,making X X X X X X
!
Real&time&Data&Enrichment
Using&metadata&to&enrich,&denormalize,&etc.,&
incoming&event&streams X X X X X X
!
Apply&OLAP&results&to&real&time&data&stream X X X
! X X
!
ScaleCout&architecture
! ! X
! ! ! !
Reliability:&ability&to&persist&data X X X X X
!
Fault&Tolerant
! ! ! ! ! !
Requires&Zookeeper&for&HA
Reliability:&ability&to&persist&data X X
! ! X X
!
Cluster&&&Resource&Management Need&to&addCon&Zookeeper
Need&to&addCon&Zookeeper;&
supports&YARN BuiltCIn BuiltCIn BuiltCIn BuiltCIn BuiltCIn
Support Cloudera Hortonworks TIBCO IBM Google Amazon VoltDB
Output&(OLAP&Integration) HDFS,&Flume,&Kafka,,&ZeroMQ HDFS,&Kafka,&Redis,&RDBMS
HDFS,&CSV,&IBM&Netezza,&HP&
Vertica,&&Microsoft,&Oracle,&
Sybase
HDFS,&CSV,&IBM&Netezza,&HP&
Vertica,&&Microsoft,&Oracle,&
Sybase Google Amazon
HDFS,&Kafka,&RabbitMQ,&CSV,&
Netezza,&HP&Vertica,&JDBC
Available&as&Open&Source Yes,&Apache&license Yes,&Apache&license X X X X Yes,&AGPL&License
Comparing,Fast,Data,Application,Platforms:,From,Simple,Streaming,to,RealKTime,Interaction,with,Decision,Making
Ingestion&&&&C>&&&Analytics&&w/o&Queries&&&&&C>&&&&&Analytics&with&queries&&&&&C&>&&&&Data&Enrichment&C>&&&Real&time&Decisions
Fast,data,applications,three,unique,requirements:,rapid,data,ingestion,,realKtime,analytics,on,streaming,data,,and,per,event,realKtime,decisions

View Slide

How to Build Streaming Data Applications: Evaluating the Top Contenders 1

How to Build Streaming Data Applications: Evaluating the Top Contenders 1

VeryFatBoy

More Decks by VeryFatBoy

Other Decks in Technology

Featured

Transcript