(SQL, No SQL) • Data warehousing/Reporting requirements for all our customers • Enable different visualization tools for customers from unified data mart • Replicate data from one system to other system Traditionally old problem populating data from OTLP data bases to OLAP database to meet reporting, BI analytics requirements and potentially machine learning.
OLAP databases at scheduled time • Scale up ETL infrastructure as more and more data needs to be ingested • Streams - Replicate data as a stream of Events concept called Event streaming, very similar to data streaming in Big data world • Data treated as unbounded stream of events flowing through a dump pipe. (Remember microservices Smart endpoints, dump pipes)
world – Stream processing, Complex event processing, Event sourcing, Streaming analytics • Event Streams enables to achieve all the above applications • Streaming solution doesn’t impact the performance of OLTP applications unlike ETL which puts load on the OLTP databases. • More importantly data lake (Repository of all types of data in single distributed file system, buzz word in Enterprises). • Microservice and Devops principles in development for scalability and reliability engineering.
Extract transform and Load where as ELT stands for Extract Load and Transform workflows. • Majorly achieved through data lake repository where the data is then transformed and reshaped to support multiple use cases. • Reshaped for bespoke ad hoc Reporting, Machine learning, Datamarts with high dimensional facts and measures to enabled OLAPI type queries(How much, How many?)
MQ, Google Cloud Pub/Sub • Evaluated Kafka and Kinesis. Decision made to stick to Kinesis • Advantages of Kinesis – Ultra low latency stream processing, high throughput, less management, enables serverless microservices ecosystem using AWS Lambda functions • Disadvantages – Every consumer has to read every message published to Kinesis and discard if not needed.
• Microservices, use of Streaming frameworks like Apache Spark streaming, Apache Flink, Serverless choices using Lambda functions. • Started with Microservices for event processing and currently migrating to Apache Spark streaming and serverless architecture using AWS Lambda functions. • Apache spark has suite of functionalities to do streaming analytics especially using Spark SQL and Streaming SQL.
but comes with its own challenges as follows • Manage fast publisher/producer and slow consumer. Consumers can fall behind reading from shards, how to keep up to provide sub second latency in stream processing) • Scale up consumers to do distributed stream processing as more data is ingested into streams. • Handling Out of order data, duplicates handling, missing data • Hard to data reconciliation with source
consuming rate upfront to avoid high latency in processing events • Any fast data processing hungry consumer can slow down other potential consumer due to the streaming nature of events • Handle backpressure from consumers, streamline the publishing rate. Don’t assume consumers can cope with any load • Do extensive performance testing to determine the throughput of Publisher and Consumer. Very useful to handle load
stream processing applications. • Streaming analytics enables running analytics on streams and can provide real time insights • ETL replication from OLTP to OLAP is messy and has scalability problems.
persolserverworks
jyouryuusui
genda
pfn
koukihosaka
kaminashi
techtekt
mohab1
bitkey
nttcom
riyaamemiya
mlcsv
katiecoart
stephaniewalter
helmedeiros
asonas
zakiyama
mza
rkendrick25
frankvandijk
jeffersonlam
csswizardry
lara
