Visualizing Streams

[email protected] www.rittmanmead.com @rittmanmead !1 Visualizing Streams

[email protected] www.rittmanmead.com @rittmanmead Francesco Tisiot BI Tech Lead at Rittman
Mead Verona, Italy Rittman Mead Blog 10 Years Experience in BI/Analytics [email protected] @FTisiot Oracle ACE !2

[email protected] www.rittmanmead.com @rittmanmead About Rittman Mead !3 Rittman Mead is
a data and analytics company who specialise in data visualisation, predictive analytics, enterprise reporting and data engineering. We use our skill, experience and know-how to work with organisations across the world to interpret their data. We enable the business, the consumers, the data providers and IT to work towards a common goal, delivering innovative and cost-effective solutions based on our core values of thought leadership, hard work and honesty. We work across multiple verticals on projects that range from mature, large scale implementations to proofs of concept and can provide skills in development, architecture, delivery, training and support.

[email protected] www.rittmanmead.com @rittmanmead Let Me Know My Audience !5 Who
Likes + ? +

[email protected] www.rittmanmead.com @rittmanmead Let Me Know My Audience !6 Who
Likes + ? +

[email protected] www.rittmanmead.com @rittmanmead !7 The Good Old Days… Photo by
Bruno Martins on Unsplash

[email protected] www.rittmanmead.com @rittmanmead !8 I Need This “Precise” KPI! Ok!
Let’s Create the Model and ETL the Data! Photo by Cristina Gottardi on Unsplash

[email protected] www.rittmanmead.com @rittmanmead !9 Predefined KPIs • Structured Reporting Database
Batch Processing • Overnight Load

[email protected] www.rittmanmead.com @rittmanmead !10 Self Service Analytics Photo by Dominik
Scythe on Unsplash

[email protected] www.rittmanmead.com @rittmanmead !11 Python R Data Scientist Photo by
Lucas Vasques on Unsplash

[email protected] www.rittmanmead.com @rittmanmead !12 Visualize Business Analyst Extract Calculate Photo
by Craig Garner on Unsplash

[email protected] www.rittmanmead.com @rittmanmead !13 IT Driven Organised Pre-Defined OBIEE Photo
by Tiago Muraro on Unsplash

[email protected] www.rittmanmead.com @rittmanmead Business Intelligence Tools !14 Oracle Analytics

[email protected] www.rittmanmead.com @rittmanmead !15 Business Driven Data Discovery No Prebuilt
Model Data Visualization Access To Raw Data Photo by Samuel Zeller on Unsplash

[email protected] www.rittmanmead.com @rittmanmead Data Visualization !16 • Information Exploration and
Discovery - Single Panel Analytics - Data Mashup - Integrated with OBIEE - DataFlow Component

[email protected] www.rittmanmead.com @rittmanmead DataFlow Component !17 • Transform/Enrich Data -
Filter - Aggregate - Join - Store Locally or Push Back - V4 Release • Machine Learning • Essbase Cube

[email protected] www.rittmanmead.com @rittmanmead !19 Volume Velocity Variety $$$ BIG DATA

[email protected] www.rittmanmead.com @rittmanmead !20

[email protected] www.rittmanmead.com @rittmanmead !22 Real Time Analytics Photo by Genessa
Panainte on Unsplash Batch vs Stream

[email protected] www.rittmanmead.com @rittmanmead !23 Any Source Any Target Open Formats
Scalable

[email protected] www.rittmanmead.com @rittmanmead !24 https://www.confluent.io/product/confluent-platform/ Data Hub

[email protected] www.rittmanmead.com @rittmanmead !25 https://www.confluent.io/product/confluent-platform/

[email protected] www.rittmanmead.com @rittmanmead !26 Hub!

[email protected] www.rittmanmead.com @rittmanmead !27 https://www.confluent.io/product/confluent-platform/ Client Library

[email protected] www.rittmanmead.com @rittmanmead !28 https://www.confluent.io/product/confluent-platform/ SQL!

[email protected] www.rittmanmead.com @rittmanmead !31 Currently Limited SQL functions Can be
Extended with UDF! GA Since March 2018! Enhancements expected KSQL

[email protected] www.rittmanmead.com @rittmanmead !32 Sources Targets Transformations Kafka ?

[email protected] www.rittmanmead.com @rittmanmead !33 Stream

[email protected] www.rittmanmead.com @rittmanmead !34 Time Series Visualization

[email protected] www.rittmanmead.com @rittmanmead !35 https://www.rittmanmead.com/blog/2017/11/taking-ksql-for-a-spin-using-real-time-device-data/

[email protected] www.rittmanmead.com @rittmanmead !36 Sources Targets Transformations Kafka

[email protected] www.rittmanmead.com @rittmanmead Photo by Alexandre Debiève on Unsplash !37
How do I Visualise the Data in Kafka?

[email protected] www.rittmanmead.com @rittmanmead !38 KSQL Kafka Consumer

[email protected] www.rittmanmead.com @rittmanmead !39 KSQL Limited set of functions Can’t
be called from “outside” Kafka (Need special adapter)

[email protected] www.rittmanmead.com @rittmanmead !41 https://www.rittmanmead.com/blog/2017/04/sql-on-hadoop-impala-vs-drill/ Big Data Traditional BI Tools
SQL-on-Hadoop ODBC JDBC

[email protected] www.rittmanmead.com @rittmanmead !43 SQL-on-(almost)Everything

[email protected] www.rittmanmead.com @rittmanmead !44 https://www.rittmanmead.com/blog/2017/07/analyzing-wimbledon-twitter-feeds-in-real-time-with-kafka-presto-and-oracle-dvd-v3/

[email protected] www.rittmanmead.com @rittmanmead !45 Static List of Streams No AVRO
Support Limitations

[email protected] www.rittmanmead.com @rittmanmead !46 Dynamic List of Streams No AVRO
Support

[email protected] www.rittmanmead.com @rittmanmead !47 Define a KSQL Stream as JSON
Format CREATE STREAM STREAM_NAME WITH ( VALUE_FORMAT=‘JSON’ ) AS SELECT … FROM …;

[email protected] www.rittmanmead.com @rittmanmead !48 Use Kafka Show Tables

[email protected] www.rittmanmead.com @rittmanmead !49 Query Data

[email protected] www.rittmanmead.com @rittmanmead !50 KSQL vs SQL-on-Hadoop Continuous Query Static
Query Data Resides in Kafka External (Static) Tables can be created Limited SQL Rich set of SQL Functions (Views can be created) Can be Accessed via ODBC No JDBC/ODBC access officially Supported Examining Data Streams and Stream Processing Ad-hoc Random Access

[email protected] www.rittmanmead.com @rittmanmead !51 Latest Apache Drill Release - Kafka
Enhancements Filter Pushdown •PartitionId •MsgOffset •MsgTimestamp

Why Should I use DV to Visualize Streams?

[email protected] www.rittmanmead.com @rittmanmead !53 It’s Data! Self Service Analytics Real
Time Insights Mash-up Test and Apply ML

Does DV Provide a Kafka Connection?

[email protected] www.rittmanmead.com @rittmanmead !55 Photo by Gemma Evans on Unsplash
But We Can Use Drill!

[email protected] www.rittmanmead.com @rittmanmead !56 #obihackers Kafka Connect DVD Example

[email protected] www.rittmanmead.com @rittmanmead !57 Use the Tools for What they
are Good at! •Stream Keys and Timestamps •Stream in JSON Format •Aggregated Tables •Complex SQL Functions •Combining Data •Ranking/Ordering •Visualize Data •Mashup •Machine Learning/Advanced Analytics

[email protected] www.rittmanmead.com @rittmanmead !58 DVD and Drill • Install MapR
ODBC Driver • Create Connection • Select Storage • Select Table • In case of Errors replace “…” with `…` "dfs.tmp".localkafka `dfs.tmp`.localkafka

[email protected] www.rittmanmead.com @rittmanmead !59 DVD and Drill • Remove Caching
• No Data Flows • Logical SQL transformations

[email protected] www.rittmanmead.com @rittmanmead !61 Limits • Creation of a Kafka
Consumer for each Query • Reads all Data from Topic (now with pushdown filters) • JSON format (until AVRO support) Not Suitable for Massive Dashboard Style Reporting

[email protected] www.rittmanmead.com @rittmanmead !62 Suggestion …Use the Tools for What
they are Good at! Kafka Connect • Data/Insights Discovery • Self Service Analytics • Data Mashup • Small Datasets • Prototype Building • Monitoring/Alerting • Dashboards • Massive Datasets • Consolidated KPIs • Complex Calculations

[email protected] www.rittmanmead.com @rittmanmead !63 Photo by Jason Blackeye on Unsplash
#Cloud

[email protected] www.rittmanmead.com @rittmanmead !65 Final Suggestions….

[email protected] www.rittmanmead.com @rittmanmead !66 • Kafka ‣ Separate Topics for
direct Reporting ‣ Include Discovery back to Kafka • Drill ‣ Prototype and Data Mashup • DVD ‣ Visualizations and Personal Data Mashup • Standardised Reporting ‣ Kafka Connect Sink and “Old Days” Tools Modern Analytical Platform Photo by Drew Patrick Miller on Unsplash

[email protected] www.rittmanmead.com @rittmanmead !67 Life Is Too Short to Use
the WRONG Technology

Visualizing Streams

Visualizing Streams

More Decks by FTisiot

Other Decks in Technology

Featured

Transcript