Slide 1
Slide 1 text
Processing Streaming Data
with KSQL
@gamussa
#SQLSaturday
Slide 2
Slide 2 text
@gamussa #SQLSaturday @confluentinc
Declarative
Stream
Language
Processing
KSQL
is a
Slide 3
Slide 3 text
@gamussa #SQLSaturday @confluentinc
KSQL
is the
Streaming
SQL Engine
for
Apache Kafka
Slide 4
Slide 4 text
@
@gamussa #SQLSaturday @confluentinc
Solutions Architect
Developer Advocate
@gamussa in internetz
Hey you, yes, you,
go follow me in twitter ©
Who am I?
Slide 5
Slide 5 text
@gamussa #SQLSaturday @confluentinc
Stream Processing by Analogy
Kafka Cluster
Connect API Stream Processing Connect API
$ cat < in.txt | grep “ksql” | tr a-z A-Z > out.txt
Slide 6
Slide 6 text
@
@gamussa #SQLSaturday @confluentinc
Kafka is a Streaming Platform
The Log Connectors
Connectors
Producer Consumer
Streaming Engine
Slide 7
Slide 7 text
@
@gamussa #SQLSaturday @confluentinc
Streaming
is the toolset for dealing
with events
as they move!
Slide 8
Slide 8 text
@
@gamussa #SQLSaturday @confluentinc
authorization_attempts possible_fraud
What exactly is Stream Processing?
Slide 9
Slide 9 text
@
@gamussa #SQLSaturday @confluentinc
CREATE STREAM possible_fraud AS
SELECT card_number, count(*)
FROM authorization_attempts
WINDOW TUMBLING (SIZE 5 MINUTE)
GROUP BY card_number
HAVING count(*) > 3;
authorization_attempts possible_fraud
What exactly is Stream Processing?
Slide 10
Slide 10 text
@
@gamussa #SQLSaturday @confluentinc
CREATE STREAM possible_fraud AS
SELECT card_number, count(*)
FROM authorization_attempts
WINDOW TUMBLING (SIZE 5 MINUTE)
GROUP BY card_number
HAVING count(*) > 3;
authorization_attempts possible_fraud
What exactly is Stream Processing?
Slide 11
Slide 11 text
@
@gamussa #SQLSaturday @confluentinc
CREATE STREAM possible_fraud AS
SELECT card_number, count(*)
FROM authorization_attempts
WINDOW TUMBLING (SIZE 5 MINUTE)
GROUP BY card_number
HAVING count(*) > 3;
authorization_attempts possible_fraud
What exactly is Stream Processing?
Slide 12
Slide 12 text
@
@gamussa #SQLSaturday @confluentinc
CREATE STREAM possible_fraud AS
SELECT card_number, count(*)
FROM authorization_attempts
WINDOW TUMBLING (SIZE 5 MINUTE)
GROUP BY card_number
HAVING count(*) > 3;
authorization_attempts possible_fraud
What exactly is Stream Processing?
Slide 13
Slide 13 text
@
@gamussa #SQLSaturday @confluentinc
CREATE STREAM possible_fraud AS
SELECT card_number, count(*)
FROM authorization_attempts
WINDOW TUMBLING (SIZE 5 MINUTE)
GROUP BY card_number
HAVING count(*) > 3;
authorization_attempts possible_fraud
What exactly is Stream Processing?
Slide 14
Slide 14 text
@
@gamussa #SQLSaturday @confluentinc
CREATE STREAM possible_fraud AS
SELECT card_number, count(*)
FROM authorization_attempts
WINDOW TUMBLING (SIZE 5 MINUTE)
GROUP BY card_number
HAVING count(*) > 3;
authorization_attempts possible_fraud
What exactly is Stream Processing?
Slide 15
Slide 15 text
No content
Slide 16
Slide 16 text
@
@gamussa #SQLSaturday @confluentinc
Table-Stream Duality
Slide 17
Slide 17 text
Do you think that’s a
table you are querying ?
Slide 18
Slide 18 text
@
@gamussa #SQLSaturday @confluentinc
Streams to Tables
Slide 19
Slide 19 text
@
@gamussa #SQLSaturday @confluentinc
Slide 20
Slide 20 text
@
@gamussa #SQLSaturday @confluentinc
Stream/Table Duality
Slide 21
Slide 21 text
@
@gamussa #SQLSaturday @confluentinc
Stream/Table Duality
Slide 22
Slide 22 text
@
@gamussa #SQLSaturday @confluentinc
Gary 1
Gary 1
Viktor 1
Gary 2
Viktor 1
Gary 2
Viktor 1
Soby 1
TABLE STREAM TABLE
(“Gary”, 1)
(“Viktor”, 1)
(“Gary”, 2)
(“Soby”, 1)
Gary 1
Gary 1
Viktor 1
Gary 2
Viktor 1
Gary 2
Viktor 1
Soby 1
Slide 23
Slide 23 text
@
@gamussa #SQLSaturday @confluentinc
Join Streams and Tables
Compacted
Topic
Join
Stream
Table
Kafka Kafka Streams
Topic
Slide 24
Slide 24 text
Demo
Slide 25
Slide 25 text
@gamussa #SQLSaturday @confluentinc
Where is KSQL not such a great fit?
BI reports (Tableau etc.)
•No indexes
•No JDBC (most BI
tools are not good with
continuous results!)
Ad-hoc queries
•Limited span of time
usually retained in
Kafka
•No indexes
Slide 26
Slide 26 text
@gamussa #SQLSaturday @confluentinc
Resources and Next Steps
https://github.com/confluentinc/ksql
http://confluent.io/ksql
https://slackpass.io/confluentcommunity #ksql
Slide 27
Slide 27 text
@
@gamussa #SQLSaturday @confluentinc
Thanks!
@gamussa
[email protected]
We are hiring!
https://www.confluent.io/careers/