From ksqlDB with LOVE: Detecting 007 with a Dash of Machine Learning @ Graz Kafka Meetup #5

* Abstract:

While lots of people are fancying stream processing technologies, many are hesitant when it comes to adoption because the landscape and ecosystem feel somewhat intimidating at first sight. Fear not, ksqlDB (the event streaming database for Apache Kafka) takes away a lot of burden so that application developers can focus on solving business needs.After a brief introduction to ksqlDB, this session continues with a memorable and fun example featuring the fictional Secret Service agent 007. In a step-by- step fashion, you are walked through an end-to-end streaming scenario which is about near real-time face identification in synthetic CCTV images. The example helps you to gain a better understanding about ksqlDB’s learning curve characteristics, extensibility options as well as integration paths with machine learning related tools and services.

* Recording: https://videos.confluent.io/watch/zkbpHyW6ECKvfJdV1prpib

Transcript

1. From ksqlDB with ♥ LOVE ♥ Detecting 007 with a dash of machine learning
2. Hans-Peter Grahsl ‣ technical trainer ‣ independent engineer & consultant ‣ associate lecturer ‣ occasional conference speaker @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 2
3. @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 3
4. @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 4
5. @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 5
6. @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 6
7. Kafka's streaming SQL engine @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 7
8. declarative stream processing language @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 8
9. KSQL in a Nutshell ‣ ANSI SQL inspired ‣familiar syntax & semantics ‣concise & expressive ‣ built on top of Kafka Streams ‣ NO(!) coding skills required ‣ entry barrier? "none" @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 9
10. KSQL in a Nutshell ‣ usual suspects OOTB: ‣projections, filters ‣joins, aggregations ‣windowing @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 10
11. Query Examples @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 11
12. Criteria 1 - Learning Curve ‣ SQL == widespread & successful 4GL ‣ algorithms & data structures handled for us ‣ KSQL ‣ very easy to learn ‣ quick implementation cycles ‣ productivity-wise hard to beat @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 12
13. "Lingua Franca" effect @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 13
14. Streaming Architecture @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 14
15. Streaming Architecture @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 15
16. Streaming Architecture @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 16
17. Streaming Architecture @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 17
18. Streaming Architecture @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 18
19. NOT the ETL of our ancestors @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 19
20. Complex Streaming Architecture @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 20
21. KSQL ➔ originally the [T] only @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 21
22. What if I told you it's as easy to build a STREAMING app as it is to build a CRUD app? @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 22
23. Unified Streaming Architecture @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 23
24. Unified Streaming Architecture @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 24
25. Unified Streaming Architecture @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 25
26. Connecting with Sources HINT: connector examples shown in demo later @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 26
27. Connecting with Sinks HINT: connector examples shown in demo later @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 27
28. Unified Streaming Architecture @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 28
29. Unified Streaming Architecture @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 29
30. Unified Streaming Architecture @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 30
31. Unified Streaming Architecture @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 31
32. Unified Streaming Architecture @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 32
33. Instead, only try to realize the truth... ksqlDB is NO DATABASE as we know it. @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 33
34. Data Concepts in ksqlDB STREAM ‣ immutable append-only sequence ‣ captures events representing a series of facts @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 34
35. Data Concepts in ksqlDB @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 35
36. Data Concepts in ksqlDB TABLE ‣ mutable collection of events ‣ holds the last known value for each key ‣ also result from stateful operations @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 36
37. Data Concepts in ksqlDB @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 37
38. Data Concepts in ksqlDB @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 38
39. ksqlDB: PUSH Queries ‣ act as subscription to query results ‣ fit asynchronous & reactive data flows ‣ run indefinitely ‣ new data causes continuous updates @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 39
40. ksqlDB: PUSH Query @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 40
41. ksqlDB: PULL Queries ‣ fetch point-in-time results ‣ fit request / response data flows ‣ terminate immediately ‣ lookup current state of materialized views @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 41
42. ksqlDB: PULL Query @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 42
43. ksqlDB Functions ‣ choose from three categories ‣ scalar ➔ UDF ‣ aggregation ➔ UDAF ‣ table ➔ UDTF @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 43
44. @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 44
45. Criteria 2: Extensibility Options ‣ custom functions (UDFs, UDFAs & UDTFs) ‣ enable flexbile & powerful capabilities ‣ but Java code needed HINT: custom UDF example shown in demo later @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 45
46. Fictional Use Case @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 46
47. ksqlDB Use Case @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 47
48. ksqlDB Use Case @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 48
49. ksqlDB Use Case @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 49
50. ksqlDB Use Case @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 50
51. Data Architecture @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 51
52. Data Architecture @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 52
53. Data Architecture @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 53
54. Data Architecture @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 54
55. Data Architecture @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 55
56. Data Architecture @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 56
57. Data Architecture @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 57
58. Criteria 3: ML Integration Paths ‣ call fully-managed ML services (external) ‣ run your own model server (co-located) ‣ package home-brewed model into UDF (embedded) ‣ completely separated: integrate ML results via Connectors @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 58
59. Everywhere... ksqlDB runs everywhere @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 59
60. Criteria 4: Deployment Options ‣ run however you want ‣ bare metal, VMs, containers ‣ deploy wherever you need ‣ on-premises, private / public cloud, hybrid ‣ something fully-managed? @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 60
61. @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 61
62. Unfortunately, no one can be told what ksqlDB is...You have to TRY IT for yourselves! https://ksqldb.io @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 62
63. @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 63
64. reach out to me @hpgrahsl