Introducing SQLStreamBuilder: Rich Streaming SQL Interface for Creating and Scaling Stateful Stream Processing Jobs

Discover how SQLStreamBuilder enables you to run streaming SQL against unbounded streams of data and create new, persistent streaming jobs.

Transcript

1. Eventador SQLStreamBuilder A rich streaming SQL interface for creating and scaling stateful stream processing jobs June 2019 Eventador.io
2. Quick Background • Streaming data systems like Apache Kafka have become massively popular for a variety of workloads. • Stream processing frameworks like Apache Flink have been widely adopted for performing computations on these streams. • But, there is no simple way to author, iterate, execute, and manage jobs against boundless streams of data without writing lots of Java or Scala code. Using SQL makes building stream processing jobs EASY and FAST.
3. Why SQL? Structured Query Language (SQL) is a tried-and-true language for managing structured data, so it is a mature and ubiquitous language with a massive user base. With streaming SQL engines for Apache Kafka, Apache Flink and other stream processing technologies, enable organizations without Java and Scala to benefit from rapid and simple streaming application development.
4. SQLStreamBuilder Overview Eventador SQLStreamBuilder (SSB) is an interface to create and manage stream processing jobs using SQL. SSB provides: • An Interactive Streaming SQL Console • A SQL Parser • Compatible with a wide variety of sources and sinks • Schema definition and management • SQL job management • A production grade engine
5. SQLStreamBuilder Feature: Console Interface SSB includes a rich editing interface for authoring, validating and launching streaming SQL jobs and includes a number of unique features. Theming and color coding for SQL syntax SQL history VIM Mode SQL editor with context sensitive menu Apache Calcite-standard SQL syntax SQL comment support
6. SQLStreamBuilder Feature: Built-in SQL Parser SSB delivers a built-in Apache Calcite SQL parser to give immediate feedback on syntax and validity. SSB’s SQL parser: • doesn’t launch the job if you’re still iterating on your syntax, • guides you with helpful feedback • enables you to interactively author SQL by iterating on the statement until it produces the result you want • enables quick trial and error of SQL and speeds time to market for streaming jobs.
7. SQLStreamBuilder Feature: Comprehensive Sources and Sinks SSB enables you to create a virtual table for a source or sink, then specify them in SQL statements (source) and in the console itself (sink)—all in a compact interface. SSB supports: • Kafka-based sources and sinks via ◦ the Eventador platform, ◦ Amazon MSK, or ◦ your own endpoint • The comprehensive Flink ecosystem of sources and sinks ◦ including ones from projects like Apache Bahir
8. SQLStreamBuilder Feature: Schema Management SSB simplifies schema management by: • defining a schema for each source at creation time • exposing that schema using metadata commands and SQL queries in the editor interface. SSB is currently compatible with JSON data, and the schema is defined using JSON-schema compatible syntax. Avro support is coming soon.
9. SQLStreamBuilder Feature: Simple Apache Flink Job Management SSB can seamlessly scale from one to hundreds of concurrent processes all servicing your one SQL job. With SSB: • each statement becomes a job • a collection of one or more jobs makes up some business need/process/feature-a streaming topology • managing jobs is done via the SQL Jobs tab where jobs can be observed, canceled, and edited. • clusters can be expanded to serve as many concurrent SQL jobs as required.
10. SQLStreamBuilder: Production-grade Stream Processing SSB being production grade was a core design tenant. One unique capability of the SSB engine is that users can iterate and perfect SQL, then when ready, simply run against production data sources. Additionally, SSB is a robust platform that: • runs in a highly scalable and fault-tolerant manner by default • utilizes checkpoints to ensure job restart-ability, • is a rich streaming SQL UI fully integrated with Apache Kafka such that restarting picks up from the proper consumer offset, and the state store itself scales and is fault tolerant • runs redundant Flink management components and a best-in-breed stack for Flink SSB also comes with Eventador’s 24x7x365 hands-on support.
11. SQLStreamBuilder: Eventador Architecture
12. Getting Started with SQLStreamBuilder To request access to SQLStreamBuilder, please visit: https://eventador.io/sql-streambuilder/ 1. Create a SSB deployment: • Click on SQLStreamBuilder on the side menu • Set a unique name for the SQL Job Name. Each query is a job and requires a name. If you are iterating quickly on your SQL statement, just use the default name. • If you already have a deployment, you can select it and skip down to the next section. • If you will be creating a new deployment, select Add Deployment from the Target Deployment dropdown box. • In the Add Deployment box, choose a logical Name for this deployment. • Give it a good Description. • Select a Region/Zone. • Click Create. Over the next few moments, The deployment will provision a cluster for running SSB jobs. 2. Create virtual tables for sources and sinks 3. Run SQL It’s as simple as that. To see more detail on getting started, creating sources and sinks, etc. with SQLStreamBuilder, please visit: https://docs.eventador.io/streambuilder/
13. SQLStreamBuilder Streaming SQL Examples Metadata Commands Example SQL Statement for Fraud Detection: Aggregation Timestamps, intervals and time Union ALL Math Misc SQL Tricks
14. Visit https://eventador.io For More Information hello@eventador.io