Upgrade to Pro — share decks privately, control downloads, hide ads and more …

The F1 Demo: Streaming Real-time Telemetry Using Apache Kafka and StreamSets

OmniSci
September 05, 2019

The F1 Demo: Streaming Real-time Telemetry Using Apache Kafka and StreamSets

The F1 Demo: Streaming Real-time Telemetry Using Kafka and StreamSets
By Randy Zwitch, Senior Director of Developer Relations at OmniSci

We have created a demo that streams telemetry data from an F1 video game into OmniSci (a GPU-accelerated relational database) for use in our booth at conferences. Visitors can drive a lap, and onlookers can watch the data flow into OmniSci via StreamSets as well as see a custom Python Dash app to visualize the data in real-time. We'd like to present the StreamSets portion at your conference.

Here is our GitHub repo with the StreamSets portion explained: https://github.com/omnisci/vehicle-telematics-analytics-demo/tree/master/dataengineering.

OmniSci

September 05, 2019
Tweet

More Decks by OmniSci

Other Decks in Technology

Transcript

  1. The F1 Demo: Streaming Real-time Telemetry Using Apache Kafka and

    StreamSets DataOps Summit SF - September 5, 2019
  2. OmniSciDB: Compiled, Columnar and (Lots of) Cores Traditional DBs can

    be highly inefficient - Each operator in SQL treated as a separate function - Incurs tremendous overhead and prevents vectorization OmniSci compiles queries w/ LLVM to create one custom function - Queries run at speeds approaching hand-written functions - LLVM enables generic targeting of different architectures (GPUs, X86, ARM, etc.) - Code can be generated to run query on CPU and GPU simultaneously
  3. What I Learned - Build large pipelines as a series

    of smaller pipelines - Watch your defaults when developing! - Avoid serializing to plain text to improve throughput - Watch out for Jython issues in multi-threaded pipelines / Use Groovy instead