Building a Real-Time IoT Application with Apache Pulsar and Apache Pinot (Timothy Spann, Cloudera) | RTA Summit 2023

We will walk step-by-step with live code and demos on how to build a real-time IoT application with Pinot + Pulsar.

First, we stream sensor data from an edge device monitoring location conditions to Pulsar via a Python application.

We have our Apache Pinot “realtime” table connected to Pulsar via the pinot-pulsar stream ingestion connector.

Our data streams into the stream, and we visualize it with Superset.

https://medium.com/@tspann/building-a-real-time-iot-application-with-apache-pulsar-and-apache-pinot-1e3baf8c1824

Source Code
https://github.com/tspannhw/pulsar-thermal-pinot

Reference
https://docs.pinot.apache.org/basics/data-import/pinot-stream-ingestion/apache-pulsar
https://dev.startree.ai/docs/pinot/recipes/pulsar

Transcript

1. Building a Real-Time IoT Application Tim Spann Principal Developer Advocate 26-April-2023
2. 2 Notes We will walk step-by-step with live code and demos on how to build a real-time IoT application with Pinot + Pulsar. First, we stream sensor data from an edge device monitoring location conditions to Pulsar via a Python application. We have our Apache Pinot “realtime” table connected to Pulsar via the pinot-pulsar stream ingestion connector. Our data streams into the stream, and we visualize it with Superset. https://medium.com/@tspann/building-a-real-time-iot-application-with-apache-pulsar-and-apach e-pinot-1e3baf8c1824 https://github.com/tspannhw/pulsar-thermal-pinot
3. FLiPN-FLaNK Stack Tim Spann @PaasDev // Blog: www.datainmotion.dev Principal Developer Advocate, Cloudera Princeton Future of Data Meetup. ex-Pivotal, ex-Hortonworks, ex-StreamNative, ex-PwC https://github.com/tspannhw/EverythingApacheNiFi https://medium.com/@tspann Apache NiFi x Apache Kafka x Apache Flink x Java
4. https://attend.cloudera.com/nificommitters0503
5. 5 • Introduction to Pinot • Introduction to Apache Pulsar • NiFi to Pulsar to Pinot (FLiPN) • NiFi to Kafka to Pinot (P-FLaNK) • FLaNK Ingest • Demos
6. 6 Assets Apache NiFi: Flows Apache Pinot: Real-Time Tables Apache Kafka: Topics Apache Pulsar: Topics Apache Flink SQL: Virtual Tables
7. APACHE PULSAR
8. 101 Unified Messaging Platform Guaranteed Message Delivery Resiliency Infinite Scalability
9. Streaming Consumer Consumer Consumer Subscription Shared Failover Consumer Consumer Subscription In case of failure in Consumer B-0 Consumer Consumer Subscription Exclusive X Consumer Consumer Key-Shared Subscription Pulsar Topic/Partition Messaging
10. Kafka On Pulsar (KoP)
11. MQTT On Pulsar (MoP)
12. AMQP On Pulsar (AoP)
13. STREAMING
14. 14 STREAMING FROM … TO .. WHILE .. Data distribution as a first class citizen IOT Devices LOG DATA SOURCES ON-PREM DATA SOURCES BIG DATA CLOUD SERVICES CLOUD BUSINESS PROCESS SERVICES * CLOUD DATA* ANALYTICS /SERVICE (Cloudera DW) App Logs Laptops /Servers Mobile Apps Security Agents CLOUD WAREHOUSE UNIVERSAL DATA DISTRIBUTION (Ingest, Transform, Deliver) Ingest Processors Ingest Gateway Router, Filter & Transform Processors Destination Processors
15. 15 End to End Streaming Pipeline Example Enterprise sources Weather Errors Aggregates Alerts Stocks ETL Analytics Clickstream Market data Machine logs Social SQL
16. APACHE KAFKA
17. © 2019 Cloudera, Inc. All rights reserved. 17 Apache Kafka • Highly reliable distributed messaging system • Decouple applications, enables many-to-many patterns • Publish-Subscribe semantics • Horizontal scalability • Efficient implementation to operate at speed with big data volumes • Organized by topic to support several use cases Source System Source System Source System Kafka Fraud Detection Security Systems Real-Time Monitoring Source System Source System Source System Fraud Detection Security Systems Real-Time Monitoring Many-To-Many Publish-Subscribe Point-To-Point Request-Response
18. STREAM TEAM
19. 19 CSP Community Edition • Kafka, KConnect, SMM, SR, Flink, and SSB in Docker • Runs in Docker • Try new features quickly • Develop applications locally • Docker compose file of CSP to run from command line w/o any dependencies, including Flink, SQL Stream Builder, Kafka, Kafka Connect, Streams Messaging Manager and Schema Registry ◦ $> docker compose up • Licensed under the Cloudera Community License • Unsupported • Community Group Hub for CSP • Find it on docs.cloudera.com under Applications
20. DATAFLOW APACHE NIFI
21. 21 Cloudera Flow and Edge Management Enable easy ingestion, routing, management and delivery of any data anywhere (Edge, cloud, data center) to any downstream system with built in end-to-end security and provenance Advanced tooling to industrialize flow development (Flow Development Life Cycle) ACQUIRE • Over 300 Prebuilt Processors • Easy to build your own • Parse, Enrich & Apply Schema • Filter, Split, Merger & Route • Throttle & Backpressure FTP SFTP HL7 UDP XML HTTP EMAIL HTML IMAGE SYSLOG PROCESS HASH MERGE EXTRACT DUPLICATE SPLIT ENCRYPT TALL EVALUATE EXECUTE GEOENRICH SCAN REPLACE TRANSLATE CONVERT ROUTE TEXT ROUTE CONTENT ROUTE CONTEXT ROUTE RATE DISTRIBUTE LOAD DELIVER • Guaranteed Delivery • Full data provenance from acquisition to delivery • Diverse, Non-Traditional Sources • Eco-system integration FTP SFTP HL7 UDP XML HTTP EMAIL HTML IMAGE SYSLOG
22. 22 Cloudera DataFlow: Universal Data Distribution Service Process Route Filter Enrich Transform Distribute Connectors Any destination Deliver Ingest Active Passive Connectors Gateway Endpoint Connect & Pull Send Data born in the cloud Data born outside the cloud Universal Data Distribution Connect to Any Data Source Anywhere then Process and Deliver to Any Destination
23. 23 © 2023 Cloudera, Inc. All rights reserved. What is Apache NiFi? Apache NiFi is a scalable, real-time streaming data platform that collects, curates, and analyzes data so customers gain key insights for immediate actionable intelligence.
24. 24 Apache NiFi Enable easy ingestion, routing, management and delivery of any data anywhere (Edge, cloud, data center) to any downstream system with built in end-to-end security and provenance ACQUIRE PROCESS DELIVER • Over 300 Prebuilt Processors • Easy to build your own • Parse, Enrich & Apply Schema • Filter, Split, Merger & Route • Throttle & Backpressure • Guaranteed Delivery • Full data provenance from acquisition to delivery • Diverse, Non-Traditional Sources • Eco-system integration Advanced tooling to industrialize flow development (Flow Development Life Cycle) FTP SFTP HL7 UDP XML HTTP EMAIL HTML IMAGE SYSLO G FTP SFTP HL7 UDP XML HTTP EMAIL HTML IMAGE SYSLO G HASH MERGE EXTRACT DUPLICATE SPLIT ROUTE TEXT ROUTE CONTENT ROUTE CONTEXT CONTROL RATE DISTRIBUTE LOAD GEOENRICH SCAN REPLACE TRANSLATE CONVERT ENCRYPT TALL EVALUATE EXECUTE
25. © 2023 Cloudera, Inc. All rights reserved. 25 Apache NiFi Pulsar Connector https://streamnative.io/apache-nifi-connector/
26. APACHE FLINK
27. © 2023 Cloudera, Inc. All rights reserved. 27 Flink SQL https://www.datainmotion.dev/2021/04/cloudera-sql-stream-builder-ssb-updated.html • Streaming Analytics • Continuous SQL • Continuous ETL • Complex Event Processing • Standard SQL Powered by Apache Calcite
28. 28 Flink SQL -- specify Kafka partition key on output SELECT foo AS _eventKey FROM sensors -- use event time timestamp from kafka -- exactly once compatible SELECT eventTimestamp FROM sensors -- nested structures access SELECT foo.’bar’ FROM table; -- must quote nested column -- timestamps SELECT * FROM payments WHERE eventTimestamp > CURRENT_TIMESTAMP-interval '10' second; -- unnest SELECT b.*, u.* FROM bgp_avro b, UNNEST(b.path) AS u(pathitem) -- aggregations and windows SELECT card, MAX(amount) as theamount, TUMBLE_END(eventTimestamp, interval '5' minute) as ts FROM payments WHERE lat IS NOT NULL AND lon IS NOT NULL GROUP BY card, TUMBLE(eventTimestamp, interval '5' minute) HAVING COUNT(*) > 4 -- >4==fraud -- try to do this ksql! SELECT us_west.user_score+ap_south.user_score FROM kafka_in_zone_us_west us_west FULL OUTER JOIN kafka_in_zone_ap_south ap_south ON us_west.user_id = ap_south.user_id; Key Takeaway: Rich SQL grammar with advanced time and aggregation tools
29. 29 © 2023 Cloudera, Inc. All rights reserved. SQL Stream Builder (SSB) SQL STREAM BUILDER allows developers, analysts, and data scientists to write streaming applications with industry standard SQL. No Java or Scala code development required. Simplifies access to data in Kafka & Flink. Connectors to batch data in HDFS, Kudu, Hive, S3, JDBC, CDC and more Enrich streaming data with batch data in a single tool Democratize access to real-time data with just SQL
30. DEMO
31. © 2023 Cloudera, Inc. All rights reserved. 31
32. Device -> Pulsar -> Pinot
40. Schemas Build a schema https://github.com/startreedata/pinot-recipes/tree/main/recipes/infer-schema-j son-data
41. https://docs.pinot.apache.org/basics/data-import/pinot-stream-ingestion/apache-pulsar https://dev.startree.ai/docs/pinot/recipes/pulsar https://github.com/startreedata/pinot-recipes/tree/main/recipes/pulsar Development Resources
42. Easy Docker Demo docker exec -it pinot-controller /bin/bash docker exec -it pinot-controller bin/pinot-admin.sh JsonToPinotSchema \ -timeColumnName ts \ -metrics "temperature,humidity,co2,totalvocppb,equivalentco2ppm,pressure,temperatureicp,cputempf"\ -dimensions "host,ipaddress" \ -pinotSchemaName=thermal \ -jsonFile=/data/thermal.json \ -outputDir=/config docker exec -it pinot-controller bin/pinot-admin.sh AddSchema \ -schemaFile /config/thermalschema.json \ -exec
43. Local Apache Pinot Admin curl -X DELETE "http://localhost:9000/tables/thermal?type=realtime" -H "accept: application/json" curl -X DELETE "http://localhost:9000/schemas/thermal" -H "accept: application/json" docker exec -it pinot-controller bin/pinot-admin.sh AddSchema \ -schemaFile /config/thermalschema.json \ -exec curl -X POST "http://localhost:9000/tables" -H "accept: application/json" -H " ….
47. NiFi -> Kafka -> Pinot
48. © 2023 Cloudera, Inc. All rights reserved. 48 Reference Architecture Microservices ETL
49. RESOURCES AND WRAP-UP
50. 50 Resources
51. 51 TH N Y U