Beyond the Tweeting Toaster: IoT Analytics with Apache Storm, Kafka and Arduino

Beyond the Tweeting Toaster: IoT Analytics with Apache Storm, Kafka and Arduino

Slides from my talk at Data Day Texas 2015.

423abccbed514c8464e12ae1479003a9?s=128

P. Taylor Goetz

January 10, 2015
Tweet

Transcript

  1. 1.

    Beyond the Tweeting Toaster (I)IoT Streaming Analytics with Apache Storm,

    Kafka and Arduino P. Taylor Goetz, Hortonworks @ptgoetz
  2. 6.

    Is that a sensor in your pocket? • GPS •

    Proximity Sensor • Ambient Light Sensor • 3-Axis Accelerometer • Magnetometer • Gyroscopic Sensor • Wifi • Camera(s) • UI (senses user interaction) • iBeacon
  3. 10.

    2014 South Napa Earthquake August 24, 3:20 AM (no, the

    earthquake was not caused by people waking up)
  4. 11.
  5. 12.

    –Wikipedia A sensor is a device that detects events or

    changes in quantities and provides a corresponding output, generally as an electrical or optical signal; for example, a thermocouple converts temperature to an output voltage.
  6. 13.
  7. 16.
  8. 19.

    Hotel Room Monitoring and Automation • Why heat/cool an unoccupied

    room? • Detect occupancy. • Anticipate occupancy. • React to occupancy.! • Added benefits to customer experience. • Analyze customer behavior.
  9. 20.

    Quikie Auto Lube • Manage inventory in response to demand.

    • Detect inventory. • Anticipate customer demand. • React accordingly.
  10. 21.

    Hospital Infection Control • CDC: Hospital acquired infections cost $30B

    per year and lead to 100K patient deaths. • Inadequate hand washing a big cause.
  11. 22.

    Hospital Infection Control • Ensure proper hygiene of medical staff

    • Detect staff presence. • Anticipate hand washing. • React to inadequate hygiene.
  12. 24.

    Auto Insurance • Rethinking traditional risk assessment • Detect unsafe

    driving practices. • Anticipate who is most at risk. • React to risk assessment.
  13. 28.

    What is Arduino? • Open Source Microcontroller Hardware + Software

    • Geared toward prototyping • “Physical Computing:” interacting with the environment "Arduino is an open-source electronics platform based on easy-to-use hardware and software. It's intended for anyone making interactive projects.”
  14. 31.

    Programming Arduino • Official cross-platform IDE written in java •

    C/C++ with some sugar • Program referred to as “Sketch” • Many open source libraries available for various hardware (sensors, etc.)
  15. 33.

    What is XBee? • Radio modules that support wireless point-to-point

    communication • Serial communication • Minimal connections required — power, ground, data in, data out (UART) • 2 power options (1 mW/100 mW) • Support for multiple network topologies (Mesh, star, tree, etc.)
  16. 34.

    Architecture/Data Flow Transmit raw sensor data. Receive data, Add timestamp,

    Publish. Reliable queue. Analytics, persistence, alerting. Sensor (XBee/Arduino) Collector (XBee/R-Pi) Kafka Storm 2.4 GHz TCP/IP
  17. 36.

    Architecture/Data Flow Sensor Ouput Kafka Storm Persist/ETL Analysis Alerting Persist/archive

    raw/intermediate data for batch/interactive flows (e.g. Lambda, etc.) Detect : Anticipate : React } }
  18. 37.

    Architecture/Data Flow Sensor Ouput Kafka Storm Persist/ETL Analysis Alerting Persist/archive

    raw/intermediate data for batch/interactive flows (e.g. Lambda, etc.) Detect : Anticipate : React } } Feed your model
  19. 39.
  20. 40.
  21. 41.
  22. 44.

    The Dog Ate My Demo! Okay, he just chewed part

    of it. But all is not lost…
  23. 45.

    Arduino Sketch (Sensor) #include <stdio.h>! #include <Esplora.h>! ! void setup()

    {! // initialize the serial communication:! Serial.begin(9600);! }! ! void loop() {! // read sensor variables! int loudness = Esplora.readMicrophone();! int light = Esplora.readLightSensor();! int temp = Esplora.readTemperature(DEGREES_F);! int slider = Esplora.readSlider();! int joystickButton = Esplora.readJoystickSwitch();! int xAxis = Esplora.readAccelerometer(X_AXIS);! // … ! ! Serial.print("{");! // Misc. Sensors! printAttribute("temperature", temp, false);! printAttribute("loudness", loudness, false);! // …! Serial.println(“}");! ! delay(1000); ! }! ! Initialize serial communication Read sensor values Dump JSON to serial port (XBee) Include Esplora convenience lib
  24. 46.

    Serial Monitor (Collector) • Read serial data, parse JSON •

    Add timestamp • Add device/sensor UID (“Sector 7-G”) • Publish to Kafka
  25. 47.

    Radiation Leak Topology Raw Sensor Output (JSON) Extract Req’d Fields

    Evaluate Threshold Raise Hell! Kafka Spout Parse Bolt Threshold Bolt Alert Bolt Shuffle Grouping Fields Grouping Shuffle Grouping
  26. 48.
  27. 50.

    Resources • Apache Storm
 http://storm.apache.org • Apache Kafka
 http://kafka.apache.org •

    Arduino
 http://arduino.cc • Adafruit
 https://www.adafruit.com • SparkFun
 https://www.sparkfun.com