Beyond the Tweeting Toaster: IoT Analytics with Apache Storm, Kafka and Arduino

Slides from my talk at Data Day Texas 2015.

Transcript

1. Beyond the Tweeting Toaster
   (I)IoT Streaming Analytics with Apache Storm, Kafka and Arduino
   P. Taylor Goetz, Hortonworks
   @ptgoetz
   

   View Slide

2. Credit where credit is due…
   

   View Slide

3. @mytoaster
   • Created by Hans Scharler in 2008
   • http://nothans.tumblr.com
   

   View Slide

4. 26 billion (I)IoT devices by 2020
   -Gartner
   http://www.gartner.com/newsroom/id/2636073
   

   View Slide

5. IPv4 Address Space: 4.6 billion
   

   View Slide

6. Is that a sensor in your pocket?
   • GPS
   • Proximity Sensor
   • Ambient Light Sensor
   • 3-Axis Accelerometer
   • Magnetometer
   • Gyroscopic Sensor
   • Wifi
   • Camera(s)
   • UI (senses user interaction)
   • iBeacon
   

   View Slide

7. In your car?
   sensormag.com
   

   View Slide

8. In your car?
   

   View Slide

9. On your wrist?
   Jawbone UP
   FitBit
   …
   

   View Slide

10. 2014 South Napa Earthquake
    August 24, 3:20 AM
    (no, the earthquake was not caused by people waking up)
    

    View Slide

11. Sensors
    

    View Slide

12. –Wikipedia
    A sensor is a device that detects events or changes in quantities
    and provides a corresponding output, generally as an electrical or
    optical signal; for example, a thermocouple converts temperature
    to an output voltage.
    

    View Slide

13. Sensors
    

    View Slide

14. http://adafruit.com
    

    View Slide

15. http://adafruit.com
    

    View Slide

16. Use Cases
    

    View Slide

17. Detect : Anticipate : React
    

    View Slide

18. Detect : Anticipate : React
    Detect behavior.
    Anticipate behavior.
    React to behavior.
    

    View Slide

19. Hotel Room Monitoring and Automation
    • Why heat/cool an unoccupied room?
    • Detect occupancy.
    • Anticipate occupancy.
    • React to occupancy.!
    • Added benefits to customer
    experience.
    • Analyze customer behavior.
    

    View Slide

20. Quikie Auto Lube
    • Manage inventory in response to
    demand.
    • Detect inventory.
    • Anticipate customer demand.
    • React accordingly.
    

    View Slide

21. Hospital Infection Control
    • CDC: Hospital acquired infections
    cost $30B per year and lead to 100K
    patient deaths.
    • Inadequate hand washing a big
    cause.
    

    View Slide

22. Hospital Infection Control
    • Ensure proper hygiene of medical
    staff
    • Detect staff presence.
    • Anticipate hand washing.
    • React to inadequate hygiene.
    

    View Slide

23. Hospital Infection Control
    

    View Slide

24. Auto Insurance
    • Rethinking traditional risk assessment
    • Detect unsafe driving practices.
    • Anticipate who is most at risk.
    • React to risk assessment.
    

    View Slide

25. Use your imagination.
    

    View Slide

26. How can you use and combine
    sensors to better serve your
    [user’s] needs?
    

    View Slide

27. What is Arduino?
    

    View Slide

28. What is Arduino?
    • Open Source Microcontroller
    Hardware + Software
    • Geared toward prototyping
    • “Physical Computing:” interacting
    with the environment
    "Arduino is an open-source electronics
    platform based on easy-to-use hardware
    and software. It's intended for anyone
    making interactive projects.”
    

    View Slide

29. http://adafruit.com
    

    View Slide

30. What is Arduino?
    

    View Slide

31. Programming Arduino
    • Official cross-platform IDE
    written in java
    • C/C++ with some sugar
    • Program referred to as
    “Sketch”
    • Many open source libraries
    available for various
    hardware (sensors, etc.)
    

    View Slide

32. Going Wireless with XBee
    

    View Slide

33. What is XBee?
    • Radio modules that support wireless
    point-to-point communication
    • Serial communication
    • Minimal connections required —
    power, ground, data in, data out
    (UART)
    • 2 power options (1 mW/100 mW)
    • Support for multiple network
    topologies (Mesh, star, tree, etc.)
    

    View Slide

34. Architecture/Data Flow
    Transmit raw
    sensor data.
    Receive data,
    Add timestamp,
    Publish.
    Reliable
    queue.
    Analytics,
    persistence,
    alerting.
    Sensor
    (XBee/Arduino)
    Collector
    (XBee/R-Pi)
    Kafka Storm
    2.4 GHz TCP/IP
    

    View Slide

35. Architecture/Data Flow
    Storm
    Zone 2
    Zone 1
    Kafka
    

    View Slide

36. Architecture/Data Flow
    Sensor Ouput Kafka Storm
    Persist/ETL
    Analysis
    Alerting
    Persist/archive raw/intermediate
    data for batch/interactive flows
    (e.g. Lambda, etc.)
    Detect : Anticipate : React
    }
    }
    

    View Slide

37. Architecture/Data Flow
    Sensor Ouput Kafka Storm
    Persist/ETL
    Analysis
    Alerting
    Persist/archive raw/intermediate
    data for batch/interactive flows
    (e.g. Lambda, etc.)
    Detect : Anticipate : React
    }
    } Feed your model
    

    View Slide

38. Demo
    Twitter: @ptgoetz #ddtx15
    

    View Slide

39. View Slide

40. View Slide

41. View Slide

42. “Three eyes are better than two!”
    

    View Slide

43. Arduino Esplora
    Several sensors included.
    (no soldering required)
    

    View Slide

44. The Dog Ate My Demo!
    Okay, he just chewed part of it. But all is not lost…
    

    View Slide

45. Arduino Sketch (Sensor)
    #include !
    #include !
    !
    void setup() {!
    // initialize the serial communication:!
    Serial.begin(9600);!
    }!
    !
    void loop() {!
    // read sensor variables!
    int loudness = Esplora.readMicrophone();!
    int light = Esplora.readLightSensor();!
    int temp = Esplora.readTemperature(DEGREES_F);!
    int slider = Esplora.readSlider();!
    int joystickButton = Esplora.readJoystickSwitch();!
    int xAxis = Esplora.readAccelerometer(X_AXIS);!
    // … !
    !
    Serial.print("{");!
    // Misc. Sensors!
    printAttribute("temperature", temp, false);!
    printAttribute("loudness", loudness, false);!
    // …!
    Serial.println(“}");!
    !
    delay(1000); !
    }!
    !
    Initialize serial communication
    Read sensor values
    Dump JSON to serial port (XBee)
    Include Esplora convenience lib
    

    View Slide

46. Serial Monitor (Collector)
    • Read serial data, parse JSON
    • Add timestamp
    • Add device/sensor UID (“Sector 7-G”)
    • Publish to Kafka
    

    View Slide

47. Radiation Leak Topology
    Raw Sensor Output
    (JSON)
    Extract Req’d
    Fields
    Evaluate
    Threshold
    Raise Hell!
    Kafka Spout Parse Bolt Threshold Bolt Alert Bolt
    Shuffle Grouping Fields Grouping Shuffle Grouping
    

    View Slide

48. Swag Time
    

    View Slide

49. Swag Time
    Be the 9th person to retweet the last message from the demo.
    

    View Slide

50. Resources
    • Apache Storm

    http://storm.apache.org
    • Apache Kafka

    http://kafka.apache.org
    • Arduino

    http://arduino.cc
    • Adafruit

    https://www.adafruit.com
    • SparkFun

    https://www.sparkfun.com

    

    View Slide

51. Thank You!
    P. Taylor Goetz, Hortonworks
    @ptgoetz
    

    View Slide