Pro Yearly is on sale from $80 to $50! »

If it moves, measure it! - Logging IoT with ELK

If it moves, measure it! - Logging IoT with ELK

Logs are everywhere! Join Chris as he explores using ELK to collect and analyze time-series data gathered from networked sensors. They've tracked temperature, humidity, motion, power, and other metrics around his home for a month. How did he do it? What did he learn?

This talk will show you all about some of the fun sensor devices he used, as well as showing how he used ELK to collect and analyze the data.

Dd9d954997353b37b4c2684f478192d3?s=128

Elastic Co

March 11, 2015
Tweet

Transcript

  1. CC-BY-ND 4.0 If it moves, measure it! or how I

    drove my wife absolutely crazy with sensors. Chris Cowan Kibana Engineer
  2. CC-BY-ND 4.0 CC-BY-ND 4.0 Measure all the things! • Presence

    Sensors • Motion Detectors • Power Usage • Bandwidth Usage • Contact Switches • Noise Sensors • Temperature and Humidity • Illuminance • Vibrations
  3. CC-BY-ND 4.0 CC-BY-ND 4.0 Architecture 3 SmartThings Hub Logstash Elasticsearch

    SSL Proxy
  4. CC-BY-ND 4.0 CC-BY-ND 4.0 SmartThings Hub • Customizable with Groovy

    • Works with Z-Wave Devices • Works with Zigbee Devices • Provides Mobile Interface
 to System • Their objective it to be open and extensible 4
  5. CC-BY-ND 4.0 CC-BY-ND 4.0 Aeon Labs Home Energy Meter 5

    • Clamps  to  Power  Mains   • Exports  power  usage  in   real  time   • Uses  Z-­‐Wave   • Exports  Watts  and  kWh   • Difficult  to  Install   • You  have  to  risk  your   life  for  installation
  6. CC-BY-ND 4.0 CC-BY-ND 4.0 Behind this panel lives certain death!

  7. CC-BY-ND 4.0 CC-BY-ND 4.0 According to the guy at hardware

    store = SAFETY! +
  8. CC-BY-ND 4.0 CC-BY-ND 4.0 What I thought was going to

    happen...
  9. CC-BY-ND 4.0 CC-BY-ND 4.0 What actually happened...

  10. CC-BY-ND 4.0 CC-BY-ND 4.0 What actually happened...

  11. CC-BY-ND 4.0 CC-BY-ND 4.0 Watt Usage Chart 10 Breakfast Dinner

  12. CC-BY-ND 4.0 CC-BY-ND 4.0 Energy Usage Chart 11

  13. CC-BY-ND 4.0 CC-BY-ND 4.0 Our HVAC Fan Cycles Every Hour

    Fan  Cycling
  14. CC-BY-ND 4.0 CC-BY-ND 4.0 Power By Hour for Last 30

    Days 13
  15. CC-BY-ND 4.0 CC-BY-ND 4.0 Aeon Labs Multisensor • Temperature •

    Humidity • Motion • Illuminance • Uses Z-Wave • Runs on 4 AAA Batteries • Indoor/Outdoors • Looks like the Eye of Sauron • Terrifies Children Most cost effective solution given everything these sensors do along with the SmartThings integration 14
  16. CC-BY-ND 4.0 CC-BY-ND 4.0 Motion Sensor Events 15

  17. CC-BY-ND 4.0 CC-BY-ND 4.0 Counting Stairs When the sensor at

    the top is triggered it looks back 25 seconds for a trigger event at the bottom of the stairs and counts that as 18 stairs.
  18. CC-BY-ND 4.0 CC-BY-ND 4.0 Illuminance Events 17

  19. CC-BY-ND 4.0 CC-BY-ND 4.0 Humidity Events 18 It  rained  last

     night!
  20. CC-BY-ND 4.0 CC-BY-ND 4.0 Temperature Events 19

  21. CC-BY-ND 4.0 CC-BY-ND 4.0 Temperature Percentiles 20

  22. CC-BY-ND 4.0 CC-BY-ND 4.0 Washing Machine Sensor Wait 50 minutes

    after a close event to start watching the for the accelerometer events to stop moving for 2 minutes. • SmartThings Multisensor • Detects Open and Close • Detects Temperature • Detects Movement 21
  23. CC-BY-ND 4.0 CC-BY-ND 4.0 Washing Machine Events 22

  24. CC-BY-ND 4.0 CC-BY-ND 4.0 Raspberry Pi Garage Door Opener 23

    • Raspberry Pi A+ • Relay Switch • Magnetic Relay Switch • Cat 5 Cable • Integrated with
 SmartThings Hub • Door Automatically
 Opens and Closes Code available at: https://github.com/simianhacker/rpi-garage-door
  25. CC-BY-ND 4.0 CC-BY-ND 4.0 Raspberry Pi Garage Door Events 24

  26. CC-BY-ND 4.0 CC-BY-ND 4.0 DIY Sound Sensor • Arduino Uno

    • Analog Sound Sensor • 433mHz Transmitter • 433mHz Receiver on an Arduino Uno hooked up to a Raspberry Pi • RPI logs directly to Elasticsearch 25
  27. CC-BY-ND 4.0 CC-BY-ND 4.0 Sound Sensor: Carpet Cleaning 26 Cleaning

     Carpets  Upstairs
  28. CC-BY-ND 4.0 CC-BY-ND 4.0 Dashboards! 27

  29. CC-BY-ND 4.0 CC-BY-ND 4.0 Energy Dashboard 28

  30. CC-BY-ND 4.0 CC-BY-ND 4.0 Temperature Dashboard 29

  31. CC-BY-ND 4.0 CC-BY-ND 4.0 Hodgepodge Dashboard 30

  32. CC-BY-ND 4.0 CC-BY-ND 4.0 Technical Stuffs

  33. CC-BY-ND 4.0 Data Enrichment with Logstash • Add a time

    parts object to each record – “timeParts” with attributes for hour, minute, day, weekday, year, month,week year, quarter • Calculate time difference between “presence” event. – When an event with the “valueAsString” attribute set to “present” query Elasticsearch and find the previous “not present” event • Calculate difference between “energy” events – Store the value for the last “energy” event and calculate the “delta” 32
  34. 33 if [displayName] =~ /^Top|^Bottom/ and [valueAsString] == "active" {

    }
  35. 33 if [displayName] =~ /^Top|^Bottom/ and [valueAsString] == "active" {

    } # If the event is Top of Stairway then search for Bottom of Stairway if [displayName] =~ /^Top/ { mutate { add_field => { "tempDisplayName" => "Bottom of Stairway” } } } else { mutate { add_field => { "tempDisplayName" => "Top of Stairway” } } }
  36. 33 if [displayName] =~ /^Top|^Bottom/ and [valueAsString] == "active" {

    } # If the event is Top of Stairway then search for Bottom of Stairway if [displayName] =~ /^Top/ { mutate { add_field => { "tempDisplayName" => "Bottom of Stairway” } } } else { mutate { add_field => { "tempDisplayName" => "Top of Stairway” } } } # Find the corresponding event in Elasticsearch elasticsearch { query => "valueAsString.raw:active \ AND displayName.raw:\"%{[tempDisplayName]}\" \ AND isoDate:[* TO %{[isoDate]}]" sort => "isoDate:desc" new_field => "previousDate" }
  37. 33 if [displayName] =~ /^Top|^Bottom/ and [valueAsString] == "active" {

    } # If the event is Top of Stairway then search for Bottom of Stairway if [displayName] =~ /^Top/ { mutate { add_field => { "tempDisplayName" => "Bottom of Stairway” } } } else { mutate { add_field => { "tempDisplayName" => "Top of Stairway” } } } # Find the corresponding event in Elasticsearch elasticsearch { query => "valueAsString.raw:active \ AND displayName.raw:\"%{[tempDisplayName]}\" \ AND isoDate:[* TO %{[isoDate]}]" sort => "isoDate:desc" new_field => "previousDate" } # If the duration between events is less then 25 seconds then add 18 steps to the value ruby { code => "event['value'] = 18 if (event['@timestamp'] - event['previousDate']) > 25" }
  38. 33 if [displayName] =~ /^Top|^Bottom/ and [valueAsString] == "active" {

    } # If the event is Top of Stairway then search for Bottom of Stairway if [displayName] =~ /^Top/ { mutate { add_field => { "tempDisplayName" => "Bottom of Stairway” } } } else { mutate { add_field => { "tempDisplayName" => "Top of Stairway” } } } # Find the corresponding event in Elasticsearch elasticsearch { query => "valueAsString.raw:active \ AND displayName.raw:\"%{[tempDisplayName]}\" \ AND isoDate:[* TO %{[isoDate]}]" sort => "isoDate:desc" new_field => "previousDate" } # If the duration between events is less then 25 seconds then add 18 steps to the value ruby { code => "event['value'] = 18 if (event['@timestamp'] - event['previousDate']) > 25" } # Remove the temporary attributes mutate { remove => ["tempDisplayName", “previousDate"] }
  39. 34 { "isoDate": "2015-03-08T17:38:46.936Z", "displayName": "Top of Stairway", "descriptionText": "Top

    of Stairway detected motion", "name": "motion", "valueAsString": "active", "value": 18, "timeParts": { "hour": "17", "minute": "38", "weekday": "Sunday", "week": "10", "day": "8", "month": "March", "year": "2015", "quarter": "1", "weekYear": "2015" }, "unit": "stairs" }
  40. CC-BY-ND 4.0 CC-BY-ND 4.0 Use Scripted Fields to Make Your

    Data Interesting 35
  41. CC-BY-ND 4.0 CC-BY-ND 4.0 Questions?

  42. CC-BY-ND 4.0 This work is licensed under the Creative Commons

    Attribution-NoDerivatives 4.0 International License. To view a copy of this license, visit: http://creativecommons.org/licenses/by-nd/4.0/ or send a letter to: Creative Commons PO Box 1866 Mountain View, CA 94042 USA CC-BY-ND 4.0