If it moves, measure it! - Logging IoT with ELK

Logs are everywhere! Join Chris as he explores using ELK to collect and analyze time-series data gathered from networked sensors. They've tracked temperature, humidity, motion, power, and other metrics around his home for a month. How did he do it? What did he learn?

This talk will show you all about some of the fun sensor devices he used, as well as showing how he used ELK to collect and analyze the data.

Transcript

1. CC-BY-ND 4.0 If it moves, measure it! or how I drove my wife absolutely crazy with sensors. Chris Cowan Kibana Engineer
2. CC-BY-ND 4.0 CC-BY-ND 4.0 Measure all the things! • Presence Sensors • Motion Detectors • Power Usage • Bandwidth Usage • Contact Switches • Noise Sensors • Temperature and Humidity • Illuminance • Vibrations
3. CC-BY-ND 4.0 CC-BY-ND 4.0 Architecture 3 SmartThings Hub Logstash Elasticsearch SSL Proxy
4. CC-BY-ND 4.0 CC-BY-ND 4.0 SmartThings Hub • Customizable with Groovy • Works with Z-Wave Devices • Works with Zigbee Devices • Provides Mobile Interface
 to System • Their objective it to be open and extensible 4
5. CC-BY-ND 4.0 CC-BY-ND 4.0 Aeon Labs Home Energy Meter 5 • Clamps  to  Power  Mains   • Exports  power  usage  in   real  time   • Uses  Z-­‐Wave   • Exports  Watts  and  kWh   • Difficult  to  Install   • You  have  to  risk  your   life  for  installation
6. CC-BY-ND 4.0 CC-BY-ND 4.0 Behind this panel lives certain death!
7. CC-BY-ND 4.0 CC-BY-ND 4.0 According to the guy at hardware store = SAFETY! +
8. CC-BY-ND 4.0 CC-BY-ND 4.0 What I thought was going to happen...
9. CC-BY-ND 4.0 CC-BY-ND 4.0 What actually happened...
10. CC-BY-ND 4.0 CC-BY-ND 4.0 What actually happened...
11. CC-BY-ND 4.0 CC-BY-ND 4.0 Watt Usage Chart 10 Breakfast Dinner
12. CC-BY-ND 4.0 CC-BY-ND 4.0 Energy Usage Chart 11
13. CC-BY-ND 4.0 CC-BY-ND 4.0 Our HVAC Fan Cycles Every Hour Fan  Cycling
14. CC-BY-ND 4.0 CC-BY-ND 4.0 Power By Hour for Last 30 Days 13
15. CC-BY-ND 4.0 CC-BY-ND 4.0 Aeon Labs Multisensor • Temperature • Humidity • Motion • Illuminance • Uses Z-Wave • Runs on 4 AAA Batteries • Indoor/Outdoors • Looks like the Eye of Sauron • Terrifies Children Most cost effective solution given everything these sensors do along with the SmartThings integration 14
16. CC-BY-ND 4.0 CC-BY-ND 4.0 Motion Sensor Events 15
17. CC-BY-ND 4.0 CC-BY-ND 4.0 Counting Stairs When the sensor at the top is triggered it looks back 25 seconds for a trigger event at the bottom of the stairs and counts that as 18 stairs.
18. CC-BY-ND 4.0 CC-BY-ND 4.0 Illuminance Events 17
19. CC-BY-ND 4.0 CC-BY-ND 4.0 Humidity Events 18 It  rained  last  night!
20. CC-BY-ND 4.0 CC-BY-ND 4.0 Temperature Events 19
21. CC-BY-ND 4.0 CC-BY-ND 4.0 Temperature Percentiles 20
22. CC-BY-ND 4.0 CC-BY-ND 4.0 Washing Machine Sensor Wait 50 minutes after a close event to start watching the for the accelerometer events to stop moving for 2 minutes. • SmartThings Multisensor • Detects Open and Close • Detects Temperature • Detects Movement 21
23. CC-BY-ND 4.0 CC-BY-ND 4.0 Washing Machine Events 22
24. CC-BY-ND 4.0 CC-BY-ND 4.0 Raspberry Pi Garage Door Opener 23 • Raspberry Pi A+ • Relay Switch • Magnetic Relay Switch • Cat 5 Cable • Integrated with
 SmartThings Hub • Door Automatically
 Opens and Closes Code available at: https://github.com/simianhacker/rpi-garage-door
25. CC-BY-ND 4.0 CC-BY-ND 4.0 Raspberry Pi Garage Door Events 24
26. CC-BY-ND 4.0 CC-BY-ND 4.0 DIY Sound Sensor • Arduino Uno • Analog Sound Sensor • 433mHz Transmitter • 433mHz Receiver on an Arduino Uno hooked up to a Raspberry Pi • RPI logs directly to Elasticsearch 25
27. CC-BY-ND 4.0 CC-BY-ND 4.0 Sound Sensor: Carpet Cleaning 26 Cleaning  Carpets  Upstairs
28. CC-BY-ND 4.0 CC-BY-ND 4.0 Dashboards! 27
29. CC-BY-ND 4.0 CC-BY-ND 4.0 Energy Dashboard 28
30. CC-BY-ND 4.0 CC-BY-ND 4.0 Temperature Dashboard 29
31. CC-BY-ND 4.0 CC-BY-ND 4.0 Hodgepodge Dashboard 30
32. CC-BY-ND 4.0 CC-BY-ND 4.0 Technical Stuffs
33. CC-BY-ND 4.0 Data Enrichment with Logstash • Add a time parts object to each record – “timeParts” with attributes for hour, minute, day, weekday, year, month,week year, quarter • Calculate time difference between “presence” event. – When an event with the “valueAsString” attribute set to “present” query Elasticsearch and find the previous “not present” event • Calculate difference between “energy” events – Store the value for the last “energy” event and calculate the “delta” 32
34. 33 if [displayName] =~ /^Top|^Bottom/ and [valueAsString] == "active" { }
35. 33 if [displayName] =~ /^Top|^Bottom/ and [valueAsString] == "active" { } # If the event is Top of Stairway then search for Bottom of Stairway if [displayName] =~ /^Top/ { mutate { add_field => { "tempDisplayName" => "Bottom of Stairway” } } } else { mutate { add_field => { "tempDisplayName" => "Top of Stairway” } } }
36. 33 if [displayName] =~ /^Top|^Bottom/ and [valueAsString] == "active" { } # If the event is Top of Stairway then search for Bottom of Stairway if [displayName] =~ /^Top/ { mutate { add_field => { "tempDisplayName" => "Bottom of Stairway” } } } else { mutate { add_field => { "tempDisplayName" => "Top of Stairway” } } } # Find the corresponding event in Elasticsearch elasticsearch { query => "valueAsString.raw:active \ AND displayName.raw:\"%{[tempDisplayName]}\" \ AND isoDate:[* TO %{[isoDate]}]" sort => "isoDate:desc" new_field => "previousDate" }
37. 33 if [displayName] =~ /^Top|^Bottom/ and [valueAsString] == "active" { } # If the event is Top of Stairway then search for Bottom of Stairway if [displayName] =~ /^Top/ { mutate { add_field => { "tempDisplayName" => "Bottom of Stairway” } } } else { mutate { add_field => { "tempDisplayName" => "Top of Stairway” } } } # Find the corresponding event in Elasticsearch elasticsearch { query => "valueAsString.raw:active \ AND displayName.raw:\"%{[tempDisplayName]}\" \ AND isoDate:[* TO %{[isoDate]}]" sort => "isoDate:desc" new_field => "previousDate" } # If the duration between events is less then 25 seconds then add 18 steps to the value ruby { code => "event['value'] = 18 if (event['@timestamp'] - event['previousDate']) > 25" }
38. 33 if [displayName] =~ /^Top|^Bottom/ and [valueAsString] == "active" { } # If the event is Top of Stairway then search for Bottom of Stairway if [displayName] =~ /^Top/ { mutate { add_field => { "tempDisplayName" => "Bottom of Stairway” } } } else { mutate { add_field => { "tempDisplayName" => "Top of Stairway” } } } # Find the corresponding event in Elasticsearch elasticsearch { query => "valueAsString.raw:active \ AND displayName.raw:\"%{[tempDisplayName]}\" \ AND isoDate:[* TO %{[isoDate]}]" sort => "isoDate:desc" new_field => "previousDate" } # If the duration between events is less then 25 seconds then add 18 steps to the value ruby { code => "event['value'] = 18 if (event['@timestamp'] - event['previousDate']) > 25" } # Remove the temporary attributes mutate { remove => ["tempDisplayName", “previousDate"] }
39. 34 { "isoDate": "2015-03-08T17:38:46.936Z", "displayName": "Top of Stairway", "descriptionText": "Top of Stairway detected motion", "name": "motion", "valueAsString": "active", "value": 18, "timeParts": { "hour": "17", "minute": "38", "weekday": "Sunday", "week": "10", "day": "8", "month": "March", "year": "2015", "quarter": "1", "weekYear": "2015" }, "unit": "stairs" }
40. CC-BY-ND 4.0 CC-BY-ND 4.0 Use Scripted Fields to Make Your Data Interesting 35
41. CC-BY-ND 4.0 CC-BY-ND 4.0 Questions?
42. CC-BY-ND 4.0 This work is licensed under the Creative Commons Attribution-NoDerivatives 4.0 International License. To view a copy of this license, visit: http://creativecommons.org/licenses/by-nd/4.0/ or send a letter to: Creative Commons PO Box 1866 Mountain View, CA 94042 USA CC-BY-ND 4.0