Big data analytics and our IoT Gateway

The slides of the talk we gave for CodeMotionAmsterdam techPills meetup about our consumer IoT Product (Toon) and the Data collection platform.

Transcript

1. 30 March, 2017 Aemro Amare, Freek van Gool 1 Bigdata Analytics and our IoT Gateway :Toon
2. Agenda • A little bit about us • Who is Quby & what is Toon? • Our big data journey • Data collection & services • Architecture & technologies • Demo • Questions @javafreekNL @twoxey
3. A little bit about us Freek Aemro @javafreekNL @twoxey
4. First attempt Home control box @javafreekNL @twoxey
5. Pivot Energy stick
6. What is Toon?
7. Meter Adapter Gas Sensor Electricity Sensor Z-Wave Phillips Hue WiFi Router Z-wave Smart plugand smoke detectors Central heating boiler Boiler adaptor Toon® Toon connections - Data sources
8. The service center Service Center Toon Displays Mobile devices Product Applications Back Office Applications Mobile Backend Internet Internet VPN Access Data collector
9. First partnership
10. Exponential growth 2013 30.000 2014 90.000 2015 190.000 2016 300.000 2017 (1.000.000) 2013 2014 2015 2016 2017
11. Our journey towards bigdata 30 March, 2017 12 2014 • 30K customers • 40 types of sensor data from 3K customers • up to 1 minutes resolution 2012 • 4K customers • Aggregated Energy data collected(DAY,WE EK,MONTH) • Benchmarking functionality for customers 2013 • 90K customers • Data collection from 30K customers • Event and Measurement data
12. Steps moving towards bigdata(cont’d) 30 March, 2017 13 2017 • 300K customers • Near Real Time aggregation with segmentation • 400 signal/5 minutes • Sharing Measurement data with tenants 2015 • 190K customers • Apache storm for Near Real Time Energy consumption (3K customers) • Data analysis using Cloudera stack • Up to 10 seconds resolution 2016 • 330K customers • Collect data from 230K customers anonymously • Data as a service • Analytics platform as a service
13. Toon® Types of data More than 300 types of sensor and user interaction data. If you want to know more please contact us : dteam@quby.com 14 Energy CH-boiler (OT) Thermostat Gas consumption Electricity consumption Water temperature Service messages Burner information Room temperature Thermostat program Manual settings Plug-data Electricity consumption On/off Water pressure Energy production Energy feed-in Boiler status Touch events Thermostat Mobile 1 2 3 4 5 Thermostat setting Hue data
14. Anonymous Data Collection 30 March, 2017 15 Service Center Customer Always anonymous data collected Customer may opt in for services and send keys Customer may opt out and change keys Data can’t be de-anonymized again
15. Sample Services with Toon’s Data? 30 March, 2017 16 1. Benchmarking with daily energy meter data
16. Sample Services with Toon’s Data? ( cont’d) 30 March, 2017 17 2. Nearly Real Time Forecasting : Energy data Aggregation
17. Sample Services with Toon’s Data? ( cont’d) 30 March, 2017 18 3. Home Appliances Energy Consumption Detection: Energy Data disaggregation
18. Sample Services with Toon’s Data ?(cont’d) 30 March, 2017 19 4. Boiler Maintenance Prediction (KetelIQ)
19. Sample Services with Toon’s Data? ( cont’d) 30 March, 2017 20 5. Customer’s Behavioral studies Click data Thermostat programs and co2 emission
20. What are we collecting?  24K measurements / day / customer  600KB (zipped)  230K customers 21
21. Arch. Principle : lambda Arch. 22
22. Messaging Data warehouse Tools and Technologies 30 March, 2017 23 Analytic Clusters Tools and Frameworks Containers & micro services Monitoring and alert
23. Data flow (partial view) 24
24. Things to mention • ssl mutual authentication • raw-data stored partitioned in better way • Customers can control access to their own data 30 March, 2017 27
25. What are we doing with all that data? • Firehose – data as a service for B2B (and researchers) • Analytic platform as a service • New data driven services for Toon users • Improve existing services • Marketing strategy (understand your customers) 30 March, 2017 29
26. 30 March, 2017 30
27. 30 March, 2017 31