All Things Open 2018

Transcript

1. IoT Data Collection Architecture, Approaches, and an Edge Device Build
2. @noahcrowley GitHub, Twitter
4. A time series is a series of data points indexed (or listed or graphed) in time order.
5. Telegraf Collect InfluxDB Store Kapacitor Process Chronograf Visualize
6. I’m a Developer Advocate & member of the Developer Relations team @noahcrowley
7. Sense Process Communicate & Actuate
8. Physical World -> Sensor (Transducer) -> Digital Signal -> Communications Protocols -> Processing
9. Processing -> Communications Protocols -> Digital Signal -> Actuator Physical World
10. +20 billion devices 11.5 zettabytes of data
11. Smartphones
12. Hardware evolution & reduced cost • Gains in process efficiencies: • More chips, cheaper chips, lower power chips • Large numbers of inexpensive, fairly accurate sensors • Increased availability of SoCs
13. IoT data must be: • Timely – ingestion rates and query efficiency is key • Accurate – data integrity and platform reliability is important • Actionable – data visualization, anomaly detection & alerting are essential • Deployable – in the datacenter and at the edge
14. IoT deployments are struggling to find an efficient, scalable data platform that meets all of these criteria.
15. Architecture Considerations
16. Where to collect your data • Distributed Collection vs. Centralized Collection • How is the data used? • Who uses the data?
17. Collect in the cloud • All data flows directly to the cloud • Analysis and visualization from anywhere • Take advantage of Infrastructure-as-a-Service • At the mercy of your network connection…
18. Collect at the device • Data is collected close to the sensor • Local processing and analysis • Unreliable back-haul network • No requirement for centralized storage or analysis
19. Distributed data collection • Data is fed to multiple collection points • Collection points feed back-end system of record • Distributes data collection load • More tolerant of network outages
20. Data layer architecture • Data is collected at the edge, close to the sensor • Edge collectors are also capable of processing the data • Edge collectors can handle local events & trigger actuation • Data forwarded to a centralized back-end • Can be down-sampled to reduce network costs
21. Data Layer Architecture
22. Edge Device Hardware
23. IoT Platform Hardware Needs • Extensive protocol support for data ingestion • BLE, Zigbee, Zwave, CoAP, MQTT, MODBUS • Storage for local data • Ideally runs a full-fledged OS
24. Advancements in processes and manufacturing that enable IoT deployments also enable data layer architectures.
26. An Edge Device Build https://www.influxdata.com/blog/building-an-influxdb-iot-edge-data-collection-device/
27. Edge Device Components • Pine-64 LTS Main Board ($32.00) • WiFi/BLE card ($9.99) • 7” TFT Touchscreen ($35.99) • Pine64 Playbox Enclosure ($9.99) • LiPo Battery ($21.99) • LoRa Board ($19.95)
28. Pine64 • Quad-core ARM A53 64-bit • 2GB LPDDR3 RAM • GPIO • UART • I2C • SPI • LTS version committed to supply until 2022
30. Sensors
31. WiFi Sensor Node • Particle Photon • Bosch BME280 • TI TSL2561
32. BLE Sensor Node • Nordic nRF52DK Developer Kit • SenseAir K30 CO2 sensor • 4700µF Capacitor • 9v Boost Converter
33. LoRa Sensor Node • Pocket Geiger Radiation Sensor • Wemos D1 Mini • LoRa Radio BoardA White LED
35. Software
36. IoT Platform Software Needs • Hardware-agnostic, scalable architecture • Comprehensive analytics and visualization tools • Robust Offline Data Capability
37. Robust Offline Data Capability • Normalization & storage of data • Downsample before forwarding to reduce bandwidth usage • Local Event Processing Engine
38. Telegraf Collect InfluxDB Store Kapacitor Process Chronograf Visualize
39. Open Source Core https://www.influxdata.com/blog/its-time-for-the-open-source-community-to-get-real/
40. Telegraf • 100% Open Source • Plugin-driven architecture • Over 150 plugins: Inputs, Outputs, Processors, Aggregators • Easy to extend • Majority of development is community driven
41. InfluxDB • Architected for Time Series Data • High rate of ingest • Efficient compression and storage on disk • Time-based functionality built-in • Downsampling
42. Installing the InfluxData Platform $ curl -sL https://repos.influxdata.com/influxdb.key | sudo apt-key add - $ source /etc/lsb-release $ echo "deb https://repos.influxdata.com/${DISTRIB_ID,,} ${DISTRIB_CODENAME} stable" | sudo tee /etc/apt/sources.list.d/influxdb.list sudo apt-get update $ sudo apt-get install influxdb chronograf telegraf kapacitor
43. Kapacitor • Data processing engine — stream or batch • Downsampling, etc. • Anomaly detection • Alerting
44. TICKscript • AlertNode • BarrierNode • BatchNode • ChangeDetectNode • CombineNode • DefaultNode • DeleteNode • DerivativeNode • EC2AutoscaleNode • EvalNode • FlattenNode • FromNode • GroupByNode • HTTPOutputNode • HTTPPostNode • InfluxDBOutputNode • InfluxQLNode • JoinNode • K8sAutoscaleNode • Kapacitor LoopbackNode • LogNode • NoOpNode • QueryNode • SampleNode • ShiftNode • SideloadNode • StateCountNode • StateDurationNode • StatsNode • StreamNode • SwarmAutoscaleNode • UDF (User Defined Function)Node • UnionNode • WhereNode • WindowNode
45. Kapacitor Alerting • Alert Types: • Threshold Alert • Deadman Alert • Alerts based on Calculated Value • Alert Mechanisms: • TCP/HTTP • Pager Duty • MQTT
46. Chronograf • Open Source Dashboarding & Visualization • Administrative Functions for InfluxDB • GUI for writing TICK Scripts
48. Join Our Community! • On the Web: https://community.influxdata.com • On GitHub: https://github.com/influxdata • On Slack: https://gophersinvite.herokuapp.com & join #influxdb • On Twitter: @InfluxDB and @noahcrowley