IoT Barcamp - Data Formats for the Internet of Things

http://www.iotbarcamp.de/ - Düsseldorf/Germany 04.10.2016

Transcript

1. Data Formats for the IoT @aschmidt75 Düsseldorf, 04.10.16 IoT Barcamp
2. Andreas Schmidt @aschmidt75 Software Development & Architecture Web Operations Linux Embedded - @thingforward www.thingforward.io
3. 3 04.10.16 - Small Devices with some Constraints & Limits i.e. on CPU, Memory, Networking ... Depending on type of Device ...
4. 4 04.10.16 - Constrained Devices Constrained Networks
5. 5 04.10.16 - 3 Categories of Constrained Devices
6. Class „Tiny“ 6 04.10.16 - § Typically 8-bit MCUs § Few KBs of Flash RAM and/or SRAM § Lower speed, small processing power § Networking very "basic" § Security too, if any § Much fun though
7. Class „Small“ 7 04.10.16 - § 32-bit MCUs, i.e. Cortex M § Some KBs of Flash RAM and SRAM § Speed 40..120MHz § Networking often included (i.E. WiFi, BLE) § Typically no full OS (i.e. Linux) at most embedded OS due to lack of MMUs (C) Particle, Redbear, Arduino
8. Class „Medium“ 8 04.10.16 - § 32-bit or 64-bit MCUs, i.e. Cortex A § Lots of Memory § Speed 400..>1GHz § Multiple networking capabilities, USB, ... § OS New&Shiny (C) Raspberry Pi, Pine64, Ordoid, Arrow
9. 9 04.10.16 - Limitations for the Use of Protocols & Data Formats depending on the Class of Device! topic of this talk!
10. Simple Example 10 04.10.16 § "DIY feel-good sensor" § Measures temperature, sends it via WiFi to a station w/ database § UDP-packets with simple temperature values § i.E. RedBear Duo + RaspberryPi
11. 11 04.10.16 redbear Pi "21" § 21% humidity? § 21 Hours until EoD? § Ah, temperature: 21 Degrees Celsius or Fahrenheit?
12. From raw data to information 12 04.10.16 § Structure § Data Formats! So we could need some
13. JSON YAML XML 13 04.10.16 I'm a web guy
14. 14 04.10.16 redbear Pi <sensor id="lvngroom"> <v type="degree" unit="celsius"> 21 </v> </sensor> § "Serialization" of mostly static data § Constrained Devices just writes out strings/bytes ✔
15. 15 04.10.16 redbear Pi § What if devices receives complex data? § i.e. Actions, Configuration, Setup ? ✗
16. Why Data Formats from the web are not well suited for IoT 16 04.10.16 Larger size of complex data formats (esp. XML) Sometimes low bandwidth on network channels (i.e. LoraWAN) or cost-intensive (i.e. GSM) Network Packet sizes (fragmentation issues) Size of request & its structure is not known before
17. Some more technical Why's 17 04.10.16 Parsers can be complex > Code size Complex parsing takes time & power from battery Dynamic memory Availability of C library
18. Some requirements 18 04.10.16 Space-efficient, both in transit & intermediate buffering Small code size of parser (at first: availability of parser for embedded..) Safe & robust handling of malformed input Standards Support schemas for a data format & its parser, on a constrained device
19. 1 0 0 1 0 0 1 0 1 0 0 1 0 0 1 0 1 0 1 0 0 1 0 0 1 0 0 1 0 0 0 1 0 0 0 1 0 0 0 1 0 1 1 0 0 1 1 1 0 0 1 0 0 1 0 0 1 0 0 0 1 0 1 0 0 1 0 1 0 1 1 1 0 0 1 0 1 0 1 0 1 1 0 0 1 0 0 1 0 1 0 0 1 0 1 0 1 0 1 0 0 1 0 0 1 0 0 1 0 0 0 1 0 0 0 1 0 0 0 1 0 1 1 0 0 1 1 1 1 0 0 1 0 0 1 0 0 1 0 0 0 0 1 0 0 1 0 1 0 1 1 1 0 0 1 0 1 0 1 0 1 1 0 0 1 0 0 1 0 1 0 0 1 0 0 1 0 1 0 1 0 0 1 0 0 1 0 0 0 0 0 1 0 0 0 1 0 0 0 1 0 1 1 0 0 1 1 1 1 0 0 1 0 0 1 0 0 1 0 0 0 1 0 1 0 0 1 0 1 0 1 1 1 0 0 0 1 0 1 0 1 1 0 0 1 0 0 1 0 1 0 0 1 0 0 1 0 1 0 1 0 0 1 0 0 1 0 0 1 0 0 0 1 0 0 0 1 0 0 0 1 0 1 0 0 1 1 1 1 0 0 1 0 0 1 0 0 1 0 0 0 1 0 1 0 0 1 0 1 0 1 1 1 0 0 1 0 1 0 1 0 1 1 1 1 0 1 0 0 1 0 0 1 0 0 1 0 1 0 0 1 0 0 1 0 1 0 1 0 0 1 0 0 1 0 0 1 0 0 0 1 0 0 0 1 0 0 0 1 0 1 1 0 0 1 1 1 0 0 1 0 0 1 0 0 1 0 0 0 1 0 1 0 0 1 0 1 0 1 1 1 0 0 1 0 1 0 1 0 1 1 0 0 1 0 0 1 0 1 0 0 1 0 1 0 1 0 1 0 0 1 0 0 1 0 0 1 0 0 0 1 0 0 0 1 0 0 0 1 0 1 1 0 0 1 1 1 1 0 0 1 0 0 1 0 0 1 0 0 0 0 1 0 0 1 0 1 0 1 1 1 0 0 1 0 1 0 1 0 1 1 0 0 1 0 0 1 0 1 0 0 1 0 0 1 0 1 0 1 0 0 1 0 0 1 0 0 0 0 0 1 0 0 0 1 0 0 0 1 0 1 1 0 0 1 1 1 1 0 0 1 0 0 1 0 0 1 0 0 0 1 0 1 0 0 1 0 1 0 1 1 1 0 0 0 1 0 1 0 1 1 0 0 1 0 0 1 0 1 0 0 1 0 0 1 0 1 0 1 0 0 1 0 0 1 0 0 1 0 0 0 1 0 0 0 1 0 0 0 1 0 1 0 0 1 1 1 1 0 0 1 0 0 1 0 0 1 0 0 0 1 0 1 0 0 1 0 1 0 1 1 1 0 0 1 0 1 0 1 0 1 1 1 1 0 1 0 0 1 0 0 1 0 0 1 0 1 0 0 1 0 0 1 0 1 0 1 0 0 1 0 0 1 0 0 1 0 0 0 1 0 0 0 1 0 0 0 1 0 1 1 0 0 1 1 1 0 0 1 0 0 1 0 0 1 0 0 0 1 0 1 0 0 1 0 1 0 1 1 1 0 0 1 0 1 0 1 0 1 1 0 0 1 0 0 1 0 1 0 0 1 0 1 0 1 0 1 0 0 1 0 0 1 0 0 1 0 0 0 1 0 0 0 1 0 0 0 1 0 1 1 0 0 1 1 1 1 0 0 1 0 0 1 0 0 1 0 0 0 0 1 0 0 1 0 1 0 1 1 1 0 0 1 0 1 0 1 0 1 1 0 0 1 0 0 1 0 1 0 0 1 0 0 1 0 1 0 1 0 0 1 0 0 1 0 0 0 0 0 1 0 0 0 1 0 0 0 1 0 1 1 0 0 1 1 1 1 0 0 1 0 0 1 0 0 1 0 0 0 1 0 1 0 0 1 0 1 0 1 1 1 0 0 0 1 0 1 0 1 1 0 0 1 0 0 1 0 1 0 0 1 0 0 1 0 1 0 1 0 0 1 0 0 1 0 0 1 0 0 0 1 0 0 0 1 0 0 0 1 0 1 0 0 1 1 1 1 0 0 1 0 0 1 0 0 1 0 0 0 1 0 1 0 0 1 0 1 0 1 1 1 0 0 1 0 1 0 1 0 1 1 1 1 0 1 0 0 Binary Formats 19 04.10.16 ... to the rescue
20. Why Binary Formats? 20 04.10.16 Space-efficient, both in transit & intermediate buffering Small code size of parser (at first: availability of parser for embedded..) Safe & robust handling of malformed input Standards Support schemas compression of up to 90% possible most do direct processing of binary data some do, some don't
21. JSON YAML XML 21 04.10.16 msgpack cbor thrift protobuf EXI
22. MessagePack 22 04.10.16 § msgpack.org https://github.com/msgpack § Apache license § Designed for efficient transmission over the wire § "supported by over 50 languages" § Compatible to JSON "MessagePack is an efficient binary serialization format" § Implementation suitable for embedded (ludocode/mpack) § Compression § No Schema § Parser with Expect-style API § Specified, not standardized
23. CBOR 23 04.10.16 § cbor.io https://github.com/cbor § Bases on MessagePack, Compatible to JSON § RFC 7049 § Implementations for >15 languages Concise Binary Object Representation "...design goals include the possibility for (...) small code sizes, (...) small message size and extensibility (...)" § Suitable for embedded (TinyCBOR) § CDDL as spec & validation language/tool § Compression rates comparable to msgpack
24. Protocol Buffers (protobuf) 24 04.10.16 § developers.google.com/protocol- buffers § Implementations for C, C#, C++, Go, Java, Py, Ruby, ... § Schema-based (".proto file") § Parser/Serializer Code is generated by the protoc "... a language-neutral, platform-neutral extensible mechanism for serializing structured data." § embedded implementations as 3rd party lib: proto-c § Size efficient binary representations
25. Thrift 25 04.10.16 § thrift.apache.org § Implementations for C, C#, C++, Java, Py, PHP, Ruby, Erlang, Javascript, ... § Schema-based (".thrift file") § Parser/Serializer Code is generated by the Thrift Compiler § .. is a full protocol stack including a service layer "... for scalable cross-language service development" § embedded implementations? § Size efficient binary representations
26. Efficient XML Interchange (EXI) 26 04.10.16 § Standardized: https://www.w3.org/TR/exi § Implementations for C++, Java, .NET, JS § Schema-aware (XSD) § Entropy-encoded binary representation of XML data as stream of events Binary representation of XML Information Sets § No OSS embedded implementations (as of May 2016) § Highest compression rates, typically >90% (!)
27. BSON 27 04.10.16 § optimized not for space-efficiency, but for lookup and traversal speed § BSON structures are typically larger than RAW data § Sometimes binary data is larger than textual JSON (!) Binary JSON
28. Size comparison 28 04.10.16 example (structured)
29. Size comparison 29 04.10.16 RAW only raw data items, separated by a single byte. Think "CSV” This is “the 100%” when comparing sizes
30. Size comparison: Web Formats 30 04.10.16 Relative sizes [%], compared to RAW format
31. Size comparison: Binary 31 04.10.16 binary representations Relative sizes [%], compared to RAW format
32. § © 2016 Digital Incubation and Growth GmbH Andreas Schmidt @aschmidt75, @thingforward 32 05.10.16 -