Upgrade to Pro — share decks privately, control downloads, hide ads and more …

IoT Barcamp - Data Formats for the Internet of Things

IoT Barcamp - Data Formats for the Internet of Things

http://www.iotbarcamp.de/ - Düsseldorf/Germany 04.10.2016

F28d4f1634bce85c90b76b197b1413d4?s=128

Andreas Schmidt

October 04, 2016
Tweet

Transcript

  1. Data Formats for the IoT @aschmidt75 Düsseldorf, 04.10.16 IoT Barcamp

  2. Andreas Schmidt @aschmidt75 Software Development & Architecture Web Operations Linux

    Embedded - @thingforward www.thingforward.io
  3. 3 04.10.16 - Small Devices with some Constraints & Limits

    i.e. on CPU, Memory, Networking ... Depending on type of Device ...
  4. 4 04.10.16 - Constrained Devices Constrained Networks

  5. 5 04.10.16 - 3 Categories of Constrained Devices

  6. Class „Tiny“ 6 04.10.16 - § Typically 8-bit MCUs §

    Few KBs of Flash RAM and/or SRAM § Lower speed, small processing power § Networking very "basic" § Security too, if any § Much fun though
  7. Class „Small“ 7 04.10.16 - § 32-bit MCUs, i.e. Cortex

    M § Some KBs of Flash RAM and SRAM § Speed 40..120MHz § Networking often included (i.E. WiFi, BLE) § Typically no full OS (i.e. Linux) at most embedded OS due to lack of MMUs (C) Particle, Redbear, Arduino
  8. Class „Medium“ 8 04.10.16 - § 32-bit or 64-bit MCUs,

    i.e. Cortex A § Lots of Memory § Speed 400..>1GHz § Multiple networking capabilities, USB, ... § OS New&Shiny (C) Raspberry Pi, Pine64, Ordoid, Arrow
  9. 9 04.10.16 - Limitations for the Use of Protocols &

    Data Formats depending on the Class of Device! topic of this talk!
  10. Simple Example 10 04.10.16 § "DIY feel-good sensor" § Measures

    temperature, sends it via WiFi to a station w/ database § UDP-packets with simple temperature values § i.E. RedBear Duo + RaspberryPi
  11. 11 04.10.16 redbear Pi "21" § 21% humidity? § 21

    Hours until EoD? § Ah, temperature: 21 Degrees Celsius or Fahrenheit?
  12. From raw data to information 12 04.10.16 § Structure §

    Data Formats! So we could need some
  13. JSON YAML XML 13 04.10.16 I'm a web guy

  14. 14 04.10.16 redbear Pi <sensor id="lvngroom"> <v type="degree" unit="celsius"> 21

    </v> </sensor> § "Serialization" of mostly static data § Constrained Devices just writes out strings/bytes ✔
  15. 15 04.10.16 redbear Pi § What if devices receives complex

    data? § i.e. Actions, Configuration, Setup ? ✗
  16. Why Data Formats from the web are not well suited

    for IoT 16 04.10.16 Larger size of complex data formats (esp. XML) Sometimes low bandwidth on network channels (i.e. LoraWAN) or cost-intensive (i.e. GSM) Network Packet sizes (fragmentation issues) Size of request & its structure is not known before
  17. Some more technical Why's 17 04.10.16 Parsers can be complex

    > Code size Complex parsing takes time & power from battery Dynamic memory Availability of C library
  18. Some requirements 18 04.10.16 Space-efficient, both in transit & intermediate

    buffering Small code size of parser (at first: availability of parser for embedded..) Safe & robust handling of malformed input Standards Support schemas for a data format & its parser, on a constrained device
  19. 1 0 0 1 0 0 1 0 1 0

    0 1 0 0 1 0 1 0 1 0 0 1 0 0 1 0 0 1 0 0 0 1 0 0 0 1 0 0 0 1 0 1 1 0 0 1 1 1 0 0 1 0 0 1 0 0 1 0 0 0 1 0 1 0 0 1 0 1 0 1 1 1 0 0 1 0 1 0 1 0 1 1 0 0 1 0 0 1 0 1 0 0 1 0 1 0 1 0 1 0 0 1 0 0 1 0 0 1 0 0 0 1 0 0 0 1 0 0 0 1 0 1 1 0 0 1 1 1 1 0 0 1 0 0 1 0 0 1 0 0 0 0 1 0 0 1 0 1 0 1 1 1 0 0 1 0 1 0 1 0 1 1 0 0 1 0 0 1 0 1 0 0 1 0 0 1 0 1 0 1 0 0 1 0 0 1 0 0 0 0 0 1 0 0 0 1 0 0 0 1 0 1 1 0 0 1 1 1 1 0 0 1 0 0 1 0 0 1 0 0 0 1 0 1 0 0 1 0 1 0 1 1 1 0 0 0 1 0 1 0 1 1 0 0 1 0 0 1 0 1 0 0 1 0 0 1 0 1 0 1 0 0 1 0 0 1 0 0 1 0 0 0 1 0 0 0 1 0 0 0 1 0 1 0 0 1 1 1 1 0 0 1 0 0 1 0 0 1 0 0 0 1 0 1 0 0 1 0 1 0 1 1 1 0 0 1 0 1 0 1 0 1 1 1 1 0 1 0 0 1 0 0 1 0 0 1 0 1 0 0 1 0 0 1 0 1 0 1 0 0 1 0 0 1 0 0 1 0 0 0 1 0 0 0 1 0 0 0 1 0 1 1 0 0 1 1 1 0 0 1 0 0 1 0 0 1 0 0 0 1 0 1 0 0 1 0 1 0 1 1 1 0 0 1 0 1 0 1 0 1 1 0 0 1 0 0 1 0 1 0 0 1 0 1 0 1 0 1 0 0 1 0 0 1 0 0 1 0 0 0 1 0 0 0 1 0 0 0 1 0 1 1 0 0 1 1 1 1 0 0 1 0 0 1 0 0 1 0 0 0 0 1 0 0 1 0 1 0 1 1 1 0 0 1 0 1 0 1 0 1 1 0 0 1 0 0 1 0 1 0 0 1 0 0 1 0 1 0 1 0 0 1 0 0 1 0 0 0 0 0 1 0 0 0 1 0 0 0 1 0 1 1 0 0 1 1 1 1 0 0 1 0 0 1 0 0 1 0 0 0 1 0 1 0 0 1 0 1 0 1 1 1 0 0 0 1 0 1 0 1 1 0 0 1 0 0 1 0 1 0 0 1 0 0 1 0 1 0 1 0 0 1 0 0 1 0 0 1 0 0 0 1 0 0 0 1 0 0 0 1 0 1 0 0 1 1 1 1 0 0 1 0 0 1 0 0 1 0 0 0 1 0 1 0 0 1 0 1 0 1 1 1 0 0 1 0 1 0 1 0 1 1 1 1 0 1 0 0 1 0 0 1 0 0 1 0 1 0 0 1 0 0 1 0 1 0 1 0 0 1 0 0 1 0 0 1 0 0 0 1 0 0 0 1 0 0 0 1 0 1 1 0 0 1 1 1 0 0 1 0 0 1 0 0 1 0 0 0 1 0 1 0 0 1 0 1 0 1 1 1 0 0 1 0 1 0 1 0 1 1 0 0 1 0 0 1 0 1 0 0 1 0 1 0 1 0 1 0 0 1 0 0 1 0 0 1 0 0 0 1 0 0 0 1 0 0 0 1 0 1 1 0 0 1 1 1 1 0 0 1 0 0 1 0 0 1 0 0 0 0 1 0 0 1 0 1 0 1 1 1 0 0 1 0 1 0 1 0 1 1 0 0 1 0 0 1 0 1 0 0 1 0 0 1 0 1 0 1 0 0 1 0 0 1 0 0 0 0 0 1 0 0 0 1 0 0 0 1 0 1 1 0 0 1 1 1 1 0 0 1 0 0 1 0 0 1 0 0 0 1 0 1 0 0 1 0 1 0 1 1 1 0 0 0 1 0 1 0 1 1 0 0 1 0 0 1 0 1 0 0 1 0 0 1 0 1 0 1 0 0 1 0 0 1 0 0 1 0 0 0 1 0 0 0 1 0 0 0 1 0 1 0 0 1 1 1 1 0 0 1 0 0 1 0 0 1 0 0 0 1 0 1 0 0 1 0 1 0 1 1 1 0 0 1 0 1 0 1 0 1 1 1 1 0 1 0 0 Binary Formats 19 04.10.16 ... to the rescue
  20. Why Binary Formats? 20 04.10.16 Space-efficient, both in transit &

    intermediate buffering Small code size of parser (at first: availability of parser for embedded..) Safe & robust handling of malformed input Standards Support schemas compression of up to 90% possible most do direct processing of binary data some do, some don't
  21. JSON YAML XML 21 04.10.16 msgpack cbor thrift protobuf EXI

  22. MessagePack 22 04.10.16 § msgpack.org https://github.com/msgpack § Apache license §

    Designed for efficient transmission over the wire § "supported by over 50 languages" § Compatible to JSON "MessagePack is an efficient binary serialization format" § Implementation suitable for embedded (ludocode/mpack) § Compression § No Schema § Parser with Expect-style API § Specified, not standardized
  23. CBOR 23 04.10.16 § cbor.io https://github.com/cbor § Bases on MessagePack,

    Compatible to JSON § RFC 7049 § Implementations for >15 languages Concise Binary Object Representation "...design goals include the possibility for (...) small code sizes, (...) small message size and extensibility (...)" § Suitable for embedded (TinyCBOR) § CDDL as spec & validation language/tool § Compression rates comparable to msgpack
  24. Protocol Buffers (protobuf) 24 04.10.16 § developers.google.com/protocol- buffers § Implementations

    for C, C#, C++, Go, Java, Py, Ruby, ... § Schema-based (".proto file") § Parser/Serializer Code is generated by the protoc "... a language-neutral, platform-neutral extensible mechanism for serializing structured data." § embedded implementations as 3rd party lib: proto-c § Size efficient binary representations
  25. Thrift 25 04.10.16 § thrift.apache.org § Implementations for C, C#,

    C++, Java, Py, PHP, Ruby, Erlang, Javascript, ... § Schema-based (".thrift file") § Parser/Serializer Code is generated by the Thrift Compiler § .. is a full protocol stack including a service layer "... for scalable cross-language service development" § embedded implementations? § Size efficient binary representations
  26. Efficient XML Interchange (EXI) 26 04.10.16 § Standardized: https://www.w3.org/TR/exi §

    Implementations for C++, Java, .NET, JS § Schema-aware (XSD) § Entropy-encoded binary representation of XML data as stream of events Binary representation of XML Information Sets § No OSS embedded implementations (as of May 2016) § Highest compression rates, typically >90% (!)
  27. BSON 27 04.10.16 § optimized not for space-efficiency, but for

    lookup and traversal speed § BSON structures are typically larger than RAW data § Sometimes binary data is larger than textual JSON (!) Binary JSON
  28. Size comparison 28 04.10.16 example (structured)

  29. Size comparison 29 04.10.16 RAW only raw data items, separated

    by a single byte. Think "CSV” This is “the 100%” when comparing sizes
  30. Size comparison: Web Formats 30 04.10.16 Relative sizes [%], compared

    to RAW format
  31. Size comparison: Binary 31 04.10.16 binary representations Relative sizes [%],

    compared to RAW format
  32. § © 2016 Digital Incubation and Growth GmbH Andreas Schmidt

    @aschmidt75, @thingforward 32 05.10.16 -