IoT data ingestion pipelines and Clojure transducers

Yodit Stanton, Founder @OpenSensors.io talk at @ds_ldn meetup

Transcript

1. @YoditStanton
2. Open data is data that is made available by organisations, businesses and individuals for anyone to access, use and share. Open data has to have a licence that says it is open data. Without a licence, the data can’t be reused. The licence might also say: • that people who use the data must credit whoever is publishing it (this is called attribution) • that people who mix the data with other data have to also release the results as open data (this is called share-alike)
3. Real Time data commons
4. Context of Scale - Traditional tech architecture will not scale to these volumes - People and devices need to find data easily “20 Billion IoT devices to be connected by 2020” — Gartner Expected no of Daily Messages - 333x the size of Twitter - 94x the size of WhatsApp IoT
6. ▪ HTTP like ▪ Over UDP ▪ Use GET, POST, PUT, DELETE ▪ coap://example.se:5683/~sensors./temp1.xml CoAP
8. MQTT ▪ PUB/SUB Protocol ▪ Tiny Packets on the Wire ▪ Suitable for low bandwidth devices
9. Topics /users/yods/home/livingroom/light1 /users/yods/work/fridge
10. Topic Matching Show me all data for my office ▪ /users/yods/myoffice/* Show me all data from the lights in my office ▪ /users/yods/myoffice/+/lights
11. MQTT Security - TLS - Device Authentication - Provenance (**Only OpenSensors)
12. Machine to Machine
13. Acme Car Park
14. Machine to Web
15. Transducers “composable algorithmic transformations”
16. Take one reducing fn and return another Do something with a seq of data to reuse functions on different data sources
17. (defn xform [xs] (->> xs (map #(+ 2 %)) (filter odd?))) (reduce + (xform (range 0 10)) (def xform2 (comp (map #(+ 2 %)) (filter odd?))) (transduce xform2 + (range 0 10))
18. Build pipes of channels
19. Why? Faster - XML parsing example (defn dothread [doc] (->> [doc] (mapcat :content) (filter #(= :chapter (:tag %))) (filter #(= "Introduction" (get-in % [:attrs :name]))) (mapcat :content) (filter #(= :para (:tag %))) (mapcat :content) (filter string?))) Threading Benchmark Evaluation count : 22646520 in 60 samples of 377442 calls. Execution time mean : 2.685036 µs Transducer Benchmark Evaluation count : 50155860 in 60 samples of 835931 calls. Execution time mean : 1.276018 µs
20. Why? Lightweight - Ability to create sophisticated analytics on the fly lumbering frameworks
21. Thank You @yoditstanton