Comparison of Open Source IoT Integration Frameworks (powered by Java, node.js, Golang)

IoT and Edge Integration with Open Source Frameworks:

Internet of Things (IoT) and edge integration is getting more important than ever before due to the massively growing number of connected devices year by year.

This session shows open source frameworks built to develop very lightweight microservices, which can be deployed on small devices or in serverless architectures with very low resources and wire together all different kinds of hardware devices, APIs and online services.

The focus of this session lies on showing open source projects such as Eclipse Kura, Node-RED or Flogo, which offer a framework plus zero-code environment with web IDE for building and deploying integration and data processing directly onto connected devices using IoT standards such as MQTT, WebSockets or CoaP, but also other interfaces such as Twitter feeds or REST services.

The end of the session discusses the relation to other components in a IoT architecture including cloud IoT platforms and big data respectively streaming analytics solutions (such as Apache Storm, Flink, Spark Streaming, Samza, StreamBase, Apama).

Keywords:
Internet of Things, IoT, Microservices, Edge Computing, Fog Computing, Open Source, Go, Golang, Flogo, Project Flogo, Node-RED, node red, IBM, Bluemix, TIBCO, TIBCO Software, Eclipse, Kura, Apache Camel, Serverless, Cloud, AWS, Azure, Google, GCP, Core, Edge, API Management, Integration, Cloud native, Java, JavaScript, OSGi, node.js, node js, Consumer IoT, Industrial IoT, IIoT, IoT Gateway, Gateway, Github, Apache, BSD, License, Microcontroller, Design Bot, Chat Bot, Nifi, Minifi, StreamSets, Cask, Hydrator, Dataflow, Data Pipeline, Process Engine, Stream Processing, Apache, Storm, Flink, Samza, Spark, Spark Streaming, Streaming Analytics, StreamBase, TIBCO, IBM, Software AG, Apama

Transcript

1. Kai Wähner Technology Evangelist kontakt@kai-waehner.de LinkedIn @KaiWaehner www.kai-waehner.de November 2016 Comparison of Open Source Frameworks for Integrating the Internet of Things
2. © Copyright 2000-2016 TIBCO Software Inc. • used to develop lightweight IoT edge apps and microservices • part of a hybrid integration architecture • complementary to IoT cloud platforms and streaming analytics Key Takeaways IoT integration frameworks are...
3. © Copyright 2000-2016 TIBCO Software Inc. 1) IoT Trends and Challenges 2) Hybrid Integration Architecture 3) IoT Integration o Dataflow Pipeline o Stream Processing o Process Engine 4) Open Source IoT Process Engines o Eclipse Kura (+ Apache Camel) o Node-RED o Flogo 5) Cloud IoT Platforms Agenda
4. © Copyright 2000-2016 TIBCO Software Inc. 1) IoT Trends and Challenges 2) Hybrid Integration Architecture 3) IoT Integration o Dataflow Pipeline o Stream Processing o Process Engine 4) Open Source IoT Process Engines o Eclipse Kura (+ Apache Camel) o Node-RED o Flogo 5) Cloud IoT Platforms Agenda
5. © Copyright 2000-2016 TIBCO Software Inc. Internet of Things (IoT) … refers to the ever-growing network of physical objects that feature an IP address for internet connectivity, and the communication that occurs between these objects and other Internet-enabled devices and systems.
6. © Copyright 2000-2016 TIBCO Software Inc. Forecast for Year 2020 à 20+ Billion Devices
7. © Copyright 2000-2016 TIBCO Software Inc. IoT Landscape 2016 The IoT Landscape 2016 Edition - Matt Turck & David Rogg, First Mark Capital
8. © Copyright 2000-2016 TIBCO Software Inc. Integration is Key for Success of IoT Projects Without Integration there is no Internet of Things! Gartner Market Guide for IoT Integration “Through 2018, half the cost of implementing IoT solutions will be spent on integration. “ Strategic Planning Assumption, Benoit J. Lheureux, Massimo Pezzini, Alfonso Velosa !
9. © Copyright 2000-2016 TIBCO Software Inc. Processes Analytics APIs Streams & Events Interconnect Everything Devices Equipment CORE Run the Business EDGE Change the Business People
10. © Copyright 2000-2016 TIBCO Software Inc. IoT raises several new challenges • Devices are not connected to the cloud • Devices have low bandwidth to connect • Latency of connectivity is significant • Connectivity is not reliable • Connectivity is not cost-effective Great IoT standards available • MQTT, CoaP, OPC US, many more • Different abstraction levels • Not the one single standard Challenges
11. © Copyright 2000-2016 TIBCO Software Inc. 1) IoT Trends and Challenges 2) Hybrid Integration Architecture 3) IoT Integration o Dataflow Pipeline o Stream Processing o Process Engine 4) Open Source IoT Process Engines o Eclipse Kura (+ Apache Camel) o Node-RED o Flogo 5) Cloud IoT Platforms Agenda
12. © Copyright 2000-2016 TIBCO Software Inc. Integration Frameworks Decision Data Conversion Transitions Process Call Process Design Tools Integration Processes Connectors to vendor applications Connectors to technologies Connectors (e.g. JMS, MQTT, FTP) (e.g. SF.com, SAP)
13. © Copyright 2000-2016 TIBCO Software Inc. Enterprise Integration Patterns “The goal of EIPs is to document technology- independent design guidance that helps developers and architects describe and develop robust integration solutions.” http://www.eaipatterns.com
14. © Copyright 2000-2016 TIBCO Software Inc. Pervasive Integration with different Technologies and User Roles https://www.linkedin.com/pulse/how-implement-hybrid-integration-platform-tackle-massimo-pezzini There is no “one-size-fits-all” IoT integration!
15. © Copyright 2000-2016 TIBCO Software Inc. Hybrid Integration Platform (HIP) Application Integration Application Integration (on a PaaS) iPaaS Integration Platform as a Service iSaaS Integration Software as a Service Cloud-Ready Cloud-Native API Management On Premise Public Cloud Public Cloud Public Cloud On Premise Public Cloud Process Integration Business Process Management Platform Streaming Analytics Fog Computing Edge Devices Edge Integration Analysts are sure: HIP is “the new default” in most enterprises!
16. © Copyright 2000-2016 TIBCO Software Inc. 1) IoT Trends and Challenges 2) Hybrid Integration Architecture 3) IoT Integration o Dataflow Pipeline o Stream Processing o Process Engine 4) Open Source IoT Process Engines o Eclipse Kura (+ Apache Camel) o Node-RED o Flogo 5) Cloud IoT Platforms Agenda
17. © Copyright 2000-2016 TIBCO Software Inc. Fog Computing http://www.slideshare.net/Angelo.Corsaro/20141210-fog … keep data closer ‘to the edge’ !
18. Typical IoT Integration Scenario What’s wrong with Cloud-only IoT Integration? • Connectivity is unreliable • Non-trivial latency in hops • Networks costs add up quickly
19. A better IoT Integration Blueprint Benefits: • Local control = more reliable • Less traffic = Lower TCO • Edge integration & event processing
20. © Copyright 2000-2016 TIBCO Software Inc. 1) IoT Trends and Challenges 2) Hybrid Integration Architecture 3) IoT Integration o Dataflow Pipeline o Stream Processing o Process Engine 4) Open Source IoT Process Engines o Eclipse Kura (+ Apache Camel) o Node-RED o Flogo 5) Cloud IoT Platforms Agenda
21. © Copyright 2000-2016 TIBCO Software Inc. Dataflow Pipeline – Extract, Transform, Load https://www.linkedin.com/pulse/data-pipeline-hadoop-part-1-2-birender-saini
22. © Copyright 2000-2016 TIBCO Software Inc. • Flow-based Programming • Ingest data from various sources • Extract (E) – Transform (T) – Load (L) • Transform / Filter / Route / Aggregate / Enrich … • High-throughput straight-through data flows • Data lineage • Synchronous and asynchronous communication • Batch or stream processing • Visual coding with flow editor • Not to be mixed-up with stream processing for event correlation Dataflow Pipeline - Characteristics
23. © Copyright 2000-2016 TIBCO Software Inc. Dataflow Pipeline – Examples All except Informatica use open source / open core business model
24. © Copyright 2000-2016 TIBCO Software Inc. 1) IoT Trends and Challenges 2) Hybrid Integration Architecture 3) IoT Integration o Dataflow Pipeline o Stream Processing o Process Engine 4) Open Source IoT Process Engines o Eclipse Kura (+ Apache Camel) o Node-RED o Flogo 5) Cloud IoT Platforms Agenda
25. © Copyright 2000-2016 TIBCO Software Inc. Stream Processing time 1 2 3 4 5 6 7 8 9 Event Streams • Continuous Queries • Sliding Windows • Filter • Aggregation • Correlation • … Stream Processing H20.ai Open Source R TERR Spark ML MATLAB SAS PMML
26. © Copyright 2000-2016 TIBCO Software Inc. Stream Processing - Examples OPEN SOURCE CLOSED SOURCE PRODUCT FRAMEWORK Azure Microsoft Stream Analytics
27. © Copyright 2000-2016 TIBCO Software Inc. Streaming Analytics à Predictive Maintenance Voltage Temperature Vibration Device history Temporal analytic: “If vibration spike is followed by temp spike then voltage spike [within 4 hours] then flag high severity alert.”
28. © Copyright 2000-2016 TIBCO Software Inc. 1) IoT Trends and Challenges 2) Hybrid Integration Architecture 3) IoT Integration o Dataflow Pipeline o Stream Processing o Process Engine 4) Open Source IoT Process Engines o Eclipse Kura (+ Apache Camel) o Node-RED o Flogo 5) Cloud IoT Platforms Agenda
29. © Copyright 2000-2016 TIBCO Software Inc. • Integrate and orchestrate various data sources • Wire together hardware devices, APIs and online services • Transform / Filter / Route / Aggregate / Enrich … • Error handling, Re-Try, Re-Routing, Wait, Resume, ... • Not just Extract-Transform-Load • Synchronous and asynchronous communication • Typically real time processing • Visual coding with flow editor • Deployed at the edge Process Engine
30. © Copyright 2000-2016 TIBCO Software Inc. Real World IoT Use Cases for a Process Engine
31. © Copyright 2000-2016 TIBCO Software Inc. 1) IoT Trends and Challenges 2) Hybrid Integration Architecture 3) IoT Integration o Dataflow Pipeline o Stream Processing o Process Engine 4) Open Source IoT Process Engines o Eclipse Kura (+ Apache Camel) o Node-RED o Flogo 5) Cloud IoT Platforms Agenda
32. © Copyright 2000-2016 TIBCO Software Inc. Commonalities: • Open source frameworks • Connectivity to IoT technologies (MQTT, CoaP, REST, …) • Web UI for visual coding, testing, debugging • Deployable “at the edge” • For developers / integration specialists / citizen integrators • Extendable SDKs / APIs Open Source IoT Process Engines
33. © Copyright 2000-2016 TIBCO Software Inc. 1) IoT Trends and Challenges 2) Hybrid Integration Architecture 3) IoT Integration o Dataflow Pipeline o Stream Processing o Process Engine 4) Open Source IoT Process Engines o Eclipse Kura (+ Apache Camel) o Node-RED o Flogo 5) Cloud IoT Platforms Agenda
34. © Copyright 2000-2016 TIBCO Software Inc. Eclipse Kura http://www.eclipse.org/kura
35. © Copyright 2000-2016 TIBCO Software Inc. • Focus on IoT Gateway • Set of Java and OSGi services • including I/O services, Data Services, Cloud Services, Networking, etc. • Web UI for Configuration (Devices, Network, Protocols, etc.) • Eclipse Public License 1.0 • Mature framework (~3 years old, code based committed in December 2013) • Best of breed with focus on coding instead of a visual designer • Eclipse IoT Projects • IoT Gateway --> Eclipse Kura • Services like Eclipse SmartHome • Standard implementations like Mosquitto (MQTT Server) • Connectivity via Eclipse Paho (MQTT Client) • Apache Camel Connector • Integration Framework (connectivity, enterprise integration patterns) • Visual coding via JBoss or Talend (both available as open source edition) • For Developers / Integration Specialists only • More complex to install and build a first IoT integration flow • Write source code • Can be run on a variety of platforms (on premise, cloud, edge devices, container) Eclipse Kura (+ Apache Camel) à Facts
36. © Copyright 2000-2016 TIBCO Software Inc. Eclipse Kura – Functionality Overview and Architecture https://www.youtube.com/watch?v=ia8cLnR1uFI http://www.slideshare.net/Eurotechchannel/kuram2miotgateway
37. © Copyright 2000-2016 TIBCO Software Inc. Eclipse Kura - Architecture
38. © Copyright 2000-2016 TIBCO Software Inc. Eclipse Kura – Hello World https://eclipse.github.io/kura/doc/hello-example.html
39. © Copyright 2000-2016 TIBCO Software Inc. Apache Camel http://www.kai-waehner.de/blog/2012/05/04/apache-camel-tutorial-introduction/
40. © Copyright 2000-2016 TIBCO Software Inc. Apache Camel (in conjunction with Eclipse Kura) https://dzone.com/articles/creating-camel-routes-for-eclipse-kura
41. © Copyright 2000-2016 TIBCO Software Inc. Live Demo Kura (+ Apache Camel) in Action
42. © Copyright 2000-2016 TIBCO Software Inc. 1) IoT Trends and Challenges 2) Hybrid Integration Architecture 3) IoT Integration o Dataflow Pipeline o Stream Processing o Process Engine 4) Open Source IoT Process Engines o Eclipse Kura (+ Apache Camel) o Node-RED o Flogo 5) Cloud IoT Platforms Agenda
43. © Copyright 2000-2016 TIBCO Software Inc. Node-RED https://nodered.org/
44. © Copyright 2000-2016 TIBCO Software Inc. Node-RED http://industrialinternet.co.uk/category/node-red/
45. © Copyright 2000-2016 TIBCO Software Inc. • Focus on Integration on an IoT Gateway • Built on Node.js and Javascript • Very easy to install and build a first IoT integration flow • For developers / integration specialists and ciziten integrators • Visual coding via web designer • Apache 2.0 License • Mature framework (~3 years old) with many examples and docs • Native integration into IBM Bluemix cloud platform • Leverage other Bluemix services • Can be run on a variety of platforms (on premise, cloud, edge devices, container) • Flows can be shared as JSON strings / files • No binaries to share and deploy on devices Node-RED à Facts
46. © Copyright 2000-2016 TIBCO Software Inc. Live Demo Node-RED in Action
47. © Copyright 2000-2016 TIBCO Software Inc. 1) IoT Trends and Challenges 2) Hybrid Integration Architecture 3) IoT Integration o Dataflow Pipeline o Stream Processing o Process Engine 4) Open Source IoT Process Engines o Eclipse Kura (+ Apache Camel) o Node-RED o Flogo 5) Cloud IoT Platforms Agenda
48. © Copyright 2000-2016 TIBCO Software Inc. Flogo http://www.flogo.io
49. Project Flogo IoT Example
50. © Copyright 2000-2016 TIBCO Software Inc. • Focus on Integration on an IoT Gateway and very lightweight edge applications • Powered by Golang • Very easy to install and build a first IoT integration flow • Ultra-lightweight footprint with zero dependency model • For developers / integration specialists and ciziten integrators • Visual coding via web designer • BSD-style License • Developer Preview (released in October 2016) • Can be run on a variety of platforms (on premise, cloud, edge devices, container) • Flows can be shared as JSON strings / files • Lightweight binaries with zero dependencies can be shared and deployed on devices Flogo à Facts
51. Web-Native Step-Back Debugger • Step back in a flow • Interactively design/debug your process • Simulate sensor events • Change data or configuration and run from Trigger or any Activity • Underlying technology could be used to support remote ops debugging of failed flows
52. © Copyright 2000-2016 TIBCO Software Inc. Live Demo Flogo in Action
53. What technology to build edge apps? Java • Heavy (>200MB) • Not agile • Robust • Licensing risk Node.js • Lighter (~70MB) • Agile • Not IoT robust • Clear licensing C/C++ • Lightest • Not Agile • Robust • Clear licensing
54. © Copyright 2000-2016 TIBCO Software Inc. Why Go? Modern programming language Getting a lot of traction due to usage in frameworks like Docker or Kubernetes Concurrency Built into the language (Goroutines, channels, no explicit thread programming) Memory management Modern Garbage collector Statically typed Type-safe development with no “surprises” during runtime “Partly” object-oriented Simple and flexible type system, composition instead of inheritance Zero-dependency programming model All-included binary file, no dependent “external” libraries Speed Ultra fast compilation and startup time, very lightweight footprint
55. Infrastructure Layers Hardware OS VM - JVM Framework - OSGi App Eclipse Kura Hardware OS VM – V8 Framework – Node.js App Node-RED Hardware OS App Flogo Typically 100s of dependencies – 100MB+ “All included” zero- dependency model enabled by Go
56. Hardware OS Security Logging Metrics Hardware OS V8 Protocol Bridges Diagnostics Sensor Processing V8 Node.js Logging Node.js Security Whoa! Overhead for every app means you run out of room for your apps Node-RED Flogo
57. Comparison – Resource Requirements of IoT Frameworks Eclipse Kura Node-RED Flogo VM JVM V8 Golang Base Disk Space 59 MB 56 MB ~ 0 MB Base Runtime Memory ~ 170 MB (with Open JDK) ~ 20 MB (with Oracle Embedded Java) > 50 MB ~ 5 MB Startup time Slow (~8 sec) Slow (~5 sec) Fast (~1 sec) Application Build Slow Not Applicable Fast (~2 secs)
58. Outlook: Flogo Nanoservices on Microcontrollers Flogo nanoservices that target microcontrollers like ARM M0 Offload I/O centric compute farther off to the edge.
59. © Copyright 2000-2016 TIBCO Software Inc. 1) IoT Trends and Challenges 2) Hybrid Integration Architecture 3) IoT Integration o Dataflow Pipeline o Stream Processing o Process Engine 4) Open Source IoT Process Engines o Eclipse Kura (+ Apache Camel) o Node-RED o Flogo 5) Cloud IoT Platforms Agenda
60. Intel IoT Reference Architecture Potential use cases • Event-driven flows & rules • Device Lifecycle Mgmt • User Experience Core Integration Agile Microservices API Management
61. Amazon IoT Reference Architecture Potential use cases • Event-driven Flows • Device Lifecycle Mgmt • Orchestration Core Integration Agile Microservices API Management
62. Google IoT Reference Architecture Potential edge apps
63. © Copyright 2000-2016 TIBCO Software Inc. Serverless Architecture with IBM’s OpenWhisk (Open Source) https://apiguy.tokyo/2016/03/01/openwhisk-a-world-first-in-open-serverless-architecture/ Edge App Edge Gateway Ultra- Lightweight Serverless App
64. © Copyright 2000-2016 TIBCO Software Inc. Hybrid Integration Architecture is the New Default
65. © Copyright 2000-2016 TIBCO Software Inc. • used to develop lightweight IoT edge apps and microservices • part of a hybrid integration architecture • complementary to IoT cloud platforms and streaming analytics Key Takeaways IoT integration frameworks are...
66. Questions? Please contact me! Kai Wähner Technology Evangelist kontakt@kai-waehner.de @KaiWaehner www.kai-waehner.de LinkedIn