Slide 1

Slide 1 text

Kai Wähner Technology Evangelist [email protected] LinkedIn @KaiWaehner www.kai-waehner.de November 2016 Comparison of Open Source Frameworks for Integrating the Internet of Things

Slide 2

Slide 2 text

© Copyright 2000-2016 TIBCO Software Inc. • used to develop lightweight IoT edge apps and microservices • part of a hybrid integration architecture • complementary to IoT cloud platforms and streaming analytics Key Takeaways IoT integration frameworks are...

Slide 3

Slide 3 text

© Copyright 2000-2016 TIBCO Software Inc. 1) IoT Trends and Challenges 2) Hybrid Integration Architecture 3) IoT Integration o Dataflow Pipeline o Stream Processing o Process Engine 4) Open Source IoT Process Engines o Eclipse Kura (+ Apache Camel) o Node-RED o Flogo 5) Cloud IoT Platforms Agenda

Slide 4

Slide 4 text

© Copyright 2000-2016 TIBCO Software Inc. 1) IoT Trends and Challenges 2) Hybrid Integration Architecture 3) IoT Integration o Dataflow Pipeline o Stream Processing o Process Engine 4) Open Source IoT Process Engines o Eclipse Kura (+ Apache Camel) o Node-RED o Flogo 5) Cloud IoT Platforms Agenda

Slide 5

Slide 5 text

© Copyright 2000-2016 TIBCO Software Inc. Internet of Things (IoT) … refers to the ever-growing network of physical objects that feature an IP address for internet connectivity, and the communication that occurs between these objects and other Internet-enabled devices and systems.

Slide 6

Slide 6 text

© Copyright 2000-2016 TIBCO Software Inc. Forecast for Year 2020 à 20+ Billion Devices

Slide 7

Slide 7 text

© Copyright 2000-2016 TIBCO Software Inc. IoT Landscape 2016 The IoT Landscape 2016 Edition - Matt Turck & David Rogg, First Mark Capital

Slide 8

Slide 8 text

© Copyright 2000-2016 TIBCO Software Inc. Integration is Key for Success of IoT Projects Without Integration there is no Internet of Things! Gartner Market Guide for IoT Integration “Through 2018, half the cost of implementing IoT solutions will be spent on integration. “ Strategic Planning Assumption, Benoit J. Lheureux, Massimo Pezzini, Alfonso Velosa !

Slide 9

Slide 9 text

© Copyright 2000-2016 TIBCO Software Inc. Processes Analytics APIs Streams & Events Interconnect Everything Devices Equipment CORE Run the Business EDGE Change the Business People

Slide 10

Slide 10 text

© Copyright 2000-2016 TIBCO Software Inc. IoT raises several new challenges • Devices are not connected to the cloud • Devices have low bandwidth to connect • Latency of connectivity is significant • Connectivity is not reliable • Connectivity is not cost-effective Great IoT standards available • MQTT, CoaP, OPC US, many more • Different abstraction levels • Not the one single standard Challenges

Slide 11

Slide 11 text

© Copyright 2000-2016 TIBCO Software Inc. 1) IoT Trends and Challenges 2) Hybrid Integration Architecture 3) IoT Integration o Dataflow Pipeline o Stream Processing o Process Engine 4) Open Source IoT Process Engines o Eclipse Kura (+ Apache Camel) o Node-RED o Flogo 5) Cloud IoT Platforms Agenda

Slide 12

Slide 12 text

© Copyright 2000-2016 TIBCO Software Inc. Integration Frameworks Decision Data Conversion Transitions Process Call Process Design Tools Integration Processes Connectors to vendor applications Connectors to technologies Connectors (e.g. JMS, MQTT, FTP) (e.g. SF.com, SAP)

Slide 13

Slide 13 text

© Copyright 2000-2016 TIBCO Software Inc. Enterprise Integration Patterns “The goal of EIPs is to document technology- independent design guidance that helps developers and architects describe and develop robust integration solutions.” http://www.eaipatterns.com

Slide 14

Slide 14 text

© Copyright 2000-2016 TIBCO Software Inc. Pervasive Integration with different Technologies and User Roles https://www.linkedin.com/pulse/how-implement-hybrid-integration-platform-tackle-massimo-pezzini There is no “one-size-fits-all” IoT integration!

Slide 15

Slide 15 text

© Copyright 2000-2016 TIBCO Software Inc. Hybrid Integration Platform (HIP) Application Integration Application Integration (on a PaaS) iPaaS Integration Platform as a Service iSaaS Integration Software as a Service Cloud-Ready Cloud-Native API Management On Premise Public Cloud Public Cloud Public Cloud On Premise Public Cloud Process Integration Business Process Management Platform Streaming Analytics Fog Computing Edge Devices Edge Integration Analysts are sure: HIP is “the new default” in most enterprises!

Slide 16

Slide 16 text

© Copyright 2000-2016 TIBCO Software Inc. 1) IoT Trends and Challenges 2) Hybrid Integration Architecture 3) IoT Integration o Dataflow Pipeline o Stream Processing o Process Engine 4) Open Source IoT Process Engines o Eclipse Kura (+ Apache Camel) o Node-RED o Flogo 5) Cloud IoT Platforms Agenda

Slide 17

Slide 17 text

© Copyright 2000-2016 TIBCO Software Inc. Fog Computing http://www.slideshare.net/Angelo.Corsaro/20141210-fog … keep data closer ‘to the edge’ !

Slide 18

Slide 18 text

Typical IoT Integration Scenario What’s wrong with Cloud-only IoT Integration? • Connectivity is unreliable • Non-trivial latency in hops • Networks costs add up quickly

Slide 19

Slide 19 text

A better IoT Integration Blueprint Benefits: • Local control = more reliable • Less traffic = Lower TCO • Edge integration & event processing

Slide 20

Slide 20 text

© Copyright 2000-2016 TIBCO Software Inc. 1) IoT Trends and Challenges 2) Hybrid Integration Architecture 3) IoT Integration o Dataflow Pipeline o Stream Processing o Process Engine 4) Open Source IoT Process Engines o Eclipse Kura (+ Apache Camel) o Node-RED o Flogo 5) Cloud IoT Platforms Agenda

Slide 21

Slide 21 text

© Copyright 2000-2016 TIBCO Software Inc. Dataflow Pipeline – Extract, Transform, Load https://www.linkedin.com/pulse/data-pipeline-hadoop-part-1-2-birender-saini

Slide 22

Slide 22 text

© Copyright 2000-2016 TIBCO Software Inc. • Flow-based Programming • Ingest data from various sources • Extract (E) – Transform (T) – Load (L) • Transform / Filter / Route / Aggregate / Enrich … • High-throughput straight-through data flows • Data lineage • Synchronous and asynchronous communication • Batch or stream processing • Visual coding with flow editor • Not to be mixed-up with stream processing for event correlation Dataflow Pipeline - Characteristics

Slide 23

Slide 23 text

© Copyright 2000-2016 TIBCO Software Inc. Dataflow Pipeline – Examples All except Informatica use open source / open core business model

Slide 24

Slide 24 text

© Copyright 2000-2016 TIBCO Software Inc. 1) IoT Trends and Challenges 2) Hybrid Integration Architecture 3) IoT Integration o Dataflow Pipeline o Stream Processing o Process Engine 4) Open Source IoT Process Engines o Eclipse Kura (+ Apache Camel) o Node-RED o Flogo 5) Cloud IoT Platforms Agenda

Slide 25

Slide 25 text

© Copyright 2000-2016 TIBCO Software Inc. Stream Processing time 1 2 3 4 5 6 7 8 9 Event Streams • Continuous Queries • Sliding Windows • Filter • Aggregation • Correlation • … Stream Processing H20.ai Open Source R TERR Spark ML MATLAB SAS PMML

Slide 26

Slide 26 text

© Copyright 2000-2016 TIBCO Software Inc. Stream Processing - Examples OPEN SOURCE CLOSED SOURCE PRODUCT FRAMEWORK Azure Microsoft Stream Analytics

Slide 27

Slide 27 text

© Copyright 2000-2016 TIBCO Software Inc. Streaming Analytics à Predictive Maintenance Voltage Temperature Vibration Device history Temporal analytic: “If vibration spike is followed by temp spike then voltage spike [within 4 hours] then flag high severity alert.”

Slide 28

Slide 28 text

© Copyright 2000-2016 TIBCO Software Inc. 1) IoT Trends and Challenges 2) Hybrid Integration Architecture 3) IoT Integration o Dataflow Pipeline o Stream Processing o Process Engine 4) Open Source IoT Process Engines o Eclipse Kura (+ Apache Camel) o Node-RED o Flogo 5) Cloud IoT Platforms Agenda

Slide 29

Slide 29 text

© Copyright 2000-2016 TIBCO Software Inc. • Integrate and orchestrate various data sources • Wire together hardware devices, APIs and online services • Transform / Filter / Route / Aggregate / Enrich … • Error handling, Re-Try, Re-Routing, Wait, Resume, ... • Not just Extract-Transform-Load • Synchronous and asynchronous communication • Typically real time processing • Visual coding with flow editor • Deployed at the edge Process Engine

Slide 30

Slide 30 text

© Copyright 2000-2016 TIBCO Software Inc. Real World IoT Use Cases for a Process Engine

Slide 31

Slide 31 text

© Copyright 2000-2016 TIBCO Software Inc. 1) IoT Trends and Challenges 2) Hybrid Integration Architecture 3) IoT Integration o Dataflow Pipeline o Stream Processing o Process Engine 4) Open Source IoT Process Engines o Eclipse Kura (+ Apache Camel) o Node-RED o Flogo 5) Cloud IoT Platforms Agenda

Slide 32

Slide 32 text

© Copyright 2000-2016 TIBCO Software Inc. Commonalities: • Open source frameworks • Connectivity to IoT technologies (MQTT, CoaP, REST, …) • Web UI for visual coding, testing, debugging • Deployable “at the edge” • For developers / integration specialists / citizen integrators • Extendable SDKs / APIs Open Source IoT Process Engines

Slide 33

Slide 33 text

© Copyright 2000-2016 TIBCO Software Inc. 1) IoT Trends and Challenges 2) Hybrid Integration Architecture 3) IoT Integration o Dataflow Pipeline o Stream Processing o Process Engine 4) Open Source IoT Process Engines o Eclipse Kura (+ Apache Camel) o Node-RED o Flogo 5) Cloud IoT Platforms Agenda

Slide 34

Slide 34 text

© Copyright 2000-2016 TIBCO Software Inc. Eclipse Kura http://www.eclipse.org/kura

Slide 35

Slide 35 text

© Copyright 2000-2016 TIBCO Software Inc. • Focus on IoT Gateway • Set of Java and OSGi services • including I/O services, Data Services, Cloud Services, Networking, etc. • Web UI for Configuration (Devices, Network, Protocols, etc.) • Eclipse Public License 1.0 • Mature framework (~3 years old, code based committed in December 2013) • Best of breed with focus on coding instead of a visual designer • Eclipse IoT Projects • IoT Gateway --> Eclipse Kura • Services like Eclipse SmartHome • Standard implementations like Mosquitto (MQTT Server) • Connectivity via Eclipse Paho (MQTT Client) • Apache Camel Connector • Integration Framework (connectivity, enterprise integration patterns) • Visual coding via JBoss or Talend (both available as open source edition) • For Developers / Integration Specialists only • More complex to install and build a first IoT integration flow • Write source code • Can be run on a variety of platforms (on premise, cloud, edge devices, container) Eclipse Kura (+ Apache Camel) à Facts

Slide 36

Slide 36 text

© Copyright 2000-2016 TIBCO Software Inc. Eclipse Kura – Functionality Overview and Architecture https://www.youtube.com/watch?v=ia8cLnR1uFI http://www.slideshare.net/Eurotechchannel/kuram2miotgateway

Slide 37

Slide 37 text

© Copyright 2000-2016 TIBCO Software Inc. Eclipse Kura - Architecture

Slide 38

Slide 38 text

© Copyright 2000-2016 TIBCO Software Inc. Eclipse Kura – Hello World https://eclipse.github.io/kura/doc/hello-example.html

Slide 39

Slide 39 text

© Copyright 2000-2016 TIBCO Software Inc. Apache Camel http://www.kai-waehner.de/blog/2012/05/04/apache-camel-tutorial-introduction/

Slide 40

Slide 40 text

© Copyright 2000-2016 TIBCO Software Inc. Apache Camel (in conjunction with Eclipse Kura) https://dzone.com/articles/creating-camel-routes-for-eclipse-kura

Slide 41

Slide 41 text

© Copyright 2000-2016 TIBCO Software Inc. Live Demo Kura (+ Apache Camel) in Action

Slide 42

Slide 42 text

© Copyright 2000-2016 TIBCO Software Inc. 1) IoT Trends and Challenges 2) Hybrid Integration Architecture 3) IoT Integration o Dataflow Pipeline o Stream Processing o Process Engine 4) Open Source IoT Process Engines o Eclipse Kura (+ Apache Camel) o Node-RED o Flogo 5) Cloud IoT Platforms Agenda

Slide 43

Slide 43 text

© Copyright 2000-2016 TIBCO Software Inc. Node-RED https://nodered.org/

Slide 44

Slide 44 text

© Copyright 2000-2016 TIBCO Software Inc. Node-RED http://industrialinternet.co.uk/category/node-red/

Slide 45

Slide 45 text

© Copyright 2000-2016 TIBCO Software Inc. • Focus on Integration on an IoT Gateway • Built on Node.js and Javascript • Very easy to install and build a first IoT integration flow • For developers / integration specialists and ciziten integrators • Visual coding via web designer • Apache 2.0 License • Mature framework (~3 years old) with many examples and docs • Native integration into IBM Bluemix cloud platform • Leverage other Bluemix services • Can be run on a variety of platforms (on premise, cloud, edge devices, container) • Flows can be shared as JSON strings / files • No binaries to share and deploy on devices Node-RED à Facts

Slide 46

Slide 46 text

© Copyright 2000-2016 TIBCO Software Inc. Live Demo Node-RED in Action

Slide 47

Slide 47 text

© Copyright 2000-2016 TIBCO Software Inc. 1) IoT Trends and Challenges 2) Hybrid Integration Architecture 3) IoT Integration o Dataflow Pipeline o Stream Processing o Process Engine 4) Open Source IoT Process Engines o Eclipse Kura (+ Apache Camel) o Node-RED o Flogo 5) Cloud IoT Platforms Agenda

Slide 48

Slide 48 text

© Copyright 2000-2016 TIBCO Software Inc. Flogo http://www.flogo.io

Slide 49

Slide 49 text

Project Flogo IoT Example

Slide 50

Slide 50 text

© Copyright 2000-2016 TIBCO Software Inc. • Focus on Integration on an IoT Gateway and very lightweight edge applications • Powered by Golang • Very easy to install and build a first IoT integration flow • Ultra-lightweight footprint with zero dependency model • For developers / integration specialists and ciziten integrators • Visual coding via web designer • BSD-style License • Developer Preview (released in October 2016) • Can be run on a variety of platforms (on premise, cloud, edge devices, container) • Flows can be shared as JSON strings / files • Lightweight binaries with zero dependencies can be shared and deployed on devices Flogo à Facts

Slide 51

Slide 51 text

Web-Native Step-Back Debugger • Step back in a flow • Interactively design/debug your process • Simulate sensor events • Change data or configuration and run from Trigger or any Activity • Underlying technology could be used to support remote ops debugging of failed flows

Slide 52

Slide 52 text

© Copyright 2000-2016 TIBCO Software Inc. Live Demo Flogo in Action

Slide 53

Slide 53 text

What technology to build edge apps? Java • Heavy (>200MB) • Not agile • Robust • Licensing risk Node.js • Lighter (~70MB) • Agile • Not IoT robust • Clear licensing C/C++ • Lightest • Not Agile • Robust • Clear licensing

Slide 54

Slide 54 text

© Copyright 2000-2016 TIBCO Software Inc. Why Go? Modern programming language Getting a lot of traction due to usage in frameworks like Docker or Kubernetes Concurrency Built into the language (Goroutines, channels, no explicit thread programming) Memory management Modern Garbage collector Statically typed Type-safe development with no “surprises” during runtime “Partly” object-oriented Simple and flexible type system, composition instead of inheritance Zero-dependency programming model All-included binary file, no dependent “external” libraries Speed Ultra fast compilation and startup time, very lightweight footprint

Slide 55

Slide 55 text

Infrastructure Layers Hardware OS VM - JVM Framework - OSGi App Eclipse Kura Hardware OS VM – V8 Framework – Node.js App Node-RED Hardware OS App Flogo Typically 100s of dependencies – 100MB+ “All included” zero- dependency model enabled by Go

Slide 56

Slide 56 text

Hardware OS Security Logging Metrics Hardware OS V8 Protocol Bridges Diagnostics Sensor Processing V8 Node.js Logging Node.js Security Whoa! Overhead for every app means you run out of room for your apps Node-RED Flogo

Slide 57

Slide 57 text

Comparison – Resource Requirements of IoT Frameworks Eclipse Kura Node-RED Flogo VM JVM V8 Golang Base Disk Space 59 MB 56 MB ~ 0 MB Base Runtime Memory ~ 170 MB (with Open JDK) ~ 20 MB (with Oracle Embedded Java) > 50 MB ~ 5 MB Startup time Slow (~8 sec) Slow (~5 sec) Fast (~1 sec) Application Build Slow Not Applicable Fast (~2 secs)

Slide 58

Slide 58 text

Outlook: Flogo Nanoservices on Microcontrollers Flogo nanoservices that target microcontrollers like ARM M0 Offload I/O centric compute farther off to the edge.

Slide 59

Slide 59 text

© Copyright 2000-2016 TIBCO Software Inc. 1) IoT Trends and Challenges 2) Hybrid Integration Architecture 3) IoT Integration o Dataflow Pipeline o Stream Processing o Process Engine 4) Open Source IoT Process Engines o Eclipse Kura (+ Apache Camel) o Node-RED o Flogo 5) Cloud IoT Platforms Agenda

Slide 60

Slide 60 text

Intel IoT Reference Architecture Potential use cases • Event-driven flows & rules • Device Lifecycle Mgmt • User Experience Core Integration Agile Microservices API Management

Slide 61

Slide 61 text

Amazon IoT Reference Architecture Potential use cases • Event-driven Flows • Device Lifecycle Mgmt • Orchestration Core Integration Agile Microservices API Management

Slide 62

Slide 62 text

Google IoT Reference Architecture Potential edge apps

Slide 63

Slide 63 text

© Copyright 2000-2016 TIBCO Software Inc. Serverless Architecture with IBM’s OpenWhisk (Open Source) https://apiguy.tokyo/2016/03/01/openwhisk-a-world-first-in-open-serverless-architecture/ Edge App Edge Gateway Ultra- Lightweight Serverless App

Slide 64

Slide 64 text

© Copyright 2000-2016 TIBCO Software Inc. Hybrid Integration Architecture is the New Default

Slide 65

Slide 65 text

© Copyright 2000-2016 TIBCO Software Inc. • used to develop lightweight IoT edge apps and microservices • part of a hybrid integration architecture • complementary to IoT cloud platforms and streaming analytics Key Takeaways IoT integration frameworks are...

Slide 66

Slide 66 text

Questions? Please contact me! Kai Wähner Technology Evangelist [email protected] @KaiWaehner www.kai-waehner.de LinkedIn