Roadmap to 1.0 17/07/17 Nick O’Leary @knolleary 

2 These slides reflect the direction we see Node-RED heading over the next few months to get us to a version 1.0 release. Getting to 1.0 will be a significant milestone for the project, and one that reflects the fact many people are already using Node-RED in production. Please send any feedback to the Node-RED mailing list or Slack team https://groups.google.com/forum/#!forum/node-red https://nodered.org/slack 

3 Workflow How an individual developer can be more productive. Support modern development practices. Integration with CI/CD pipelines. Collaboration Working within a team of developers. Sharing reusable assets. Reviewing changes. Scaling Handling workloads of increasing size. Supporting both horizontal and vertical scaling. Providing the tools to manage a system at scale. API Stability Giving confidence the API exposed can be developer against for the long-term. Extensibility Ensuring both the runtime and editor provide the necessary extension points for them to be customised and built upon Themes These are high level themes this roadmap focuses on. They reflect the priorities we identified through feedback from the community and our own experiences. 

4 Single User on a device A Team collaborating on a single instance A Team collaborating but developing locally Hit deploy, test locally, push to remote git when ready Hit deploy, straight to running in production Push to remote, trigger CI that pushes to runtimes Push to remote, trigger CI that pushes to testing env Promote flows from testing to production Deploy to 10,000 devices Cloud hosted editor, multiple runtimes Usage Scenarios to enable These are the sorts of usage scenarios we want to enable through this roadmap. Node-RED won’t solve these scenarios entirely, but it will be much easier to integrate with existing solutions to achieve them. 

5 Workflow Collaboration Extensibility API Stability Scaling Projects x x Version Control x x Library Redesign x x Split Editor/Runtime Packaging x x x Subflow Instance Properties x x Persistable Context x x x Pluggable Routing x x x Node Messaging API x x Epic Features and their Themes We have identified the following features as being the main priorities in order to successfully address the themes. 

6 • A project is a set of files that represent a complete, redistributable Node-RED application. • The Runtime runs a single project at any one time, but the Editor provides an easy way to switch between projects. • A project exists locally on disk, but can be linked to a remote git repository Feature: Projects .node-red/ ├── node_modules ├── package.json ├── projects │ └── my-project │ ├── .git │ ├── flow.json │ ├── flow_cred.json │ ├── package.json │ ├── README.md │ └── settings.json └── settings.js Workflow • Collaboration • Extensibility • API Stability • Scaling 

7 • Allow a developer to take regular snapshots of their work and to revert back to previous versions easily. • Allow a team of developers to collaborate on a single set of flows in a controlled manner; each developing locally, but sharing changes and maintaining a history. Feature: Version Control Workflow • Collaboration • Extensibility • API Stability • Scaling 

8 • Provide a more flexible library system within the editor. • Allow different library sources to be added; both local-file system based and git repository based. • Allow a team of developers to share reusable assets – provide a common set of flows. Feature: Library Redesign Workflow • Collaboration • Extensibility • API Stability • Scaling 

9 node-red • Split the packaging of node-red into three parts: • node-red-core-runtime • node-red-core-nodes • node-red-core-editor • The existing node-red package will become a meta-package that pulls them together to deliver Node-RED as it is used today. Feature: Split Editor/Runtime Packaging runtime editor HTTP Admin API Editor Runtime API core-nodes Workflow • Collaboration • Extensibility • API Stability • Scaling 

10 • Enable the customisation of individual instances of a subflow, whilst retaining the common subflow template. • Building on that, a subflow can be saved to a library and becomes a versioned thing. If the library copy is updated, the editor notifies the user and lets them substitute the updated version into their flow – with their customisations preserved. Feature: Subflow Instance Properties Workflow • Collaboration • Extensibility • API Stability • Scaling 

11 • Provide an external API to enable an external key/value store to be used to persist context variables with a flow. • The current get/set context API is synchronous. Need to consider how that maps to a possibly asychronous external API. • The get/set API may also need to be extended to provide a core set of atomic actions such as ‘increment value’ Feature: Persistable Context count: 23 fruit: [“apple”, “banana”] flow context count: 23 fruit: [“apple”, “banana”] Workflow • Collaboration • Extensibility • API Stability • Scaling 

12 • Currently if a node is passed a message, the runtime has no way to know when the node has finished processing the message. • This poses some challenges for systems that require some level of message handling assurance – knowing that a message has been properly handled and not silently dropped. • This feature updates the API nodes implement so they can notify the runtime when they have finished with a message. • The runtime will continue to support the existing API as it will take some time for all 1000+ nodes to be updated. Feature: Node Messaging API Workflow • Collaboration • Extensibility • API Stability • Scaling https://github.com/node-red/node-red/wiki/Design:-Node-Messaging-API 

13 • The mechanism by which messages are passed from one node to the next will become a pluggable component of the runtime. • This will enable a number of use cases, such as: • Flow Debugger – a router that allows breakpoints to be set • Custom low-level logging of messages/events • Distributing flows across multiple runtime instances Feature: Pluggable Routing Workflow • Collaboration • Extensibility • API Stability • Scaling 

14 0.18 0.19 1.0 Themes: - Collaboration - Workflow Features: - Projects - Version control - Library redesign Themes: - Extensibility - Collaboration - API Stability - Scaling Features: - Split editor/runtime packaging - Subflow Instance properties - Node Messaging API Themes: - Scaling - Extensibility Features: - Persistable context - Pluggable routing 0.20 This timeline proposes how the features will be staged across releases. The precise timings of these releases is not yet known, but the goal is to get to v1.0 by the end of 2017. 

15 The following slides look at some of these features in greater depth, including some design proposals for the user experience within the editor. 

16 Library Redesign 

17 • The current library UX in Node-RED is limited. It stores saved flows on disk but doesn’t provide any way for a team of developers to automatically share those assets. • Flow JSON can be exported/imported via the clipboard, but that is an entirely manual process • If an asset is updated, there is no convenient way to update instances that use it Library Redesign : As-Is Issues Workflow • Collaboration • Extensibility • API Stability • Scaling 

18 • An individual developer wants to save useful flows snippets for later reuse • A team of developers want to have a common catalog of reusable assets that can be versioned. • A developer wants to be able to pick-up new versions of an asset as easily as possible. • A developer wants to customise instances of an asset but retain the ability to update the asset – for example, setting developer-specific credentials in a node. Library Redesign : Use Cases Workflow • Collaboration • Extensibility • API Stability • Scaling 

19 • Provide a more flexible library system within the editor. • Allow different library sources to be added; both local-file system based and git repository based. • Allow a team of developers to share reusable assets – provide a common set of flows. Library Redesign Workflow • Collaboration • Extensibility • API Stability • Scaling 

20 Library The Library is accessed as a sidebar tab. The drop-down menu version is removed. The sidebar lets the user pick which library to view, what type of library object and a listing of the contents. TBC: whether the listing is a complete tree or a single folder (with ‘..’ to return up a level). Library Sidebar Examples ├── inject │ └── repeated injects ├── projects │ └── my examples │ ├── get-sensor-data │ ├── convert-temperature │ └── something-else └── another folder Local Flows 

21 Flows Library The first drop-down is a list of the libraries that have been added. They can be local-file-system based, or remote git repositories. TBC: also support a read-only HTTP based catalogue Library Sidebar Examples ├── inject │ └── repeated injects ├── projects │ └── my examples │ ├── get-sensor-data │ ├── convert-temperature │ └── something-else └── another folder Local Local Cookbook flows My Team flows Community library 

22 Flows Library The library can contain different types of things. Aside from Flows, any node can register a library type in the editor – for example the Function and Template nodes. Selecting the Type then provides a filtered view of the library contents. Library Sidebar Examples ├── inject │ └── repeated injects ├── projects │ └── my examples │ ├── get-sensor-data │ ├── convert-temperature │ └── something-else └── another folder Local Flows Functions Templates UI-Templates 

23 Library Selecting an entry in the library shows a preview below. The preview will include any useful meta-data about the entry. Clicking the Import button will add an instance of this entry to the workspace. Library Sidebar Local Convert Temperature Last updated: 22:03 8/7/2017 Import Flows Examples ├── inject │ └── repeated injects ├── projects │ └── my examples │ ├── get-sensor-data │ ├── convert-temperature │ └── something-else └── another folder │ ├── convert-temperature 

24 Library TBC: dragging the flow icon can also be used to add an instance. Library Sidebar Local Convert Temperature Last updated: 22:03 8/7/2017 Import Flows Examples ├── inject │ └── repeated injects ├── projects │ └── my examples │ ├── get-sensor-data │ ├── convert-temperature │ └── something-else └── another folder │ ├── convert-temperature 

Close User Settings View Keyboard Palette 25 Library A new Library tab in settings is where a user can configure the available libraries. The drop-down lists the configured libraries. Selecting each one shows the relevant configuration information about the library. For example, the default local file system library. Library Settings Local Type: local file system Location: /home/nol/.node-red/lib Add New… 

Close User Settings View Keyboard Palette 26 Library For example, a git repository Library Settings My Team Flows Type: git repository Location: /home/nol/.node-red/.lib/my-team-flows Add New… Repository: git://example.com/my-team-lib.git 

Close User Settings View Keyboard Palette 27 Library To add a new library, the Add New button is clicked. Library Settings My Team Flows Type: git repository Location: /home/nol/.node-red/.lib/my-team-flows Add New… Repository: git://example.com/my-team-lib.git 

Close User Settings View Keyboard Palette 28 Library The type of library is selected Library Settings Local File System Library Type Add new Library Cancel Add Close Local File System Git Repository HTTP Catalogue 

Close User Settings View Keyboard Palette 29 Library For a local file system library, it is given a name and location on disk. The location defaults to $HOME/.node-red/.lib/ with the name field appended (after being made ‘safe’) Library Settings Local File System Library Type Add new Library Cancel Add Close My second library Name $HOME/.node-red/.lib/my-new-lib Path 

Close User Settings View Keyboard Palette 30 Library For a git repository, the location and credentials are provided. TBC: more work around authentication. If the underlying git is configured properly, no credentials are needed. An option is to only present credentials challenge if git clone fails due to auth issue. Library Settings Git repository Library Type Add new Library Cancel Add Close git://example.com/my-team-lib.git Repository Username Password 

31 Save to Library 

32 Saving a flow to the library starts with the ‘Export dialog’. A new button is added to Save to Library. Save to Library Export flow Export to Clipboard Save to Library Cancel selected nodes current flow all flows Export [{"id":"3dbdbbb5.233fc4","type":"change","z":"423676fd.589fc8","na me":"","rules":[{"t":"set","p":"testGlobal","pt":"global","to":"payload","t ot":"msg"}],"action":"","property":"","from":"","to": "", "reg ":false,"x":340 ,"y":260,"wires":[["c53655c3.147608"]]},{"id":"792edc09.641874","typ e":"change","z":"423676fd.589fc8","name":"","rules":[{"t":"set","p":"pa yload","pt":"msg","to":"$flowContext('flowTest') & 'hi'","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":fals e,"x":400,"y":200,"wires":[["16488989.2bc3c6"]]}] 

33 The user picks the library and enters a path/filename for the entry. A button is added to the path field that can expand a browser. Save to Library Save to Library Save to Library Cancel Local Examples / path / … 

34 The expanded browser allows the user to pick a path, or select an existing entry to overwrite. Attempting overwrite will prompt a confirmation. Save to Library Save to Library Save to Library Cancel .. inject projects my-flow-one my-flow-two Local ... Examples / path / … 

35 For a git-backed library, an extra input is used to provide a commit message. Save to Library Save to Library Save to Library Cancel My Team Library Examples / path / … Change description 

36 Projects + Version Control 

37 • Node-RED does not provide any integration with version control. This means developers have to manually manage their flow files if they want to put them under versioning. • Manually merging changes from multiple developers at the raw flow file level is not trivial as it involves comparing JSON changes without any context. The Flow Merge tool added in 0.16 is not available to them. • A local developer’s user data directory can become littered with flow files as they switch between different projects. Switching involves manually restarting Node-RED. Projects + Version Control : As-Is Issues Workflow • Collaboration • Extensibility • API Stability • Scaling 

38 • An individual developer wants to develop their flows under proper version control. They want to be able to commit versions of the flows and rollback to earlier levels. • A team of developers want to collaborate on a set of flows – each developing locally, but using a shared git repository. • A developer wants to quickly switch between different sets of flows they are working on. Projects + Version Control : Use Cases Workflow • Collaboration • Extensibility • API Stability • Scaling 

39 • A project is a set of files that represent a complete, redistributable Node-RED application. • The Runtime runs a single project at any one time, but the Editor provides an easy way to switch between projects. • A project exists locally on disk, but can be linked to a remote git repository Projects .node-red/ ├── node_modules ├── package.json ├── projects │ └── my-project │ ├── .git │ ├── flow.json │ ├── flow_cred.json │ ├── package.json │ ├── README.md │ └── settings.json └── settings.js Workflow • Collaboration • Extensibility • API Stability • Scaling 

40 • A default project consists of: • flow.json / flow_cred.json – the flow & credential files • package.json – lists any node module dependencies • README.md – documentation for the flow • settings.json – project specific settings • The editor will provide facilities to edit each of these files in the appropriate context. • TBD: A project could contain other files (such as public/html content). Should NR allows you to edit/manage them as well? We don’t want to become a generic IDE. Projects .node-red/ ├── node_modules ├── package.json ├── projects │ └── my-project │ ├── .git │ ├── flow.json │ ├── flow_cred.json │ ├── package.json │ ├── README.md │ └── settings.json └── settings.js Workflow • Collaboration • Extensibility • API Stability • Scaling 

41 • The package.json file is the standard npm format. • It lists the npm dependencies and any other project metadata. • The dependencies do not get installed in the project directory – they get installed at the top level of the user directory (as they would today). Projects .node-red/ ├── node_modules ├── package.json ├── projects │ └── my-project │ ├── .git │ ├── flow.json │ ├── flow_cred.json │ ├── package.json │ ├── README.md │ └── settings.json └── settings.js Workflow • Collaboration • Extensibility • API Stability • Scaling 

42 • The settings.json file contains project specific settings. They are a subset of those available in the top-level settings.js file. TBD: exactly what settings are supported. • The settings file is strictly JSON. It cannot contain code as it must be reloadable without side-effects and be writeable by the runtime. • Introduces the concept of flow-variables: • Extend the existing $(ENV_VAR) support to include properties defined in the settings file. • This makes it easier to deploy a flow to two different environments with environment-specific flow properties. • TBD: describe the workflow for that. Projects .node-red/ ├── node_modules ├── package.json ├── projects │ └── my-project │ ├── .git │ ├── flow.json │ ├── flow_cred.json │ ├── package.json │ ├── README.md │ └── settings.json └── settings.js Workflow • Collaboration • Extensibility • API Stability • Scaling 

43 • Allow a developer to take regular snapshots of their work and to revert back to previous versions easily. • Allow a team of developers to collaborate on a single set of flows in a controlled manner; each developing locally, but sharing changes and maintaining a history. Version Control Workflow • Collaboration • Extensibility • API Stability • Scaling 

44 • Both the Projects and Version Control features will be hidden behind a feature flag so they can disabled if required. Need to define the likely scenarios for where these features may need to be disabled – for example the existing cloud-hosted offerings. • The first time NR runs with Projects enabled, if there is an existing flow-file, the editor will offer to migrate the files to a project. If there is no existing flow-file, the editor offers to create a new project or open an existing one Projects + Version Control Workflow • Collaboration • Extensibility • API Stability • Scaling 

45 The first time Node-RED runs, if there is no flow-file provided, users are presented with a new Welcome screen. They can either: • Create a new Project • Open an existing Project Welcome Screen Create a new project Open an existing project Welcome to Node-RED 

46 Create a new project 

47 Creating a project requires a project name. They can optionally point at a git repository that should be used to clone the initial project from. Create a new project my-project Create a new project Project name Create Clone from repository Cancel 

48 Creating a project requires a project name. They can optionally point at a git repository that should be used to clone the initial project from. Create a new project – clone from repository my-project Create a new project Project name Create https://example.com/my-project.git Clone from repository Cancel 

49 After clicking ‘Create’, a new project is created. Create a new project – clone from repository Creating project Create Cancel 

50 If a remote repository was provided and the clone failed for authentication reasons, the user is prompted for their credentials. The ‘Remember me’ option will save these details, encrypted on disk, and remembered for subsequent requests. Notes: What about SSH Key auth? Do we require the user to configure that so git Just Works? Clone from repository - credentials Clone from repository Username Create Password Cancel Remember me 

51 After clicking ‘Create’, a new project is created. This time using the newly provided credentials. Clone from repository Creating project Create Cancel 

52 If the repository was cloned from a remote repository and the credentials were encrypted, it will then prompt for the encryption key. This key is stored in runtime settings, not the project’s settings file (as that will be version controlled alongside the credential files). Notes: What if the user doesn’t know it? Clone from repository – credentials key Clone from repository Flow Credentials Key Create Cancel 

53 If the cloned project includes nodes that are not installed in the runtime, it will prompt the user to install them. Skipping this step will leave the flows stopped with ‘unknown’ nodes. Clone from repository – credentials key Clone from repository This project contains nodes you do not have installed. Install them now? Install Skip 

54 In the users .node-red directory, a new directory is created under the top-level projects directory. A project contains the flow configuration (including credentials) and a project-specific settings file. A default README.md is added The settings file is a .json file – not .js. The is an important distinction as it must be reloadable without any side- effects of executed code. That restricts what project-specific settings can be used. For example, functionGlobalContext cannot be used on a per-project basis. But it can be used to define flow- variables Project create on disk .node-red/ ├── node_modules ├── package.json ├── projects │ └── my-project │ ├── .git │ ├── flow.json │ ├── flow_cred.json │ ├── package.json │ ├── README.md │ └── settings.json └── settings.js 

55 my-project 

56 Open an existing project 

57 Selecting ‘Open an existing project’ presents the user with a list of available projects. The user selects one and hits Open Open an existing project Open an existing project Open Select a project my-project another-project top-secret-project my-project Cancel 

58 my-project 

59 Editing a project’s non-flow content 

60 Project Project Sidebar This is the project’s README content. It’s a place to document what the project is about and any other useful information. It uses markdown formatting. Description edit Dependencies Settings 

61 Project The README can be edited within the sidebar. Project Sidebar # This is the project’s README content. It’s a place to document what the project is about and any other useful information. It uses markdown formatting. Description save Settings cancel Dependencies 

62 Project The dependencies section lists the node modules that the project declares in its package.json. Project Sidebar Description Dependencies edit node-red-node-random 0.1.1 node-red-contrib-another-node 0.10.1 node-red-contrib-lower-case 7.1.1 Settings 

63 Project Editing the dependencies allows the user to edit the relevant section of package.json. Project Sidebar Description Dependencies { “node-red-node-random”: “0.1.1”, “node-red-contrib-another-node”: “0.10.1”, ”node-red-contrib-lower-case”: “7.1.1” } save cancel Settings 

64 Project If a node is dragged into the flow that comes of a module not in the project’s dependencies list, it will be flagged up with an option to add it. Equally, any dependencies found to be unused can be removed. Project Sidebar Description Dependencies edit node-red-node-random 0.1.1 node-red-contrib-another-node 0.10.1 node-red-contrib-lower-case 7.1.1 base64 Missing dependencies node-red-node-base64 0.1.1 ! add to project node-red-node-unused 0.1.1 remove Unused dependencies Settings 

65 Project The Settings section allows flow properties to be configured. TBD: a workflow for setting ‘local-only’ values for the flow properties – values that are stored locally, but not saved to the projects settings file. This would allow per-developer customisation of the properties whilst developing locally. Project Sidebar Description Dependencies Settings Flow Properties MQTT_HOSTNAME example.com WS_API_KEY ABC_DEF_123 DB_TABLE production_data add 

66 Version Control of flows 

67 Changes are deployed as usual. The changes are written to the project’s flow file. In git terminology, they are un-staged changes. A new option is added to the Deploy menu – ‘Commit changes’ Selecting that option opens ’Version Control’ sidebar (naming tbc). Version Control my-project Commit changes View history 

68 Version Control The ‘Version Control’ tab allows the user to: - commit unstaged changes - browse local history - view diff - revert to earlier commit - push local commits to remote - browse remote history - pull changes down The ‘Unsaved’ section contains the uncommitted changes. Hovering over each reveals a ‘diff’ and ‘revert’ button. The user selects the files to commit, then enters a message and clicks commit. Version Control Unsaved Local Remote You have uncommitted changes Enter a commit message… Commit flow.json flow_cred.json package.json README.md settings.json 

69 Any previous commit can be viewed using the flow diff tool. Version Control Close 

70 Version Control The Local section shows the history of the local repository and shows where the remote repo is up to. The local changes can be pushed to the remote. Version Control You have no uncommitted changes Local – 2 commits to push Remote The latest commit a-user 10:23 7/7/17 A previous commit a-user 10:23 7/7/17 A previous commit a-user 10:23 7/7/17 A previous commit a-user 10:23 7/7/17 A previous commit a-user 10:23 7/7/17 remote push to remote 

71 Version Control The Remote section shows the commit history of the remote repository, and provides a button to pull those changes down. Version Control You have no uncommitted changes Local Remote – 1 commit to pull Someone else’s work ano-user 10:23 7/7/17 A previous commit a-user 10:23 7/7/17 A previous commit a-user 10:23 7/7/17 A previous commit a-user 10:23 7/7/17 A previous commit a-user 10:23 7/7/17 remote pull changes local 

72 Please send any feedback to the Node-RED mailing list or Slack team https://groups.google.com/forum/#!forum/node-red https://nodered.org/slack Nick O’Leary - @knolleary