Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Node-RED: Roadmap to 1.0

Node-RED: Roadmap to 1.0

A presentation that sets out the project roadmap to get to a 1.0 release.

Nick O'Leary

July 17, 2017
Tweet

More Decks by Nick O'Leary

Other Decks in Technology

Transcript

  1. 2 These slides reflect the direction we see Node-RED heading

    over the next few months to get us to a version 1.0 release. Getting to 1.0 will be a significant milestone for the project, and one that reflects the fact many people are already using Node-RED in production. Please send any feedback to the Node-RED mailing list or Slack team https://groups.google.com/forum/#!forum/node-red https://nodered.org/slack
  2. 3 Workflow How an individual developer can be more productive.

    Support modern development practices. Integration with CI/CD pipelines. Collaboration Working within a team of developers. Sharing reusable assets. Reviewing changes. Scaling Handling workloads of increasing size. Supporting both horizontal and vertical scaling. Providing the tools to manage a system at scale. API Stability Giving confidence the API exposed can be developer against for the long-term. Extensibility Ensuring both the runtime and editor provide the necessary extension points for them to be customised and built upon Themes These are high level themes this roadmap focuses on. They reflect the priorities we identified through feedback from the community and our own experiences.
  3. 4 Single User on a device A Team collaborating on

    a single instance A Team collaborating but developing locally Hit deploy, test locally, push to remote git when ready Hit deploy, straight to running in production Push to remote, trigger CI that pushes to runtimes Push to remote, trigger CI that pushes to testing env Promote flows from testing to production Deploy to 10,000 devices Cloud hosted editor, multiple runtimes Usage Scenarios to enable These are the sorts of usage scenarios we want to enable through this roadmap. Node-RED won’t solve these scenarios entirely, but it will be much easier to integrate with existing solutions to achieve them.
  4. 5 Workflow Collaboration Extensibility API Stability Scaling Projects x x

    Version Control x x Library Redesign x x Split Editor/Runtime Packaging x x x Subflow Instance Properties x x Persistable Context x x x Pluggable Routing x x x Node Messaging API x x Epic Features and their Themes We have identified the following features as being the main priorities in order to successfully address the themes.
  5. 6 • A project is a set of files that

    represent a complete, redistributable Node-RED application. • The Runtime runs a single project at any one time, but the Editor provides an easy way to switch between projects. • A project exists locally on disk, but can be linked to a remote git repository Feature: Projects .node-red/ ├── node_modules ├── package.json ├── projects │ └── my-project │ ├── .git │ ├── flow.json │ ├── flow_cred.json │ ├── package.json │ ├── README.md │ └── settings.json └── settings.js Workflow • Collaboration • Extensibility • API Stability • Scaling
  6. 7 • Allow a developer to take regular snapshots of

    their work and to revert back to previous versions easily. • Allow a team of developers to collaborate on a single set of flows in a controlled manner; each developing locally, but sharing changes and maintaining a history. Feature: Version Control Workflow • Collaboration • Extensibility • API Stability • Scaling
  7. 8 • Provide a more flexible library system within the

    editor. • Allow different library sources to be added; both local-file system based and git repository based. • Allow a team of developers to share reusable assets – provide a common set of flows. Feature: Library Redesign Workflow • Collaboration • Extensibility • API Stability • Scaling
  8. 9 node-red • Split the packaging of node-red into three

    parts: • node-red-core-runtime • node-red-core-nodes • node-red-core-editor • The existing node-red package will become a meta-package that pulls them together to deliver Node-RED as it is used today. Feature: Split Editor/Runtime Packaging runtime editor HTTP Admin API Editor Runtime API core-nodes Workflow • Collaboration • Extensibility • API Stability • Scaling
  9. 10 • Enable the customisation of individual instances of a

    subflow, whilst retaining the common subflow template. • Building on that, a subflow can be saved to a library and becomes a versioned thing. If the library copy is updated, the editor notifies the user and lets them substitute the updated version into their flow – with their customisations preserved. Feature: Subflow Instance Properties Workflow • Collaboration • Extensibility • API Stability • Scaling
  10. 11 • Provide an external API to enable an external

    key/value store to be used to persist context variables with a flow. • The current get/set context API is synchronous. Need to consider how that maps to a possibly asychronous external API. • The get/set API may also need to be extended to provide a core set of atomic actions such as ‘increment value’ Feature: Persistable Context count: 23 fruit: [“apple”, “banana”] flow context count: 23 fruit: [“apple”, “banana”] Workflow • Collaboration • Extensibility • API Stability • Scaling
  11. 12 • Currently if a node is passed a message,

    the runtime has no way to know when the node has finished processing the message. • This poses some challenges for systems that require some level of message handling assurance – knowing that a message has been properly handled and not silently dropped. • This feature updates the API nodes implement so they can notify the runtime when they have finished with a message. • The runtime will continue to support the existing API as it will take some time for all 1000+ nodes to be updated. Feature: Node Messaging API Workflow • Collaboration • Extensibility • API Stability • Scaling https://github.com/node-red/node-red/wiki/Design:-Node-Messaging-API
  12. 13 • The mechanism by which messages are passed from

    one node to the next will become a pluggable component of the runtime. • This will enable a number of use cases, such as: • Flow Debugger – a router that allows breakpoints to be set • Custom low-level logging of messages/events • Distributing flows across multiple runtime instances Feature: Pluggable Routing Workflow • Collaboration • Extensibility • API Stability • Scaling
  13. 14 0.18 0.19 1.0 Themes: - Collaboration - Workflow Features:

    - Projects - Version control - Library redesign Themes: - Extensibility - Collaboration - API Stability - Scaling Features: - Split editor/runtime packaging - Subflow Instance properties - Node Messaging API Themes: - Scaling - Extensibility Features: - Persistable context - Pluggable routing 0.20 This timeline proposes how the features will be staged across releases. The precise timings of these releases is not yet known, but the goal is to get to v1.0 by the end of 2017.
  14. 15 The following slides look at some of these features

    in greater depth, including some design proposals for the user experience within the editor.
  15. 17 • The current library UX in Node-RED is limited.

    It stores saved flows on disk but doesn’t provide any way for a team of developers to automatically share those assets. • Flow JSON can be exported/imported via the clipboard, but that is an entirely manual process • If an asset is updated, there is no convenient way to update instances that use it Library Redesign : As-Is Issues Workflow • Collaboration • Extensibility • API Stability • Scaling
  16. 18 • An individual developer wants to save useful flows

    snippets for later reuse • A team of developers want to have a common catalog of reusable assets that can be versioned. • A developer wants to be able to pick-up new versions of an asset as easily as possible. • A developer wants to customise instances of an asset but retain the ability to update the asset – for example, setting developer-specific credentials in a node. Library Redesign : Use Cases Workflow • Collaboration • Extensibility • API Stability • Scaling
  17. 19 • Provide a more flexible library system within the

    editor. • Allow different library sources to be added; both local-file system based and git repository based. • Allow a team of developers to share reusable assets – provide a common set of flows. Library Redesign Workflow • Collaboration • Extensibility • API Stability • Scaling
  18. 20 Library The Library is accessed as a sidebar tab.

    The drop-down menu version is removed. The sidebar lets the user pick which library to view, what type of library object and a listing of the contents. TBC: whether the listing is a complete tree or a single folder (with ‘..’ to return up a level). Library Sidebar Examples ├── inject │ └── repeated injects ├── projects │ └── my examples │ ├── get-sensor-data │ ├── convert-temperature │ └── something-else └── another folder Local Flows
  19. 21 Flows Library The first drop-down is a list of

    the libraries that have been added. They can be local-file-system based, or remote git repositories. TBC: also support a read-only HTTP based catalogue Library Sidebar Examples ├── inject │ └── repeated injects ├── projects │ └── my examples │ ├── get-sensor-data │ ├── convert-temperature │ └── something-else └── another folder Local Local Cookbook flows My Team flows Community library
  20. 22 Flows Library The library can contain different types of

    things. Aside from Flows, any node can register a library type in the editor – for example the Function and Template nodes. Selecting the Type then provides a filtered view of the library contents. Library Sidebar Examples ├── inject │ └── repeated injects ├── projects │ └── my examples │ ├── get-sensor-data │ ├── convert-temperature │ └── something-else └── another folder Local Flows Functions Templates UI-Templates
  21. 23 Library Selecting an entry in the library shows a

    preview below. The preview will include any useful meta-data about the entry. Clicking the Import button will add an instance of this entry to the workspace. Library Sidebar Local Convert Temperature Last updated: 22:03 8/7/2017 Import Flows Examples ├── inject │ └── repeated injects ├── projects │ └── my examples │ ├── get-sensor-data │ ├── convert-temperature │ └── something-else └── another folder │ ├── convert-temperature
  22. 24 Library TBC: dragging the flow icon can also be

    used to add an instance. Library Sidebar Local Convert Temperature Last updated: 22:03 8/7/2017 Import Flows Examples ├── inject │ └── repeated injects ├── projects │ └── my examples │ ├── get-sensor-data │ ├── convert-temperature │ └── something-else └── another folder │ ├── convert-temperature
  23. Close User Settings View Keyboard Palette 25 Library A new

    Library tab in settings is where a user can configure the available libraries. The drop-down lists the configured libraries. Selecting each one shows the relevant configuration information about the library. For example, the default local file system library. Library Settings Local Type: local file system Location: /home/nol/.node-red/lib Add New…
  24. Close User Settings View Keyboard Palette 26 Library For example,

    a git repository Library Settings My Team Flows Type: git repository Location: /home/nol/.node-red/.lib/my-team-flows Add New… Repository: git://example.com/my-team-lib.git
  25. Close User Settings View Keyboard Palette 27 Library To add

    a new library, the Add New button is clicked. Library Settings My Team Flows Type: git repository Location: /home/nol/.node-red/.lib/my-team-flows Add New… Repository: git://example.com/my-team-lib.git
  26. Close User Settings View Keyboard Palette 28 Library The type

    of library is selected Library Settings Local File System Library Type Add new Library Cancel Add Close Local File System Git Repository HTTP Catalogue
  27. Close User Settings View Keyboard Palette 29 Library For a

    local file system library, it is given a name and location on disk. The location defaults to $HOME/.node-red/.lib/ with the name field appended (after being made ‘safe’) Library Settings Local File System Library Type Add new Library Cancel Add Close My second library Name $HOME/.node-red/.lib/my-new-lib Path
  28. Close User Settings View Keyboard Palette 30 Library For a

    git repository, the location and credentials are provided. TBC: more work around authentication. If the underlying git is configured properly, no credentials are needed. An option is to only present credentials challenge if git clone fails due to auth issue. Library Settings Git repository Library Type Add new Library Cancel Add Close git://example.com/my-team-lib.git Repository Username Password
  29. 32 Saving a flow to the library starts with the

    ‘Export dialog’. A new button is added to Save to Library. Save to Library Export flow Export to Clipboard Save to Library Cancel selected nodes current flow all flows Export [{"id":"3dbdbbb5.233fc4","type":"change","z":"423676fd.589fc8","na me":"","rules":[{"t":"set","p":"testGlobal","pt":"global","to":"payload","t ot":"msg"}],"action":"","property":"","from":"","to": "", "reg ":false,"x":340 ,"y":260,"wires":[["c53655c3.147608"]]},{"id":"792edc09.641874","typ e":"change","z":"423676fd.589fc8","name":"","rules":[{"t":"set","p":"pa yload","pt":"msg","to":"$flowContext('flowTest') & 'hi'","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":fals e,"x":400,"y":200,"wires":[["16488989.2bc3c6"]]}]
  30. 33 The user picks the library and enters a path/filename

    for the entry. A button is added to the path field that can expand a browser. Save to Library Save to Library Save to Library Cancel Local Examples / path / …
  31. 34 The expanded browser allows the user to pick a

    path, or select an existing entry to overwrite. Attempting overwrite will prompt a confirmation. Save to Library Save to Library Save to Library Cancel .. inject projects my-flow-one my-flow-two Local ... Examples / path / …
  32. 35 For a git-backed library, an extra input is used

    to provide a commit message. Save to Library Save to Library Save to Library Cancel My Team Library Examples / path / … Change description
  33. 37 • Node-RED does not provide any integration with version

    control. This means developers have to manually manage their flow files if they want to put them under versioning. • Manually merging changes from multiple developers at the raw flow file level is not trivial as it involves comparing JSON changes without any context. The Flow Merge tool added in 0.16 is not available to them. • A local developer’s user data directory can become littered with flow files as they switch between different projects. Switching involves manually restarting Node-RED. Projects + Version Control : As-Is Issues Workflow • Collaboration • Extensibility • API Stability • Scaling
  34. 38 • An individual developer wants to develop their flows

    under proper version control. They want to be able to commit versions of the flows and rollback to earlier levels. • A team of developers want to collaborate on a set of flows – each developing locally, but using a shared git repository. • A developer wants to quickly switch between different sets of flows they are working on. Projects + Version Control : Use Cases Workflow • Collaboration • Extensibility • API Stability • Scaling
  35. 39 • A project is a set of files that

    represent a complete, redistributable Node-RED application. • The Runtime runs a single project at any one time, but the Editor provides an easy way to switch between projects. • A project exists locally on disk, but can be linked to a remote git repository Projects .node-red/ ├── node_modules ├── package.json ├── projects │ └── my-project │ ├── .git │ ├── flow.json │ ├── flow_cred.json │ ├── package.json │ ├── README.md │ └── settings.json └── settings.js Workflow • Collaboration • Extensibility • API Stability • Scaling
  36. 40 • A default project consists of: • flow.json /

    flow_cred.json – the flow & credential files • package.json – lists any node module dependencies • README.md – documentation for the flow • settings.json – project specific settings • The editor will provide facilities to edit each of these files in the appropriate context. • TBD: A project could contain other files (such as public/html content). Should NR allows you to edit/manage them as well? We don’t want to become a generic IDE. Projects .node-red/ ├── node_modules ├── package.json ├── projects │ └── my-project │ ├── .git │ ├── flow.json │ ├── flow_cred.json │ ├── package.json │ ├── README.md │ └── settings.json └── settings.js Workflow • Collaboration • Extensibility • API Stability • Scaling
  37. 41 • The package.json file is the standard npm format.

    • It lists the npm dependencies and any other project metadata. • The dependencies do not get installed in the project directory – they get installed at the top level of the user directory (as they would today). Projects .node-red/ ├── node_modules ├── package.json ├── projects │ └── my-project │ ├── .git │ ├── flow.json │ ├── flow_cred.json │ ├── package.json │ ├── README.md │ └── settings.json └── settings.js Workflow • Collaboration • Extensibility • API Stability • Scaling
  38. 42 • The settings.json file contains project specific settings. They

    are a subset of those available in the top-level settings.js file. TBD: exactly what settings are supported. • The settings file is strictly JSON. It cannot contain code as it must be reloadable without side-effects and be writeable by the runtime. • Introduces the concept of flow-variables: • Extend the existing $(ENV_VAR) support to include properties defined in the settings file. • This makes it easier to deploy a flow to two different environments with environment-specific flow properties. • TBD: describe the workflow for that. Projects .node-red/ ├── node_modules ├── package.json ├── projects │ └── my-project │ ├── .git │ ├── flow.json │ ├── flow_cred.json │ ├── package.json │ ├── README.md │ └── settings.json └── settings.js Workflow • Collaboration • Extensibility • API Stability • Scaling
  39. 43 • Allow a developer to take regular snapshots of

    their work and to revert back to previous versions easily. • Allow a team of developers to collaborate on a single set of flows in a controlled manner; each developing locally, but sharing changes and maintaining a history. Version Control Workflow • Collaboration • Extensibility • API Stability • Scaling
  40. 44 • Both the Projects and Version Control features will

    be hidden behind a feature flag so they can disabled if required. Need to define the likely scenarios for where these features may need to be disabled – for example the existing cloud-hosted offerings. • The first time NR runs with Projects enabled, if there is an existing flow-file, the editor will offer to migrate the files to a project. If there is no existing flow-file, the editor offers to create a new project or open an existing one Projects + Version Control Workflow • Collaboration • Extensibility • API Stability • Scaling
  41. 45 The first time Node-RED runs, if there is no

    flow-file provided, users are presented with a new Welcome screen. They can either: • Create a new Project • Open an existing Project Welcome Screen Create a new project Open an existing project Welcome to Node-RED
  42. 47 Creating a project requires a project name. They can

    optionally point at a git repository that should be used to clone the initial project from. Create a new project my-project Create a new project Project name Create Clone from repository Cancel
  43. 48 Creating a project requires a project name. They can

    optionally point at a git repository that should be used to clone the initial project from. Create a new project – clone from repository my-project Create a new project Project name Create https://example.com/my-project.git Clone from repository Cancel
  44. 49 After clicking ‘Create’, a new project is created. Create

    a new project – clone from repository Creating project Create Cancel
  45. 50 If a remote repository was provided and the clone

    failed for authentication reasons, the user is prompted for their credentials. The ‘Remember me’ option will save these details, encrypted on disk, and remembered for subsequent requests. Notes: What about SSH Key auth? Do we require the user to configure that so git Just Works? Clone from repository - credentials Clone from repository Username Create Password Cancel Remember me
  46. 51 After clicking ‘Create’, a new project is created. This

    time using the newly provided credentials. Clone from repository Creating project Create Cancel
  47. 52 If the repository was cloned from a remote repository

    and the credentials were encrypted, it will then prompt for the encryption key. This key is stored in runtime settings, not the project’s settings file (as that will be version controlled alongside the credential files). Notes: What if the user doesn’t know it? Clone from repository – credentials key Clone from repository Flow Credentials Key Create Cancel
  48. 53 If the cloned project includes nodes that are not

    installed in the runtime, it will prompt the user to install them. Skipping this step will leave the flows stopped with ‘unknown’ nodes. Clone from repository – credentials key Clone from repository This project contains nodes you do not have installed. Install them now? Install Skip
  49. 54 In the users .node-red directory, a new directory is

    created under the top-level projects directory. A project contains the flow configuration (including credentials) and a project-specific settings file. A default README.md is added The settings file is a .json file – not .js. The is an important distinction as it must be reloadable without any side- effects of executed code. That restricts what project-specific settings can be used. For example, functionGlobalContext cannot be used on a per-project basis. But it can be used to define flow- variables Project create on disk .node-red/ ├── node_modules ├── package.json ├── projects │ └── my-project │ ├── .git │ ├── flow.json │ ├── flow_cred.json │ ├── package.json │ ├── README.md │ └── settings.json └── settings.js
  50. 57 Selecting ‘Open an existing project’ presents the user with

    a list of available projects. The user selects one and hits Open Open an existing project Open an existing project Open Select a project my-project another-project top-secret-project my-project Cancel
  51. 60 Project Project Sidebar This is the project’s README content.

    It’s a place to document what the project is about and any other useful information. It uses markdown formatting. Description edit Dependencies Settings
  52. 61 Project The README can be edited within the sidebar.

    Project Sidebar # This is the project’s README content. It’s a place to document what the project is about and any other useful information. It uses markdown formatting. Description save Settings cancel Dependencies
  53. 62 Project The dependencies section lists the node modules that

    the project declares in its package.json. Project Sidebar Description Dependencies edit node-red-node-random 0.1.1 node-red-contrib-another-node 0.10.1 node-red-contrib-lower-case 7.1.1 Settings
  54. 63 Project Editing the dependencies allows the user to edit

    the relevant section of package.json. Project Sidebar Description Dependencies { “node-red-node-random”: “0.1.1”, “node-red-contrib-another-node”: “0.10.1”, ”node-red-contrib-lower-case”: “7.1.1” } save cancel Settings
  55. 64 Project If a node is dragged into the flow

    that comes of a module not in the project’s dependencies list, it will be flagged up with an option to add it. Equally, any dependencies found to be unused can be removed. Project Sidebar Description Dependencies edit node-red-node-random 0.1.1 node-red-contrib-another-node 0.10.1 node-red-contrib-lower-case 7.1.1 base64 Missing dependencies node-red-node-base64 0.1.1 ! add to project node-red-node-unused 0.1.1 remove Unused dependencies Settings
  56. 65 Project The Settings section allows flow properties to be

    configured. TBD: a workflow for setting ‘local-only’ values for the flow properties – values that are stored locally, but not saved to the projects settings file. This would allow per-developer customisation of the properties whilst developing locally. Project Sidebar Description Dependencies Settings Flow Properties MQTT_HOSTNAME example.com WS_API_KEY ABC_DEF_123 DB_TABLE production_data add
  57. 67 Changes are deployed as usual. The changes are written

    to the project’s flow file. In git terminology, they are un-staged changes. A new option is added to the Deploy menu – ‘Commit changes’ Selecting that option opens ’Version Control’ sidebar (naming tbc). Version Control my-project Commit changes View history
  58. 68 Version Control The ‘Version Control’ tab allows the user

    to: - commit unstaged changes - browse local history - view diff - revert to earlier commit - push local commits to remote - browse remote history - pull changes down The ‘Unsaved’ section contains the uncommitted changes. Hovering over each reveals a ‘diff’ and ‘revert’ button. The user selects the files to commit, then enters a message and clicks commit. Version Control Unsaved Local Remote You have uncommitted changes Enter a commit message… Commit flow.json flow_cred.json package.json README.md settings.json
  59. 69 Any previous commit can be viewed using the flow

    diff tool. Version Control Close
  60. 70 Version Control The Local section shows the history of

    the local repository and shows where the remote repo is up to. The local changes can be pushed to the remote. Version Control You have no uncommitted changes Local – 2 commits to push Remote The latest commit a-user 10:23 7/7/17 A previous commit a-user 10:23 7/7/17 A previous commit a-user 10:23 7/7/17 A previous commit a-user 10:23 7/7/17 A previous commit a-user 10:23 7/7/17 remote push to remote
  61. 71 Version Control The Remote section shows the commit history

    of the remote repository, and provides a button to pull those changes down. Version Control You have no uncommitted changes Local Remote – 1 commit to pull Someone else’s work ano-user 10:23 7/7/17 A previous commit a-user 10:23 7/7/17 A previous commit a-user 10:23 7/7/17 A previous commit a-user 10:23 7/7/17 A previous commit a-user 10:23 7/7/17 remote pull changes local
  62. 72 Please send any feedback to the Node-RED mailing list

    or Slack team https://groups.google.com/forum/#!forum/node-red https://nodered.org/slack Nick O’Leary - @knolleary