Link Call Node

[knolleary]

Ref https://trello.com/c/4UkvoNM1/199-link-add-link-call-node

The Link nodes can be used to add virtual links between different flows. That's useful if there is a common flow you want to be able to call from multiple places.

But you can't used them to create a 'subroutine' flow that can be called from other flows and have the result returned to the calling flow. (In theory, that's what subflows are for - but they also have their drawbacks.)

This design proposal covers how the link nodes could be extended to support the call-and-return mode of operation.

There are two parts to this design - a new node in the palette and a new option in the link out node.

link-call node

This is a regular flow node with an input and output. Its configuration lets you pick a Link In node as its destination. When it receives a message it passes it to the Link In node.

Before passing the message on, it sets msg._linkSource (better name needed...) to an array containing the node's id. If msg._linkSource already exists, it pushes its id to the end of the array. This will allow for flows that make nested calls.

link out 'return' mode

The link out node gets a new option to put it into return mode. In that mode, if it receives a message with msg._linkSource set, it gets the last entry from the msg._linkSource array, gets the corresponding link-call node and passes it the message directly. It will do this using a private API between the link out and link-call nodes - allowing the link-call node to send that message on as normal. This has the downside of bypassing the pluggable message routing layer, but as it will only ever be returning the message to one caller, the message won't need cloning.

[dceejay]

Could the link out node not auto detect the _linksource property ? So the user didn’t have to select that mode.

It also assumes that no nodes in the linked segment create new msgs and drop that property. Could the link out node flag an error if a message arrives and it’s output is not wired and the _linksource is missing ?

[knolleary]

Could the link out node not auto detect the _linksource property ?

Yes, but then you wouldn't be able to use link nodes 'within' the overall called flow.

Also, a link-out node in 'return' mode cannot also be connected to link-in nodes - it cannot do both things.

So an explicit 'return' mode is desirable.

It also assumes that no nodes in the linked segment create new msgs and drop that property.

Yes. That is a requirement, just as with HTTP In/Response node pairs. The message is the only thing that can carry the context of where it came from.

[dceejay]

Makes sense (re links inside links) and return. So yes a ) visually can have port removed and b) could indeed flag an error just as per http

[dceejay]

Then again may want port to hang some sort of visualisation of where it links back to (link call node name - click to jump there etc)

[Steve-Mcl]

Some thoughts...

Will the link out node, when set for return mode hide it's output pin?

Should the special property be removed from the msg before it's returned to the link call node (to clean up the msg / prevent round trips)?

[knolleary]

Will the link out node, when set for return mode hide it's output pin?

I think so. It doesn't get wired to anything.

Should the special property be removed from the msg before it's returned to the link call node (to clean up the msg / prevent round trips)?

If it finds the property is an empty array, then it has reached the top most layer of the call stack. In that instance, removing the property makes sense.

But if the array isn't empty, then we're in a nested call loop so we need to leave the property in place so the calls can all be unwound.

[rozek]

Well, I'm a complete beginner with respect to node implementation and may therefore be completely wrong with my thoughts but...

...is it wise to use a property (namely return) to completely change behaviour of link nodes? What if the "wrong" mode is set?

Let me illustrate. Right now

• link out nodes start a "goto" - and link in nodes define the jump target (let's call this kind of link out node a "goto node")
• in short: the flow is link out -> link in

With the extension

• link in nodes may also define the entry point of a separate flow expecting a link out node (one or multiple) with mode return (lets call it "return" node)
• link call nodes "call" these link in nodes and wait for return nodes to be triggered
• in short: the flow is link call -> link in -> link out (nota bene: semantic order of link in and link out is reversed!)

What if

• link in nodes are not wired to return nodes (thats perfectly valid because of today's definition of link nodes!), but are triggered by link call nodes?
• link in nodes are triggered by "goto nodes" (perfectly valid) but reach a "return node"?
• link in nodes are wired both to link out nodes with and without "return" mode set? Such nodes could be "called" as well as "jumped to" and it could even be the case that everything works fine if a developer properly routes the message (behind the link in node) depending on some other data - resulting in a complete hacker mess (but working and, thus, not obviously wrong!)

Node-RED is a visual development environment. How do you visualize that a link in node should be "called" rather than jumped to? How do you visualize that a link out node acts as a "return"? How do you handle the link in behaviour in the list of selectable nodes for link out (e.g., you could skip them - but this my surprize the flow programmer)?

Wouldn't it be wiser to implement a completely new node triple (namely xxx-in, xxx-out and xxx - just replace "xxx" by s.th. you like) that clearly reflects the new functionality? This would also avoid having to touch well tested and battle-proven node implementations...

Again, I'm completely new and may thus be completely wrong - if so, please accept my apologies for having disturbed you!

[dceejay]

re - what-ifs

1. then the link call node and downstream would never see any return and the flow would stop there. This would be fairly easy to spot for a developer.
2. if a message arrives at a node in return mode without an _linkSource property we can flag an error.
3. if you mean split the flow and wire to two out nodes, one with return set and one without then as per above - if the return one receives something without a _linkSource it will error. - If not then yes indeed it would be possible for the developer to get into knots - but they can with current link nodes (or indeed other nodes with or without wires - eg passing thing in context etc).

Visualising a link out being a return - the "obvious" answer is to remove the output pin so it can't be wired... and/or giving it a default name of "return".
Visualising the link-call to link-in... yes I think this is one reason the idea has sat on the ideas wall for a good while.

[rozek]

You are right...

...but important will be that these checks will be made at design time and mark faulty flows/nodes as such - rather than at run time (since people rarely write tests with full code coverage - if they write them at all)

[dceejay]

well I don't think it would be possible to check at edit time - but it is something that nrlint could add a rule for.
(again similar to the http in -> response node pair)

[knolleary]

Having implement this design and played with it, I'm coming to the conclusion it will be better to have a separate link-return node.

The issues with overloading the link-out node are:

• visually, there's no difference to show what mode it is in. We could make the icon dynamic... something we've resisted before, but justifiable if its done right. But I'm not sure we want to do that just for this scenario.
• if someone exploring the options puts a link-out node into return mode, it will remove all the existing links on that node.
• it isn't "discoverable" that the node has this mode. It will be clearer to have an explicit return node.

[Steve-Mcl]

Hi Nick makes sense what you are saying.

Some thing that came to mind...

I wonder if there is any mileage in considering replacing the link in node also? Would reduce the paths of execution in the link-in node and would provide a visually separated set nodes of nodes that could be named similarly that relate to each other. Perhaps...

• call, call-start, call-return
  Or
• subroutine-call, subroutine-start, subroutine-end
  Or
• Something else ???

Just musing - food for thought.

[knolleary]

Yes, that is a consideration. But we're really trying not to flood the default palette with lots of nodes. I'm also keen to not introduce lots of ways of doing almost the same thing - and to keep the link nodes as a set of nodes that deal with this sort of thing.

The real alternative is to forget this feature entirely in this form and double-down on subflows being the proper way of doing this - which was my intention. That would mean introducing the 'Singleton Subflow' option - so that all instances of a subflow point to the same concrete instance in the runtime. But we know people don't like using subflows for this - and why there has been the proliferation of contrib nodes that implement this type of feature rather than anyone engaging with us to implement something in the core.

[Steve-Mcl]

No, please don't drop this 🙏

I personally have a vested interest in subroutine alike feature over subflow or singleton approach for reasons I won't get into too deeply but I believe once others grasp the usefulness, they'll wonder how they managed before.

If I absolutely must, I will go to a contrib node or write one myself but that would only create more problems for the ecosystem - built in is best by far I hope you agree?

The way I justify the need for a subroutine alike feature is similar to PLC and to some degree, industrial Robot programming. We have what some manufacturers call a "function" (similar to singleton), "function block" (similar to subflow, with instance variables etc) and "subroutine".

By far, the most confusing aspect for support technicians is the function-block then the function and subroutine.

Hope that helps?

(Apologies if that comes across any other way than intended)

[knolleary]

No intention of dropping the concept of making it easier to do subroutines. My point was that maybe link nodes aren't the right way to do it and that we ought to refocus on improving the user experience around subflows.

That said, I'm currently back to looking at making the link-out node work in both scenarios so we don't have to introduce an explicit return node. Slowly chipping away at making the UX work.