Basic pipe support for inter-task communication

[TerryE]

Missing feature

#2797 introduced a baseline implementation of a basic pipe feature for multi-record inter-task communication. This enables one task to write records into pipe automatically triggering a receiver task to process them. This is going to be used within the Lua VM for buffering UART input into the Lua interactive loop, and also for marshaling node.ouput() records for sending over the internet. It just seemed sensible to include this as a general core library as a first step.

Workarounds

Doing RAM and processing efficient alternatives in Lua are difficult. I did an example in my telnet module which Nathaniel @nwf has refined and abstracted into the Lua fifo module. Using a simple Lua array can quickly run into memory exhaustion if the average string lengths are small. (This is case with node.output() redirection where the average string length is one print field, say 10 characters.) Hence a two-tier approach of aggregating small strings by concatenation into larger ~LUAL_BUFFERSIZE chunks in needed, but this approach generates a very high Lua GC load.

General discussion

Doing the general pipe implementation in C using a mix of the Lua and C worlds seemed the best approach. In this pipe implementation, the pipe itself is stored as a Lua array of LUAL_BUFFERSIZE userdata chunks. This is RAM-efficient and also avoids the Lua GC load on all of the concatenation products. The bottom line is that this algo runs fast, and takes just about the minimum memory possible. Also as Nathaniel has pointed out encapsulating all this in a standard library makes it easy for average Lua developers to use.

At the moment the baseline version implements the following functions:

• pipe.create()
• p:write(rec)
• p:read(delim/len)
• p:reader(delim/len) a Lua iterator based on the p:read() to simplify coding for loops
• p:unread(rec) to backout a read

One of the pragmatic compromises of the current implementation is not to do delimiter look-ahead, so if you want to read a record up to the next \n, say, and the last record in the fifo doesn't include one then the last record read will be incomplete. Of course, this is only the case where the writer emits part records. Gregor @HHHartmann has quite correctly pointed out the weakness here, which is why I have included the unread method so that the application can unread and ignore any part complete last records. The alternative is to do do multiblock lookahead for the next delimiter, though the unread implenetaion is easier for implementing the Lua interactive loop.

However this features set in subject to consensus. If we think that getting a last part complete record is this is an unnecessary complication then I add the extra logic to implement milti-block delimiter lookahead. Feedback and views requested.

Proposed Enhancements

The pipe.create() can take a optional function argument. Writing to the pipe automatically will trigger a node.task.post(function) and this post logic has debounce logic so that only one post occurs before the function runs. The function, once the CB fires, can then consume all of the pipe, part and repost itself, or just end and it will be reposted on the next write. Adding this sort of functionality makes implementing asynchrous tasks such as spooling output over a sock really simple to code.

[nwf]

Only tangentially related, but, I wonder if it's worth moving fifosock over to pipe instead of fifo, now that it's available?

[TerryE]

Nathaniel, I want to get it working with lua.c etc., first. Give me a couple of days 😊.

[TerryE]

Implemented fully in #2836