RFC - Sorting out the input handling #2787
Comments
|
Off the top of my head I can't remember whether the esp32 port uses the same approach, or whether I got away with the regular stdio setup there. |
|
@jmattsson, I want this |
|
On thing that we (at least I) have always taken as given is that the Lua VM just works as-is and dovetails into the NodeMCU task oriented paradigm. But I have never questioned why and how. We have three layers of execution:
In general the Lua call interface handles stack cleanup on exit, so this works well as long as the individual task masters run atomically and are single threaded, and that the stack is balanced and doesn't have any leaking growth. IMO there are two weaknesses here:
In fact the Lua interpreter polling readline uses this first feature to build up the Lua chunk from line to line. In my view both of these are highly undesirable. Ideally we should just wrap the task callback code in a pcall, but that is going to be another bulk editing exercise. PS. Given that the task code itself rarely throws an error, and nearly all tasks immediately exit, it might be better to bulk edit all CBs to include a |
|
Returning to the original issue I would also sort out the node.input () issue which does not collect input until it is compiled correctly but fails on partial commands which breaks all telnet and serial forwarder examples out there including our own. |
|
Gregor, we are a little constrained by the interpreter architecture and the way that it defines a "chunk": that is the smallest number of complete lines that does not raise an incomplete error when parsed (parser hit end of stream when still looking for lexical components), and also that we can have multiple source streams. The current implementation is a botch. The Lua VM should also be in a well determined state when doing this parsing, again which it isn't for I also need to be careful not to waste too many RAM resources whilst doing all of this. Let me do the first cut and I'll post it as a PR |
|
The standard Lua interpreter sits at call level 0 looping around the We also want to be able to redirect output (both The standard interpreter accumulates lines on the Lua stack, but this model does not work with multiple input sources. What I have done so far is to create a lineQ resource which is hung off a Lua registry entry, and this:
This may seem a little convoluted, but under normal circumstances the Q will bounce between empty and 1 entry, so this strategy avoids needing to create Lua array resources unless necessary. However, the Q is filled by a high priority task which then posts a low priority task to do the compile, so burst input will cause the queue to grow. Another advantage of this approach is that we can simply maintain two Qs instead of 1 (one for each of the API and the UART), and build the chunk in the Q. We can also use a similar approach to tidy up output handling -- which currently breaks the Lua architectural rules as the NodeMCU function runs in a single task, but can generate many field-level The correct approach is to do is either direct output to the UART or to buffer output, but to handle this buffering properly in the VM. The output from a single task will be buffered, and the handler to clear the buffer will be posted as a separate task to run on completion of the outputing task. If the application is redirecting output then the application tasks and the logging tasks will interleave within normal SDK task priorities. node.output()Syntax
Parameters
Returns
Examplelocal function tonet(q)
if type(q) == 'table' and #q > 0 then
sk:send(table.remove(q,1), tonet)
end
node.output(tonet, 1, 1400) -- serial also get the Lua output.Note that even though the (maximum) buffer size can be specified, the output task is still scheduled to run after every application task that does at least one |
|
As I mentioned about, I've decided to back-port a lot of these changes into Lua 5.1. My reasoning is that I want to keep everything outside of
I've been using Node.js and Node-RED for my own Home Automation work, I am struck by the architectural similarities between the Node.js and NodeMCU execution models. The non-blocking atomic task architecture is common to both. Since this is core to NodeMCU, I've started to use the tasking model within some VM aspects, and it really does make like a lot easier. |
|
@TerryE does that buffering part mean you could stream input from another task too? I have a queue for receiving received data, once I get a \n I put the partial string inside the lua_Load thing, I wonder if this is why any function that ends up calling vfs_open() triggers a wdt reset |
IMO, that's the correct way to invoke Lua code; the a=someFunc( |
|
I have also added the complimentary output stuff, and especially a new do
local p = pipe.create()
local fp = p:reader(1400)
local running = true
local function despooler() -- Upvals: fp, sk, despooler, running
local buf = fp()
if buf then
sk:send(buf, despooler)
elseif not running then
p, fp = nil
-- you might also want to close/dispose sk here as well
end
end
local function spool_rec(rec) -- Upvals: p, despooler
if #p == 0 then
node.task.post(node.task.HIGH_PRIORITY, despooler)
end
p:write(rec)
end
function close_output() -- Upvals: p, fp, running
node.output()
if #p then
running = false
else
p, fp = nil
end
end
function spool_output() -- Upvals: spool_rec
node.output(spool_rec)
end
endHere, the despooler CBs interleave as separate tasks between the application tasks. All of the marshalling and blocking of the network sends is handled through the |
|
Sometimes you go to sleep pleased that a new piece of functionality like my |
|
OK, the new version is all working. Since it's a new module and won't break existing code, then I'll push and commit it tomorrow. @nwf perhaps you might want to have a play. |
|
Coming back to this thought of putting the NodeMCU and Node.js architectures along side each other, really the main difference between standard Lua and NodeMCU Lua (whether on the ESP8266 or the ESP32 variants) is that the standard However the current NodeMCU Let me do the PR but this is one that will need careful review and comment before we commit it. |
|
Having just tried using |
|
Usecases such as in COAP where the string is supposed to be a complete Lua chunk should definitely use a load(string) implementation. The We are getting there 😊😬 |
|
Fixed in #2836 |
Another case of "I am currently doing my Lua 5.3 port and there are a number of issues on which I would like to give a heads-up / get feedback from the contributors." This one relates to how we integrate input handing into
lua.c.I have never really taken a close look at this bit of
lua.c+user_main.cuart.cbefore, but it is really all a bit of a botch -- the sort of thing that might have evolved if you were hacking an application together against a deadline rather than designing it properly -- which was the case for the early versions 1.x of Lua51, but now I am trying to srt out Lua53 for the long term so it's time to design it properly, IMO.Architecturally, the standard
src/lua.cassumes that an underlying OS will provide some form of API for a synchronousreadline()function that handles stuff like terminal echo; the Lua interpreter sits in thedoREPL()function looping around this readline call. This approach doesn't map onto an event-driven task-orientated system, where the uart driver or posts a receive events for each line received which must then be handled and an atomic task. Hence thisdoREPL()logic must be inverted, so that the interactive processing is all triggered from a line-in callback.Note that the way the Lua interpreter does chunking is a bit like a sledge hammer. Each source chunk is assembled from sequence of successive input lines. The entire chunk is recompiled each time an extra line is added .
<eof>error then the chunk is assumed incomplete and the interpreter waits for another line.The means that if a 20 line chunk is received then the 1st line compiles a 1-line chunk, the 2nd a 2-line chunk and so on so the 20th line compiles a 20 line chunk. Hence the per-line the compiler overhead goes uniformly as the chunk size increases. This isn't really an issue if the lines are really being typed interactively. However, if some IDE such as Esplorer or even if you are just pasting a chunk of code into PuTTY, then it is quite easy to get to the point where the input overruns the ability of the interpreter to flush input buffer.
Another complication in the case of NodeMCU is that there is nothing stopping other application callbacks firing whilst you are typing and thus threading other Lua processing tasks in between the lines in a chunk.
Given then everything outside the
app/luaorapp/lua53directories is going to be shared then its going to be simpler for me to packport some of this tidy-up intoapp/lua/lua.cso that we can keep the rest common.In terms of input tasks, I want to split the emptying of the fixed size input FIFO which should be short duration high-priority task from the activities relating to processing it, in the case of compiling, the input lines should be queued at high priority and should run at low priority. IMO, the best way to do this is to introduce a Lua string array to as a buffer of complete input lines. The UART event/readline function will be posted at high-priority and fills this line array. The interactive loop the processor-intensive compilation task will empty it at low priority. This high / low split priority spilt will largely mitigate some of the overrun issues can can occur during startup and for burst input.
The text was updated successfully, but these errors were encountered: