Sync timing issue for unit tests

[atownley]

Hi,

I finally got around to trying to integrate LMDB as a drop-in replacement for some key/value memory and file stores I've been using. So far, the easy stuff has gone pretty well—apart from some messages about not being able to serialize keys for multi-valued keys. I've solved that for now. I'm really just trying to see how to make it work with the least amount of surgery to the overall architecture. It was essentially designed for this model from the beginning, so it should be fine.

So far, I'm very impressed.

However, the problem I'm having now that I've moved past the testing of the pieces into actually testing the bigger parts is that even using putSync to write in the same transaction, my unit tests – and probably my command line tools once we get there – aren't able to reliably detect when the data has been fully loaded and ready to use. It writes across several database indexes in chunks, and then expects to be able to use the data once the write completes.

Instrumenting the adapters, it's clear that the writes are happening, but not until after the tests have completed. The tests are intended to be driven from the command line, and the store is used by a whole lot of command line tools, so I'm perfectly comfortable dealing with potential thread blocking and all the rest for the moment.

I am surprised that even in sync mode that the writes aren't available to subsequent reads when the methods return, but I guess I understand why this is the case. However, I'm really wondering how this will deal with bulk loads once I get this problem figured out.

I get I'm probably missing something pretty obvious here, but I'm a bit stumped on how to proceed. I tried to see if there was anything similar already posted, but I didn't see anything relevant.

Anyone have any suggestions apart from rewriting 10+ years worth of code?

Cheers,

ast

[kriszyp]

When putSync returns or put resolves, the written data should be visible. If you are able to reproduce a case where that is not true, let me know.
However, one thing to be aware of with multi-threaded (or multi-process) use of lmdb-js is that read transactions (a snapshot of the database) are reused for multiple reads, and once used are reset on a setImmediate callback (next queued event turn). This means that if thread 1 makes a write (and finishes the write), thread 2 make continue to read with an older read snapshot until the next queued reset of the read transaction. If thread 1 makes a write and wants to notify thread 2 of that data, then thread 2, once receiving notification that there is new data (new completed transaction), should call store.resetReadTxn() prior to attempting to read the latest written data.

[atownley]

Playing with it a bit more, it seems to be related to transactions. If I take out the transactions, this doesn't happen. With the transactions, there's a timing issue.

Also somewhat interesting to observe is that with the transactions, it seems like somehow the internal node this references are disrupted. I have many areas where I iterate over arrays using custom code defining an #each where the callback is invoked via the #apply method to set the 'this' pointer to the array element.

There are several cases I observed where the 'this' reference is corrupted.

After reading your reply, I started to wonder about the transactions, because the structure of what I'm doing is that I have a root db with several child dbs acting as indices. I want to make sure these are updated together, hence the transaction.

And there are other scenarios where I do bulk loads of chunks that I either want to commit or revert as a whole, each of which effectively spawns a separate transaction to write each element given the data structure.

I eliminated the transactions, changed nothing else, and the tests pass as I'd expect, with the same results as before. On one hand, this is fantastic news, because it proves my architecture was right since I'd been planning for this all along. I had just thought it would be harder than it was, and, at the time, I'd have had to write the wrappers myself—which was more work than I had the bandwidth for at the time.

If I enable the transactions, all hell breaks loose: the this pointers get corrupted, the transactions don't complete as a synchronous unit, and nothing works.

The way the architecture is, the LMDB aspect is abstracted to effectively two core data structures, a hash and a multi map backed by LMDB. I've poked some holes in this for the transaction support, but otherwise, I'm not quite sure what to do to reproduce or diagnose the underlying issue. I'm certainly happy to help do so, however, because this is the next essential phase in the evolution of my project.

For now, I'm going to go without the explicit transactions because my environment is pretty stable, and I can recover if something really goes wrong. The speed boost alone is worth it to move forward with other things.

However, from a production perspective, the transactions are essential—even if they're in synchronous mode, because actually, given the architecture, I can already segment the writes to an external process queue. Ultimately, that's not what I want, but I can make it work.

I've written C/C++ extensions for Java and Ruby in the dim and distant past, but I've never done it for node, so I'm not quite sure where to start.

What do you suggest?

[kriszyp]

What is this in this context? Are you using sync or async transactions?

[atownley]

Ok, thanks. You got me: operator error on the transactions. I didn't notice the #transaction vs. #transactionSync in the API docs. I have updated it to switch based on the type of write I'm doing, and everything works swimmingly now.

The this pointer is being used inside a callback, e.g.,:

for(var i = 0; i < arr.length; ++i)
{
  cb.apply(arr[i], i);
}

items.each(function() {
  var val = this;
});

During the iteration, the values in the LMDB-backed data structures are being accessed.

However, given that I was using async transactions by mistake, who knows what it was really pointing to as far as the stack frame and local memory assignments.

The superbly awesome news is that it was all down to operator error, and with using the right transaction method, I now have a highly-performant, truly amazing drop-in replacement for my in-memory and alternatively file-backed stores.

You rock, man! Really great work, and my apologies for wasting your time because I couldn't correctly RTFM.

Cheers,

ast

[atownley]

What I've discovered is that there's a pretty subtle bug in the way the query results are being processed. Migrating to LDBM and the fact that it's complaining about invalid keys highlighted the issue. Everything's fine—except my code. I knew it was begging for some refactoring, but I wasn't expecting this.

Again, great work on the library. Thanks for your patience. I'll go fix some bugs of my own now.

Cheers,

ast