Memory limits?

[Llorx]

I've seen than lmdb is a memory mapped database. If I get this correctly, it means that the files are mapped to a memory location and you read-write to those mappings, which they end in the files.

As far as I understand, if you have a 16GB database, then the memory usage would be also 16GB, but I've seen that the files are not increasing the same way as the database files.

Does it have any improvements, like "demapping" and "remapping" different files on-the-fly, to keep the RAM usage low? I'm asking because I may use lmdb-js on a project where RAM could be limited. I mean, can a 16GB database be used within a 512MB device and grow "infinitely"?

Thank you.

[kriszyp]

can a 16GB database be used within a 512MB device and grow "infinitely"?

Yes, it can! In our application/servers we currently have about 150GB of database on a server with 64GB of RAM (and lots of other processes running). More precisely memory mapping is mapping to virtual memory. And on most (64-bit) OSes this has a maximum of 256TB (so not quite infinite, but pretty huge and vastly larger than typical physical RAM amounts). And mapping virtual memory to physical memory is handled by OS, and is an sophisticated dynamic algorithm that balances on-demand loading from disk with read-ahead, caching, paging etc. Essentially memory mapping means that the database is directly accessing the OS file cache memory. And so accessing data in the database is subject to paging. If you access a part of the database that is not in the OS file cache, it will be loaded from disk. And parts of the database that are not accessed for long periods of time are subject to being paged out and removed from the file cache (and would be paged in/read from disk on next access).

Of course if your database is larger than your physical memory, and you have pretty random access to data, you will frequently be accessing pages that aren't cached in physical memory, and will require disk I/O to load, which can significantly reduce performance. This is universally true with all databases though, and is one of the basic constraints of database performance. If you are in this situation, and are concerned about slower disk-bound gets slowing down your main thread, you can use the new async prefetch API to asynchronously preload data to ensure it can be later be accessed quickly.

Does it have any improvements, like "demapping" and "remapping" different files on-the-fly, to keep the RAM usage low?

Actually yes, LMDB does support dynamic remapping of database chunks. You can simply set the remapChunks flag to true (on open) if you want to try this. And this will indeed lead to lower reported memory. However, this can be misleading since it does not typically change the amount of actual/real memory in use for the OS file cache, unmapping just means it won't be attributed to your process. Generally, I would discourage using remapChunks (unless needed to run large DBs on a 32-bit machine, but that is automatic), as it has extra overhead, and it doesn't typically improve management/usage of real physical memory.

[Llorx]

That's a really insightful answer! Thank you!

[Llorx]

Another question is, so in your 150GB db you need to set a 150GB mapSize property, right? I mean, if I set a 1GB mapSize and my database grows to a bunch of GBs, what' can be the problem?

I'm asking because for another project I'm comparing LMDB to SDLite3 with memory-mapped files enabled and they have this explanation:

If mmap_size is set to N then all current implementations map the first N bytes of the database file and use legacy xRead() calls for any content beyond N bytes.

So you always want a map size equal or bigger than the database size. I don't know how LMDB deals with the map size. Is this also true for LMDB or it deals this in another way? How it deals creating a too high map size? Will it create a file with that size or will the file increase over time with usage?

[kriszyp]

Another question is, so in your 150GB db you need to set a 150GB mapSize property, right?

No, we do not. lmdb-js automatically handles setting and growing the mapSize for you.

I mean, if I set a 1GB mapSize and my database grows to a bunch of GBs, what' can be the problem?

If the mapSize is set to 1GB and your database is about to grow beyond 1GB, internally lmdb-js detects this condition and automatically remaps the database to a memory map that is twice as big (so in this case, it would automatically expand the memory map to 2GB). The memory map size is stored in the database itself, and so on restart LMDB uses the latest database's stored map size.

By default, lmdb-js actually starts with a memory map size of 128KB, and then just automatically grows the memory map by doubling map as necessary. So by default, LMDB will always have a memory map that is as big or bigger than the database (with no "legacy" read/writes to worry about). I'd recommend leaving the default, there is no reason to set it any higher, since lmdb-js automatically makes it as big as necessary (in our 150GB we never manually set any map sizes). (These changes are also automatically detected and propagated across processes, if you are running multiple processes as well)

How it deals creating a too high map size?

Since memory maps are automatically grown in response to the actual database size, the memory maps are pretty efficient in terms of using the virtual memory address space. In most 64-bit OSes the maximum virtual memory space is 256TB (a lot!). But if you really need more than that, LMDB actually does have an option, remapChunks that will dynamically map blocks of memory from the database rather than the whole database. This reduces performance though, so I wouldn't recommend using this unless your database really does surpass virtual memory limits.

Will it create a file with that size or will the file increase over time with usage?

On almost all modern OSes, the file size will increase over time based on the number of actual pages that have been written/used in the database. I believe there are some old OS/memory map implementations that allocate the file size based on the memory map size, and I believe MacOS allocates disk space for the full memory map, while it is open. So even in these cases the dynamic memory maps ensure that the map size is never more than the twice the database size to protect against excessive allocation. But in general, your file size should correspond to the actual used database size, less than the full memory map.

[Llorx]

As always, very insightful. Thank you!

[kriszyp]

One more note here, LMDB is not capable of shrinking a database. If you delete a bunch of entries, that space is converted to free space, to be reused by future writes. For most db applications this is fine, writes usually outnumber deletes over time, and free space is typically used quickly. However, if shrinking is really needed, libmdbx (and my corresponding lmdbx-js library) does add support for shrinking database files. However, lmdbx-js does not support automatic memory map growth, so it may be more cumbersome to use for most situations.

[Llorx]

Oh I get it. Nice points to take into account. SQLite has the same problem with memory mapped files when shrinking. It neither has the autogrow feature so has the bad part of both actually.