LUCENE-10408 Better encoding of doc Ids in vectors

[mayya-sharipova]

Better encoding of doc Ids in Lucene91HnswVectorsFormat
for a dense case where all docs have vectors.

Currently we write doc Ids of all documents that have vectors
not very efficiently.
This improve their encoding by for a case when all documents
have vectors, we don't write document IDs, but just write a
single short value – a dense marker.

[jtibshirani]

Great that you are looking into these TODOs while it's easy to make changes to the format (before the 9.1 release). I am curious if you noticed any search performance improvement, or if the motivation is more on space/ memory savings.

[jpountz]

Optimizing for the case when all docs have a value makes sense to me.

for a case when only certain documents have vectors, we do delta encoding of doc Ids.

In the past we rejected changes that would consist of having the data written in a compressed fashion on disk but still uncompressed in memory.

I wonder if it would be a better trade-off to keep ints uncompressed, but read them from disk directly instead of loading giant arrays in memory? Or possibly switch to something like DirectMonotonicReader if it doesn't slow down searches.

[mayya-sharipova]

@jtibshirani @jpountz Thanks for your feedback. I've tried to address in 6bf1aea. I've decided to focus this PR only on optimizing the dense case and keeping the sparse case as was before – uncompressed way.

I wonder if it would be a better trade-off to keep ints uncompressed, but read them from disk directly instead of loading giant arrays in memory? Or possibly switch to something like DirectMonotonicReader if it doesn't slow down searches.

@jpountz Thank you for the suggestion, Adrien. I've put this as TODO in the code to explore. I am also wondering since we use binarySearch on docIds array, would it still be acceptable to have this array on disk? Do we have a precedent for such use case for reading from disk?

[jpountz]

I am also wondering since we use binarySearch on docIds array, would it still be acceptable to have this array on disk?

I think so. It's a bit like the terms index, the access pattern is random, but this data should be small enough that it would generally be in the FS cache.

[jpountz]

Of course if we can have a less random access pattern (e.g. maybe by using exponential search, having skip lists like postings or jump tables like doc values) it would be even better.

[jtibshirani]

The new changes make sense to me (I'm not an expert in these encoding decisions though).

[jtibshirani]

Additional motivation for this PR: it could help with performance of exact search (in #656). When all docs have vectors, we can avoid a binary search in VectorValues#advance.

[mayya-sharipova]

@jpountz @jtibshirani Thank you for your suggestions. Are you ok if we keep this PR as it is now to optimize the dense case only, and explore Adrien's suggestions for a sparse case in subsequent work?

[jtibshirani]

@mayya-sharipova that plan makes sense to me. This looks good to me, I just left some tiny last comments.

[jtibshirani]

Thanks for including me! I'm also fine if you omit me when I'm a reviewer.

[LuXugang]

I wonder if it would be a better trade-off to keep ints uncompressed, but read them from disk directly instead of loading giant arrays in memory? Or possibly switch to something like DirectMonotonicReader if it doesn't slow down searches.

@jpountz, could we use IndexedDISI to store docIds and DirectMonotonicWriter to store ordToDoc mapping like doc values did.

[jpountz]

Yes, something like that sounds like a good fit to store the ordToDoc mapping indeed 👍