[BUG] Space in knn_vector index type index.mapping property name prevents subsequent snapshot to be taken

[axeoman]

Describe the bug

Discovered that second and all subsequent snapshots could not be taken for an index with type: knn_vector, presumably because of incorrect escaping.

OpenSearch complains with an error: org.opensearch.OpenSearchParseException: missing or invalid physical file name [_0_2011_my vector.hnswc if mapping property name contain space, e.g. my vector (my_vector would work fine).

Was able to replicate on OpenSearch 2.15 release on GCP and S3.

Full stack trace:

Jul 11 13:02:01 os-knn-test-1 opensearch[87]: [2024-07-11T13:02:01,536][WARN ][o.o.r.b.BlobStoreRepository] [os-knn-test-1] [opensearch-20240711t125920832386z-frequent/IjyRjHELRT-9SQYOUCq5nQ] failed to delete shard data for shard [knn-index][0] org.opensearch.OpenSearchParseException: missing or invalid physical file name [_0_2011_my vector.hnswc]|        at org.opensearch.index.snapshots.blobstore.BlobStoreIndexShardSnapshot$FileInfo.fromXContent(BlobStoreIndexShardSnapshot.java:344) ~[opensearch-2.15.0.jar:2.15.0]|        at org.opensearch.index.snapshots.blobstore.BlobStoreIndexShardSnapshots.fromXContent(BlobStoreIndexShardSnapshots.java:298) ~[opensearch-2.15.0.jar:2.15.0]|        at org.opensearch.repositories.blobstore.ChecksumBlobStoreFormat.deserialize(ChecksumBlobStoreFormat.java:144) ~[opensearch-2.15.0.jar:2.15.0]|        at org.opensearch.repositories.blobstore.ChecksumBlobStoreFormat.read(ChecksumBlobStoreFormat.java:119) ~[opensearch-2.15.0.jar:2.15.0]|        at org.opensearch.repositories.blobstore.BlobStoreRepository.buildBlobStoreIndexShardSnapshots(BlobStoreRepository.java:3587) ~[opensearch-2.15.0.jar:2.15.0]|        at org.opensearch.repositories.blobstore.BlobStoreRepository$3.doRun(BlobStoreRepository.java:1388) [opensearch-2.15.0.jar:2.15.0]|        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:941) [opensearch-2.15.0.jar:2.15.0]|        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.15.0.jar:2.15.0]|        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]|        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]|        at java.base/java.lang.Thread.run(Thread.java:840) [?:?]|

Related component

Storage:Snapshots

To Reproduce

0. Configure object storage repository

1. Create index with knn_vector

curl -X PUT 'http://localhost:9200/knn-index/' -H 'Content-Type: application/json' -d '
{
  "settings": {
    "index": {
      "knn": true,
      "knn.algo_param.ef_search": 100
    }
  },
  "mappings": {
    "properties": {
      "my vector": {
        "type": "knn_vector",
        "dimension": 2
      }
    }
  }
}
'

2. Add some data

curl -X POST 'http://localhost:9200/_bulk' -H 'Content-Type: application/json' -d '{ "index": { "_index": "knn-index", "_id": "1" } }
   { "my vector": [1.5, 2.5], "price": 12.2 }
   { "index": { "_index": "knn-index", "_id": "2" } }
   { "my vector": [2.5, 3.5], "price": 7.1 }
   { "index": { "_index": "knn-index", "_id": "3" } }
   { "my vector": [3.5, 4.5], "price": 12.9 }
   { "index": { "_index": "knn-index", "_id": "4" } }
   { "my vector": [5.5, 6.5], "price": 1.2 }
   { "index": { "_index": "knn-index", "_id": "5" } }
   { "my vector": [4.5, 5.5], "price": 3.7 }\n 
'

3. Search for data

curl -X GET 'http://localhost:9200/knn-index/_search' -H 'Content-Type: application/json' -d '{
  "size": 2,
  "query": {
    "knn": {
      "my vector": {
        "vector": [2, 3],
        "k": 2
      }
    }
  }
}'

4. Create first snapshot. Should be fine

curl -X GET 'http://localhost:9200/_snapshot/my-s3-repository/my-first-snapshot'

6. Create second snapshot. Failure.

curl -X GET 'http://localhost:9200/_snapshot/my-s3-repository/my-second-snapshot'

7. Observe error in logs, snapshot could not be taken with "missing or invalid physical file name" error:

Jul 11 13:02:01 os-knn-test-1 opensearch[87]: [2024-07-11T13:02:01,536][WARN ][o.o.r.b.BlobStoreRepository] [os-knn-test-1] [opensearch-20240711t125920832386z-frequent/IjyRjHELRT-9SQYOUCq5nQ] failed to delete shard data for shard [knn-index][0] org.opensearch.OpenSearchParseException: missing or invalid physical file name [_0_2011_my vector.hnswc]|        at org.opensearch.index.snapshots.blobstore.BlobStoreIndexShardSnapshot$FileInfo.fromXContent(BlobStoreIndexShardSnapshot.java:344) ~[opensearch-2.15.0.jar:2.15.0]|        at org.opensearch.index.snapshots.blobstore.BlobStoreIndexShardSnapshots.fromXContent(BlobStoreIndexShardSnapshots.java:298) ~[opensearch-2.15.0.jar:2.15.0]|        at org.opensearch.repositories.blobstore.ChecksumBlobStoreFormat.deserialize(ChecksumBlobStoreFormat.java:144) ~[opensearch-2.15.0.jar:2.15.0]|        at org.opensearch.repositories.blobstore.ChecksumBlobStoreFormat.read(ChecksumBlobStoreFormat.java:119) ~[opensearch-2.15.0.jar:2.15.0]|        at org.opensearch.repositories.blobstore.BlobStoreRepository.buildBlobStoreIndexShardSnapshots(BlobStoreRepository.java:3587) ~[opensearch-2.15.0.jar:2.15.0]|        at org.opensearch.repositories.blobstore.BlobStoreRepository$3.doRun(BlobStoreRepository.java:1388) [opensearch-2.15.0.jar:2.15.0]|        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:941) [opensearch-2.15.0.jar:2.15.0]|        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.15.0.jar:2.15.0]|        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]|        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]|        at java.base/java.lang.Thread.run(Thread.java:840) [?:?]|

Expected behavior

All subsequent snapshots should be taken regardless space in the mappings property name or space should be forbidden to use in the property names.

Additional Details

No response

[reta]

I believe this belongs to https://github.com/opensearch-project/k-NN and needs more strict validation of field names. The k-NN codec uses PerFieldKnnVectorsFormat but snapshot / restore has restrictions on file (key/container) names, one of them is character.

[vamshin]

@axeoman thanks for pointing. As a workaround could you avoid having space in the field name. Let us know if you are willing to contribute or we can take up this issue in future release

[axeoman]

@vamshin thanks, yeah, this is exactly the recommendation we gave our customer.
Can't say we have free hands to contribute at the moment, so whoever ready to take this into work - please do :)

[0ctopus13prime]

Hi @axeoman
I'm working on fixing the bug, and you can track it in here
Thank you

[dblock]

Catch All Triage - 1, 2, 3