[FEATURE] Hybrid search and collapse compatibility

[qmauret]

Describe the bug

Using collapse feature in an hybrid search did not collapse documents.

Related component

Search

To Reproduce

I’m trying to combine hybrid search (semantic + keyword) with collapse feature to deduplicate products from same visual.

I have tried collapsed search on a basic search, which works great.

With hybrid search, the behaviour is a bit different. It places products from the same visual in the inner_hits field but did not collapse them (they are still present in the root level of the search results) which is not the expected behaviour.

Anyone’s aware of a problem of compatibility between hybrid and collapse ?

Expected behavior

I expect the same behaviour as performing a collapse on non hybrid search

Additional Details

Host/Environment (please complete the following information):

• OS: AWS
• Version : 2.11

Additional context
Basic search with collapse (working as expected) :

GET /product_1/_search
{
“_source”: {
“includes”: [“_id”, “name”, “category_name”, “visual.id_visual”]
},
“query”: {
“match”: {
“name”: {
“query”: “Ski”
}
}
},
“collapse”: {
“field”: “visual.id_visual”,
“inner_hits”: {
“size”: 1,
“name”: “from_same_visual”,
“sort”: [
{
“_score”: “desc”
}
]
}
}
}

---

Hybrid search with collapse (not working) :

GET /product_1/_search?search_pipeline=search_pipeline
{
“_source”: {
“includes”: [“_id”, “name”, “category_name”, “visual.id_visual”]
},
“query”: {
“hybrid”: {
“queries”: [
{
“neural”: {
“fullname_v”: {
“query_text”: “Ski”,
“model_id”: “xxx”,
“k”: 200
}
}
},
{
“multi_match”: {
“query”: “Ski”,
“type”: “most_fields”,
“fields”: [“category.name^2”, “name^4”, “tags.name^3”],
“fuzziness”: “AUTO”,
“prefix_length”: 0,
“max_expansions”: 10
}
}
]
}
},
“collapse”: {
“field”: “visual.id_visual”,
“inner_hits”: {
“size”: 1,
“name”: “from_same_visual”,
“sort”: [
{
“_score”: “desc”
}
]
}
}
}

[peternied]

[Triage - attendees 1 2 3 4 5 6 7 8]
@opensearch-project/admin Could you transfer this to the neural search repository, this seems related to its functionality.

[martin-gaievski]

@qmauret functionality of collapse is not supported by the hybrid query. Team will look into the feasibility of adding it.

[sonic182]

Hi, I'm having the same issue, for certain customers with products + product variants, it is ugly to have the same result repeated sometimes (eg: for the product size so the image is the same)

Would be nice to have this feature for hybrid queries 👍

[navneet1v]

cc: @minalsha , @vamshin.

One more customer request: https://opensearch.slack.com/archives/C0539F41Z5X/p1722335776347039