Fix parsing of flat object fields with dots in keys #11425

msfroh · 2023-11-30T23:37:17Z

Description

We have a bug where a flat object field with inner fields that contain dots will "push" the dotted name onto the dot-path from the root, but then would just "pop" off the last part of the dotted name.

This change adds more robust support for flat object keys and subkeys that contain dots (i.e. it pops off the entirety of the latest key, regardless of how many dots it contains).

Related Issues

Resolves #11402

Check List

New functionality includes testing.
- All tests pass
~~New functionality has been documented.~~
- ~~New functionality has javadoc added~~
~~[ ] Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)~~
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)
~~[ ] Public documentation issue/PR created~~

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-11-30T23:52:11Z

❌ Gradle check result for 7dbfa8c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2023-11-30T23:59:39Z

Compatibility status:

Checks if related components are compatible with change 46dfd28

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/performance-analyzer.git]

noCharger

Thanks @msfroh for adding UT for JsonToStringXContentParser

server/src/test/java/org/opensearch/common/xcontent/JsonToStringXContentParserTests.java

server/src/main/java/org/opensearch/common/xcontent/JsonToStringXContentParser.java

mingshl · 2023-12-21T19:00:01Z

Thanks @msfroh for taking up this bug fix!!! so the documents can have as many dots as they wants in the document field name and can be mapped as flat-object 😆

We have a bug where a flat object field with inner fields that contain dots will "push" the dotted name onto the dot-path from the root, but then would just "pop" off the last part of the dotted name. This change adds more robust support for flat object keys and subkeys that contain dots (i.e. it pops off the entirety of the latest key, regardless of how many dots it contains). Fixes opensearch-project#11402 Signed-off-by: Michael Froh <froh@amazon.com>

github-actions · 2024-01-04T19:14:03Z

❕ Gradle check result for 46dfd28: UNSTABLE

TEST FAILURES:

      1 org.opensearch.search.SearchWeightedRoutingIT.testShardRoutingWithNetworkDisruption_FailOpenEnabled

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

opensearch-trigger-bot · 2024-01-04T21:14:25Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-11425-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 22b628bacaab56c145538898ca190ffce9778c6e
# Push it to GitHub
git push --set-upstream origin backport/backport-11425-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-11425-to-2.x.

reta · 2024-01-04T21:14:52Z

@msfroh could you please backport to 2.x manually? thank you!

…ct#11425) We have a bug where a flat object field with inner fields that contain dots will "push" the dotted name onto the dot-path from the root, but then would just "pop" off the last part of the dotted name. This change adds more robust support for flat object keys and subkeys that contain dots (i.e. it pops off the entirety of the latest key, regardless of how many dots it contains). Fixes opensearch-project#11402 Signed-off-by: Michael Froh <froh@amazon.com> (cherry picked from commit 22b628b)

msfroh · 2024-01-09T18:39:18Z

@msfroh could you please backport to 2.x manually? thank you!

Backport PR: #11825

…ct#11425) We have a bug where a flat object field with inner fields that contain dots will "push" the dotted name onto the dot-path from the root, but then would just "pop" off the last part of the dotted name. This change adds more robust support for flat object keys and subkeys that contain dots (i.e. it pops off the entirety of the latest key, regardless of how many dots it contains). Fixes opensearch-project#11402 Signed-off-by: Michael Froh <froh@amazon.com> (cherry picked from commit 22b628b)

We have a bug where a flat object field with inner fields that contain dots will "push" the dotted name onto the dot-path from the root, but then would just "pop" off the last part of the dotted name. This change adds more robust support for flat object keys and subkeys that contain dots (i.e. it pops off the entirety of the latest key, regardless of how many dots it contains). Fixes #11402 Signed-off-by: Michael Froh <froh@amazon.com> (cherry picked from commit 22b628b)

…ct#11425) We have a bug where a flat object field with inner fields that contain dots will "push" the dotted name onto the dot-path from the root, but then would just "pop" off the last part of the dotted name. This change adds more robust support for flat object keys and subkeys that contain dots (i.e. it pops off the entirety of the latest key, regardless of how many dots it contains). Fixes opensearch-project#11402 Signed-off-by: Michael Froh <froh@amazon.com>

…ct#11425) We have a bug where a flat object field with inner fields that contain dots will "push" the dotted name onto the dot-path from the root, but then would just "pop" off the last part of the dotted name. This change adds more robust support for flat object keys and subkeys that contain dots (i.e. it pops off the entirety of the latest key, regardless of how many dots it contains). Fixes opensearch-project#11402 Signed-off-by: Michael Froh <froh@amazon.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>

github-actions bot added bug Something isn't working Indexing Indexing, Bulk Indexing and anything related to indexing labels Nov 30, 2023

msfroh force-pushed the flat_object_dot_key_fix branch from 7dbfa8c to 6f1e477 Compare November 30, 2023 23:45

msfroh marked this pull request as ready for review December 1, 2023 00:12

noCharger approved these changes Dec 1, 2023

View reviewed changes

msfroh commented Dec 1, 2023

View reviewed changes

server/src/test/java/org/opensearch/common/xcontent/JsonToStringXContentParserTests.java Show resolved Hide resolved

kkmr reviewed Dec 1, 2023

View reviewed changes

server/src/main/java/org/opensearch/common/xcontent/JsonToStringXContentParser.java Show resolved Hide resolved

reta approved these changes Dec 12, 2023

View reviewed changes

kkmr approved these changes Dec 13, 2023

View reviewed changes

mingshl mentioned this pull request Dec 21, 2023

FlatObjectField will benefit from using IndexOrDocValuesQuery #11537

Open

msfroh force-pushed the flat_object_dot_key_fix branch from 6f1e477 to 46dfd28 Compare January 4, 2024 18:23

github-actions bot added the Search Search query, autocomplete ...etc label Jan 4, 2024

reta self-requested a review January 4, 2024 19:33

reta approved these changes Jan 4, 2024

View reviewed changes

reta merged commit 22b628b into opensearch-project:main Jan 4, 2024
33 checks passed

reta added v3.0.0 Issues and PRs related to version 3.0.0 backport 2.x Backport to 2.x branch v2.12.0 Issues and PRs related to version 2.12.0 labels Jan 4, 2024

opensearch-trigger-bot bot added the backport-failed label Jan 4, 2024

msfroh mentioned this pull request Jan 9, 2024

[Backport 2.x] Fix parsing of flat object fields with dots in keys #11825

Merged

8 tasks

BrewTestBot mentioned this pull request Feb 21, 2024

opensearch 2.12.0 Homebrew/homebrew-core#163463

Merged

reta mentioned this pull request Apr 4, 2024

Fix ClassCastException when using dynamic template with flat_object mapping and dots in field names #13081

Closed

8 tasks

dblock mentioned this pull request Apr 16, 2024

[BUG] Since OpenSearch 2.12, when using flat_object, the error Preview of field's value: 'null' occurs. #13184

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix parsing of flat object fields with dots in keys #11425

Fix parsing of flat object fields with dots in keys #11425

msfroh commented Nov 30, 2023 •

edited

Loading

github-actions bot commented Nov 30, 2023

github-actions bot commented Nov 30, 2023 •

edited

Loading

noCharger left a comment

mingshl commented Dec 21, 2023

github-actions bot commented Jan 4, 2024

opensearch-trigger-bot bot commented Jan 4, 2024

reta commented Jan 4, 2024

msfroh commented Jan 9, 2024

Fix parsing of flat object fields with dots in keys #11425

Fix parsing of flat object fields with dots in keys #11425

Conversation

msfroh commented Nov 30, 2023 • edited Loading

Description

Related Issues

Check List

github-actions bot commented Nov 30, 2023

github-actions bot commented Nov 30, 2023 • edited Loading

Compatibility status:

Incompatible components

Skipped components

Compatible components

noCharger left a comment

Choose a reason for hiding this comment

mingshl commented Dec 21, 2023

github-actions bot commented Jan 4, 2024

opensearch-trigger-bot bot commented Jan 4, 2024

reta commented Jan 4, 2024

msfroh commented Jan 9, 2024

msfroh commented Nov 30, 2023 •

edited

Loading

github-actions bot commented Nov 30, 2023 •

edited

Loading