Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce remote store hash algorithm in customData in IndexMetadata #12986

Merged
merged 1 commit into from
Apr 2, 2024

Conversation

ashking94
Copy link
Member

@ashking94 ashking94 commented Mar 30, 2024

Description

This PR is a sub task of feature request #12567 and being tracked in meta issue - #12589.

This PR consists refactoring to introduce path_hash_algorithm in remote_store customData in IndexMetadata. This will allow user to change the hashing algorithm used for generating the optimised remote store prefix once we add more algorithms. We also add FNV1a hash which we are planning to use for generating hashes for remote store optimised prefix.

Related Issues

Resolves #12985

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
  • Commits are signed per the DCO using --signoff
  • Commit changes are listed out in CHANGELOG.md file (See: Changelog)
  • [ ] Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@github-actions github-actions bot added enhancement Enhancement or improvement to existing feature or request Storage:Performance Storage:Resiliency Issues and PRs related to the storage resiliency v2.14.0 labels Mar 30, 2024
Copy link
Contributor

github-actions bot commented Mar 30, 2024

Compatibility status:

Checks if related components are compatible with change a91ed4d

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/flow-framework.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/performance-analyzer.git]

Copy link
Contributor

✅ Gradle check result for 631694e: SUCCESS

Copy link

codecov bot commented Mar 30, 2024

Codecov Report

Attention: Patch coverage is 86.90476% with 22 lines in your changes are missing coverage. Please review.

Project coverage is 71.41%. Comparing base (b15cb0c) to head (a91ed4d).
Report is 107 commits behind head on main.

Files Patch % Lines
.../org/opensearch/index/remote/RemoteStoreEnums.java 77.08% 11 Missing ⚠️
...h/cluster/metadata/MetadataCreateIndexService.java 80.00% 1 Missing and 1 partial ⚠️
.../main/java/org/opensearch/index/IndexSettings.java 71.42% 0 Missing and 2 partials ⚠️
...ensearch/index/remote/RemoteStorePathStrategy.java 94.59% 1 Missing and 1 partial ⚠️
...rc/main/java/org/opensearch/common/hash/FNV1a.java 92.30% 1 Missing ⚠️
...c/main/java/org/opensearch/index/IndexService.java 0.00% 1 Missing ⚠️
...in/java/org/opensearch/index/shard/IndexShard.java 50.00% 1 Missing ⚠️
...ndex/store/RemoteSegmentStoreDirectoryFactory.java 94.44% 0 Missing and 1 partial ⚠️
...rg/opensearch/index/translog/RemoteFsTranslog.java 94.44% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main   #12986      +/-   ##
============================================
- Coverage     71.42%   71.41%   -0.01%     
- Complexity    59978    60331     +353     
============================================
  Files          4985     5022      +37     
  Lines        282275   284178    +1903     
  Branches      40946    41157     +211     
============================================
+ Hits         201603   202937    +1334     
- Misses        63999    64399     +400     
- Partials      16673    16842     +169     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Signed-off-by: Ashish Singh <ssashish@amazon.com>
Copy link
Contributor

❕ Gradle check result for a91ed4d: UNSTABLE

  • TEST FAILURES:
      1 org.opensearch.remotestore.multipart.RemoteStoreMultipartIT.testAsyncDurabilityThrowsExceptionWhenRestrictSettingTrue

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

@gbbafna gbbafna merged commit 170ea7a into opensearch-project:main Apr 2, 2024
33 of 37 checks passed
ashking94 added a commit to ashking94/OpenSearch that referenced this pull request Apr 23, 2024
shiv0408 pushed a commit to Gaurav614/OpenSearch that referenced this pull request Apr 25, 2024
…search-project#12986)

Signed-off-by: Ashish Singh <ssashish@amazon.com>
Signed-off-by: Shivansh Arora <hishiv@amazon.com>
@ashking94 ashking94 added the backport 2.x Backport to 2.x branch label Apr 26, 2024
opensearch-trigger-bot bot pushed a commit that referenced this pull request Apr 26, 2024
Signed-off-by: Ashish Singh <ssashish@amazon.com>
(cherry picked from commit 170ea7a)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
ashking94 added a commit to ashking94/OpenSearch that referenced this pull request Apr 26, 2024
sachinpkale pushed a commit that referenced this pull request Apr 26, 2024
…) (#13407)

(cherry picked from commit 170ea7a)

Signed-off-by: Ashish Singh <ssashish@amazon.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
harshavamsi pushed a commit to harshavamsi/OpenSearch that referenced this pull request Apr 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport 2.x Backport to 2.x branch enhancement Enhancement or improvement to existing feature or request skip-changelog Storage:Performance Storage:Resiliency Issues and PRs related to the storage resiliency v2.14.0
Projects
Status: ✅ Done
Status: Planned work items
Development

Successfully merging this pull request may close these issues.

[Remote Store] Add support for hashing algorithm extensibility in remote custom data in IndexMetadata
2 participants