Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve processing OpenSearch data types. Fix using subfields for text type. #2003

Open
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

Yury-Fridlyand
Copy link
Collaborator

Description

See doc for technical details: https://github.com/Bit-Quill/opensearch-project-sql/blob/dev-add-text-type/docs/dev/text-type.md
See also this comment describing some changes.

Please, see team review and discussion in Bit-Quill#299.

Issues Resolved

  1. Pass full types (instances of OpenSearchDataType) though :core module instead of simplified ones (ExprCoreType).
    This unblocks access to important mapping info such as text fields or date formats. This info is required to build proper DSL queries to OpenSearch.
  2. Use text fields instead of hardcoded keyword subfield name.

Check List

  • New functionality includes testing.
    • All tests pass, including unit test, integration test and doctest
  • New functionality has been documented.
    • New functionality has javadoc added
    • New functionality has user manual doc added
  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

…xt` type. (#299)

* Add `text` type.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Simplify implementation.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Add docs.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Add IT.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix type resolution in IT.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Code clean up and comment update.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Minor clean up.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Update `ExprType` and tests.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Minor fix in docs.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Finalize implementation and tests.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Smart subfield selection.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Address PR feedback.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

---------

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>
Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>
Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>
Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>
@codecov
Copy link

codecov bot commented Aug 21, 2023

Codecov Report

Merging #2003 (64bdbc0) into main (6055c7e) will increase coverage by 0.00%.
Report is 1 commits behind head on main.
The diff coverage is 100.00%.

@@            Coverage Diff            @@
##               main    #2003   +/-   ##
=========================================
  Coverage     97.30%   97.30%           
- Complexity     4622     4645   +23     
=========================================
  Files           407      407           
  Lines         11934    11966   +32     
  Branches        827      837   +10     
=========================================
+ Hits          11612    11644   +32     
  Misses          315      315           
  Partials          7        7           
Flag Coverage Δ
sql-engine 97.30% <100.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed Coverage Δ
...in/java/org/opensearch/sql/data/type/ExprType.java 100.00% <100.00%> (ø)
...org/opensearch/sql/data/type/WideningTypeRule.java 100.00% <100.00%> (ø)
...h/sql/opensearch/data/type/OpenSearchDataType.java 100.00% <100.00%> (ø)
...h/sql/opensearch/data/type/OpenSearchDateType.java 100.00% <100.00%> (ø)
...h/sql/opensearch/data/type/OpenSearchTextType.java 100.00% <100.00%> (ø)
...nsearch/data/value/OpenSearchExprValueFactory.java 100.00% <100.00%> (ø)
...search/sql/opensearch/storage/OpenSearchIndex.java 100.00% <100.00%> (ø)
...ript/aggregation/dsl/AggregationBuilderHelper.java 100.00% <100.00%> (ø)
...ensearch/storage/script/core/ExpressionScript.java 100.00% <100.00%> (ø)
...search/storage/script/filter/lucene/LikeQuery.java 100.00% <100.00%> (ø)
... and 2 more

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant