-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[feat] Support columns_sorted
in row_filters with pageIndex
#3477
Conversation
Signed-off-by: yangjiang <yangjiang@ebay.com>
@@ -79,6 +79,7 @@ object_store = "0.5.0" | |||
ordered-float = "3.0" | |||
parking_lot = "0.12" | |||
parquet = { version = "22.0.0", features = ["arrow", "async"] } | |||
parquet-format = "4.0.0" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we move this logic to arrow-rs
to avoid import this parquet-format
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think that makes sense
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think parquet 23.0.0
will have properly exposed arrow definitions (added by @tustvold in apache/arrow-rs#2626)
Though now that I see this, I see the proposal is probably to move the functions columns_sorted
and check_is_ordered
into the parquet
crate which makes sense to me (and will likely get reviewed by some others with more parquet knowledge)
@Ted-Jiang can you file a ticket / PR to do so?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sure!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
@@ -79,6 +79,7 @@ object_store = "0.5.0" | |||
ordered-float = "3.0" | |||
parking_lot = "0.12" | |||
parquet = { version = "22.0.0", features = ["arrow", "async"] } | |||
parquet-format = "4.0.0" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think parquet 23.0.0
will have properly exposed arrow definitions (added by @tustvold in apache/arrow-rs#2626)
Though now that I see this, I see the proposal is probably to move the functions columns_sorted
and check_is_ordered
into the parquet
crate which makes sense to me (and will likely get reviewed by some others with more parquet knowledge)
@Ted-Jiang can you file a ticket / PR to do so?
marking this PR as a draft pending resolution of @liukun4515 's review: #3477 (comment) |
Closing as this PR is over a year old. Please feel free to reopen it / rebase it if you plan to keep working on it |
Which issue does this PR close?
Closes #3476.
I think we can set true with only one col with pageIndex which is ordered. 🤔
Rationale for this change
What changes are included in this PR?
Are there any user-facing changes?