Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Iceberg] Refine the partition specs that really need to be checked #22753

Merged
merged 2 commits into from
May 24, 2024

Conversation

hantangwangd
Copy link
Member

@hantangwangd hantangwangd commented May 15, 2024

Description

Iceberg table may contains empty partition specs with data all be deleted or rewritten to another partition spec, so that it may contains some manifest files with spec id of these partition specs just to record the information about data files deletion. If we don't filter them out during judgement, we might in some cases prohibit metadata deletion or predicate thoroughly pushdown, even if it's feasible.

This PR refine the partition specs that really need to be checked in an Iceberg table when determining whether supports metadata deletion or predicate thoroughly pushdown.

Test Plan

  • Test that metadata deletion and predicate thoroughly pushdown could be applied on iceberg table with empty unsupported partition spec
  • Test that metadata deletion and predicate thoroughly pushdown could be applied on iceberg table with unsupported partition spec whose data files are all deleted
  • To do: test that metadata deletion and predicate thoroughly pushdown could be applied on iceberg table with multiple partition specs after whole table's data files rewriting (optimization)

Contributor checklist

  • Please make sure your submission complies with our development, formatting, commit message, and attribution guidelines.
  • PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced.
  • Documented new properties (with its default value), SQL syntax, functions, or other functionality.
  • If release notes are required, they follow the release notes guidelines.
  • Adequate tests were added if applicable.
  • CI passed.

Release Notes

== RELEASE NOTES ==


Iceberg Connector Changes
* Improve the partition specs that must be checked to determine if the partition supports metadata deletion or predicate thoroughly pushdown :pr:`22753`

@steveburnett
Copy link
Contributor

Thanks for the release note entry! A nit rephrasing suggestion to consider:

== RELEASE NOTES ==


Iceberg Connector Changes
* Refine the partition specs that must be checked to determine if the partition supports metadata deletion or predicate thoroughly pushdown :pr:`22753`

@hantangwangd
Copy link
Member Author

@steveburnett Thanks for the suggestion, fixed!

@steveburnett
Copy link
Contributor

A final nit: I'm sorry but I didn't think about the Order of changes recommendations in the Release Notes Guidelines when I commented. Here's another revision suggestion, apologies for the churn:

== RELEASE NOTES ==


Iceberg Connector Changes
* Improve the partition specs that must be checked to determine if the partition supports metadata deletion or predicate thoroughly pushdown :pr:`22753`

@hantangwangd
Copy link
Member Author

@steveburnett So grateful for your help in standardizing the release notes, fixed! I also need to learn the Guidelines :-).

Copy link
Contributor

@ZacBlanco ZacBlanco left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the main change is fine. I do think we should probably add tests for the different table format versions+modes though to verify the expected behavior.

@hantangwangd hantangwangd force-pushed the refine_partition_specs branch 3 times, most recently from 5caced9 to 67e089f Compare May 19, 2024 12:15
@hantangwangd
Copy link
Member Author

@ZacBlanco, to add test case for V2 MOR table, I found that previously we didn't take delete files into consider when executing metadata-deletion/truncate. So I added a commit at the front to fix the behavior of metadata deletion and table truncate, then add the test cases we mentioned above for V2 MOR table. Please take a look when convenient, thanks!

Copy link
Contributor

@ZacBlanco ZacBlanco left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two questions regarding the new change

ZacBlanco
ZacBlanco previously approved these changes May 21, 2024
tdcmeehan
tdcmeehan previously approved these changes May 24, 2024
{
String tableName = "test_empty_partition_spec_table";
try {
// Create a table with on partition
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// Create a table with on partition
// Create a table with no partition

tdcmeehan
tdcmeehan previously approved these changes May 24, 2024
@hantangwangd
Copy link
Member Author

Hi @tdcmeehan, I just special handled some test cases in TestIcebergSmokeRest because of the known Iceberg issue 8756 mentioned by @kiersten-stokes, or else they would fail. Please take a look again when available, thanks!

Copy link
Contributor

@tdcmeehan tdcmeehan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @hantangwangd

@hantangwangd hantangwangd merged commit e17d22d into prestodb:master May 24, 2024
56 checks passed
@hantangwangd hantangwangd deleted the refine_partition_specs branch May 24, 2024 19:22
@wanglinsong wanglinsong mentioned this pull request Jun 25, 2024
36 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants