Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: fixedsizelist in delta #2607

Merged
merged 8 commits into from
Feb 7, 2024
Merged

Conversation

universalmind303
Copy link
Contributor

@universalmind303 universalmind303 commented Feb 7, 2024

fixes the issue of delta converting fsl -> list when writing delta tables.

This PR adds some metadata containing the original field if it was downcast when writing to delta, then compares those fields. If they are different, it applies a projection from the delta field to the original field.

If we need further performance, we'll likely have to read and write directly instead of using delta's arrow/datafusion logic as they have no intention on supporting FixedSizeList.

testdata/sqllogictests/datatypes.slt Outdated Show resolved Hide resolved
crates/datasources/src/native/access.rs Show resolved Hide resolved
crates/datasources/Cargo.toml Outdated Show resolved Hide resolved
@universalmind303 universalmind303 enabled auto-merge (squash) February 7, 2024 23:06
@universalmind303 universalmind303 merged commit 3ec75e6 into main Feb 7, 2024
22 checks passed
@universalmind303 universalmind303 deleted the universalmind303/fsl-delta branch February 7, 2024 23:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants