-
Notifications
You must be signed in to change notification settings - Fork 758
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Panic when writing Parquet from non-nullable ListArray #385
Comments
I think there may be a more fundamental issue with
Notice
|
Hi @hohav I missed this, thanks for looking further. I'll take a look at this |
#270 fixed the initial behaviour that you observed with the panics, so we correctly roundtrip even though the file is technically incorrect. We do this because we independently count the nulls from the definition, instead of relying on what the metadata says. The issue is with the column writer at https://github.com/apache/arrow-rs/blob/master/parquet/src/column/writer.rs#L471. It effectively says "if a value is not populated, then it's null", which is incorrect with the empty list case. |
Thanks for taking a look. I'm still seeing the initial panic when I update to latest master of arrow-rs, so I don't think #270 fixed it unfortunately. But I think there's something else going on, because I get the same crash from |
I wonder if this is still an issue after the recent work from @tustvold and others to clean up nested struct / null handling? |
I'll check this too |
Closed by #1746 |
Possibly related: #282, #270.
Minimal reproducing code here.
Trying to write a Parquet file containing a variable-length array with non-nullable items results in this panic:
The text was updated successfully, but these errors were encountered: