Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Ensure parquet schema arg is propagated to IR #19084

Merged
merged 7 commits into from
Oct 3, 2024

Conversation

nameexhaustion
Copy link
Collaborator

Fixes #19081

@github-actions github-actions bot added fix Bug fix python Related to Python Polars rust Related to Rust Polars labels Oct 3, 2024
#[cfg(feature = "serde")]
use serde::{Deserialize, Serialize};

#[derive(Clone, Debug, PartialEq, Eq, Hash)]
#[cfg_attr(feature = "serde", derive(Serialize, Deserialize))]
pub struct ParquetOptions {
pub schema: Option<ArrowSchemaRef>,
pub schema: Option<SchemaRef>,
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change ParquetOptions::schema to hold our native schema type, this reduces excess copy/conversions as that is the type we receive at input, and also the type we give to the IR. The conversion to arrow schema is now done below in to_alp_impl and stored directly to the FileInfo::reader_schema field

Copy link

codecov bot commented Oct 3, 2024

Codecov Report

Attention: Patch coverage is 85.71429% with 1 line in your changes missing coverage. Please review.

Project coverage is 79.76%. Comparing base (35946cf) to head (48bd3a3).
Report is 2 commits behind head on main.

Files with missing lines Patch % Lines
...ates/polars-stream/src/nodes/parquet_source/mod.rs 0.00% 1 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main   #19084   +/-   ##
=======================================
  Coverage   79.76%   79.76%           
=======================================
  Files        1531     1531           
  Lines      208521   208546   +25     
  Branches     2913     2913           
=======================================
+ Hits       166332   166357   +25     
  Misses      41638    41638           
  Partials      551      551           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@ritchie46 ritchie46 merged commit 0cef5b7 into pola-rs:main Oct 3, 2024
25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
fix Bug fix python Related to Python Polars rust Related to Rust Polars
Projects
None yet
Development

Successfully merging this pull request may close these issues.

allow_missing_columns combined with passing schema fails if columns are missing from first file
2 participants