Skip to content

Commit

Permalink
[SPARK-48724][SQL][TESTS] Fix incorrect conf settings of `ignoreCorru…
Browse files Browse the repository at this point in the history
…ptFiles` related tests case in `ParquetQuerySuite`

### What changes were proposed in this pull request?

This PR aims to fix incorrect conf settings of `ignoreCorruptFiles` related tests case in `ParquetQuerySuite`. The inner `withSQLConf (SQLConf.IGNORE_CORRUPT_FILES.key -> "false")` will overwrite the outer configuration, making it impossible to test the situation where `sqlConf` is true.

### Why are the changes needed?

Fix test coverage logic.

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Pass GA.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #47088 from wayneguow/parquet_query_suite.

Authored-by: Wei Guo <guow93@gmail.com>
Signed-off-by: yangjie01 <yangjie01@baidu.com>
  • Loading branch information
wayneguow authored and LuciferYang committed Jun 26, 2024
1 parent e23d69b commit a474b88
Showing 1 changed file with 8 additions and 10 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -369,16 +369,14 @@ abstract class ParquetQuerySuite extends QueryTest with ParquetTest with SharedS
}

withSQLConf(SQLConf.IGNORE_CORRUPT_FILES.key -> sqlConf) {
withSQLConf(SQLConf.IGNORE_CORRUPT_FILES.key -> "false") {
val exception = intercept[SparkException] {
testIgnoreCorruptFiles(options)
}.getCause
assert(exception.getMessage().contains("is not a Parquet file"))
val exception2 = intercept[SparkException] {
testIgnoreCorruptFilesWithoutSchemaInfer(options)
}.getCause
assert(exception2.getMessage().contains("is not a Parquet file"))
}
val exception = intercept[SparkException] {
testIgnoreCorruptFiles(options)
}.getCause
assert(exception.getMessage().contains("is not a Parquet file"))
val exception2 = intercept[SparkException] {
testIgnoreCorruptFilesWithoutSchemaInfer(options)
}.getCause
assert(exception2.getMessage().contains("is not a Parquet file"))
}
}
}
Expand Down

0 comments on commit a474b88

Please sign in to comment.