Skip to content

Commit

Permalink
[SPARK-48709][SQL] Fix varchar type resolution mismatch for DataSourc…
Browse files Browse the repository at this point in the history
…eV2 CTAS

### What changes were proposed in this pull request?

This PR fixes varchar type resolution mismatch for DataSourceV2 CTAS. For example:
```sql
set spark.sql.storeAssignmentPolicy=LEGACY;
CREATE TABLE testcat.ns.t1 (d1 string, d2 varchar(200)) USING parquet;
CREATE TABLE testcat.ns.t2 USING foo as select * from testcat.ns.t1
```
Error message:
```
org.apache.spark.sql.AnalysisException: LEGACY store assignment policy is disallowed in Spark data source V2. Please set the configuration spark.sql.storeAssignmentPolicy to other values.
```

### Why are the changes needed?

Avoid query failures.

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Unit test.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #47082 from wangyum/SPARK-48709.

Authored-by: Yuming Wang <yumwang@ebay.com>
Signed-off-by: Kent Yao <yao@apache.org>
  • Loading branch information
wangyum authored and yaooqinn committed Jun 26, 2024
1 parent 313479c commit e23d69b
Show file tree
Hide file tree
Showing 2 changed files with 12 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -64,10 +64,11 @@ trait V2WriteCommand extends UnaryCommand with KeepAnalyzedQuery with CTEInChild
table.skipSchemaResolution || (query.output.size == table.output.size &&
query.output.zip(table.output).forall {
case (inAttr, outAttr) =>
val inType = CharVarcharUtils.getRawType(inAttr.metadata).getOrElse(inAttr.dataType)
val outType = CharVarcharUtils.getRawType(outAttr.metadata).getOrElse(outAttr.dataType)
// names and types must match, nullability must be compatible
inAttr.name == outAttr.name &&
DataType.equalsIgnoreCompatibleNullability(inAttr.dataType, outType) &&
DataType.equalsIgnoreCompatibleNullability(inType, outType) &&
(outAttr.nullable || !inAttr.nullable)
})
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1739,6 +1739,16 @@ class DataSourceV2SQLSuiteV1Filter
}
}

test("SPARK-48709: varchar resolution mismatch for DataSourceV2 CTAS") {
withSQLConf(
SQLConf.STORE_ASSIGNMENT_POLICY.key -> SQLConf.StoreAssignmentPolicy.LEGACY.toString) {
withTable("testcat.ns.t1", "testcat.ns.t2") {
sql("CREATE TABLE testcat.ns.t1 (d1 string, d2 varchar(200)) USING parquet")
sql("CREATE TABLE testcat.ns.t2 USING foo as select * from testcat.ns.t1")
}
}
}

test("ShowCurrentNamespace: basic tests") {
def testShowCurrentNamespace(expectedCatalogName: String, expectedNamespace: String): Unit = {
val schema = new StructType()
Expand Down

0 comments on commit e23d69b

Please sign in to comment.