Don't error in simplify_expressions rule #8957

haohuaijin · 2024-01-23T05:00:19Z

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

before this pr, in simplify_espression we try to fold the expression, but this can result some error, for example

select case 1 when 2 then 4/0 end;

4/0 never reach in runtime.

so in this PR, if we encounter an error during fold expression, return the origin expression, and the error will occur in runtime if it is a real error(not the short-circuit problem).

Note: this pr doesn't solve #8910 , due to #8927

Are these changes tested?

test by existing tests

Are there any user-facing changes?

datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs

datafusion/physical-expr/src/expressions/case.rs

haohuaijin · 2024-01-23T05:28:26Z

cc @jackwener @alamb. This is the first step to solving #8910; the main change in this PR is that it throws an error in runtime, not in simplify_expression. I open #8959 to discuss how to deal with the short-circuit function.
before this pr

❯ select 1/0;
Optimizer rule 'simplify_expressions' failed
caused by
Arrow error: Divide by zero error

after this pr

❯ select 1/0;
Arrow error: Divide by zero error

alamb

Thank you @haohuaijin -- I think this is looking really good to me.

I think we should avoid the clone if possible, add a few more tests, and update the expected error messages, but all in all this PR is looking really close.

cc @jackwener

alamb · 2024-01-23T11:35:12Z

datafusion/optimizer/src/simplify_expressions/simplify_exprs.rs

@@ -138,28 +138,6 @@ mod tests {
        ExprSchemable, JoinType,
    };

-    /// A macro to assert that one string is contained within another with


💯 for reusing the existing one

datafusion/sqllogictest/test_files/arrow_typeof.slt

datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs

datafusion/physical-expr/src/expressions/case.rs

alamb · 2024-01-23T11:54:38Z

datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs

@@ -281,7 +280,13 @@ impl<'a> TreeNodeRewriter for ConstEvaluator<'a> {

    fn mutate(&mut self, expr: Expr) -> Result<Expr> {
        match self.can_evaluate.pop() {
-            Some(true) => Ok(Expr::Literal(self.evaluate_to_scalar(expr)?)),
+            Some(true) => {
+                let lit = self.evaluate_to_scalar(expr.clone());


It seems a real shame that this has to clone the expression even for the common case where there is no error

maybe we could use something like the following:

enum SimplifyResult { // expr was simplifed and contains the new expression Simplified(Expr), // Evalaution encountered an error, contains the original expression SimplifyRuntimeError(DataFusionError, Expr), }

haohuaijin · 2024-01-23T14:02:18Z

Thanks @alamb for the great reviews; I apply your suggestions, re-added the deleted tests and add some new tests.

haohuaijin · 2024-01-23T14:14:22Z

datafusion/sqllogictest/test_files/scalar.slt

 SELECT not(1), not(0)

 query ?B
 SELECT null, not(null)
 ----
 NULL NULL

-query error DataFusion error: Optimizer rule 'simplify_expressions' failed\ncaused by\nInternal error: NOT 'Literal \{ value: Utf8\("hi"\) \}' can't be evaluated because the expression's type is Utf8, not boolean or NULL
+query error


remove the detailed error message because this query return internal error, that make ci failed

❯ select not('hi'); Internal error: NOT 'Literal { value: Utf8("hi") }' can't be evaluated because the expression's type is Utf8, not boolean or NULL. This was likely caused by a bug in DataFusion's code and we would welcome that you file an bug report in our issue tracker ❯ select 1 and 2; Error during planning: Cannot infer common argument type for logical boolean operation Int64 AND Int64

I think the return error message should like And operator, they both only work for boolean value.

track in #8965

alamb

Thank you @haohuaijin -- this PR looks great to me. Thank you

FYI @jackwener

alamb · 2024-01-23T20:04:46Z

datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs

+    #[test]
+    fn test_simplify_divide_zero_by_zero() {
+        // because divide by 0 maybe occur in short-circuit expression
+        // so we should not simplify this, and throw error in runtime


alamb · 2024-01-23T20:05:03Z

datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs

+
+    #[test]
+    fn test_expression_partial_simplify_2() {
+        // (1 > 2) and (4 / 0) -> false


jackwener

A very nice job! Thanks @haohuaijin @alamb ❤️

jackwener · 2024-01-24T03:33:48Z

datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs

+            // Certain expressions such as `CASE` and `COALESCE` are short circuiting
+            // and may not evalute all their sub expressions. Thus if
+            // if any error is countered during simplification, return the original
+            // so that normal evaluation can occur


Don't error in simplify_expressions rule

710f5e4

github-actions bot added physical-expr Physical Expressions optimizer Optimizer rules sqllogictest SQL Logic Tests (.slt) labels Jan 23, 2024

haohuaijin commented Jan 23, 2024

View reviewed changes

datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs Outdated Show resolved Hide resolved

datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs Show resolved Hide resolved

datafusion/physical-expr/src/expressions/case.rs Show resolved Hide resolved

alamb mentioned this pull request Jan 23, 2024

DataFusion weekly project plan (Andrew Lamb) - Jan 22, 2024 #8933

Closed

9 tasks

alamb reviewed Jan 23, 2024

View reviewed changes

haohuaijin added 2 commits January 23, 2024 21:22

apply reviews

e284400

apply reviers

5c575a5

haohuaijin commented Jan 23, 2024

View reviewed changes

haohuaijin mentioned this pull request Jan 23, 2024

NOT operator should not return internal error when args are not boolean value #8965

Closed

haohuaijin requested a review from alamb January 23, 2024 14:18

alamb approved these changes Jan 23, 2024

View reviewed changes

jackwener approved these changes Jan 24, 2024

View reviewed changes

alamb merged commit 78ca43c into apache:main Jan 24, 2024
22 checks passed

haohuaijin deleted the fix_expression branch January 24, 2024 11:32

universalmind303 mentioned this pull request Apr 5, 2024

regression in casting between 35 and 36. #9951

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Don't error in simplify_expressions rule #8957

Don't error in simplify_expressions rule #8957

haohuaijin commented Jan 23, 2024

haohuaijin commented Jan 23, 2024 •

edited

Loading

alamb left a comment

alamb Jan 23, 2024

alamb Jan 23, 2024

haohuaijin commented Jan 23, 2024

haohuaijin Jan 23, 2024

haohuaijin Jan 23, 2024

alamb left a comment

alamb Jan 23, 2024

alamb Jan 23, 2024

jackwener left a comment

jackwener Jan 24, 2024

Don't error in simplify_expressions rule #8957

Don't error in simplify_expressions rule #8957

Conversation

haohuaijin commented Jan 23, 2024

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

haohuaijin commented Jan 23, 2024 • edited Loading

alamb left a comment

Choose a reason for hiding this comment

alamb Jan 23, 2024

Choose a reason for hiding this comment

alamb Jan 23, 2024

Choose a reason for hiding this comment

haohuaijin commented Jan 23, 2024

haohuaijin Jan 23, 2024

Choose a reason for hiding this comment

haohuaijin Jan 23, 2024

Choose a reason for hiding this comment

alamb left a comment

Choose a reason for hiding this comment

alamb Jan 23, 2024

Choose a reason for hiding this comment

alamb Jan 23, 2024

Choose a reason for hiding this comment

jackwener left a comment

Choose a reason for hiding this comment

jackwener Jan 24, 2024

Choose a reason for hiding this comment

haohuaijin commented Jan 23, 2024 •

edited

Loading