fix(SQL Lab): hang when result set size is too big #30522
Draft
+134
−1
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
anamitraadhikari
changed the title
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #30522 +/- ##
===========================================
+ Coverage 60.48% 77.32% +16.83%
===========================================
Files 1931 533 -1398
Lines 76236 38559 -37677
Branches 8568 0 -8568
===========================================
- Hits 46114 29814 -16300
+ Misses 28017 8745 -19272
+ Partials 2105 0 -2105
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
anamitraadhikari
changed the title
villebro
requested changes
Oct 8, 2024
| @@ -408,3 +408,4 @@ zipp==3.19.0 | |||
| # via importlib-metadata | |||
| zstandard==0.22.0 | |||
| # via flask-compress | |||
| PyHive==0.7.0 | |||
There was a problem hiding this comment.
Unrelated change?
Suggested change
| PyHive==0.7.0 | |
| PyHive==0.7.0 |
| @@ -955,6 +955,9 @@ class D3TimeFormat(TypedDict, total=False): | |||
| SQLLAB_SAVE_WARNING_MESSAGE = None | |||
| SQLLAB_SCHEDULE_WARNING_MESSAGE = None | |||
|
|
|||
| # Max payload size (MB) for SQL Lab to prevent browser hangs with large results. | |||
| SQL_LAB_PAYLOAD_MAX_MB = None | |||
There was a problem hiding this comment.
To keep the config keys in line with the pre-existing ones above, let's remove the underscore
Suggested change
| SQL_LAB_PAYLOAD_MAX_MB = None | |
| SQLLAB_PAYLOAD_MAX_MB = None |
| @@ -214,7 +215,7 @@ def execute_sql_statement( # pylint: disable=too-many-statements, too-many-loca | |||
| insert_rls = ( | |||
| insert_rls_as_subquery | |||
| if database.db_engine_spec.allows_subqueries | |||
| and database.db_engine_spec.allows_alias_in_select | |||
| and database.db_engine_spec.allows_alias_in_select | |||
There was a problem hiding this comment.
I assume this will be flagged by the linter
Suggested change
| and database.db_engine_spec.allows_alias_in_select | |
| and database.db_engine_spec.allows_alias_in_select |
Comment on lines
+608
to
+610
| if config.get("SQL_LAB_PAYLOAD_MAX_MB"): | ||
| serialized_payload_size = sys.getsizeof(serialized_payload) | ||
| sql_lab_payload_max_mb = config["SQL_LAB_PAYLOAD_MAX_MB"] |
There was a problem hiding this comment.
nit: walrus operator can be used to simplify:
Suggested change
| if config.get("SQL_LAB_PAYLOAD_MAX_MB"): | |
| serialized_payload_size = sys.getsizeof(serialized_payload) | |
| sql_lab_payload_max_mb = config["SQL_LAB_PAYLOAD_MAX_MB"] | |
| if sql_lab_payload_max_mb := config.get("SQL_LAB_PAYLOAD_MAX_MB"): | |
| serialized_payload_size = sys.getsizeof(serialized_payload) |
| logger.info("Result size exceeds the allowed limit.") | ||
| raise SupersetErrorException( | ||
| SupersetError( | ||
| message=f"Result size ({serialized_payload_size / (1024 * 1024):.2f} MB) exceeds the allowed limit of {sql_lab_payload_max_mb} MB.", |
There was a problem hiding this comment.
nit: we're rewriting 1024 * 1024 here and in many places after this. To keep the codebase DRY, let's define it only once as a const.
Comment on lines
+133
to
+137
| def test_execute_sql_statement_exceeds_payload_limit_log_check(mocker: MockerFixture, caplog) -> None: | ||
| """ | ||
| Test for `execute_sql_statements` when the result payload size exceeds the limit, | ||
| and check if the correct log message is captured without raising the exception. | ||
| """ |
There was a problem hiding this comment.
What is the purpose of this test? Shouldn't we always raise an exception if the payload size exceeds the threshold?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
SUMMARY
Users were encountering performance issues in SQL Lab, where large query result payloads caused the browser to hang. This issue was noticeable even when result sets contained only a few rows (e.g., 1-10 rows), but the payload was quite large. The large payloads overwhelmed the browser’s ability to render the data, resulting in crashes or freezing.
BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
Before:
After:
TESTING INSTRUCTIONS
Test by running a query that generates a large-sized result (e.g., > 100 MB), and verify that an appropriate error message is triggered if the result size becomes too large to handle efficiently. The size limit, after which the handling logic is triggered, can be defined in the Superset config file under SQL_LAB_PAYLOAD_MAX_MB.
ADDITIONAL INFORMATION
NA