-
Notifications
You must be signed in to change notification settings - Fork 13.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix display limit in sql lab #5392
Conversation
Can we also remove the setting in the config? |
Also worth pointing out, the documentation/comments in |
@timifasubaa I think so, but I'm not familiar with the history of the variables, especially why |
I see. There have been 2 attempts at limiting query results. One at the query level and the other at the display level. The second attempt is to only get a subset of the results from the results backend to send to the frontend. That's the display limit. As seen here (https://github.com/apache/incubator-superset/blame/a17f7141b75f4aca2458d3190ad92f96840ccbcd/superset/views/core.py#L2480) Now most of the work is done on the backend by modifying the query so it should be fine using the same variable. I recommend we use |
Agreed @timifasubaa, this looks much better. |
There is a small js issue breaking the build rn. A team mate is working on resolving it. Once that's done, I will merge. |
@villebro you'll need to rebase |
Thanks for the help @timifasubaa , should now be good to go. |
Codecov Report
@@ Coverage Diff @@
## master #5392 +/- ##
==========================================
- Coverage 77.15% 77.15% -0.01%
==========================================
Files 44 44
Lines 8892 8891 -1
==========================================
- Hits 6861 6860 -1
Misses 2031 2031
Continue to review full report at Codecov.
|
* Fix display limit in sql lab * Add redundant variable from config.py * Fix test config * Change to SQL_MAX_ROW * Remove last remaining occurence of DISPLAY_SQL_MAX_ROW
I think this isn't right. I got a user saying that their CSV export today was just 1k rows and I was confused as I thought the limit was much higher like 1M for CSVs. I got confused because I read the code and thought "mmh I thought there was 2 different settings for this..." but couldn't find evidence of that since this PR had changed that. So when is In non-async mode, we should apply Sorry this is all a bit confusing and clearly needs to be documented better. We also need to allow the user to know what is happening here and maybe allow them to change their limits, though we shouldn't let them crash their browser too easily. |
@mistercrunch Ok that makes sense. Sounds fairly trivial, if needed I can make this fix. |
@villebro let's see what @timifasubaa and @john-bodley were going with this. The main question is around allowing users to export large CSVs when in async mode. There could be solutions around allowing CSV exports of entire tables instead of queries (capped at say 1M), and forcing users to run the |
Taking a look at this |
@timifasubaa ping |
Sorry for the delay. Currently working on this. |
* Fix display limit in sql lab * Add redundant variable from config.py * Fix test config * Change to SQL_MAX_ROW * Remove last remaining occurence of DISPLAY_SQL_MAX_ROW
Just a comment from a confused user: I was trying queries with ORDER BY and they either were failing (Order by is not allowed in sub-queries message) or just getting things out of order. It took me some time to understand there is this select * from (yourquery) limit SQL_MAX_ROW thing, as I think it is not documented anywhere. Now I set SQL_MAX_ROW to 0 to turn that thing off and I am happy again. |
@ofajardo which database engine were you using? The routine that "squeezes in" the limit clause is different from a database engine to the next. |
Teradata. Maybe the version for teradata needs adjustement? This is how the query after transformation:
two not so good things here: |
Can I get the answer here because I want that only 500 rows are displayed in the table visualization and I could download the original amount of rows returned from the DB. What config values should I use to help me achieve this. @villebro @mistercrunch |
Limits query by
DISPLAY_SQL_MAX_ROW
(default value 1,000) instead ofSQL_MAX_ROW
(default value 1,000,000). Fixes #5390. This could also be fixed by changing the default values ofSQL_MAX_ROW
to something more manageable, say 1,000. Raises the question of the need for having a separateSQL_MAX_ROW
as opposed toDISPLAY_SQL_MAX_ROW
inconfig.py
, as there doesn't seem to be any other reference toSQL_MAX_ROW
in the code base.