[get_df] Updating multi-statement logic #5517
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR fixes an issue we've been running into with multi-statements in Presto which is asynchronous. When using the cursor associated with a raw connection using multiple statements, i.e.,
for Presto it wouldn't poll to check that the first statement had finished before executing the second. It seems that some DB-API's have the
nextset
method (though this isn't universal) but reading through code snippets it seems the best way to ensure that multi-statements execute sequentially for all engine types is to simply callfetchall
from the cursor which waits until the statement is done, i.e., here's the logic for Presto.The other option is to call
db_engine_spec.handle_cursor
after every statement (which is what SQL Lab does) which polls when necessary, however this logic is very much configured for SQL Lab queries.Note that when using the
sqlalchemy.engine.base.Connection
(without a cursor) the following would works. I believe the reason is becauseexecute
returns a result set which enforces syncronicity:However this approach isn't viable as we need to use the cursor to execute queries via
db_engine_spec.execute
.to: @graceguo-supercat @michellethomas @mistercrunch @timifasubaa