Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: integrate jobs.query and stateless query for faster queries #1337

Merged
merged 13 commits into from
Mar 27, 2024

Conversation

alvarowolfx
Copy link
Contributor

Integrate with jobs.query endpoint for faster small queries. Keep using jobs.getQueryResults when pagination is needed. Fallback to current method of creating a job + jobs.getQueryResults when not possible to use jobs.query.

@product-auto-label product-auto-label bot added size: m Pull request size is medium. api: bigquery Issues related to the googleapis/nodejs-bigquery API. labels Feb 16, 2024
@alvarowolfx alvarowolfx added the owlbot:run Add this label to trigger the Owlbot post processor. label Feb 29, 2024
@gcf-owl-bot gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Feb 29, 2024
@product-auto-label product-auto-label bot added size: l Pull request size is large. and removed size: m Pull request size is medium. labels Mar 11, 2024
@alvarowolfx alvarowolfx changed the title feat: integrate jobs.query and jobless query for faster queries feat: integrate jobs.query and stateless query for faster queries Mar 15, 2024
@alvarowolfx alvarowolfx marked this pull request as ready for review March 18, 2024 19:17
@alvarowolfx alvarowolfx requested review from a team as code owners March 18, 2024 19:17
@alvarowolfx alvarowolfx added the owlbot:run Add this label to trigger the Owlbot post processor. label Mar 18, 2024
@gcf-owl-bot gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Mar 18, 2024
@alvarowolfx alvarowolfx added the owlbot:run Add this label to trigger the Owlbot post processor. label Mar 19, 2024
@gcf-owl-bot gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Mar 19, 2024
@alvarowolfx alvarowolfx added the owlbot:run Add this label to trigger the Owlbot post processor. label Mar 19, 2024
@gcf-owl-bot gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Mar 19, 2024
Copy link
Contributor

@feywind feywind left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Someone with more BQ experience should give this a look too, but I don't see major Node issues :)

src/bigquery.ts Show resolved Hide resolved
src/bigquery.ts Outdated
params: Query['params'],
types: Query['types']
): {
parameterMode: 'positional' | 'named' | undefined;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might be good to pull this type out into an interface, but YMMV.

src/bigquery.ts Outdated
const queryParameters: bigquery.IQueryParameter[] = [];
if (parameterMode === 'named') {
const namedParams = params as {[param: string]: any};
for (const namedParameter in namedParams) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might want to use for (const namedParameter of Object.getOwnPropertyNames(namedParams)) or check each namedParameter against namedParams.hasOwnProperty().

src/bigquery.ts Show resolved Hide resolved
src/bigquery.ts Outdated
let rows: any = [];
if (res.schema && res.rows) {
rows = BigQuery.mergeSchemaWithRows_(res.schema, res.rows, {
wrapIntegers: options.wrapIntegers!, // TODO: fix default value
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above (re: not sure if this was moved from elsewhere or new code) but I wanted to ping in case the TODO isn't intentional.

src/bigquery.ts Outdated
// The Job is important for the `queryAsStream_` method, so a new query
// isn't created each time results are polled for.
options = extend({job}, queryOpts, options);
if (options.timeoutMs) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can just delete, no need for the if.

src/bigquery.ts Show resolved Hide resolved
src/bigquery.ts Outdated
requestId: uuid.v4(),
jobCreationMode: 'JOB_CREATION_OPTIONAL',
};
if (req.maxResults) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Temporary workaround?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gonna remove that

src/job.ts Show resolved Hide resolved
src/logger.ts Outdated Show resolved Hide resolved
src/bigquery.ts Outdated Show resolved Hide resolved
@alvarowolfx alvarowolfx added the owlbot:run Add this label to trigger the Owlbot post processor. label Mar 26, 2024
@gcf-owl-bot gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Mar 26, 2024
@shollyman shollyman removed the request for review from farhan0102 March 26, 2024 17:44
Copy link
Contributor

@shollyman shollyman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like reasonable coverage for the expansion here, thanks for putting it together.

Copy link
Contributor

@feywind feywind left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All still looks fine. I wonder if we have a code style standard for foo_ vs _foo. Not really something specific to this PR.

@alvarowolfx alvarowolfx added the owlbot:run Add this label to trigger the Owlbot post processor. label Mar 27, 2024
@gcf-owl-bot gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Mar 27, 2024
@alvarowolfx alvarowolfx added the owlbot:run Add this label to trigger the Owlbot post processor. label Mar 27, 2024
@gcf-owl-bot gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Mar 27, 2024
@alvarowolfx alvarowolfx merged commit 74aa150 into googleapis:main Mar 27, 2024
15 checks passed
@alex-statsig
Copy link

alex-statsig commented Aug 9, 2024

Hey @alvarowolfx , thanks for contributing this!

I tried using this out but noticed it seems like the types might be incorrect. In particular, when I use jobs.query(query, options) (or jobs.query(query)), it seems to return 3 items of the shape [ResultsArray, {job: IJob}, JobStatistics]. The types indicate the return type is [ResultsArray, IJob] (note in particular that job is not nested under a key "job" for the second value). Does that sound correct to you?

@alex-statsig
Copy link

alex-statsig commented Aug 9, 2024

Also do you know if there is a reason jobTimeoutMs cannot be set? Seems like a nice safety net for runaway queries that is unavailable here (unless timeoutMs implicitly stops the query, but it doesn't seem like it)

edit: Also timeoutMs doesn't seem to behave as I would expect. It seems like its a timeout for the initial request only, but then the option is deleted and it will wait for the results again still which may take much longer? I want to ensure my requests don't exceed 1s, but it seems like for a slow query it will still wait 10+s. The "job not complete" log fires fairly fast, but the results still don't resolve until the query actually finishes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: bigquery Issues related to the googleapis/nodejs-bigquery API. size: l Pull request size is large.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants