Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Squash GQL result logging #540

Closed
MattWellie opened this issue Sep 7, 2023 · 0 comments · Fixed by #650
Closed

Squash GQL result logging #540

MattWellie opened this issue Sep 7, 2023 · 0 comments · Fixed by #650
Assignees

Comments

@MattWellie
Copy link
Contributor

Issue

Our use of the gql library is resulting in all our sample-metadata results being written to log files on query execution.

https://github.com/populationgenomics/sample-metadata/blob/e94a4d8c1b8d33d439fb7abf0e38128911c0689b/metamist/graphql/__init__.py#L135

Can I gauge thoughts on adding an argument to query/query_async which adjusts the logging sensitivity?

By default the gql.client.execute module obtains a logger without setting an explicit level. A lot of the code we run is done with a logger keyed to INFO, which is resulting in over a MB of sample-metadata results being farted into the Hail logs every time a metadata query is executed. This pipeline invocation totalled 1.7MB of just metamist results.

Probably the lines responsible:
https://github.com/graphql-python/gql/blob/d4c975198ad617265e4abb4716e5643f31b75e08/gql/transport/requests.py#L225-L227

I've put a don't log this escape around gql calls before e.g. here, but might be a good idea to implement something similar upstream in this library. This feels like a pretty ghetto solution but could be effective:

import logging
from typing import Bool, Dict
...

def query(
    _query: str | DocumentNode, variables: Dict = None, client: Client = None, log_reponse: Bool = False
) -> Dict[str, Any]:
    """Query the metamist GraphQL API"""
    if variables is None:
        variables = {}
    
    current_level = logging.root.level  # even if not set this defaults to WARN(30) on logging import
    if current_level >= logging.INFO and not log_response:
        logging.getLogger().setLevel(logging.WARN)
    response = (client or configure_sync_client()).execute_sync(
        _query if isinstance(_query, DocumentNode) else gql(_query),
        variable_values=variables,
    )
    # reset logger to baseline
    logging.getLogger().setLevel(current_level)
    return response

A decorator to accomplish the same would be way sexier, but I can't put one of those together in an Issue text box

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants