Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Backport 1.7.latest] Fix ensuring we produce valid jsonschema artifacts for manifest, catalog, sources, and run-results #9229

Merged
merged 5 commits into from
Dec 7, 2023
Merged

[Backport 1.7.latest] Fix ensuring we produce valid jsonschema artifacts for manifest, catalog, sources, and run-results #9229

merged 5 commits into from
Dec 7, 2023

Commits on Dec 6, 2023

  1. Drop all_refs=True from jsonschema-ization build process 

    Passing `all_refs=True` makes it so that Everything is a ref, even
the top level schema. In jsonschema land, this essentially makes the
produced artifact not a full schema, but a fractal object to be included
in a schema. Thus when `$id` is passed in, jsonschema tools blow up
because `$id` is for identifying a schema, which we explicitly weren't
creating. The alternative was to drop the inclusion of `$id`. Howver, we're
intending to create a schema, and having an `$id` is recommended best
practice. Additionally since we were intending to create a schema,
not a fractal, it seemed best to create to full schema.
    @QMalcolm
    QMalcolm committed Dec 6, 2023
    Configuration menu
    Copy the full SHA
    4ac8a89 View commit details
    Browse the repository at this point in the history

  2. Explicity produce jsonschemas using DRAFT_2020_12 dialect 

    Previously were were implicitly using the `DRAFT_2020_12` dialect through
mashumaro. It felt wise to begin explicitly specifying this. First, it
is closest in available mashumaro provided dialects to what we produced
pre 1.7. Secondly, if mashumaro changes its default for whatever reason
(say a new dialect is added, and mashumaro moves to that), we don't want
to automatically inherit that.
    @QMalcolm
    QMalcolm committed Dec 6, 2023
    Configuration menu
    Copy the full SHA
    543cdd8 View commit details
    Browse the repository at this point in the history

  3. Begin including schema dialect specification in produced jsonschema 

    In jsonschema's documentation they state
> It's not always easy to tell which draft a JSON Schema is using.
> You can use the $schema keyword to declare which version of the JSON Schema specification the schema is written to.
> It's generally good practice to include it, though it is not required.

and

> For brevity, the $schema keyword isn't included in most of the examples in this book, but it should always be used in the real world.

Basically, to know how to parse a schema, it's important to include what
schema dialect is being used for the schema specification. The change in
this commit ensures we include that information.
    @QMalcolm
    QMalcolm committed Dec 6, 2023
    Configuration menu
    Copy the full SHA
    34b1dc2 View commit details
    Browse the repository at this point in the history

  4. Add change documentation for jsonschema schema production fix

    @QMalcolm
    QMalcolm committed Dec 6, 2023
    Configuration menu
    Copy the full SHA
    8258021 View commit details
    Browse the repository at this point in the history

  5. Regenerate dbt jsonschemas with fixed mashumaro jsonschema production… 

    … process

Specifically we regenerated
* catalog v1
* manifest v11
* run-results v5
* sources v3
using the command `scripts/collect-artifact-schema.py --path schemas`
    @QMalcolm
    QMalcolm committed Dec 6, 2023
    Configuration menu
    Copy the full SHA
    0d5fc3a View commit details
    Browse the repository at this point in the history