Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Increased CTL Dataset Validation on Create/Update/Upsert #2115

Closed
pattisdr opened this issue Dec 23, 2022 · 2 comments
Closed

Add Increased CTL Dataset Validation on Create/Update/Upsert #2115

pattisdr opened this issue Dec 23, 2022 · 2 comments

Comments

@pattisdr
Copy link
Contributor

Is your feature request related to a specific problem?

CTL Datasets could use some increased validation before they are saved to the database. This is coming up when we try to link an existing CTL Dataset to a DatasetConfig and it is missing expected elements.

Describe the solution you'd like

  • Before writing a CTL Dataset to the db, verify that supplied data categories match those in the database
  • Validate against the Fideslang Dataset schema, which requires that an organizational_fides_key and a data_qualifier exist.

Context:
#2096 (comment)
#2113 (comment)

Describe alternatives you've considered, if any

Note this will be a little tricky because this is a crud endpoint which is written in a generic way to work with numerous ctl-resources, beyond just Datasets.

Additional context

Add any other context or screenshots about the feature request here.

@pattisdr
Copy link
Contributor Author

pattisdr commented Jan 4, 2023

Doing some digging here -

I see in the crud routes that are programatically defined, it does look like there's a mapping for the fides lang models. The create route for example will have the resource validated against the appropriate Fideslang model, so Datasets are being validated through this endpoint.

However, the upsert endpoint just has resources as a list of Dictionaries, I assume this is where the issue is. Hopefully this is just a quick change.

@pattisdr
Copy link
Contributor Author

pattisdr commented Jan 4, 2023

That's it! It's upsert specifically that has the issue.

@pattisdr pattisdr self-assigned this Jan 4, 2023
@pattisdr pattisdr changed the title Add Increased CTL Dataset Validation on Create/Update Add Increased CTL Dataset Validation on Create/Update/Upsert Jan 5, 2023
pattisdr added a commit that referenced this issue Jan 8, 2023
- Fix crud upsert endpoints so they validate the request against the appropriate pydantic model.
- When creating, updating, or upserting Datasets, validate that the supplied data categories align with those currently in the db. (Note that this has to happen outside of typical Pydantic validation because we need access to a db session)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants