[CT-2600] Add aditional row(s) of metadata to models/sources

[jochemvandooren]

Describe the feature

Enrich dbt-docs by giving the user flexibility of adding additional rows of metadata. The content of the meta is displayed in the catalog, which is very useful. We would like to expand this even further, and be able to add anything we want. For example test results, usage statistics or just more metadata.

See the proposed solution in the image:

The corresponding model would look something like this:

models:
  - name: model_name
    meta:
      meta_foo: meta_bar
    additional_meta:
      - row1_foo: row1_bar
        row1_baz: row1_qux
      - row2_foo: row2_bar
        row2_baz: row2_qux

By providing something like this, users of dbt-docs are able to add anything they would like to models and sources!

Describe alternatives you've considered

There is no good alternative solution. We have tried expanding meta with more key:val pairs, but no new rows are created when doing this. Therefore the content of meta gets too wide.

Additional context

The information shown comes from manifest.json. Currently when adding another property next to meta, e.g. additional_meta as shown in the example, the new property is not included in manifest.json. Therefore this might also require a change to dbt-core.

Another solution would be to include additional_meta inside meta. This way, it can be solved without an additional change in dbt-core.

Who will this benefit?

Any users that want to show more information in the data catalog.

Are you interested in contributing this feature?

Definitely, and we already have an idea on how to implement it if we go for the additional solution to nest additional_meta in meta.

[dbeatty10]

Thanks for reaching out @jochemvandooren !

Our intent is that meta should be the place to add anything (and everything!) you want without the need to create and define any additional meta.

It sounds like the primary issue you are experiencing is the content that is rendered on the docs webpage getting too wide once you have a certain number of meta properties. So the solution would be to update how it displays in those cases.

We won't be able to prioritize that ourselves, we'd review a pull request that updates the display of meta to accommodate a larger number of properties (as long as all meta properties are treated equally and none have any special meaning). Accordingly, I'm labeling this as "help wanted".

[jochemvandooren]

Hi @dbeatty10! Thanks for your response.

I think I didn't describe the feature clear enough. It is true that the content that is rendered on the docs webpage is getting too wide, although that's not the problem I am trying to solve here. What I envision is that we are able to decide what properties are grouped in a row.

We would like to keep meta as is, while providing another property additional_meta, to provide more metadata that has a specific structure in order to show it in a nice manner on the docs webpage. The idea is that inside additional_meta the user can group properties which will be rendered as rows on the docs webpage. This allows for unlimited flexibility, grouping properties in whatever way and adding rows with a limited amount of properties in them.

This doesn't necessarily have to be an additional property. This could also be a sub-property of meta, it would look something like this:

  models:
  - name: model_name
    meta:
      meta_foo: meta_bar
      docs:
        - row1_foo: row1_bar
          row1_baz: row1_qux
        - row2_foo: row2_bar
          row2_baz: row2_qux

Then what would need to happen is the following:

• Everything inside meta will be rendered on the docs webpage, as it is now
• Except, everything in meta.docs will be shown in separate rows, depending on the structure of meta.docs

I hope it is clearer now what I would like to achieve, thanks in advance!

[dbeatty10]

Thanks for clarifying @jochemvandooren!

We are not willing to either add a new additional_meta property or add any special structure to meta at this time. This is because we want to be completely agnostic to the contents of meta and allow users to do whatever they want with its contents. So that means we won't be enabling users to specify which properties are grouped in a row in the dbt-docs website.

Now that I understand your proposal better, I'm going to close this as "not planned" since it doesn't fit with our approach to meta.

However, we would be willing to do the following alternative:

• Update the display of meta to accommodate a larger number of properties (as long as all meta properties are treated equally and none have any special meaning)

If you decide that you are interested in the alternative above, then we can either open a new feature request for it, or we can re-open this one.

[matthieucan]

Hi @dbeatty10,
Thanks for providing your point of view on this feature. I'd love to re-open this issue, as I'm sure we can find a way forward that is satisfying for everyone.

If I understand correctly, your main concern is to avoid enforcing any structure in meta, and any semantics in the content of its fields. I believe @jochemvandooren's proposal addresses this concern - let me expand on some of its properties:

• It is backwards-compatible with anything users have been storing in meta: it adds a way to group fields, but doesn't enforce it. Furthermore, the whole feature could be disabled by default and only enabled on a given model if that model says so in its config.
• It doesn't give any semantics or special meaning to key/value properties of meta: as it has been the case so far, users are free to store what they want without it being interpreted one way or another. I understand the structure can change for a subset of meta, but again we can do this in a way that is compatible with all past and present uses of meta.

Our intent is that meta should be the place to add anything (and everything!) you want without the need to create and define any additional meta.

I think meta is a killer feature, exactly for the reason you outline. Our proposal is one way to play with it, by giving everyone the possibility to enrich their catalogs in creative ways. We have multiple use-cases (test results, usage insights) for it, and I'm sure we will see more of those coming from diverse usages of dbt-docs. Hence I believe our proposal goes in the direction of enabling users to visualize the everything and anything they add to meta.

The implementation is simple, and we're happy to submit a PR and focus on backwards-compatibility and documentation. Please let me know if I failed to understand your concerns - I'd love to discuss and I'm sure we can find a favorable outcome.

[dbeatty10]

If I understand correctly, your main concern is to avoid enforcing any structure in meta, and any semantics in the content of its fields.

Yes, that is correct 🎯

It doesn't give any semantics or special meaning to key/value properties of meta: as it has been the case so far, users are free to store what they want without it being interpreted one way or another.

@matthieucan Can you help me understand how the proposal is not adding any semantics or special meaning to meta.docs? It seems like the proposal is that if a user were to structure the content in meta in a certain way, then it would affect its presentation on the final website. Could it be that I'm misinterpreting this?

[matthieucan]

@matthieucan Can you help me understand how the proposal is not adding any semantics or special meaning to meta.docs? It seems like the proposal is that if a user were to structure the content in meta in a certain way, then it would affect its presentation on the final website. Could it be that I'm misinterpreting this?

You're right! I think the confusion arose from the difference between structure and content. The content of the key/values within that proposed list does not hold any meaning, but indeed the structure does (as in, a list of lists).

I believe that with an appropriate key (e.g. meta_docs), it remains backwards-compatible with any other usage of meta, and it adds so many possibilities to the catalog! We're now using it to display lots of meta information, structured by groups (e.g. test results, usage insights, etc). The implementation was easy and doesn't add much maintenance burden: https://github.com/PicnicSupermarket/dbt-docs/pull/1/files#diff-e1eda06315c59a5e4ea06df4e0ae111da2577a2190cd4212ba5dea84f2ae2d17

Let me know what you think!

[dbeatty10]

Inline responses

The content of the key/values within that proposed list does not hold any meaning, but indeed the structure does (as in, a list of lists)

This is the crucial part -- our intent is that neither the values nor the structure have any meaning to dbt. That way, external systems have full flexibility to supply their own meaning without any additional rules or caveats.

I believe that with an appropriate key (e.g. meta_docs), it remains backwards-compatible with any other usage of meta

In this proposal, meta_docs would be essentially designated as a reserved keyword. Suppose a user already has a key named meta_docs. In this proposal, the behavior of dbt would change for them.

That being said, our decision is not being primarily driven by backwards compatibility concerns -- it is more about maintaining a free-form structure within meta.

Summary

I appreciate the changes you made in your fork -- your efforts in pushing the boundaries are truly commendable! 🤩

While I recognize its value and the innovative thinking behind it, I want to affirm our decision not to adopt it.

The overarching reason for this:

• Our intent is for meta to remain an open-ended collection without any special treatment or predefined structure by dbt.

[matthieucan]

Hi @dbeatty10,
All clear, and thanks a lot for expanding on your reasoning. We're happy to maintain that feature in our fork, maybe it can be useful to others. :)