TSDB for Input Packages

[ishleenk17]

We have been adding TSDB fields to OOTB Integrations where the developer defines all the field mappings beforehand.
Opening up this ticket to curate information/brainstorm on how do we deal with TSDB for Input Packages.

There is no default field mappings in Input Packages.
Are we going to provide the user with the option to add the dimension while doing his custom mappings?
Or is the stand to not support TSDB for Input Packages for now?

cc: @lalit-satapathy @rameshelastic

[lalit-satapathy]

We will need a way to have input packages (of type metrics), to have metric_types and dimensions annotated; Similar to currently being done for the integration packages. We can take this up, once the initial integration packages are rolled out to the early TSDB customers.

Let's explore, what are the gating changes in the mean time, for us to enable the same.

[lalit-satapathy]

We had a brief chat among the TSDB team for input package, we would need further discussion. Let me summarise the initial thoughts here:

• Decide which input packages will have TSDB enabled.

  • This would be simple decision. The input packages are of type metrics, can be chosen to have TSDB enabled.

• How to enable TSDB for input package

  • Can input packages follow the same principle as integrations and come pre TSDB enabled in the manifest file?
  • Note: input packages do not have any data streams, is enablement like this supported at a package manifest file?
  • Whether the user should have an option to toggle TSDB for input package
    • Initial thoughts: follow same mechanism as integrations. This should not be an user option.

• How mappings are provided for the input package?

  • Input packages by definition expects user to provide the mapping. Can we expect Fleet to provide a mechanism for the users to provide these additional mappings in the run time?
  • What is the process to define dimensions for metrics ingested by input packages?
  • What is the process to define metric_types for for input packages?
  • Can we have any package level pre-fined mappings for the above or these are defined runtime?

CC: @ruflin

[ishleenk17]

• Note: input packages do not have any data streams, will enablement like this supported at a package manifest file?

Yes, we will have to add it in the package manifest file

• What is the process to define dimensions for metrics ingested by input packages?

Currently in the custom mappings I don't see any option to add dimension, metric_types during custom mappings.

• Can we have any package level pre-fined mappings for the above or these are defined runtime?

There will be some default mapping( for fields which we know are always going to be mapped a certain way). And there will be user mapped fields as well., The way we add dimension for both will vary.

[botelastic]

Hi! We just realized that we haven't looked into this issue in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!