Updated TSDB documentation with additional details #5706

agithomas · 2023-03-28T04:51:10Z

Type of change

Enhancement

What does this PR do?

Added useful information for integration developers when working with TSDB.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

Not applicable

Related issues

Closes [TSDB] Update TSDB guidelines document for Integration developers #5705

Screenshots

Not applicable

elasticmachine · 2023-03-28T06:47:03Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-08-23T09:04:08.003+0000
Duration: 107 min 9 sec

Test stats 🧪

Test	Results
Failed	0
Passed	4920
Skipped	10
Total	4930

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2023-03-28T06:47:18Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`453/453`)	💚
Files	96.65% (`779/806`)	👎 -3.35
Classes	96.65% (`779/806`)	👎 -3.35
Methods	91.863% (`7530/8197`)	👎 -8.137
Lines	88.319% (`170879/193480`)	👎 -11.681
Conditionals	100.0% (`0/0`)	💚

agithomas · 2023-03-28T07:25:01Z

/test

lalit-satapathy

LGTM

docs/developer_tsdb_migration_guidelines.md

lalit-satapathy · 2023-04-17T04:19:11Z

@agithomas, Can this PR be merged?

agithomas · 2023-04-17T05:29:35Z

@agithomas, Can this PR be merged?

@lalit-satapathy , @jsoriano can you please approve the PR?

docs/developer_tsdb_migration_guidelines.md

Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>

botelastic · 2023-05-18T05:26:08Z

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

agithomas · 2023-06-06T04:11:02Z

/test

botelastic · 2023-07-06T05:53:28Z

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

botelastic · 2023-08-05T16:14:14Z

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

jsoriano · 2023-08-07T09:28:22Z

/test

jsoriano · 2023-08-07T09:28:31Z

@agithomas @lalit-satapathy should we merge this?

lalit-satapathy · 2023-08-07T11:03:05Z

There may be some other changes required, @agithomas do we need to add any others update here to make it up-to-date? Let's keep this doc up-to-date.

… issue-5705-docs-tsdb

agithomas · 2023-08-11T04:05:30Z

@lalit-satapathy , can you do a re-review ? I have updated the guidelines with the most recent guidelines.

lalit-satapathy

I think we need to re-look into these two sentences:

A field having type flattened cannot be selected as a dimension field and that paragraph.
An enhancement request for Kibana is created to indicate the write index. Until then, refer to the index.time_series.start_time of indices and compare with the current time to identify the write index.

At the end, Can we also refer to a cloud integration also?

docs/developer_tsdb_migration_guidelines.md

agithomas · 2023-08-11T08:11:00Z

At the end, Can we also refer to a cloud integration also?

Added link to AWS Redshift integration at the end.

agithomas · 2023-08-11T12:52:38Z

@lalit-satapathy , are you ok with the documentation so that I can merge?

docs/developer_tsdb_migration_guidelines.md

constanca-m · 2023-08-14T07:11:58Z

docs/developer_tsdb_migration_guidelines.md


+    **Warning:** Choosing an insufficient number of dimension fields may lead to data loss


Can you refer the reader to the testing section and link the TSDB migration test kit?

@constanca-m , can we take that as a separate update to this document? May be you can add that part?

docs/developer_tsdb_migration_guidelines.md

constanca-m · 2023-08-14T07:22:29Z

Git doesn't let me comment on lines that were not changed...

Additional comments:

When a package upgrade with the above mentioned change is applied, the changes are made only on the index template. This means, the user need to wait until index.time_series.end_time of the current write index before seeing the change, following a package upgrade.

Are we sure of this? I only upgrade the package and the automatic rollover happens. The only instance that did not happen was when I modified the look_ahead_time field.

Known issues section suggestion: Let's just reference the meta issue to avoid having to maintain this document.
Reference to existing package already migrated section: Can we only add the meta issue as well for this?
Testing section: Please let's just link the TSDB testing kit, this way of testing is not our current way of doing it.
Best practices section:

Use Lens as the preferred visualisation type.

Is this related to TSDB?

Co-authored-by: Constança Manteigas <113898685+constanca-m@users.noreply.github.com>

agithomas · 2023-08-14T09:12:20Z

Git doesn't let me comment on lines that were not changed...

Additional comments:

When a package upgrade with the above mentioned change is applied, the changes are made only on the index template. This means, the user need to wait until index.time_series.end_time of the current write index before seeing the change, following a package upgrade.

Are we sure of this? I only upgrade the package and the automatic rollover happens. The only instance that did not happen was when I modified the look_ahead_time field.

It was the case earlier. I shall re-test this to validate the relevancy of this point.

Known issues section suggestion: Let's just reference the meta issue to avoid having to maintain this document.

Reference to existing package already migrated section: Can we only add the meta issue as well for this?
Already present. Please refer the section "Other known issues". It will take to the meta issue.

Testing section: Please let's just link the TSDB testing kit, this way of testing is not our current way of doing it.

@constanca-m , you may add this part in a separate PR.

Best practices section:

Use Lens as the preferred visualisation type.

Is this related to TSDB?

Yes, it is. Some of the errors associated with the usage of counter type can only be seen when using Lens visualistion.

agithomas · 2023-08-14T09:19:18Z

4. Testing section: Please let's just link the TSDB testing kit, this way of testing is not our current way of doing it.

I do not think that there exists a fault in comparing the records, especially when the comparison is easy. Record count based check still happens. TSDB testkit can be a supplementary solution.

…ic rollover section

agithomas · 2023-08-23T05:40:54Z

@constanca-m , i have addressed all the review comments, except those related to test kit. Can the change be merged?

constanca-m

Yes! I will likely open a PR then to include the testing kit, but for now the changes here seem enough.

agithomas · 2023-08-23T07:52:31Z

/test

… issue-5705-docs-tsdb

* Updated TSDB documentation with additional details * Updated links to the integration examples * Update docs/developer_tsdb_migration_guidelines.md Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co> * Updated with recent guidelines * Added reference to AWS Redshift * Updated kibana.version guidelines * Update docs/developer_tsdb_migration_guidelines.md Co-authored-by: Constança Manteigas <113898685+constanca-m@users.noreply.github.com> * Updated link to default max limit of TSDB dimensions. Updated Automatic rollover section * Updated link to default max limit of TSDB dimensions. * Updated the link within the document --------- Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co> Co-authored-by: Constança Manteigas <113898685+constanca-m@users.noreply.github.com>

Updated TSDB documentation with additional details

5f76d69

agithomas requested a review from a team as a code owner March 28, 2023 04:51

This was referenced Mar 28, 2023

[Oracle] Modify dimension fields for Oracle Integration #5695

Closed

[TSDB] Update TSDB guidelines document for Integration developers #5705

Closed

[ECS] [TSDB] Centralisation of Dimension Fields #5193

Open

agithomas requested review from jsoriano and lalit-satapathy March 28, 2023 05:01

lalit-satapathy reviewed Apr 2, 2023

View reviewed changes

docs/developer_tsdb_migration_guidelines.md Outdated Show resolved Hide resolved

Updated links to the integration examples

da79ce1

lalit-satapathy approved these changes Apr 17, 2023

View reviewed changes

jsoriano reviewed Apr 17, 2023

View reviewed changes

docs/developer_tsdb_migration_guidelines.md Outdated Show resolved Hide resolved

Update docs/developer_tsdb_migration_guidelines.md

28f4653

Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>

botelastic bot added the Stalled label May 18, 2023

botelastic bot removed the Stalled label Jun 6, 2023

botelastic bot added the Stalled label Jul 6, 2023

jsoriano approved these changes Jul 6, 2023

View reviewed changes

botelastic bot removed the Stalled label Jul 6, 2023

botelastic bot added the Stalled label Aug 5, 2023

botelastic bot removed the Stalled label Aug 7, 2023

agithomas added 2 commits August 11, 2023 01:04

Merge branch 'main' of https://github.com/agithomas/integrations into…

3010f13

… issue-5705-docs-tsdb

Updated with recent guidelines

a65bc8b

lalit-satapathy reviewed Aug 11, 2023

View reviewed changes

docs/developer_tsdb_migration_guidelines.md Show resolved Hide resolved

agithomas added 2 commits August 11, 2023 08:07

Added reference to AWS Redshift

e21f5a7

Updated kibana.version guidelines

9fe020c

constanca-m reviewed Aug 14, 2023

View reviewed changes

docs/developer_tsdb_migration_guidelines.md Show resolved Hide resolved

constanca-m reviewed Aug 14, 2023

View reviewed changes

docs/developer_tsdb_migration_guidelines.md Outdated Show resolved Hide resolved

constanca-m reviewed Aug 14, 2023

View reviewed changes

docs/developer_tsdb_migration_guidelines.md Show resolved Hide resolved

constanca-m reviewed Aug 14, 2023

View reviewed changes

docs/developer_tsdb_migration_guidelines.md Outdated Show resolved Hide resolved

Update docs/developer_tsdb_migration_guidelines.md

9f925d2

Co-authored-by: Constança Manteigas <113898685+constanca-m@users.noreply.github.com>

agithomas added 2 commits August 23, 2023 05:35

Updated link to default max limit of TSDB dimensions. Updated Automat…

afe4780

…ic rollover section

Updated link to default max limit of TSDB dimensions.

d76052b

constanca-m approved these changes Aug 23, 2023

View reviewed changes

agithomas added 2 commits August 23, 2023 09:02

Merge branch 'main' of https://github.com/agithomas/integrations into…

1a44bc4

… issue-5705-docs-tsdb

Updated the link within the document

f7d7d53

agithomas merged commit d808ae0 into elastic:main Aug 23, 2023
4 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Updated TSDB documentation with additional details #5706

Updated TSDB documentation with additional details #5706

agithomas commented Mar 28, 2023 •

edited

Loading

elasticmachine commented Mar 28, 2023 •

edited

Loading

Build stats

Test stats 🧪

elasticmachine commented Mar 28, 2023 •

edited

Loading

agithomas commented Mar 28, 2023

lalit-satapathy left a comment

lalit-satapathy commented Apr 17, 2023

agithomas commented Apr 17, 2023

botelastic bot commented May 18, 2023

agithomas commented Jun 6, 2023

botelastic bot commented Jul 6, 2023

botelastic bot commented Aug 5, 2023

jsoriano commented Aug 7, 2023

jsoriano commented Aug 7, 2023

lalit-satapathy commented Aug 7, 2023

agithomas commented Aug 11, 2023

lalit-satapathy left a comment

agithomas commented Aug 11, 2023

agithomas commented Aug 11, 2023

constanca-m Aug 14, 2023

agithomas Aug 14, 2023

constanca-m commented Aug 14, 2023

agithomas commented Aug 14, 2023

agithomas commented Aug 14, 2023

agithomas commented Aug 23, 2023

constanca-m left a comment

agithomas commented Aug 23, 2023


		Warning: Choosing an insufficient number of dimension fields may lead to data loss

Updated TSDB documentation with additional details #5706

Updated TSDB documentation with additional details #5706

Conversation

agithomas commented Mar 28, 2023 • edited Loading

What does this PR do?

Checklist

How to test this PR locally

Related issues

Screenshots

elasticmachine commented Mar 28, 2023 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

elasticmachine commented Mar 28, 2023 • edited Loading

🌐 Coverage report

agithomas commented Mar 28, 2023

lalit-satapathy left a comment

Choose a reason for hiding this comment

lalit-satapathy commented Apr 17, 2023

agithomas commented Apr 17, 2023

botelastic bot commented May 18, 2023

agithomas commented Jun 6, 2023

botelastic bot commented Jul 6, 2023

botelastic bot commented Aug 5, 2023

jsoriano commented Aug 7, 2023

jsoriano commented Aug 7, 2023

lalit-satapathy commented Aug 7, 2023

agithomas commented Aug 11, 2023

lalit-satapathy left a comment

Choose a reason for hiding this comment

agithomas commented Aug 11, 2023

agithomas commented Aug 11, 2023

constanca-m Aug 14, 2023

Choose a reason for hiding this comment

agithomas Aug 14, 2023

Choose a reason for hiding this comment

constanca-m commented Aug 14, 2023

agithomas commented Aug 14, 2023

agithomas commented Aug 14, 2023

agithomas commented Aug 23, 2023

constanca-m left a comment

Choose a reason for hiding this comment

agithomas commented Aug 23, 2023

agithomas commented Mar 28, 2023 •

edited

Loading

elasticmachine commented Mar 28, 2023 •

edited

Loading

elasticmachine commented Mar 28, 2023 •

edited

Loading