Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[cli] Add command to generate index/bloom from a data file #903

Merged
merged 14 commits into from
Aug 30, 2021

Conversation

annanay25
Copy link
Contributor

@annanay25 annanay25 commented Aug 23, 2021

What this PR does:

  • Adds a command to generate the index from a data file. To test, run:
go run ./cmd/tempo-cli gen index --backend=local --bucket=./cmd/tempo-cli/test-data/ single-tenant b18beca6-4d7f-4464-9f72-f343e688a4a0

The new index file will be created in the test-data/single-tenant/b18beca6-4d7f-4464-9f72-f343e688a4a0 folder.

Diff the newly generated index with the index-copy present in the same folder. There should be no differences.

  • This PR also adds some diagrams that help with understanding the encoding package. The drawio XML file is also included. The expectation is that the xml and png file will be version controlled but it is not on the official website docs and so it doesn't have to be updated with every PR.

Tempo encoding

Which issue(s) this PR fixes:
Fixes #894

Checklist

  • Tests updated
    • Needs some form of runtime validation as well
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

Signed-off-by: Annanay annanayagarwal@gmail.com

Signed-off-by: Annanay <annanayagarwal@gmail.com>
Signed-off-by: Annanay <annanayagarwal@gmail.com>
cmd/tempo-cli/main.go Outdated Show resolved Hide resolved
cmd/tempo-cli/cmd-gen-bloom-index.go Outdated Show resolved Hide resolved
cmd/tempo-cli/cmd-gen-bloom-index.go Outdated Show resolved Hide resolved
Signed-off-by: Annanay <annanayagarwal@gmail.com>
Signed-off-by: Annanay <annanayagarwal@gmail.com>
Signed-off-by: Annanay <annanayagarwal@gmail.com>
Signed-off-by: Annanay <annanayagarwal@gmail.com>
@annanay25 annanay25 marked this pull request as ready for review August 25, 2021 12:22
Signed-off-by: Annanay <annanayagarwal@gmail.com>
@annanay25
Copy link
Contributor Author

Added a very basic index verifier and I think the first stage of should include just the local backend. As follow ups, we can implement a remote backend reader which will involve implementing an io.Reader for the S3 backend.

@joe-elliott
Copy link
Member

Also, let's improve these docs:

https://github.com/grafana/tempo/blob/main/docs/tempo/website/troubleshooting/bad-blocks.md

now that your PR adds a way to fix blocks!

Signed-off-by: Annanay <annanayagarwal@gmail.com>
Signed-off-by: Annanay <annanayagarwal@gmail.com>
Signed-off-by: Annanay <annanayagarwal@gmail.com>
Signed-off-by: Annanay <annanayagarwal@gmail.com>
Signed-off-by: Annanay <annanayagarwal@gmail.com>
@annanay25 annanay25 changed the title [cli] Add command to generate index from a data file [cli] Add command to generate index/bloom from a data file Aug 30, 2021
Signed-off-by: Annanay <annanayagarwal@gmail.com>
Signed-off-by: Annanay <annanayagarwal@gmail.com>
@annanay25 annanay25 merged commit 6f99d44 into grafana:main Aug 30, 2021
@annanay25 annanay25 deleted the cli-gen-bloom-index branch August 30, 2021 13:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add a cli command to generate bloom & index for a block from data file
3 participants