Avoid index errors for deleted tenants

[zalegrala]

What this PR does:

Here we ensure that when Tempo tries to delete a tenant index that has already been deleted, that this is not an error condition, and doesn't not log or count against the metrics. This can happen when an index was deleted because no blocks have been found, but for block paths that still contain spurious objects.

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[zalegrala]

It does not appear to be an error in s3 when you delete an object that is not present, so the s3 implementation has not been updated.

[joe-elliott]

did this have any impact besides metrics/extra logging?

[zalegrala]

Not that I'm thinking of. We still leave the spurious objects in the backend, so perhaps we can have a conversation about cleaning that up also.

[github-actions]

Hello @joe-elliott!
Backport pull requests need to be either:

• Pull requests which address bugs,
• Urgent fixes which need product approval, in order to get merged,
• Docs changes.

Please, if the current pull request addresses a bug fix, label it with the type/bug label.
If it already has the product approval, please add the product-approved label. For docs changes, please add the type/docs label.
If the pull request modifies CI behaviour, please add the type/ci label.
If none of the above applies, please consider removing the backport label and target the next major/minor release.
Thanks!

[github-actions]

The backport to release-v2.2 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new branch
git switch --create backport-2781-to-release-v2.2 origin/release-v2.2
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x e960fd03a8ee4a1208dbf4fe449a7700e61ed225
# When the conflicts are resolved, stage and commit the changes
git add . && git cherry-pick --continue

If you have the GitHub CLI installed:

# Create the PR body template
PR_BODY=$(gh pr view 2781 --json body --template 'Backport e960fd03a8ee4a1208dbf4fe449a7700e61ed225 from #2781{{ "\n\n---\n\n" }}{{ index . "body" }}')
# Push the branch to GitHub and a PR
echo "${PR_BODY}" | gh pr create --title "[release-v2.2] Avoid index errors for deleted tenants" --body-file - --label "type/bug" --label "backport" --base release-v2.2 --milestone release-v2.2 --web

Or, if you don't have the GitHub CLI installed (we recommend you install it!):

# If you don't have the GitHub CLI installed: Push the branch to GitHub and manually create a PR:
git push --set-upstream origin backport-2781-to-release-v2.2
# Remove the local backport branch
git switch main
git branch -D backport-2781-to-release-v2.2

Unless you've used the GitHub CLI above, now create a pull request where the base branch is release-v2.2 and the compare/head branch is backport-2781-to-release-v2.2.