Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: add recalculateOwnedStreams to check stream ownership if the ring is changed #13103

Merged
merged 11 commits into from
Jun 14, 2024
Merged

feat: add recalculateOwnedStreams to check stream ownership if the ring is changed #13103

merged 11 commits into from
Jun 14, 2024

Conversation

JordanRushing
Copy link
Contributor

@JordanRushing JordanRushing commented Jun 3, 2024 •
edited
Loading

What this PR does / why we need it:

re:

This PR adds a recalculateOwnedStreams service to ingesters which periodically polls in the ingester ring and updates the owned stream counts of each instance.

When Loki ingesters are auto-scaled and run in a zone-aware configuration adding replicas to an ingester statefulset can cause inadvertent stream limiting until old streams are flushed from memory. Using a new limiter that dynamically determines owned streams helps to solve this issue.

Which issue(s) this PR fixes:

N/A

Special notes for your reviewer:

N/A

Checklist

  • Reviewed the CONTRIBUTING.md guide (required)
  • Documentation added
  • Tests updated
  • Title matches the required conventional commits format, see here
    • Note that Promtail is considered to be feature complete, and future development for logs collection will be in Grafana Alloy. As such, feat PRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.
  • Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
  • For Helm chart changes bump the Helm chart version in production/helm/loki/Chart.yaml and update production/helm/loki/CHANGELOG.md and production/helm/loki/README.md. Example PR
  • If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR

@JordanRushing
Update ownedStreamService to check the ingester ring for changes befo…
959d4e0 
…re recalculating owned streams; `go mod tidy && go mod vendor`; update ingester tests

Signed-off-by: JordanRushing <rushing.jordan@gmail.com>
@JordanRushing JordanRushing self-assigned this Jun 3, 2024
@JordanRushing
Refactor ownedStreamService and break out recalculation
ac5a82d 
Signed-off-by: JordanRushing <rushing.jordan@gmail.com>
@JordanRushing
Move recalculateOwnedStreams to the ingester level
9cd3f9c 
Signed-off-by: JordanRushing <rushing.jordan@gmail.com>
@JordanRushing
Clean up instances by removing readRing since recalculateOwnedStreams…
5437852 
… is at the ingester level now; update tests

Signed-off-by: JordanRushing <rushing.jordan@gmail.com>
vlad-diachenko
vlad-diachenko reviewed Jun 6, 2024
View reviewed changes
Copy link
Contributor

@vlad-diachenko vlad-diachenko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the implementation looks really great. just minor things to address

pkg/ingester/instance.go Outdated Show resolved Hide resolved
pkg/ingester/recalculate_owned_streams.go Outdated Show resolved Hide resolved
pkg/ingester/ingester.go Outdated Show resolved Hide resolved
pkg/ingester/ingester.go Outdated Show resolved Hide resolved
@vlad-diachenko
addressed comments on PR
ade3ef8 
Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>
@vlad-diachenko
added tests to cover recalculate_owned_streams.go
115d467 
Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>
@vlad-diachenko
Merge branch 'refs/heads/main' into recalculate-owned-streams-ingeste…
0a9185a 
…r-limit

# Conflicts:
#	pkg/ingester/instance_test.go
@vlad-diachenko
fixed linter and re-generated docs
9880e4f 
Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>
@github-actions github-actions bot added the type/docs Issues related to technical documentation; the Docs Squad uses this label across many repositories label Jun 12, 2024
@vlad-diachenko vlad-diachenko changed the title [WIP] Update ownedStreamService to check the ingester ring for changes before recalculating an instance's owned streams; go mod tidy && go mod vendor; update ingester tests feat: add recalculateOwnedStreams to check stream ownership if the ring is changed Jun 12, 2024
@vlad-diachenko vlad-diachenko marked this pull request as ready for review June 12, 2024 11:29
@vlad-diachenko vlad-diachenko requested a review from a team as a code owner June 12, 2024 11:29
vlad-diachenko
vlad-diachenko approved these changes Jun 12, 2024
View reviewed changes
Copy link
Contributor

@vlad-diachenko vlad-diachenko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

JordanRushing
JordanRushing commented Jun 13, 2024
View reviewed changes
Copy link
Contributor Author

@JordanRushing JordanRushing left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes look great. Should we consider combining owned_streams and recalculate_owned_streams into a single file? I also wonder if we should re-name them ownedStreamsCountService and recalculateOwnedStreamCount for clarity? Not blocking though.

@vlad-diachenko
Copy link
Contributor

Changes look great. Should we consider combining owned_streams and recalculate_owned_streams into a single file? I also wonder if we should re-name them ownedStreamsCountService and recalculateOwnedStreamCount for clarity? Not blocking though.

I would keep these two files separately. that said, we can find better names for it ))) let's discuss it next week and rename it afterward

@vlad-diachenko vlad-diachenko enabled auto-merge (squash) June 14, 2024 07:54
@vlad-diachenko
Copy link
Contributor

merging it to have these changes included in the next weekly release.

@vlad-diachenko vlad-diachenko merged commit e7689b2 into grafana:main Jun 14, 2024
60 checks passed
@loki-gh-app loki-gh-app bot mentioned this pull request Jun 17, 2024
Merged
@vlad-diachenko vlad-diachenko mentioned this pull request Jun 18, 2024
Merged
7 tasks
@loki-gh-app loki-gh-app bot mentioned this pull request Jun 24, 2024
Closed
@JordanRushing JordanRushing mentioned this pull request Jun 25, 2024
Merged
7 tasks
@loki-gh-app loki-gh-app bot mentioned this pull request Jul 1, 2024
Closed
@grafanabot grafanabot mentioned this pull request Jul 2, 2024
Open
This was referenced Jul 3, 2024
Closed
Closed
Closed
@RodrigoCMoraes RodrigoCMoraes mentioned this pull request Jul 12, 2024
Closed
@loki-gh-app loki-gh-app bot mentioned this pull request Jul 15, 2024
Closed
JordanRushing added a commit to JordanRushing/loki that referenced this pull request Jul 15, 2024
@JordanRushing
feat: Ingester Stream Limit Improvements
1c8376b 
Ingester stream limits now take into account "owned streams" and periodically
update when the Ingester ring is changed. Non-owned streams are now also
flushed when this update takes place. The stream limit calculation has also been updated for improved
accuracy in multi-zone ingester deployments.

Relevant PRs:
- grafana#13006
- grafana#13030
- grafana#13232
- grafana#13103
- grafana#13231
- grafana#13254
- grafana#13314
- grafana#13321
@JordanRushing JordanRushing mentioned this pull request Jul 15, 2024
Merged
7 tasks
@loki-gh-app loki-gh-app bot mentioned this pull request Jul 22, 2024
Closed
@cboggs cboggs mentioned this pull request Aug 5, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@vlad-diachenko vlad-diachenko vlad-diachenko approved these changes

Assignees

@JordanRushing JordanRushing

Labels
size/XL type/docs Issues related to technical documentation; the Docs Squad uses this label across many repositories
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@JordanRushing @vlad-diachenko