Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: serve service histograms from aggregated metrics #625

Draft
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

trevorwhitney
Copy link
Contributor

This depends on grafana/loki#13621. Once that is merged, this change allows us to query those aggregated metrics to serve the service landing page histograms.

@trevorwhitney trevorwhitney requested a review from a team as a code owner July 23, 2024 13:07
Copy link
Contributor

@matyax matyax left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👏 👏

How would we handle backwards compatibility?

@trevorwhitney
Copy link
Contributor Author

@matyax I see 2 options

  1. put this behind a feature flag, turn the backend on a few weeks before enabling the frontend
  2. make a second query to the traditional query path if the first one comes back empty (would be more complicated, how to figure out which times are empty?)

const lineFilter = sceneGraph.lookupVariable(VAR_LINE_FILTER, this) as CustomVariable;
const labels = sceneGraph.lookupVariable(VAR_LABELS, this) as AdHocFiltersVariable;

this._subs.add(
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this the correct way to be tracking this state?

private getVizPanel() {
const service = this.service();
const queryString = this.hasSingleServiceSelector()
? `sum by (${LEVEL_VARIABLE_VALUE}) (sum_over_time({__aggregated_metric__=\`${service}\`} | logfmt | unwrap count [$__auto]))`
Copy link
Contributor Author

@trevorwhitney trevorwhitney Jul 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the _aggregated_metric__ query is still getting fields injected as filter statements ie service_name="foo". I'm surprised that would happen since I'm not using the $LOG_STREAM_SELECTOR_EXPR. Where's that coming from, and is there a way to disable it in this case? Or do I need to include __service_name__ in the aggregated log line?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We use service_name as an "AddHocVariable": https://github.com/grafana/explore-logs/blob/main/src/Components/IndexScene/IndexScene.tsx#L180-L189

The way scenes works, it will try to add all those variables as add hoc filters to every query run. We would need to change the variable type in order for this to work. As a workaround for your PR you could change
https://github.com/grafana/explore-logs/blob/main/src/Components/IndexScene/IndexScene.tsx#L182 to something like datasource: { type: 'fake' },.
With that you wouldn't see those service_name labels included, however it would break showing the values in the label dropdowns.

project-words.txt Outdated Show resolved Hide resolved

this._subs.add(
fields.subscribeToState((newState, prevState) => {
if (newState.filters.length !== prevState.filters.length) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Filters may have the same length but different contents. What it's usually done, which is "not great", is comparing serialized strings with JSON.parse().

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Helper function in #633 is being proposed as solution for this

return false;
}

private service(): string {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
private service(): string {
private serviceName(): string {

or

Suggested change
private service(): string {
private getServiceName(): string {

@matyax
Copy link
Contributor

matyax commented Jul 24, 2024

make a second query to the traditional query path if the first one comes back empty (would be more complicated, how to figure out which times are empty?)

I think we need to do something like this, because otherwise the app will not work for older Loki versions.

@svennergr
Copy link
Contributor

make a second query to the traditional query path if the first one comes back empty (would be more complicated, how to figure out which times are empty?)

I think we need to do something like this, because otherwise the app will not work for older Loki versions.

When we move to /detected_fields we would require a different Loki version anyways. I think it's fine to not be backwards compatible in the state we are currently, as long as we support the latest released Loki version. But that's totally up for discussion.

Copy link
Contributor

@svennergr svennergr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fyi, we should release a new version without this change, so just marking this accordingly so it doesn't get merged by accident.

@matyax
Copy link
Contributor

matyax commented Jul 24, 2024

Agree.

@matyax
Copy link
Contributor

matyax commented Aug 2, 2024

@trevorwhitney Do you mind if we clean up the frontend part of this PR or would you prefer that we add comments?

@trevorwhitney
Copy link
Contributor Author

Feel free to clean it up! Thank you

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants