[AO][SERVERLESS] Fix Custom Threshold rule tests for Serverless

[fkanout]

Summary

Revert the revert for #166942 💪🏻

Fixes #165569
Fixes #166617
Fixes #166618
Fixes #166619
Fixes #166620

[apmmachine]

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

• /oblt-deploy : Deploy a Kibana instance using the Observability test environments.
• /oblt-deploy-serverless : Deploy a serverless Kibana instance using the Observability test environments.
• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[elasticmachine]

Pinging @elastic/actionable-observability (Team: Actionable Observability)

[maryam-saeidi]

Any particular reason to remove cleaning up the event log index?

[fkanout]

@maryam-saeidi, good question. Actually, in this use case, a.k.a "no_data," we don't ingest any data; hence, there is nothing to delete

[maryam-saeidi]

But, the event log index is related to the result of executing a rule, not the ingested data.

You can check the test server with and without cleaning; you'll see without it, you still have some logs in this index.

[fkanout]

@maryam-saeidi thank you for pointing that out. There is a race condition on this index as we use the wildcard option *, which creates inconsistent results and sometimes throws an error.

"cause":{"type":"version_conflict_engine_exception","reason":"[S_D574oB8qizAIQCLOib]: version conflict, required seqNo [1], primary term [1]. but no document was found","index_uuid":"cZ1A-w-KS1qY34KrbPGHPA","shard":"0","index":".ds-.kibana-event-log-ds-2023.10.02-000001"},"status":409},{"index":".ds-.kibana-event-log-ds-2023.10.02-000001","id":"TPD574oB8qizAIQCMOiB","cause":{"type":"version_conflict_engine_exception","reason":"[TPD574oB8qizAIQCMOiB]: version conflict, required seqNo [2], primary term [1]. but no document was found","index_uuid":"cZ1A-w-KS1qY34KrbPGHPA","shard":"0","index":".ds-.kibana-event-log-ds-2023.10.02-000001"},"status":409}

So, I changed the delete term to use the rule.id instead of the alert.consumer to be more accurate and avoid this issue.

[maryam-saeidi]

Interesting, how does changing the field solve the issue?
The error says there is no data based on the query, so it should be the same case for any field (including rule.id), isn't it?

[fkanout]

@maryam-saeidi No, the error message is before I change it to rule.id

The error says there is no data based on the query,

This is why I removed the deleteByQuery in the beginning, but then I realized there is a racing condition.
If you look at the error type version_conflict_engine_exception. It means there are two operations at the same.

[maryam-saeidi]

This is why I removed the deleteByQuery in the beginning, but then I realized there is a racing condition.
If you look at the error type version_conflict_engine_exception. It means there are two operations at the same.

I still don't understand, either the document exists in the index, which should contain both fields (rule.id and kibana.alert.rule.consumer), or the document does not exist. How does checking one field instead of the other help?
In which case, do we have a document with rule.id that does not contain kibana.alert.rule.consumer?

[fkanout]

@maryam-saeidi, the test files run in parallel. So we could have two test files trying to perform CRUD operations on the same index. As the error message I shared mentioned, version conflict, required seqNo [1], primary term [1].. e.g. A rule is running and still generating events while another after from another file tries to delete everything using the wildcard *. (race condition)

So when I scoped the deletion i.e. every rule/every test file deleted ONLY its generated event using rule.id we
no longer have any issue. We should have relied on the rule.id instead of consumer from the beginning.

Is that make sense?

[maryam-saeidi]

Yes, super clear, thanks.

Then can we bring back cleaning the event log index with the rule.id condition? Otherwise, after this test, we still have data in the event log index.

await esClient.deleteByQuery({
  index: '.kibana-event-log-*',
  query: { term: { 'rule.id': ruleId } },
});

[fkanout]

Yeah, I already got it back!

[fkanout]

@elasticmachine merge upstream

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: 8c77d4f

Metrics [docs]

✅ unchanged

History

• 💚 Build #164499 succeeded bc252eb
• 💔 Build #164435 failed e94cc74
• 💔 Build #164359 failed 81af405
• 💚 Build #164085 succeeded 9bb86f6
• 💔 Build #164014 failed d0dc63b

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @fkanout