[Response Ops][Alerting] Migrate installation of context-specific component templates, index templates & indices to alerting framework #151697

ymao1 · 2023-02-21T12:55:40Z

In #145581 we started installing FAAD resources (behind a feature flag). We also chose a slightly different naming convention than what is used by the rule registry but have since decided that we should use the same convention.

This issue is for:

changing the naming convention of the context-specific component template from alerts-{context}-component-template to .alerts-{context}.alerts-mappings
for rule registry rules, creating the context specific index template (for the default space) on alerting plugin setup if enableFrameworkAlerts: true

ETA: While working on this issue, it was easier to roll in the installation of index templates & concrete write indices in addition to the component templates, so that is also included in the scope of this.

The text was updated successfully, but these errors were encountered:

elasticmachine · 2023-02-21T12:55:42Z

Pinging @elastic/response-ops (Team:ResponseOps)

…ponent templates, index templates and concrete write index to framework for alerts-as-data (#151792) Resolves #151697 ## Summary In a previous [PR](#145581) we started installing a context-specific component templates, index templates and concrete write indices for framework alerts as data when the `xpack.alerting.enableFrameworkAlerts` config flag is set to true. In that PR we used a different naming pattern than what is used by the rule registry for those resources. In this PR, we are aligning the naming of these resources with the rule registry and installing these resources on alerting plugin setup when `enableFrameworkAlerts: true`. If the flag is set to false, the rule registry will continue to handle this resource installation. In this PR we are doing the following: * Registering all rules currently registered with the rule registry with the alerting framework. This registration allows the alerting framework to build context specific component templates. Because this PR only addresses resource installation, rules will continue to be registered with the rule registry. * When `enableFrameworkAlerts: true`: * The framework installs the context specific component template with the following naming convention: `.alerts-{context}.alerts-mappings`. This matches what the rule registry currently installs so the transition should be seamless * The framework installs the context specific index template for the `default` space with the following name: `.alerts-{context}.alerts-default-index-template`. Space awareness will be addressed in a followup PR. This matches the current rule registry naming.This index template will reference (1) ECS component template (if `useEcs: true`), (2) context-specific component template, (3) legacy alert component template and (4) framework component template where the legacy alert component template + framework component template = technical component template (from the rule registry). * The framework creates or updates the concrete write index for the `default` space with the naming convention: `.internal.alerts-{context}.alerts-default-000001`. Space awareness will be addressed in a followup PR. This matches the current rule registry naming. * The installation of the index template & write index differs from the rule registry in that it occurs on alerting plugin start vs the first rule run. * We modified the rule registry resource installer to skip installation of these resources when `enableFrameworkAlerts: true`. In addition, it will wait for the alerting resource installation promise so if a rule runs before its resources are fully initialized, it will wait for initialization to complete before writing. ## To Verify The following rule registry contexts are affected: `observability.apm` `observability.logs` `observability.metrics` `observability.slo` `observability.uptime` `security` For each context, we should verify the following: `Note that if your rule context references the ECS mappings, there may be differences in those mappings between main and this branch depending on whether you're running main with enableFrameworkAlerts true or false. These differences are explained in the summary of this prior PR: #150384 but essentially we're aligning with the latest ECS fields. In the instructions, I suggest running main with enableFrameworkAlerts: true to minimize the differences caused by ECS changes` **While running `main` with `enableFrameworkAlerts: true`:** 1. Get the context specific component template `GET _component_template/.alerts-{context}.alerts-mappings` 2. Create rule for this context that creates an alert and then 3. Get the index template `GET _index_template/.alerts-{context}.alerts-default-index-template` 4. Get the index mapping for the concrete index: `GET .internal.alerts-{context}.alerts-default-000001/_mapping` **While running this branch with `xpack.alerting.enableFrameworkAlerts: true` (with a fresh ES instance):** 5. Get the context specific component template `GET _component_template/.alerts-{context}.alerts-mappings` 6. Get the index template `GET _index_template/.alerts-{context}.alerts-default-index-template` 7. Get the index mapping for the concrete index: `GET .internal.alerts-{context}.alerts-default-000001/_mapping` Note that you should not have to create a rule that generates alerts before seeing these resources installed. **Compare the component templates** Compare 1 and 5. The difference should be: * component template from this branch should have `_meta.managed: true`. This is a flag indicating to the user that these templates are system managed and should not be manually modified. **Compare the index templates** Compare 3 and 6. The differences should be: * index template from this branch should have `managed: true` in the `_meta` fields * index template from this branch should not have a `priority` field. This will be addressed in a followup PR * index template from this branch should be composed of `.alerts-legacy-alert-mappings` and `.alerts-framework-mappings` instead of `.alerts-technical-mappings` but under the hood, these mappings are equivalent. **Compare the index mappings** Compare 4 and 7. The difference should be: * index mappings from this branch should have `_meta.managed: true`. ### Verify that installed resources templates work as expected 1. Run this branch on a fresh ES install with `xpack.alerting.enableFrameworkAlerts: true`. 2. Create a rule in your context that generates alerts. 3. Verify that there are no errors during rule execution. 4. Verify that the alerts show up in your alerts table as expected. 5. (For detection rules only): Run this branch with `xpack.alerting.enableFrameworkAlerts: true` and verify rules in a non-default space continue to create resources on first rule run and run as expected. 6. (For detection rules only): Run this branch with `xpack.alerting.enableFrameworkAlerts: true` and verify rule preview continue to work as expected ### Verify that installed resources templates work with existing rule registry resources. 1. Run `main` or a previous version and create a rule in your context that generates alerts. 2. Using the same ES data, switch to this branch with `xpack.alerting.enableFrameworkAlerts: false` and verify Kibana starts with no rule registry errors and the rule continues to run as expected. 3. Using the same ES data, switch to this branch with `xpack.alerting.enableFrameworkAlerts: true` and verify Kibana starts with no alerting or rule registry errors and the rule continues to run as expected. 4. Verify the alerts show up on the alerts table as expected. 5. (For detection rules only): Run this branch with `xpack.alerting.enableFrameworkAlerts: true` and verify rules in a non-default space continue to create resources on first rule run and run as expected. 6. (For detection rules only): Run this branch with `xpack.alerting.enableFrameworkAlerts: true` and verify rule preview continue to work as expected --------- Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

…ponent templates, index templates and concrete write index to framework for alerts-as-data (elastic#151792) Resolves elastic#151697 ## Summary In a previous [PR](elastic#145581) we started installing a context-specific component templates, index templates and concrete write indices for framework alerts as data when the `xpack.alerting.enableFrameworkAlerts` config flag is set to true. In that PR we used a different naming pattern than what is used by the rule registry for those resources. In this PR, we are aligning the naming of these resources with the rule registry and installing these resources on alerting plugin setup when `enableFrameworkAlerts: true`. If the flag is set to false, the rule registry will continue to handle this resource installation. In this PR we are doing the following: * Registering all rules currently registered with the rule registry with the alerting framework. This registration allows the alerting framework to build context specific component templates. Because this PR only addresses resource installation, rules will continue to be registered with the rule registry. * When `enableFrameworkAlerts: true`: * The framework installs the context specific component template with the following naming convention: `.alerts-{context}.alerts-mappings`. This matches what the rule registry currently installs so the transition should be seamless * The framework installs the context specific index template for the `default` space with the following name: `.alerts-{context}.alerts-default-index-template`. Space awareness will be addressed in a followup PR. This matches the current rule registry naming.This index template will reference (1) ECS component template (if `useEcs: true`), (2) context-specific component template, (3) legacy alert component template and (4) framework component template where the legacy alert component template + framework component template = technical component template (from the rule registry). * The framework creates or updates the concrete write index for the `default` space with the naming convention: `.internal.alerts-{context}.alerts-default-000001`. Space awareness will be addressed in a followup PR. This matches the current rule registry naming. * The installation of the index template & write index differs from the rule registry in that it occurs on alerting plugin start vs the first rule run. * We modified the rule registry resource installer to skip installation of these resources when `enableFrameworkAlerts: true`. In addition, it will wait for the alerting resource installation promise so if a rule runs before its resources are fully initialized, it will wait for initialization to complete before writing. ## To Verify The following rule registry contexts are affected: `observability.apm` `observability.logs` `observability.metrics` `observability.slo` `observability.uptime` `security` For each context, we should verify the following: `Note that if your rule context references the ECS mappings, there may be differences in those mappings between main and this branch depending on whether you're running main with enableFrameworkAlerts true or false. These differences are explained in the summary of this prior PR: elastic#150384 but essentially we're aligning with the latest ECS fields. In the instructions, I suggest running main with enableFrameworkAlerts: true to minimize the differences caused by ECS changes` **While running `main` with `enableFrameworkAlerts: true`:** 1. Get the context specific component template `GET _component_template/.alerts-{context}.alerts-mappings` 2. Create rule for this context that creates an alert and then 3. Get the index template `GET _index_template/.alerts-{context}.alerts-default-index-template` 4. Get the index mapping for the concrete index: `GET .internal.alerts-{context}.alerts-default-000001/_mapping` **While running this branch with `xpack.alerting.enableFrameworkAlerts: true` (with a fresh ES instance):** 5. Get the context specific component template `GET _component_template/.alerts-{context}.alerts-mappings` 6. Get the index template `GET _index_template/.alerts-{context}.alerts-default-index-template` 7. Get the index mapping for the concrete index: `GET .internal.alerts-{context}.alerts-default-000001/_mapping` Note that you should not have to create a rule that generates alerts before seeing these resources installed. **Compare the component templates** Compare 1 and 5. The difference should be: * component template from this branch should have `_meta.managed: true`. This is a flag indicating to the user that these templates are system managed and should not be manually modified. **Compare the index templates** Compare 3 and 6. The differences should be: * index template from this branch should have `managed: true` in the `_meta` fields * index template from this branch should not have a `priority` field. This will be addressed in a followup PR * index template from this branch should be composed of `.alerts-legacy-alert-mappings` and `.alerts-framework-mappings` instead of `.alerts-technical-mappings` but under the hood, these mappings are equivalent. **Compare the index mappings** Compare 4 and 7. The difference should be: * index mappings from this branch should have `_meta.managed: true`. ### Verify that installed resources templates work as expected 1. Run this branch on a fresh ES install with `xpack.alerting.enableFrameworkAlerts: true`. 2. Create a rule in your context that generates alerts. 3. Verify that there are no errors during rule execution. 4. Verify that the alerts show up in your alerts table as expected. 5. (For detection rules only): Run this branch with `xpack.alerting.enableFrameworkAlerts: true` and verify rules in a non-default space continue to create resources on first rule run and run as expected. 6. (For detection rules only): Run this branch with `xpack.alerting.enableFrameworkAlerts: true` and verify rule preview continue to work as expected ### Verify that installed resources templates work with existing rule registry resources. 1. Run `main` or a previous version and create a rule in your context that generates alerts. 2. Using the same ES data, switch to this branch with `xpack.alerting.enableFrameworkAlerts: false` and verify Kibana starts with no rule registry errors and the rule continues to run as expected. 3. Using the same ES data, switch to this branch with `xpack.alerting.enableFrameworkAlerts: true` and verify Kibana starts with no alerting or rule registry errors and the rule continues to run as expected. 4. Verify the alerts show up on the alerts table as expected. 5. (For detection rules only): Run this branch with `xpack.alerting.enableFrameworkAlerts: true` and verify rules in a non-default space continue to create resources on first rule run and run as expected. 6. (For detection rules only): Run this branch with `xpack.alerting.enableFrameworkAlerts: true` and verify rule preview continue to work as expected --------- Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

…ponent templates, index templates and concrete write index to framework for alerts-as-data (#151792) Resolves #151697 ## Summary In a previous [PR](#145581) we started installing a context-specific component templates, index templates and concrete write indices for framework alerts as data when the `xpack.alerting.enableFrameworkAlerts` config flag is set to true. In that PR we used a different naming pattern than what is used by the rule registry for those resources. In this PR, we are aligning the naming of these resources with the rule registry and installing these resources on alerting plugin setup when `enableFrameworkAlerts: true`. If the flag is set to false, the rule registry will continue to handle this resource installation. In this PR we are doing the following: * Registering all rules currently registered with the rule registry with the alerting framework. This registration allows the alerting framework to build context specific component templates. Because this PR only addresses resource installation, rules will continue to be registered with the rule registry. * When `enableFrameworkAlerts: true`: * The framework installs the context specific component template with the following naming convention: `.alerts-{context}.alerts-mappings`. This matches what the rule registry currently installs so the transition should be seamless * The framework installs the context specific index template for the `default` space with the following name: `.alerts-{context}.alerts-default-index-template`. Space awareness will be addressed in a followup PR. This matches the current rule registry naming.This index template will reference (1) ECS component template (if `useEcs: true`), (2) context-specific component template, (3) legacy alert component template and (4) framework component template where the legacy alert component template + framework component template = technical component template (from the rule registry). * The framework creates or updates the concrete write index for the `default` space with the naming convention: `.internal.alerts-{context}.alerts-default-000001`. Space awareness will be addressed in a followup PR. This matches the current rule registry naming. * The installation of the index template & write index differs from the rule registry in that it occurs on alerting plugin start vs the first rule run. * We modified the rule registry resource installer to skip installation of these resources when `enableFrameworkAlerts: true`. In addition, it will wait for the alerting resource installation promise so if a rule runs before its resources are fully initialized, it will wait for initialization to complete before writing. ## To Verify The following rule registry contexts are affected: `observability.apm` `observability.logs` `observability.metrics` `observability.slo` `observability.uptime` `security` For each context, we should verify the following: `Note that if your rule context references the ECS mappings, there may be differences in those mappings between main and this branch depending on whether you're running main with enableFrameworkAlerts true or false. These differences are explained in the summary of this prior PR: #150384 but essentially we're aligning with the latest ECS fields. In the instructions, I suggest running main with enableFrameworkAlerts: true to minimize the differences caused by ECS changes` **While running `main` with `enableFrameworkAlerts: true`:** 1. Get the context specific component template `GET _component_template/.alerts-{context}.alerts-mappings` 2. Create rule for this context that creates an alert and then 3. Get the index template `GET _index_template/.alerts-{context}.alerts-default-index-template` 4. Get the index mapping for the concrete index: `GET .internal.alerts-{context}.alerts-default-000001/_mapping` **While running this branch with `xpack.alerting.enableFrameworkAlerts: true` (with a fresh ES instance):** 5. Get the context specific component template `GET _component_template/.alerts-{context}.alerts-mappings` 6. Get the index template `GET _index_template/.alerts-{context}.alerts-default-index-template` 7. Get the index mapping for the concrete index: `GET .internal.alerts-{context}.alerts-default-000001/_mapping` Note that you should not have to create a rule that generates alerts before seeing these resources installed. **Compare the component templates** Compare 1 and 5. The difference should be: * component template from this branch should have `_meta.managed: true`. This is a flag indicating to the user that these templates are system managed and should not be manually modified. **Compare the index templates** Compare 3 and 6. The differences should be: * index template from this branch should have `managed: true` in the `_meta` fields * index template from this branch should not have a `priority` field. This will be addressed in a followup PR * index template from this branch should be composed of `.alerts-legacy-alert-mappings` and `.alerts-framework-mappings` instead of `.alerts-technical-mappings` but under the hood, these mappings are equivalent. **Compare the index mappings** Compare 4 and 7. The difference should be: * index mappings from this branch should have `_meta.managed: true`. ### Verify that installed resources templates work as expected 1. Run this branch on a fresh ES install with `xpack.alerting.enableFrameworkAlerts: true`. 2. Create a rule in your context that generates alerts. 3. Verify that there are no errors during rule execution. 4. Verify that the alerts show up in your alerts table as expected. 5. (For detection rules only): Run this branch with `xpack.alerting.enableFrameworkAlerts: true` and verify rules in a non-default space continue to create resources on first rule run and run as expected. 6. (For detection rules only): Run this branch with `xpack.alerting.enableFrameworkAlerts: true` and verify rule preview continue to work as expected ### Verify that installed resources templates work with existing rule registry resources. 1. Run `main` or a previous version and create a rule in your context that generates alerts. 2. Using the same ES data, switch to this branch with `xpack.alerting.enableFrameworkAlerts: false` and verify Kibana starts with no rule registry errors and the rule continues to run as expected. 3. Using the same ES data, switch to this branch with `xpack.alerting.enableFrameworkAlerts: true` and verify Kibana starts with no alerting or rule registry errors and the rule continues to run as expected. 4. Verify the alerts show up on the alerts table as expected. 5. (For detection rules only): Run this branch with `xpack.alerting.enableFrameworkAlerts: true` and verify rules in a non-default space continue to create resources on first rule run and run as expected. 6. (For detection rules only): Run this branch with `xpack.alerting.enableFrameworkAlerts: true` and verify rule preview continue to work as expected --------- Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

ymao1 added Feature:Alerting Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) Feature:Alerting/Alerts-as-Data Issues related to Alerts-as-data and RuleRegistry labels Feb 21, 2023

mikecote assigned ymao1 Feb 21, 2023

ymao1 mentioned this issue Feb 27, 2023

[Response Ops][Alerting] Migrate installation of context-specific component templates, index templates and concrete write index to framework for alerts-as-data #151792

Merged

ymao1 changed the title ~~[Response Ops][Alerting] Migrate installation of context-specific component templates to alerting framework~~ [Response Ops][Alerting] Migrate installation of context-specific component templates, index templates & indices to alerting framework Mar 1, 2023

ymao1 closed this as completed in #151792 Mar 8, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Response Ops][Alerting] Migrate installation of context-specific component templates, index templates & indices to alerting framework #151697

[Response Ops][Alerting] Migrate installation of context-specific component templates, index templates & indices to alerting framework #151697

ymao1 commented Feb 21, 2023 •

edited

Loading

elasticmachine commented Feb 21, 2023

[Response Ops][Alerting] Migrate installation of context-specific component templates, index templates & indices to alerting framework #151697

[Response Ops][Alerting] Migrate installation of context-specific component templates, index templates & indices to alerting framework #151697

Comments

ymao1 commented Feb 21, 2023 • edited Loading

elasticmachine commented Feb 21, 2023

ymao1 commented Feb 21, 2023 •

edited

Loading