Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Fleet] Install final pipeline #100973

Merged
merged 7 commits into from
Jun 3, 2021
Merged

[Fleet] Install final pipeline #100973

merged 7 commits into from
Jun 3, 2021

Conversation

nchaulet
Copy link
Member

@nchaulet nchaulet commented May 31, 2021 •
edited
Loading

Summary

Resolve #97181

To ensure the agent do not tamper agent id when sending event, we are going to use a final pipeline that check the agent id against the agent id contained in the api key.

For that:

  • we create a pipeline .fleet_final_pipeline during the fleet setup
  • we use that final pipeline on every index template installed by Fleet

Not done in that PR:

  • we do not update the existing index template installed by fleet without final_pipeline, should we migrate these index?

Decisions made in that PR:

  • If a template define a final_pipeline we will throw an error when installing it.

What to test?

every event ingested with Elastic agent should have the field event.agent_id_status.

@nchaulet nchaulet added v8.0.0 release_note:skip Skip the PR/issue when compiling release notes Team:Fleet Team label for Observability Data Collection Fleet team v7.14.0 labels May 31, 2021
@nchaulet nchaulet self-assigned this May 31, 2021
@nchaulet nchaulet marked this pull request as ready for review May 31, 2021 13:24
@nchaulet nchaulet requested a review from a team as a code owner May 31, 2021 13:24
@elasticmachine
Copy link
Contributor

Pinging @elastic/fleet (Team:Fleet)

@nchaulet nchaulet requested a review from andrewkroh June 1, 2021 12:39
@nchaulet nchaulet requested a review from kpollich June 1, 2021 14:40
@jen-huang jen-huang changed the title [Fleet] Fleet final pipeline [Fleet] Install final pipeline Jun 1, 2021
@P1llus
Copy link
Member

P1llus commented Jun 2, 2021

Just a small question/comment on this approach. Does this mean that final_pipeline will be set for all indices used by packages/integrations?
If so, this will then remove the possibility for end-users to apply a pipeline to overwrite module fields in any way.

@nchaulet
Copy link
Member Author

nchaulet commented Jun 2, 2021

Just a small question/comment on this approach. Does this mean that final_pipeline will be set for all indices used by packages/integrations?
If so, this will then remove the possibility for end-users to apply a pipeline to overwrite module fields in any way.

Yes with that PR the final_pipeline will be used for all indices, user will still be able to define a pipeline to overwrite module fields, but not a final_pipeline

@nchaulet
Copy link
Member Author

nchaulet commented Jun 2, 2021

@elasticmachine merge upstream

@nchaulet nchaulet added the auto-backport Deprecated - use backport:version if exact versions are needed label Jun 2, 2021
andrewkroh
andrewkroh approved these changes Jun 2, 2021
View reviewed changes
Copy link
Member

@andrewkroh andrewkroh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

x-pack/test/fleet_api_integration/apis/epm/final_pipeline.ts Outdated Show resolved Hide resolved
@nchaulet nchaulet enabled auto-merge (squash) June 3, 2021 12:31
@nchaulet
Copy link
Member Author

nchaulet commented Jun 3, 2021

@elasticmachine merge upstream

@kibanamachine
Copy link
Contributor

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @nchaulet

@nchaulet nchaulet merged commit 3b1e8b0 into elastic:master Jun 3, 2021
@kibanamachine kibanamachine mentioned this pull request Jun 3, 2021
Merged
kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Jun 3, 2021
@nchaulet @kibanamachine
[Fleet] Install final pipeline (elastic#100973)
94f9b0e
@kibanamachine
Copy link
Contributor

💚 Backport successful

Status Branch Result
✅ 7.x

This backport PR will be merged automatically after passing CI.

@nchaulet nchaulet mentioned this pull request Jun 3, 2021
Closed
kibanamachine added a commit that referenced this pull request Jun 3, 2021
@kibanamachine @nchaulet
[Fleet] Install final pipeline (#100973) (#101299)
1609e13 
Co-authored-by: Nicolas Chaulet <nicolas.chaulet@elastic.co>
@andrewkroh andrewkroh mentioned this pull request Jun 21, 2021
Merged
1 task
andrewkroh added a commit that referenced this pull request Jun 21, 2021
@andrewkroh
[Fleet] Update final pipeline based on ECS event.agent_id_status (#10…
2e3d527 
…2805)

This updates the Fleet final pipeline added in #100973 to match the specification of
`event.agent_id_status` field as defined in ECS. The field was added to ECS in
elastic/ecs#1454. Basically the values of the field were simplified
from what was originally proposed and implemented.
kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Jun 21, 2021
@andrewkroh @kibanamachine
[Fleet] Update final pipeline based on ECS event.agent_id_status (ela…
29979da 
…stic#102805)

This updates the Fleet final pipeline added in elastic#100973 to match the specification of
`event.agent_id_status` field as defined in ECS. The field was added to ECS in
elastic/ecs#1454. Basically the values of the field were simplified
from what was originally proposed and implemented.
kibanamachine added a commit that referenced this pull request Jun 22, 2021
@kibanamachine @andrewkroh
[Fleet] Update final pipeline based on ECS event.agent_id_status (#10…
7f7ba0b 
…2805) (#102832)

This updates the Fleet final pipeline added in #100973 to match the specification of
`event.agent_id_status` field as defined in ECS. The field was added to ECS in
elastic/ecs#1454. Basically the values of the field were simplified
from what was originally proposed and implemented.

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>
@nchaulet nchaulet mentioned this pull request Aug 2, 2021
Closed
@ruflin ruflin mentioned this pull request Jun 17, 2022
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@andrewkroh andrewkroh andrewkroh approved these changes

@kpollich kpollich kpollich approved these changes

Assignees

@nchaulet nchaulet

Labels
auto-backport Deprecated - use backport:version if exact versions are needed release_note:skip Skip the PR/issue when compiling release notes Team:Fleet Team label for Observability Data Collection Fleet team v7.14.0 v8.0.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Fleet] Install a final_pipeline to help with host impersonation
6 participants
@nchaulet @elasticmachine @P1llus @kibanamachine @andrewkroh @kpollich