Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Swap ECS 1.7 experimental schema with non-experimental to revert wildcard changes #23673

Merged
merged 2 commits into from
Jan 26, 2021
Merged

Swap ECS 1.7 experimental schema with non-experimental to revert wildcard changes #23673

merged 2 commits into from
Jan 26, 2021

Conversation

andrewstucki
Copy link
Contributor

What does this PR do?

This PR changes the experimental ECS 1.7 schema that we were going to use in 7.11 to be the ECS 1.7 non-experimental version. This coincides with the Elasticsearch performance regressions that we saw on some widely used filebeat modules as mentioned in #23671. The fallback logic introduced to support wildcard fields can stay since the schema itself is switching back from wildcard to keyword. The intent is to proceed at a more judicious pace with wildcard adoption in coming releases.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
    - [ ] I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

@andrewstucki andrewstucki added needs_backport PR is waiting to be backported to other branches. ecs v7.11.0 Team:Security-External Integrations labels Jan 26, 2021
@andrewstucki andrewstucki requested a review from a team as a code owner January 26, 2021 04:05
@elasticmachine
Copy link
Collaborator

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jan 26, 2021
@elasticmachine
Copy link
Collaborator

elasticmachine commented Jan 26, 2021 •
edited by jenkins-beats-ci bot
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: Started by user Andrew Stucki

    • Start Time: 2021-01-26T14:36:39.979+0000

  • Duration: 49 min 44 sec

  • Commit: 8c8012f

Test stats 🧪

Test Results
Failed 0
Passed 45149
Skipped 4737
Total 49886

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test Results
Failed 0
Passed 45149
Skipped 4737
Total 49886

@andrewstucki andrewstucki requested a review from a team January 26, 2021 13:23
ebeahan
ebeahan approved these changes Jan 26, 2021
View reviewed changes
Copy link
Member

@ebeahan ebeahan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The wildcard revert LGTM 👍 .

In addition to reverting the wildcard fields, we're also removing the user.changes.*, user.effective.*, and user.target.* fields. Were those originally introduced as a side-effect of using the experimental schema from 1.7?

@andrewstucki
Copy link
Contributor Author

@ebeahan so, previous to the 1.7 experimental field incorporation, we had auditd sources both in filebeat and auditbeat leveraging user.effective.* fields. When we added the experimental fields, I stripped those out of the custom field definitions of tohse data sources. Now, as a result of the revert to the non-experimental schema, I needed to add back in user.effective.* as custom fields for those sources--so they'll still work. Nothing was previously leveraging user.target.*.

andrewkroh
andrewkroh approved these changes Jan 26, 2021
View reviewed changes
Copy link
Member

@andrewkroh andrewkroh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@andrewstucki andrewstucki merged commit 0f50eb7 into elastic:master Jan 26, 2021
@andrewstucki andrewstucki mentioned this pull request Jan 26, 2021
Merged
5 tasks
@andrewstucki andrewstucki removed the needs_backport PR is waiting to be backported to other branches. label Jan 26, 2021
andrewstucki pushed a commit to andrewstucki/beats that referenced this pull request Jan 26, 2021
Swap ECS 1.7 experimental schema with non-experimental to revert wild…
66a4bf7 
…card changes (elastic#23673)

* Swap ECS 1.7 experimental schema with non-experimental to revert wildcard changes

* Add back in user.effective.* fields to auditd integrations due to no longer using experimental schema

(cherry picked from commit 0f50eb7)
@andrewstucki andrewstucki mentioned this pull request Jan 26, 2021
Merged
5 tasks
andrewstucki pushed a commit to andrewstucki/beats that referenced this pull request Jan 26, 2021
Swap ECS 1.7 experimental schema with non-experimental to revert wild…
266127c 
…card changes (elastic#23673)

* Swap ECS 1.7 experimental schema with non-experimental to revert wildcard changes

* Add back in user.effective.* fields to auditd integrations due to no longer using experimental schema

(cherry picked from commit 0f50eb7)
@andrewstucki andrewstucki deleted the wildcard-revert branch January 26, 2021 21:41
andrewstucki pushed a commit that referenced this pull request Jan 26, 2021
Swap ECS 1.7 experimental schema with non-experimental to revert wild…
d1aa3ff 
…card changes (#23673) (#23700)

* Swap ECS 1.7 experimental schema with non-experimental to revert wildcard changes

* Add back in user.effective.* fields to auditd integrations due to no longer using experimental schema

(cherry picked from commit 0f50eb7)
andrewstucki pushed a commit that referenced this pull request Jan 26, 2021
Cherry-pick #23673 to 7.11: Swap ECS 1.7 experimental schema with non…
3f44e4c 
…-experimental to revert wildcard changes (#23699)

* Swap ECS 1.7 experimental schema with non-experimental to revert wildcard changes (#23673)

* Swap ECS 1.7 experimental schema with non-experimental to revert wildcard changes

* Add back in user.effective.* fields to auditd integrations due to no longer using experimental schema

(cherry picked from commit 0f50eb7)

* Add potential workaround using GIT_BASE_COMMIT

* Pin generator to origin/7.11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@andrewkroh andrewkroh andrewkroh approved these changes

@ebeahan ebeahan ebeahan approved these changes

@marc-gr marc-gr marc-gr approved these changes

@ruflin ruflin Awaiting requested review from ruflin

Assignees
No one assigned
Labels
ecs v7.11.0 v7.12.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Negative performance impact on wildcard field changes
5 participants
@andrewstucki @elasticmachine @andrewkroh @ebeahan @marc-gr