Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Specify an ECS version in Auditbeat/Packetbeat/Winlogbeat #19159

Merged

Conversation

andrewkroh
Copy link
Member

What does this PR do?

When we update the Beat (include all of its modules) we will then bump the ECS
version that it includes in events.

I went for a less granular approach than what is being used in Filebeat because
I think it's desirable to move a whole beat to a new ECS version "at once" and
more realistic to do so with these Beats that have fewer updates. By "at once" I
mean we won't release a version that is partially updated. This implies that if
we will be making multiple commits that we should use a feature branch to
ensure the update is atomic.

Why is it important?

We want the ecs.version to accurately represent the schema that is implemented.

Related issues

@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jun 12, 2020
@elasticmachine
Copy link
Collaborator

elasticmachine commented Jun 12, 2020

💔 Build Failed

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #19159 updated]

  • Start Time: 2020-07-13T15:46:35.431+0000

  • Duration: 81 min 33 sec

Test stats 🧪

Test Results
Failed 0
Passed 660
Skipped 68
Total 728

Steps errors

Expand to view the steps failures

  • Name: Make -C auditbeat testsuite

    • Description: make -C auditbeat testsuite

    • Duration: 7 min 28 sec

    • Start Time: 2020-07-13T16:21:27.037+0000

    • log

  • Name: Install docker-compose 1.21.0

    • Description: .ci/scripts/install-docker-compose.sh

    • Duration: 2 min 7 sec

    • Start Time: 2020-07-13T16:20:15.801+0000

    • log

  • Name: Make -C packetbeat testsuite

    • Description: make -C packetbeat testsuite

    • Duration: 7 min 44 sec

    • Start Time: 2020-07-13T16:21:41.770+0000

    • log

Log output

Expand to view the last 100 lines of log output

[2020-07-13T17:04:07.069Z] + FILE=libbeat/build/coverage/full.cov
[2020-07-13T17:04:07.069Z] + '[' -f libbeat/build/coverage/full.cov ']'
[2020-07-13T17:04:07.069Z] + for i in auditbeat filebeat heartbeat libbeat metricbeat packetbeat winlogbeat journalbeat
[2020-07-13T17:04:07.069Z] + FILE=metricbeat/build/coverage/full.cov
[2020-07-13T17:04:07.069Z] + '[' -f metricbeat/build/coverage/full.cov ']'
[2020-07-13T17:04:07.069Z] + for i in auditbeat filebeat heartbeat libbeat metricbeat packetbeat winlogbeat journalbeat
[2020-07-13T17:04:07.069Z] + FILE=packetbeat/build/coverage/full.cov
[2020-07-13T17:04:07.069Z] + '[' -f packetbeat/build/coverage/full.cov ']'
[2020-07-13T17:04:07.069Z] + for i in auditbeat filebeat heartbeat libbeat metricbeat packetbeat winlogbeat journalbeat
[2020-07-13T17:04:07.069Z] + FILE=winlogbeat/build/coverage/full.cov
[2020-07-13T17:04:07.069Z] + '[' -f winlogbeat/build/coverage/full.cov ']'
[2020-07-13T17:04:07.069Z] + for i in auditbeat filebeat heartbeat libbeat metricbeat packetbeat winlogbeat journalbeat
[2020-07-13T17:04:07.069Z] + FILE=journalbeat/build/coverage/full.cov
[2020-07-13T17:04:07.069Z] + '[' -f journalbeat/build/coverage/full.cov ']'
[2020-07-13T17:04:55.900Z] Machine "default" was started.
[2020-07-13T17:04:55.900Z] Waiting for SSH to be available...
[2020-07-13T17:04:55.900Z] Detecting the provisioner...
[2020-07-13T17:04:55.900Z] Started machines may have new IP addresses. You may need to re-run the `docker-machine env` command.
[2020-07-13T17:04:55.900Z] Client: Docker Engine - Community
[2020-07-13T17:04:55.900Z]  Version:           19.03.1
[2020-07-13T17:04:55.900Z]  API version:       1.40
[2020-07-13T17:04:55.900Z]  Go version:        go1.12.5
[2020-07-13T17:04:55.900Z]  Git commit:        74b1e89
[2020-07-13T17:04:55.900Z]  Built:             Thu Jul 25 21:18:17 2019
[2020-07-13T17:04:55.900Z]  OS/Arch:           darwin/amd64
[2020-07-13T17:04:55.900Z]  Experimental:      false
[2020-07-13T17:04:55.900Z] 
[2020-07-13T17:04:55.900Z] Server: Docker Engine - Community
[2020-07-13T17:04:55.900Z]  Engine:
[2020-07-13T17:04:55.900Z]   Version:          19.03.5
[2020-07-13T17:04:55.900Z]   API version:      1.40 (minimum version 1.12)
[2020-07-13T17:04:55.900Z]   Go version:       go1.12.12
[2020-07-13T17:04:55.900Z]   Git commit:       633a0ea838
[2020-07-13T17:04:55.900Z]   Built:            Wed Nov 13 07:28:45 2019
[2020-07-13T17:04:55.900Z]   OS/Arch:          linux/amd64
[2020-07-13T17:04:55.900Z]   Experimental:     false
[2020-07-13T17:04:55.900Z]  containerd:
[2020-07-13T17:04:55.900Z]   Version:          v1.2.10
[2020-07-13T17:04:55.900Z]   GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
[2020-07-13T17:04:55.900Z]  runc:
[2020-07-13T17:04:55.900Z]   Version:          1.0.0-rc8+dev
[2020-07-13T17:04:55.900Z]   GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
[2020-07-13T17:04:55.900Z]  docker-init:
[2020-07-13T17:04:55.900Z]   Version:          0.18.0
[2020-07-13T17:04:55.900Z]   GitCommit:        fec3683
[2020-07-13T17:06:47.281Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19159/src/github.com/elastic/beats
[2020-07-13T17:06:47.591Z] + find . -type f -name TEST*.xml -path */build/* -delete
[2020-07-13T17:06:47.604Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19159/src/github.com/elastic/beats/Lint
[2020-07-13T17:06:47.683Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19159/src/github.com/elastic/beats/Winlogbeat-oss
[2020-07-13T17:06:47.755Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19159/src/github.com/elastic/beats/Auditbeat-crosscompile
[2020-07-13T17:06:47.825Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19159/src/github.com/elastic/beats/Auditbeat-x-pack-Windows
[2020-07-13T17:06:47.897Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19159/src/github.com/elastic/beats/Auditbeat-oss-Windows
[2020-07-13T17:06:47.977Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19159/src/github.com/elastic/beats/Winlogbeat-Windows-x-pack
[2020-07-13T17:06:48.048Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19159/src/github.com/elastic/beats/Packetbeat-oss
[2020-07-13T17:06:48.132Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19159/src/github.com/elastic/beats/Auditbeat-oss-Linux
[2020-07-13T17:06:48.228Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19159/src/github.com/elastic/beats/Auditbeat-x-pack
[2020-07-13T17:06:48.310Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19159/src/github.com/elastic/beats/Winlogbeat-Windows
[2020-07-13T17:06:48.394Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19159/src/github.com/elastic/beats/Auditbeat-oss-Mac-OS-X
[2020-07-13T17:06:48.469Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19159/src/github.com/elastic/beats/Auditbeat-x-pack-Mac-OS-X
[2020-07-13T17:06:48.850Z] + cat
[2020-07-13T17:06:48.850Z] + /usr/local/bin/runbld ./runbld-script
[2020-07-13T17:06:48.850Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-07-13T17:06:55.445Z] runbld>>> runbld started
[2020-07-13T17:06:55.445Z] runbld>>> 1.6.12/f45d832f2ba0aa2722ab4ec1fda8ad140f027f8b
[2020-07-13T17:06:56.386Z] runbld>>> The following profiles matched the job 'Beats/beats/PR-19159' in order of occurrence in the config (last value wins).
[2020-07-13T17:06:57.768Z] runbld>>> Debug logging enabled.
[2020-07-13T17:06:57.768Z] runbld>>> Storing result
[2020-07-13T17:06:57.768Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-07-13T17:06:57.768Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200713170657-42CA2341
[2020-07-13T17:06:57.768Z] runbld>>> Adding system facts.
[2020-07-13T17:06:58.716Z] runbld>>> Adding vcs info for the latest commit:  6af0f9c9e9276b095405189eb31d15e4538c0aaa
[2020-07-13T17:06:58.716Z] runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>>
[2020-07-13T17:06:58.716Z] runbld>>> Adding /usr/lib/jvm/java-8-openjdk-amd64/bin to the path.
[2020-07-13T17:06:58.716Z] Processing JUnit reports with runbld...
[2020-07-13T17:06:58.716Z] + echo 'Processing JUnit reports with runbld...'
[2020-07-13T17:06:59.289Z] runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
[2020-07-13T17:06:59.289Z] runbld>>> DURATION: 14ms
[2020-07-13T17:06:59.289Z] runbld>>> STDOUT: 40 bytes
[2020-07-13T17:06:59.289Z] runbld>>> STDERR: 49 bytes
[2020-07-13T17:06:59.289Z] runbld>>> WRAPPED PROCESS: SUCCESS (0)
[2020-07-13T17:06:59.289Z] runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/Beats_beats_PR-19159/src/github.com/elastic/beats
[2020-07-13T17:07:00.230Z] runbld>>> Storing build metadata: 
[2020-07-13T17:07:00.230Z] runbld>>> Adding test report.
[2020-07-13T17:07:00.230Z] runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/Beats_beats_PR-19159/src/github.com/elastic/beats
[2020-07-13T17:07:01.172Z] runbld>>> Found 15 test output files
[2020-07-13T17:07:01.433Z] runbld>>> Test output logs contained: Errors: 0 Failures: 0 Tests: 728 Skipped: 56
[2020-07-13T17:07:01.433Z] runbld>>> Storing result
[2020-07-13T17:07:01.433Z] runbld>>> FAILURES: 0
[2020-07-13T17:07:01.694Z] runbld>>> Store result: updated {:total 2, :successful 2, :failed 0} 2
[2020-07-13T17:07:01.694Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200713170657-42CA2341
[2020-07-13T17:07:01.954Z] runbld>>> Email notification disabled by environment variable.
[2020-07-13T17:07:01.954Z] runbld>>> Slack notification disabled by environment variable.
[2020-07-13T17:07:07.553Z] Running on worker-395930 in /var/lib/jenkins/workspace/Beats_beats_PR-19159
[2020-07-13T17:07:07.670Z] [INFO] getVaultSecret: Getting secrets
[2020-07-13T17:07:07.761Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-07-13T17:07:10.049Z] + chmod 755 generate-build-data.sh
[2020-07-13T17:07:10.049Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-19159/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-19159/runs/3 FAILURE 4833029
[2020-07-13T17:07:10.049Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-19159/runs/3/steps/?limit=10000 -o steps-info.json
[2020-07-13T17:07:11.502Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-19159/runs/3/tests/?status=FAILED -o tests-errors.json
[2020-07-13T17:07:12.202Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-19159/runs/3/log/ -o pipeline-log.txt

@andrewkroh
Copy link
Member Author

run tests

@andrewkroh andrewkroh marked this pull request as ready for review June 12, 2020 17:52
@elasticmachine
Copy link
Collaborator

Pinging @elastic/siem (Team:SIEM)

@andrewkroh
Copy link
Member Author

run tests

1 similar comment
@andrewkroh
Copy link
Member Author

run tests

@webmat
Copy link
Contributor

webmat commented Jun 25, 2020

Thanks for getting this started!

I agree ideally a whole Beat should upgrade ECS versions all at once whenever possible. This is an improvement over setting it once in libbeat for all Beats, for a given stack release. Given that these 3 Beats have much fewer datasets than Metricbeat and Filebeat, I think this is an acceptable tradeoff. 👍

@andrewkroh
Copy link
Member Author

run tests

@andrewkroh andrewkroh force-pushed the feature/pin-ecs-version-wlb-ab-pb branch from f02adf6 to 593c6cf Compare June 29, 2020 13:51
@andrewkroh
Copy link
Member Author

run tests

@andrewkroh
Copy link
Member Author

Run tests

@andrewkroh andrewkroh force-pushed the feature/pin-ecs-version-wlb-ab-pb branch from 593c6cf to a135e25 Compare July 13, 2020 12:58
When we update the Beat (include all of its modules) we will then bump the ECS
version that it includes in events.

I went for a less granular approach than what is being used in Filebeat because
I think it's desirable to move a whole beat to a new ECS version "at once" and
more realistic to do so with these Beats that have fewer updates. By "at once" I
mean we won't release a version that is partially updated. This implies that if
we will be making multiple commits that we should use a feature branch to
ensure the update is atomic.

Closes elastic#17688
@andrewkroh andrewkroh force-pushed the feature/pin-ecs-version-wlb-ab-pb branch from a135e25 to 12fb849 Compare July 13, 2020 15:28
@andrewkroh
Copy link
Member Author

The kibana container is failing to launch for the integration tests due to elastic/kibana#71343, but the unit tests are all passing so I'm going to merge.

@andrewkroh andrewkroh merged commit 256b50d into elastic:master Jul 13, 2020
andrewkroh added a commit to andrewkroh/beats that referenced this pull request Jul 13, 2020
)

When we update the Beat (include all of its modules) we will then bump the ECS
version that it includes in events.

I went for a less granular approach than what is being used in Filebeat because
I think it's desirable to move a whole beat to a new ECS version "at once" and
more realistic to do so with these Beats that have fewer updates. By "at once" I
mean we won't release a version that is partially updated. This implies that if
we will be making multiple commits that we should use a feature branch to
ensure the update is atomic.

Closes elastic#17688

(cherry picked from commit 256b50d)
v1v added a commit to v1v/beats that referenced this pull request Jul 14, 2020
* upstream/master: (25 commits)
  [Elastic Agent] Send checkin payload to Fleet (elastic#19857)
  [Ingest Manager] Fixed tests across agent elastic#19877
  [Ingest Manager] Fix serialization test  elastic#19876
  Fix service start type mapping in windows/service metricset (elastic#19551)
  ci: Change comment trigger detection method (elastic#19827)
  Add 21 autogenerated filesets from rsa2elk devices (elastic#19713)
  [Ingest Manager] Agent config cleanup (elastic#19848)
  libbeat/publisher/pipeline: fix data races (elastic#19821)
  Update monitoring-internal-collection.asciidoc (elastic#19422) (elastic#19697)
  [Elastic Agent] Trust exchange endpoint must bind to 127.0.0.1 (elastic#19861)
  Specify an ECS version in Auditbeat/Packetbeat/Winlogbeat (elastic#19159)
  Add azure billing metricset (elastic#19207)
  Add support for appinsights in the metricbeat azure module (elastic#18940)
  Add MySQL query metricset with lightweight module and SQL helper (elastic#18955)
  [Ingest Manager] Refuse invalid stream values in configuration (elastic#19587)
  Do not use vendor during integration tests (elastic#19839)
  LIBBEAT: Enhancement Convert dissected values from String to other basic data types and IP (elastic#18683)
  [Elastic Agent] Remove support for "logs" and only support logfile (elastic#19761)
  [CI] support windows-2012 (elastic#19773)
  Do not update go.mod during packaging and testing (elastic#19823)
  ...
andrewkroh added a commit that referenced this pull request Jul 14, 2020
…19862)

When we update the Beat (include all of its modules) we will then bump the ECS
version that it includes in events.

I went for a less granular approach than what is being used in Filebeat because
I think it's desirable to move a whole beat to a new ECS version "at once" and
more realistic to do so with these Beats that have fewer updates. By "at once" I
mean we won't release a version that is partially updated. This implies that if
we will be making multiple commits that we should use a feature branch to
ensure the update is atomic.

Closes #17688

(cherry picked from commit 256b50d)
melchiormoulin pushed a commit to melchiormoulin/beats that referenced this pull request Oct 14, 2020
)

When we update the Beat (include all of its modules) we will then bump the ECS
version that it includes in events.

I went for a less granular approach than what is being used in Filebeat because
I think it's desirable to move a whole beat to a new ECS version "at once" and
more realistic to do so with these Beats that have fewer updates. By "at once" I
mean we won't release a version that is partially updated. This implies that if
we will be making multiple commits that we should use a feature branch to
ensure the update is atomic.

Closes elastic#17688
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Filebeat] Allow modules to set the ecs.version field value
4 participants