[SIEM][Detection Engine] Fixes skipped tests #71347
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Pinging @elastic/siem (Team:SIEM) |
FrankHassanabad
added
v9.0.0
v7.9.0
release_note:skip
spong
approved these changes
Jul 9, 2020
There was a problem hiding this comment.
LGTM! Thanks for the improvements and (hopefully 🤞) taking the flake out of these tests @FrankHassanabad! 🙂
|
@elasticmachine merge upstream |
💚 Build SucceededBuild metrics
History
To update your PR or re-run it, just comment with: |
FrankHassanabad
added a commit
to FrankHassanabad/kibana
that referenced
this pull request
Jul 10, 2020
## Summary * elastic#69632 * Adds a retry loop in case of a network outage/issue which should increase the chances of success * If there is still an issue after the 20th try, then it moves on and there is a high likelihood the tests will continue without issues. * Adds console logging statements so we know if this flakiness happens again a bit more insight into why the network is behaving the way it is. * Helps prevent the other tests from being skipped in the future due to bad networking issues. The errors that were coming back from the failed tests are in the `afterEach` and look to be network related or another test interfering: ```ts 1) detection engine api security and spaces enabled 01:59:54 find_statuses 01:59:54 "after each" hook for "should return a single rule status when a single rule is loaded from a find status with defaults added": 01:59:54 ResponseError: Response Error 01:59:54 at IncomingMessage.response.on (/dev/shm/workspace/kibana/node_modules/@elastic/elasticsearch/lib/Transport.js:287:25) 01:59:54 at endReadableNT (_stream_readable.js:1145:12) 01:59:54 at process._tickCallback (internal/process/next_tick.js:63:19) 01:59:54 01:59:54 └- ✖ fail: "detection engine api security and spaces enabled find_statuses "after each" hook for "should return a single rule status when a single rule is loaded from a find status with defaults added"" 01:59:54 │ 01:59:54 └-> "after all" hook 01:59:54 └-> "after all" hook 01:59:54 │ 01:59:54 │42 passing (2.0m) 01:59:54 │1 failing ``` So this should fix it to where the afterEach calls try up to 20 times before giving up and then on giving up they move on with the hope a different test doesn't fail. ### Checklist - [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
jloleysens
added a commit
to jloleysens/kibana
that referenced
this pull request
Jul 10, 2020
…11y-overlay * 'master' of github.com:elastic/kibana: (33 commits) address index templates feedback (elastic#71353) Upgrade EUI to v26.3.1 (elastic#70243) [build] Creates Linux aarch64 archive (elastic#69165) [SIEM][Detection Engine] Fixes skipped tests (elastic#71347) [SIEM][Detection Engine][Lists] Adds read_privileges route for lists and list items [kbn/optimizer] implement "requiredBundles" property of KP plugins (elastic#70911) [Security Solution][Exceptions] - Exceptions modal pt 2 (elastic#70886) [ML] DF Analytics: stop status polling when job stopped (elastic#71159) [SIEM][CASE] IBM Resilient Connector (elastic#66385) jenkins_xpack_saved_objects_field_metrics.sh expects to be run from the KIBANA_DIR in CI Deduplication of entries and items before sending to endpoint (elastic#71297) [services/remote/webdriver] fix eslint error (elastic#71346) send slack notifications on visual baseline failures fix visual regression job (elastic#70999) [Ingest Manager] Add schema to usageCollector. (elastic#71219) [ftr] use typed chromeOptions object, adding TEST_BROWSER_BINARY_PATH (elastic#71279) [Ingest Manager] Fix limited packages incorrect response (elastic#71292) Support multiple features declaring same properties (elastic#71106) [Security_Solution][Resolver]Add beta badge to Resolver panel (elastic#71183) [DOCS] Clarify trial subscription levels (elastic#70636) ...
FrankHassanabad
added a commit
that referenced
this pull request
Jul 10, 2020
## Summary * #69632 * Adds a retry loop in case of a network outage/issue which should increase the chances of success * If there is still an issue after the 20th try, then it moves on and there is a high likelihood the tests will continue without issues. * Adds console logging statements so we know if this flakiness happens again a bit more insight into why the network is behaving the way it is. * Helps prevent the other tests from being skipped in the future due to bad networking issues. The errors that were coming back from the failed tests are in the `afterEach` and look to be network related or another test interfering: ```ts 1) detection engine api security and spaces enabled 01:59:54 find_statuses 01:59:54 "after each" hook for "should return a single rule status when a single rule is loaded from a find status with defaults added": 01:59:54 ResponseError: Response Error 01:59:54 at IncomingMessage.response.on (/dev/shm/workspace/kibana/node_modules/@elastic/elasticsearch/lib/Transport.js:287:25) 01:59:54 at endReadableNT (_stream_readable.js:1145:12) 01:59:54 at process._tickCallback (internal/process/next_tick.js:63:19) 01:59:54 01:59:54 └- ✖ fail: "detection engine api security and spaces enabled find_statuses "after each" hook for "should return a single rule status when a single rule is loaded from a find status with defaults added"" 01:59:54 │ 01:59:54 └-> "after all" hook 01:59:54 └-> "after all" hook 01:59:54 │ 01:59:54 │42 passing (2.0m) 01:59:54 │1 failing ``` So this should fix it to where the afterEach calls try up to 20 times before giving up and then on giving up they move on with the hope a different test doesn't fail. ### Checklist - [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
gmmorris
added a commit
to gmmorris/kibana
that referenced
this pull request
Jul 13, 2020
* master: (78 commits) Bump lodash package version (elastic#71392) refactor: 💡 use allow-list in AppArch codebase (elastic#71400) improve bugfix 7198 test stability (elastic#71250) [Security Solution][Ingest Manager][Endpoint] Optional ingest manager (elastic#71198) [Metrics UI] Round metric threshold time buckets to nearest unit (elastic#71172) [Security Solution][Endpoint] Policy creation callback fixes + Improved error handling in user manifest loop (elastic#71269) [Security Solution] Allow to configure Event Renderers settings (elastic#69693) Fix a11y keyboard overlay (elastic#71214) [APM] UI text updates (elastic#71333) [Logs UI] Limit `extendDatemath` to valid ranges (elastic#71113) [SIEM] fix tooltip of notes (elastic#71342) address index templates feedback (elastic#71353) Upgrade EUI to v26.3.1 (elastic#70243) [build] Creates Linux aarch64 archive (elastic#69165) [SIEM][Detection Engine] Fixes skipped tests (elastic#71347) [SIEM][Detection Engine][Lists] Adds read_privileges route for lists and list items [kbn/optimizer] implement "requiredBundles" property of KP plugins (elastic#70911) [Security Solution][Exceptions] - Exceptions modal pt 2 (elastic#70886) [ML] DF Analytics: stop status polling when job stopped (elastic#71159) [SIEM][CASE] IBM Resilient Connector (elastic#66385) ...
MindyRS
added
the
Team: SecuritySolution
|
Pinging @elastic/security-solution (Team: SecuritySolution) |
jbudz
added
backport:skip
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
The errors that were coming back from the failed tests are in the
afterEachand look to be network related or another test interfering:So this should fix it to where the afterEach calls try up to 20 times before giving up and then on giving up they move on with the hope a different test doesn't fail.
Checklist