[Security Solution] Fix incorrect time for dns histogram #83532

angorayc · 2020-11-17T11:27:59Z

Summary

Before this PR, we had an issue about DNS hitogram, that skip the data with empty value in the expected bucket.

I think that's because the query result return empty value in the expected buckets, so when our parser iterates them,
it return nothing. Therefor when the data reaches the bar chart, some buckets are missing.

So this PR updates the query to make histogram rendered as expected (should not skip the expected bucket)
Note: The DNS histogram also aligns the option include PTR, with the table below. Means that if it is switched on in the table below, it applies to the histogram too.

Checklist

Delete any items that are not applicable to this PR.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Unit or functional tests were updated or added to match the most common scenarios

…rval

…alendar-interval

…rval

angorayc · 2020-11-18T16:18:37Z

@elasticmachine merge upstream

…alendar-interval

angorayc · 2020-11-23T09:53:05Z

@elasticmachine merge upstream

…rval

…alendar-interval

patrykkopycinski · 2020-11-24T12:50:10Z

...solution/server/search_strategy/security_solution/factory/network/details/__mocks__/index.ts

@@ -263,7 +263,101 @@ export const formattedSearchStrategyResponse = {
  ...mockSearchStrategyResponse,
  inspect: {
    dsl: [
-      '{\n  "allowNoIndices": true,\n  "index": [\n    "apm-*-transaction*",\n    "auditbeat-*",\n    "endgame-*",\n    "filebeat-*",\n    "logs-*",\n    "packetbeat-*",\n    "winlogbeat-*"\n  ],\n  "ignoreUnavailable": true,\n  "body": {\n    "aggs": {\n      "source": {\n        "filter": {\n          "term": {\n            "source.ip": "35.196.65.164"\n          }\n        },\n        "aggs": {\n          "firstSeen": {\n            "min": {\n              "field": "@timestamp"\n            }\n          },\n          "lastSeen": {\n            "max": {\n              "field": "@timestamp"\n            }\n          },\n          "as": {\n            "filter": {\n              "exists": {\n                "field": "source.as"\n              }\n            },\n            "aggs": {\n              "results": {\n                "top_hits": {\n                  "size": 1,\n                  "_source": [\n                    "source.as"\n                  ],\n                  "sort": [\n                    {\n                      "@timestamp": "desc"\n                    }\n                  ]\n                }\n              }\n            }\n          },\n          "geo": {\n            "filter": {\n              "exists": {\n                "field": "source.geo"\n              }\n            },\n            "aggs": {\n              "results": {\n                "top_hits": {\n                  "size": 1,\n                  "_source": [\n                    "source.geo"\n                  ],\n                  "sort": [\n                    {\n                      "@timestamp": "desc"\n                    }\n                  ]\n                }\n              }\n            }\n          }\n        }\n      },\n      "destination": {\n        "filter": {\n          "term": {\n            "destination.ip": "35.196.65.164"\n          }\n        },\n        "aggs": {\n          "firstSeen": {\n            "min": {\n              "field": "@timestamp"\n            }\n          },\n          "lastSeen": {\n            "max": {\n              "field": "@timestamp"\n            }\n          },\n          "as": {\n            "filter": {\n              "exists": {\n                "field": "destination.as"\n              }\n            },\n            "aggs": {\n              "results": {\n                "top_hits": {\n                  "size": 1,\n                  "_source": [\n                    "destination.as"\n                  ],\n                  "sort": [\n                    {\n                      "@timestamp": "desc"\n                    }\n                  ]\n                }\n              }\n            }\n          },\n          "geo": {\n            "filter": {\n              "exists": {\n                "field": "destination.geo"\n              }\n            },\n            "aggs": {\n              "results": {\n                "top_hits": {\n                  "size": 1,\n                  "_source": [\n                    "destination.geo"\n                  ],\n                  "sort": [\n                    {\n                      "@timestamp": "desc"\n                    }\n                  ]\n                }\n              }\n            }\n          }\n        }\n      },\n      "host": {\n        "filter": {\n          "term": {\n            "host.ip": "35.196.65.164"\n          }\n        },\n        "aggs": {\n          "results": {\n            "top_hits": {\n              "size": 1,\n              "_source": [\n                "host"\n              ],\n              "sort": [\n                {\n                  "@timestamp": "desc"\n                }\n              ]\n            }\n          }\n        }\n      }\n    },\n    "query": {\n      "bool": {\n        "should": []\n      }\n    },\n    "size": 0,\n    "track_total_hits": false\n  }\n}',


patrykkopycinski · 2020-11-24T12:56:48Z

...security_solution/server/search_strategy/timeline/factory/events/all/query.events_all.dsl.ts

@@ -58,7 +58,7 @@ export const buildTimelineEventsAllQuery = ({
    index: defaultIndex,
    ignoreUnavailable: true,
    body: {
-      ...(isEmpty(docValueFields) ? { docvalue_fields: docValueFields } : {}),
+      ...(!isEmpty(docValueFields) ? { docvalue_fields: docValueFields } : {}),


patrykkopycinski

Code-wise LGTM, Thank you for fixing logic in all DSLs

kibanamachine · 2020-11-24T14:45:41Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 5258750

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`securitySolution`	2081	2080	-1

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`securitySolution`	7.9MB	7.9MB	-504.0B

History

💔 Build #89915 failed 46936c4
💔 Build #89899 failed d5ca920
💚 Build #89801 succeeded 2bcdd8f
💚 Build #89794 succeeded c9660a7
💔 Build #89754 failed c1b9e44

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

* getSuitableUnit * update dns histogram query * update dns query * update dns histogram query * fix type error * fix lint error * remove unused comments * fix histogram query size * revert change * fix unit test * fix dns request options * clean up * cleanup types * fix dependency * review * review * revert * restore docValueFields * fix unit test * cleanup * restore docValueFields for dns histogram * review * review * lint Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

* getSuitableUnit * update dns histogram query * update dns query * update dns histogram query * fix type error * fix lint error * remove unused comments * fix histogram query size * revert change * fix unit test * fix dns request options * clean up * cleanup types * fix dependency * review * review * revert * restore docValueFields * fix unit test * cleanup * restore docValueFields for dns histogram * review * review * lint Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> # Conflicts: # x-pack/plugins/security_solution/public/network/containers/network_dns/index.tsx # x-pack/plugins/security_solution/server/search_strategy/security_solution/factory/network/dns/query.dns_network.dsl.ts

…4228) * getSuitableUnit * update dns histogram query * update dns query * update dns histogram query * fix type error * fix lint error * remove unused comments * fix histogram query size * revert change * fix unit test * fix dns request options * clean up * cleanup types * fix dependency * review * review * revert * restore docValueFields * fix unit test * cleanup * restore docValueFields for dns histogram * review * review * lint Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

…) (#84235) * [Security Solution] Fix incorrect time for dns histogram (#83532) * getSuitableUnit * update dns histogram query * update dns query * update dns histogram query * fix type error * fix lint error * remove unused comments * fix histogram query size * revert change * fix unit test * fix dns request options * clean up * cleanup types * fix dependency * review * review * revert * restore docValueFields * fix unit test * cleanup * restore docValueFields for dns histogram * review * review * lint Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> # Conflicts: # x-pack/plugins/security_solution/public/network/containers/network_dns/index.tsx # x-pack/plugins/security_solution/server/search_strategy/security_solution/factory/network/dns/query.dns_network.dsl.ts * eslint

* master: (41 commits) [Maps] fix code-owners (elastic#84265) [@kbn/utils] Clean target before build (elastic#84253) [code coverage] collect for oss integration tests (elastic#83907) [APM] Use `asTransactionRate` consistently everywhere (elastic#84213) Attempt to fix incremental build error (elastic#84152) Unskip "Copy dashboards to space" (elastic#84115) Remove expressions.legacy from README (elastic#79681) Expression: Add render mode and use it for canvas interactivity (elastic#83559) [deb/rpm] Move systemd service to /usr/lib/systemd/system (elastic#83571) [Security Solution][Resolver] Allow a configurable entity_id field (elastic#81679) [ML] Space permision checks for job deletion (elastic#83871) [build] Provide ARM build of RE2 (elastic#84163) TSVB should use "histogram:maxBars" and "histogram:barTarget" settings for auto instead of a default 100 buckets (elastic#83628) [Workplace Search] Initial rendering of Org Sources (elastic#84164) update geckodriver to 0.28 (elastic#84085) Fix timelion vis escapes single quotes (elastic#84196) [Security Solution] Fix incorrect time for dns histogram (elastic#83532) [DX] Bump TS version to v4.1 (elastic#83397) [Security Solution] Add endpoint policy revision number (elastic#83982) [Fleet] Integration Policies List view (elastic#83634) ...

angorayc added 11 commits November 9, 2020 12:26

getSuitableUnit

58f8ec0

Merge branch 'master' of github.com:elastic/kibana into calendar-inte…

ab2ee15

…rval

update dns histogram query

2eb6d17

update dns query

e5356b1

update dns histogram query

93ecd9e

fix type error

49de18d

Merge branch 'master' of github.com:elastic/kibana into calendar-inte…

8611973

…rval

fix lint error

a78a577

remove unused comments

eadd57f

fix histogram query size

987c436

revert change

b6b4490

angorayc added release_note:fix Team:Threat Hunting Security Solution Threat Hunting Team v7.10.1 Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. labels Nov 17, 2020

angorayc requested review from a team as code owners November 17, 2020 11:28

angorayc mentioned this pull request Nov 17, 2020

[Security Solution] Fix incorrect time for dns histogram #82943

Closed

2 tasks

angorayc added 5 commits November 17, 2020 12:23

Merge branch 'master' of github.com:elastic/kibana into calendar-inte…

d2c99d7

…rval

Merge branch 'calendar-interval' of github.com:angorayc/kibana into c…

491acc0

…alendar-interval

fix unit test

b49484c

Merge branch 'master' of github.com:elastic/kibana into calendar-inte…

b0a3079

…rval

fix dns request options

d934574

kibanamachine and others added 4 commits November 18, 2020 11:18

Merge branch 'master' into calendar-interval

be1c2bd

clean up

c67af56

Merge branch 'calendar-interval' of github.com:angorayc/kibana into c…

80946c6

…alendar-interval

cleanup types

cdb1b9f

Merge branch 'master' into calendar-interval

852f0db

angorayc added 13 commits November 23, 2020 13:37

Merge branch 'master' of github.com:elastic/kibana into calendar-inte…

8d52fe5

…rval

Merge branch 'calendar-interval' of github.com:angorayc/kibana into c…

ff79765

…alendar-interval

fix dependency

251e7df

review

79d087e

review

5bd67a5

revert

bec1f1d

restore docValueFields

c1b9e44

fix unit test

c9660a7

cleanup

2bcdd8f

restore docValueFields for dns histogram

3a3f7e3

review

d5ca920

review

46936c4

lint

5258750

patrykkopycinski reviewed Nov 24, 2020

View reviewed changes

patrykkopycinski approved these changes Nov 24, 2020

View reviewed changes

angorayc merged commit d80e8ca into elastic:master Nov 24, 2020

angorayc mentioned this pull request Nov 24, 2020

[7.x] [Security Solution] Fix incorrect time for dns histogram (#83532) #84228

Merged

angorayc mentioned this pull request Nov 24, 2020

[7.10] [Security Solution] Fix incorrect time for dns histogram (#83532) #84235

Merged

angorayc mentioned this pull request Nov 30, 2020

[Security Solution][Network] Histogram dates are incorrect #82728

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Security Solution] Fix incorrect time for dns histogram #83532

[Security Solution] Fix incorrect time for dns histogram #83532

angorayc commented Nov 17, 2020 •

edited

Loading

angorayc commented Nov 18, 2020

angorayc commented Nov 23, 2020

patrykkopycinski Nov 24, 2020

patrykkopycinski Nov 24, 2020

patrykkopycinski left a comment

kibanamachine commented Nov 24, 2020

[Security Solution] Fix incorrect time for dns histogram #83532

[Security Solution] Fix incorrect time for dns histogram #83532

Conversation

angorayc commented Nov 17, 2020 • edited Loading

Summary

Checklist

angorayc commented Nov 18, 2020

angorayc commented Nov 23, 2020

patrykkopycinski Nov 24, 2020

Choose a reason for hiding this comment

patrykkopycinski Nov 24, 2020

Choose a reason for hiding this comment

patrykkopycinski left a comment

Choose a reason for hiding this comment

kibanamachine commented Nov 24, 2020

💚 Build Succeeded

Metrics [docs]

Module Count

Async chunks

History

angorayc commented Nov 17, 2020 •

edited

Loading