[Security Solution][Resolver] Allow a configurable entity_id field #81679

jonathan-buttner · 2020-10-26T21:07:41Z

This PR adds a new /tree api that is not dependent on a single ID. Since this api will only return a single document per node in the resolver graph we'll need to update the /events api to support requesting all the lifecycle events for multiple nodes at a time. This PR does not include the needed /events api changes.

Once the front end is transitioned over to the new api we'll be able to remove the old /tree route that was centered around a single process.entity_id and all the supporting code and tests.

It includes a new /tree api that supports:

Specifying a timerange for the nodes being retrieved
Allows a configurable field to use as the entity_id
Configurable indices to search for the nodes
Will only return a single event for a each node
Flattens the keys to make it easier to find the specified configurable field

Testing

Generate a tree:

yarn test:generate --maxCh --ch 2 --ancSize 20 --delete -s resolver --pt 100 --related 10 --pr 100 --relAlerts 0

Make a request:

POST http://elastic:changeme@localhost:5601/api/endpoint/resolver/tree
{
    "descendantLevels": 1,
    "descendants": 100,
    "ancestors": 0,
    "timerange": {
        "from": "2020-10-21T19:49:32.095Z",
        "to": "2020-12-30T19:49:14.878Z"
    },
    "schema": {
        "id": "process.entity_id",
        "parent": "process.parent.entity_id"
    },
    "nodes": [
        "0i17c2m22c"
    ],
    "indexPatterns": ["logs-*"]
}

curl command:

curl --location --request POST 'http://elastic:changeme@localhost:5601/api/endpoint/resolver/tree' \
--header 'Accept: application/json' \
--header 'Content-Type: application/json' \
--header 'kbn-xsrf: xxx' \
--data-raw '{
    "descendantLevels": 1,
    "descendants": 100,
    "ancestors": 0,
    "timerange": {
        "from": "2020-10-21T19:49:32.095Z",
        "to": "2020-12-30T19:49:14.878Z"
    },
    "schema": {
        "id": "process.entity_id",
        "parent": "process.parent.entity_id"
    },
    "nodes": [
        "0i17c2m22c"
    ],
    "indexPatterns": ["logs-*"]
}'

Example response:

[
    {
        "data": {
            "@timestamp": [
                "2020-11-09T21:13:18.246Z"
            ],
            "process.parent.entity_id": [
                "0i17c2m22c"
            ],
            "process.entity_id": [
                "n8adot0jno"
            ]
        },
        "stats": {
            "total": 10,
            "byCategory": {
                "driver": 3,
                "authentication": 2,
                "file": 2,
                "registry": 2,
                "session": 2,
                "network": 1
            }
        }
    },
    {
        "data": {
            "@timestamp": [
                "2020-11-09T21:13:25.246Z"
            ],
            "process.parent.entity_id": [
                "0i17c2m22c"
            ],
            "process.entity_id": [
                "6k8waczi22"
            ]
        },
        "stats": {
            "total": 10,
            "byCategory": {
                "network": 4,
                "authentication": 3,
                "driver": 3,
                "session": 3
            }
        }
    }
]

...gins/security_solution/server/endpoint/routes/resolver/new_tree/queries/aggregation_utils.ts

...ck/plugins/security_solution/server/endpoint/routes/resolver/new_tree/queries/descendants.ts

…i-fields

jonathan-buttner · 2020-11-16T14:46:27Z

@elasticmachine merge upstream

michaelolo24 · 2020-11-18T15:00:50Z

x-pack/plugins/security_solution/common/endpoint/generate_data.test.ts

+      let startTime = new Date(timestampSafeVersion(tree.allEvents[0]) ?? startOfEpoch);
+      expect(startTime).not.toEqual(startOfEpoch);
+      let endTime = new Date(timestampSafeVersion(tree.allEvents[0]) ?? startOfEpoch);
+      expect(startTime).not.toEqual(startOfEpoch);


should this be endTime?

Yup, thanks, copy/paste issue 😆

michaelolo24 · 2020-11-18T15:03:50Z

x-pack/plugins/security_solution/common/endpoint/generate_data.test.ts

+          startTime = currentEventTime;
+        }
+
+        if (currentEventTime > endTime) {


Don't know how likely it is for currentEventTime to equal endTime or startTime, but maybe just use an else as a catch all?

if (currentEventTime < startTime) { startTime = currentEventTime } else { endTime = currentEventTime }

Good idea 👍

actually they need to be separate if statements haha. That's why I just got a random test failure (also not great, I'll try to fix that too). Using the if-else will mean that endTime will be reset anytime currentEventTime is greater than or equal to startTime, which isn't what we want. We want endTime to be set anytime currentEventTime is greater than endTime otherwise we could set endTime to something less than the greatest event time.

Cool, makes sense. Ignore me, I'm bad with 🕐

michaelolo24 · 2020-11-18T15:04:42Z

x-pack/plugins/security_solution/common/endpoint/generate_data.ts

+      if (eventTimestamp !== undefined) {
+        if (eventTimestamp < startTime) {
+          startTime = eventTimestamp;
+        } else if (eventTimestamp > endTime) {


same thing here? Not sure how likely it is to happen from the generated data 🤷‍♂️

michaelolo24 · 2020-11-18T15:07:25Z

x-pack/plugins/security_solution/common/endpoint/schema/resolver.ts

+export const validateTree = {
+  body: schema.object({
+    // if the ancestry field is specified this field will be ignored
+    descendantLevels: schema.number({ defaultValue: 20, min: 0, max: 1000 }),


Remind me why the ancestry affects the descendantLevels again?

Yeah it affects it because if we have the ancestry field defined than we have a much more performant way of retrieving levels so my thought was to not limit the number of levels that come back in that scenario. We could still limit it, but what we'd likely have to do is get all the levels back like we normally do with the ancestry array, bucket them together by level, and then remove the levels that exceeded the requested number which seems kind of wasteful.

I'll add this as a comment as well 👍

michaelolo24 · 2020-11-18T15:09:02Z

x-pack/plugins/security_solution/common/endpoint/types/index.ts

+ *    "process.entity_id": [
+ *      "6k8waczi22"
+ *    ]
+ *  },


might be worth adding the name to the example as well?

Yeah good point, I'll add name.

jonathan-buttner · 2020-11-18T18:03:24Z

@elasticmachine merge upstream

…i-fields

…ibana into resolver-multi-fields

This reverts commit 9e9abf6.

jonathan-buttner · 2020-11-23T14:02:52Z

@elasticmachine merge upstream

jonathan-buttner · 2020-11-24T14:34:57Z

@elasticmachine merge upstream

kibanamachine · 2020-11-24T16:45:41Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: d182ca3

Metrics [docs]

Distributable file count

id	before	after	diff
`default`	43049	43055	+6

History

💚 Build #89619 succeeded a6aa4b1
💚 Build #88771 succeeded 0694636
💔 Build #88688 failed aaad8c4
💚 Build #88654 succeeded e13bf88
💚 Build #88010 succeeded 0660d81

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

…81679) (#84251) * Trying to flesh out new tree route * Working on the descendants query * Almost working descendants * Possible solution for aggs * Working aggregations extraction * Working on the ancestry array for descendants * Making changes to the unique id for ancestr * Implementing ancestry funcitonality * Deleting the multiple edges * Fleshing out the descendants loop for levels * Writing tests for ancestors and descendants * Fixing type errors and writing more tests * Renaming validation variable and deprecating old tree routes * Renaming tree integration test file * Adding some integration tests * Fixing ancestry to handle multiple nodes in the request and writing more tests * Adding more tests * Renaming new tree to handler file * Renaming new tree directory * Adding more unit tests * Using doc value fields and working on types * Adding comments and more tests * Fixing timestamp test issue * Adding more comments * Fixing timestamp test issue take 2 * Adding id, parent, and name fields to the top level response * Fixing generator start and end time generation * Adding more comments * Revert "Fixing generator start and end time generation" This reverts commit 9e9abf6. * Adding test for time Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

* master: (41 commits) [Maps] fix code-owners (elastic#84265) [@kbn/utils] Clean target before build (elastic#84253) [code coverage] collect for oss integration tests (elastic#83907) [APM] Use `asTransactionRate` consistently everywhere (elastic#84213) Attempt to fix incremental build error (elastic#84152) Unskip "Copy dashboards to space" (elastic#84115) Remove expressions.legacy from README (elastic#79681) Expression: Add render mode and use it for canvas interactivity (elastic#83559) [deb/rpm] Move systemd service to /usr/lib/systemd/system (elastic#83571) [Security Solution][Resolver] Allow a configurable entity_id field (elastic#81679) [ML] Space permision checks for job deletion (elastic#83871) [build] Provide ARM build of RE2 (elastic#84163) TSVB should use "histogram:maxBars" and "histogram:barTarget" settings for auto instead of a default 100 buckets (elastic#83628) [Workplace Search] Initial rendering of Org Sources (elastic#84164) update geckodriver to 0.28 (elastic#84085) Fix timelion vis escapes single quotes (elastic#84196) [Security Solution] Fix incorrect time for dns histogram (elastic#83532) [DX] Bump TS version to v4.1 (elastic#83397) [Security Solution] Add endpoint policy revision number (elastic#83982) [Fleet] Integration Policies List view (elastic#83634) ...

jonathan-buttner added 6 commits October 21, 2020 09:27

Trying to flesh out new tree route

58c6ada

Working on the descendants query

3139b65

Almost working descendants

b561c3d

Possible solution for aggs

8506512

Working aggregations extraction

cf548c6

Working on the ancestry array for descendants

da4af52

jonathan-buttner requested review from oatkiller, bkimmel, michaelolo24 and kqualters-elastic October 26, 2020 21:07

jonathan-buttner added Feature:Resolver Security Solution Resolver feature release_note:skip Skip the PR/issue when compiling release notes Team:Threat Hunting Security Solution Threat Hunting Team v7.11.0 v8.0.0 labels Oct 26, 2020

Making changes to the unique id for ancestr

a9c87d0

michaelolo24 approved these changes Oct 27, 2020

View reviewed changes

michaelolo24 self-requested a review October 27, 2020 14:35

MindyRS added the Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. label Oct 27, 2020

jonathan-buttner added 2 commits October 27, 2020 19:01

Implementing ancestry funcitonality

023f119

Deleting the multiple edges

dd04e44

michaelolo24 reviewed Oct 30, 2020

View reviewed changes

...gins/security_solution/server/endpoint/routes/resolver/new_tree/queries/aggregation_utils.ts Outdated Show resolved Hide resolved

michaelolo24 reviewed Oct 30, 2020

View reviewed changes

...gins/security_solution/server/endpoint/routes/resolver/new_tree/queries/aggregation_utils.ts Outdated Show resolved Hide resolved

michaelolo24 reviewed Oct 30, 2020

View reviewed changes

...ck/plugins/security_solution/server/endpoint/routes/resolver/new_tree/queries/descendants.ts Outdated Show resolved Hide resolved

jonathan-buttner added 6 commits October 30, 2020 15:39

Fleshing out the descendants loop for levels

109c09f

Writing tests for ancestors and descendants

43423a0

Fixing type errors and writing more tests

0d45d15

Renaming validation variable and deprecating old tree routes

7e2ac9e

Renaming tree integration test file

c05c795

Merge branch 'master' of github.com:elastic/kibana into resolver-mult…

893eb46

…i-fields

jonathan-buttner added 4 commits November 10, 2020 11:25

Merge branch 'master' of github.com:elastic/kibana into resolver-mult…

b4fd18e

…i-fields

Fixing timestamp test issue take 2

400331c

Merge branch 'master' of github.com:elastic/kibana into resolver-mult…

e8ec0a0

…i-fields

Adding id, parent, and name fields to the top level response

619088c

Merge branch 'master' into resolver-multi-fields

0660d81

michaelolo24 reviewed Nov 18, 2020

View reviewed changes

michaelolo24 approved these changes Nov 18, 2020

View reviewed changes

kibanamachine and others added 7 commits November 18, 2020 13:03

Merge branch 'master' into resolver-multi-fields

e13bf88

Merge branch 'master' of github.com:elastic/kibana into resolver-mult…

d62acac

…i-fields

Merge branch 'resolver-multi-fields' of github.com:jonathan-buttner/k…

7da1579

…ibana into resolver-multi-fields

Fixing generator start and end time generation

9e9abf6

Adding more comments

aaad8c4

Revert "Fixing generator start and end time generation"

1610a48

This reverts commit 9e9abf6.

Adding test for time

0694636

Merge branch 'master' into resolver-multi-fields

a6aa4b1

jonathan-buttner mentioned this pull request Nov 23, 2020

[Security Solution][Resolver] Add support for predefined schemas for endpoint and winlogbeat #84103

Merged

Merge branch 'master' into resolver-multi-fields

d182ca3

jonathan-buttner merged commit 5e183dd into elastic:master Nov 24, 2020

jonathan-buttner mentioned this pull request Nov 24, 2020

[7.x] [Security Solution][Resolver] Allow a configurable entity_id field (#81679) #84251

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Security Solution][Resolver] Allow a configurable entity_id field #81679

[Security Solution][Resolver] Allow a configurable entity_id field #81679

jonathan-buttner commented Oct 26, 2020 •

edited

Loading

jonathan-buttner commented Nov 16, 2020

michaelolo24 Nov 18, 2020

jonathan-buttner Nov 18, 2020

michaelolo24 Nov 18, 2020

jonathan-buttner Nov 18, 2020

jonathan-buttner Nov 18, 2020

michaelolo24 Nov 18, 2020

michaelolo24 Nov 18, 2020 •

edited

Loading

michaelolo24 Nov 18, 2020

jonathan-buttner Nov 18, 2020

michaelolo24 Nov 18, 2020

jonathan-buttner Nov 18, 2020 •

edited

Loading

jonathan-buttner commented Nov 18, 2020

jonathan-buttner commented Nov 23, 2020

jonathan-buttner commented Nov 24, 2020

kibanamachine commented Nov 24, 2020

[Security Solution][Resolver] Allow a configurable entity_id field #81679

[Security Solution][Resolver] Allow a configurable entity_id field #81679

Conversation

jonathan-buttner commented Oct 26, 2020 • edited Loading

Testing

jonathan-buttner commented Nov 16, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

michaelolo24 Nov 18, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jonathan-buttner Nov 18, 2020 • edited Loading

Choose a reason for hiding this comment

jonathan-buttner commented Nov 18, 2020

jonathan-buttner commented Nov 23, 2020

jonathan-buttner commented Nov 24, 2020

kibanamachine commented Nov 24, 2020

💚 Build Succeeded

Metrics [docs]

Distributable file count

History

jonathan-buttner commented Oct 26, 2020 •

edited

Loading

michaelolo24 Nov 18, 2020 •

edited

Loading

jonathan-buttner Nov 18, 2020 •

edited

Loading