Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Security Solution][CTI] Investigation time enrichment UI #103383

Merged
merged 42 commits into from
Jun 30, 2021

Commits on Jun 24, 2021

  1. Add pure fn and consuming hook to fetch event enrichment

    It's not being invoked yet, but I've added a placeholder where it's
    going.
    rylnd committed Jun 24, 2021
    Configuration menu
    Copy the full SHA
    2b68cc3 View commit details
    Browse the repository at this point in the history
  2. Move existing enrichment tests to new spec file

    This is a rough copy/paste, I'll clean up as I flesh out the new tests.
    rylnd committed Jun 24, 2021
    Configuration menu
    Copy the full SHA
    a85aa96 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    7978796 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    18b22fc View commit details
    Browse the repository at this point in the history
  5. Extract some inline parsing logic into a helper function

    And test it!
    rylnd committed Jun 24, 2021
    Configuration menu
    Copy the full SHA
    7d6ffc9 View commit details
    Browse the repository at this point in the history
  6. Solidifying enrichment types on the backend

    * Declares an enum for our types
    * Sets type during indicator match rule enrichment
    * Sets type during investigation-time enrichment
    rylnd committed Jun 24, 2021
    Configuration menu
    Copy the full SHA
    3dafc7c View commit details
    Browse the repository at this point in the history
  7. WIP: Enrichment rows are rendered on the alerts summary

    There are lots of TODOs here, but this implements the following:
    
    * Fetching investigation-time enrichments from the backend
    * Parsing existing enrichments from timeline data
    * Merging the two enrichment types together, and rendering them in rows
      as specified
    
    Much of the data-fetching is hardcoded, and this broke the existing
    pattern with SummaryView/SummaryRow so that got a little messy; I may
    end up just using my own EuiTable but we'll see.
    
    Threat Intel tab is currently broken; that's up next.
    rylnd committed Jun 24, 2021
    Configuration menu
    Copy the full SHA
    d7e6f9d View commit details
    Browse the repository at this point in the history
  8. Updates ThreatDetailsView to accept an array of enrichments

    The investigation-time enrichments are a little messy because they
    contain all the non-ECS fields that indicators contain; other than that,
    this is looking good.
    
    Still need to add the new header, and potentially sort the fields.
    rylnd committed Jun 24, 2021
    Configuration menu
    Copy the full SHA
    7216655 View commit details
    Browse the repository at this point in the history
  9. Sort our details fields

    This promotes sanity for the user.
    rylnd committed Jun 24, 2021
    Configuration menu
    Copy the full SHA
    ebd1b9d View commit details
    Browse the repository at this point in the history
  10. Add "view threat intel data" button

    This simply opens the threat intel tab.
    rylnd committed Jun 24, 2021
    Configuration menu
    Copy the full SHA
    dcfa72c View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    1e1690f View commit details
    Browse the repository at this point in the history

Commits on Jun 25, 2021

  1. Configuration menu
    Copy the full SHA
    2a76394 View commit details
    Browse the repository at this point in the history
  2. Fix remaining tests for components we modified

    This also addresses a bug where we were not properly sorting new
    enrichments by first_seen; this is covered under the tests that were
    fixed.
    rylnd committed Jun 25, 2021
    Configuration menu
    Copy the full SHA
    6a2adf3 View commit details
    Browse the repository at this point in the history
  3. Filter out duplicate investigation-time enrichments

    Because the enrichment endpoint is dumb and doesn't know about the
    existing event or its enrichments, we need to merge these together on
    the client to reduce noise and redundant data.
    rylnd committed Jun 25, 2021
    Configuration menu
    Copy the full SHA
    c823df1 View commit details
    Browse the repository at this point in the history
  4. Add inspect button to investigation enrichments

    * Massages the response into the format that the inspect component uses
    * Moves stateful fetching of query and persisting in redux to new, more
      specialized hook
    * Moves existing enrichment hook to a more suitable location in
      containers/
    rylnd committed Jun 25, 2021
    Configuration menu
    Copy the full SHA
    57e5fe3 View commit details
    Browse the repository at this point in the history

Commits on Jun 26, 2021

  1. Fix failing unit tests

    * indicator match rule now specifies `matched.type` as coming from the
      rule
    * Inspecting the enrichment query requires use of the redux store, which
      was not previously mocked
    rylnd committed Jun 26, 2021
    Configuration menu
    Copy the full SHA
    0bc1eaf View commit details
    Browse the repository at this point in the history
  2. Fix existing CTI cypress tests

    This covers the basics of the Alert Summary and Threat Intel tabs; the
    investigation-time enrichment functionality is up next.
    rylnd committed Jun 26, 2021
    Configuration menu
    Copy the full SHA
    67033bc View commit details
    Browse the repository at this point in the history

Commits on Jun 28, 2021

  1. Adds a cypress test exercising investigation time enrichment

    * Loads more indicators (filebeat data, `threat_indicator2` archive)
      AFTER the rule has executed
    * Asserts that those indicators are also found on the alert summary.
    rylnd committed Jun 28, 2021
    Configuration menu
    Copy the full SHA
    35eb550 View commit details
    Browse the repository at this point in the history
  2. Populate event enrichment call with actual alert fields

    This was previously hardcoded during development.
    rylnd committed Jun 28, 2021
    Configuration menu
    Copy the full SHA
    09b7f7c View commit details
    Browse the repository at this point in the history
  3. Add a new field to our suspicious event to trigger enrichment

    The existing myhash field will generate an alert due to the way the rule
    is written, but the alert had no other fields that would match the
    investigation time enrichment. This gives it a source.ip, and updates
    the indicator to match.
    rylnd committed Jun 28, 2021
    Configuration menu
    Copy the full SHA
    7838a5c View commit details
    Browse the repository at this point in the history
  4. Only fetch enrichments data if there are valid event fields

    If none of the alert's fields would be relevant to the enrichment query,
    then we don't make the request at all.
    rylnd committed Jun 28, 2021
    Configuration menu
    Copy the full SHA
    ba91b57 View commit details
    Browse the repository at this point in the history
  5. Update enrichments matched.typed in integration tests

    This field was updated to reflect the source of the match, in this case:
    indicator match rules.
    rylnd committed Jun 28, 2021
    Configuration menu
    Copy the full SHA
    14a0978 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    7193073 View commit details
    Browse the repository at this point in the history
  7. Ensure draggable fields are unique in a multi-match scenario

    If a given field matched multiple indicators, then the previous
    contextId was not unique as it was based on field/value that matched.
    Adding provider to the mix would fix it, except that we're not
    guaranteed to have a provider.
    
    I've added both provider (if present) and an index value to the key to
    ensure that it's unique.
    rylnd committed Jun 28, 2021
    Configuration menu
    Copy the full SHA
    f48ffe8 View commit details
    Browse the repository at this point in the history
  8. Simplify types

    This field can never be null, as we always set it in our response.
    rylnd committed Jun 28, 2021
    Configuration menu
    Copy the full SHA
    accb629 View commit details
    Browse the repository at this point in the history
  9. Move helper functioons out of shared location and into consuming comp…

    …onent
    
    These are unlikely to be used elsewhere.
    rylnd committed Jun 28, 2021
    Configuration menu
    Copy the full SHA
    5a94f99 View commit details
    Browse the repository at this point in the history
  10. Clean up data parsing logic using reduce

    This obviates the need for our filter/guard function and the extra loop
    that it entails. We have to specify the return value of our reduce fn,
    however, but that's mostly equivalent to our type guard.
    rylnd committed Jun 28, 2021
    Configuration menu
    Copy the full SHA
    18c96fa View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    0196d72 View commit details
    Browse the repository at this point in the history
  12. Extract the concept of "enrichment identifiers"

    This was already partially codified with 'buildEnrichmentId,' which is
    used to dedup enrichments; this extends the idea to all fields that
    could uniquely identify a given indicator.
    rylnd committed Jun 28, 2021
    Configuration menu
    Copy the full SHA
    fabefb7 View commit details
    Browse the repository at this point in the history
  13. Use existing constant as the source of our enrichments query

    This is now used by both the overview card and the enrichment query.
    rylnd committed Jun 28, 2021
    Configuration menu
    Copy the full SHA
    81dd927 View commit details
    Browse the repository at this point in the history
  14. Configuration menu
    Copy the full SHA
    48d3f6b View commit details
    Browse the repository at this point in the history

Commits on Jun 29, 2021

  1. Remove unnecessary flexbox

    The generic SummaryView component previously had to deal with
    multi-valued CTI fields, representing the multiple values coming from
    the multiple nested objects with that field.
    
    However, with the new UI we no longer have that constraint, and so the
    default columnar style, and the corresponding overriding styles, are no
    longer necessary.
    rylnd committed Jun 29, 2021
    Configuration menu
    Copy the full SHA
    866cb27 View commit details
    Browse the repository at this point in the history
  2. Filter out partial responses in the event enrichment observable

    The UI does not currently handle these. We need to test the behavior of
    long-running queries with this filter, but this should simplify the
    behavior to complete/error until we handle partial responses.
    rylnd committed Jun 29, 2021
    Configuration menu
    Copy the full SHA
    f1e843c View commit details
    Browse the repository at this point in the history
  3. Display placeholders while event enrichment is loading

    Displays a loading spinner in the Threat Intel tab title, and some
    loading lines where the enrichments summary is.
    rylnd committed Jun 29, 2021
    Configuration menu
    Copy the full SHA
    d2aa4cd View commit details
    Browse the repository at this point in the history
  4. Update our indicator data to be within the last 30 days

    This fixes our cypress test, but it's going to start failing again in 30
    days. However, by that time I'll have implemented the absolute data
    picker, which will allow for a more comprehensive test in addition to us
    sidestepping this issue.
    rylnd committed Jun 29, 2021
    Configuration menu
    Copy the full SHA
    811174c View commit details
    Browse the repository at this point in the history
  5. Fix type error with our details tabs

    The name prop on a Tab will be rendered as a node, so both strings and
    elements are acceptable. This relaxes the types to inherit from the
    component itself.
    rylnd committed Jun 29, 2021
    Configuration menu
    Copy the full SHA
    3fdd33b View commit details
    Browse the repository at this point in the history
  6. Fix failing jest tests

    The addition of our filtering of the search observable broke this test,
    since we now need to implement the search observable.
    
    Rather than do that, we'll instead mock our local hook as that's more
    likely to change.
    rylnd committed Jun 29, 2021
    Configuration menu
    Copy the full SHA
    9381125 View commit details
    Browse the repository at this point in the history
  7. Merge branch 'master' into ad_hoc_enrichment_ui

     Conflicts:
    	x-pack/plugins/security_solution/cypress/integration/detection_rules/indicator_match_rule.spec.ts
    rylnd committed Jun 29, 2021
    Configuration menu
    Copy the full SHA
    ea1128d View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    52bdb2f View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    afe8d70 View commit details
    Browse the repository at this point in the history

Commits on Jun 30, 2021

  1. Skips flaky cypress test

    See details on elastic#84020
    rylnd committed Jun 30, 2021
    Configuration menu
    Copy the full SHA
    8275e05 View commit details
    Browse the repository at this point in the history
  2. Fix archive mappings to fix cypress test failure

    This override test is failing due to a mapping conflict, which causes
    the rule name override field to be invalid and not persisted.
    
    The mapping conflict is due to a typo in the threat_indicator2 archive,
    where the document declares one index but the mappings declare another.
    While es_archive happily loads the document into the index it specifies,
    the mappings already exist via the threat_indicator archive, and thus
    the new threat_indicator2 index receives dynamic mappings.
    
    Additionally, because the index created by threat_indicator2 is not the
    index specified in threat_indicator2's mappings, the behavior seems to
    be to leave that index when calling unload!
    
    Thus, on the next test that cares (override.spec.ts), we have an errant
    filebeat (threat_indicator2) index with default mappings, causing
    mapping conflicts with our other ECS indexes.
    rylnd committed Jun 30, 2021
    Configuration menu
    Copy the full SHA
    e007972 View commit details
    Browse the repository at this point in the history