Skip to content

Add logs threat intel processor #30319

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Add logs threat intel processor #30319

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 13 additions & 6 deletions content/en/logs/log_configuration/processors.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -241,7 +241,7 @@ Use the [Datadog Log Pipeline API endpoint][1] with the following log service re

## Log message remapper

`message` is a key attribute in Datadog. Its value is displayed in the **Content** column of the Log Explorer to provide context on the log. You can use the search bar to find a log by the log message.
`message` is a key attribute in Datadog. Its value is displayed in the **Content** column of the Log Explorer to provide context on the log. You can use the search bar to find a log by the log message.

Use the log message remapper processor to define one or more attributes as the official log message. Define more than one attribute for cases where the attributes might not exist and an alternative is available. For example, if the defined message attributes are `attribute1`, `attribute2`, and `attribute3`, and `attribute1` does not exist, then `attribute2` is used. Similarly, if `attribute2` does not exist, then `attribute3` is used.

Expand Down Expand Up @@ -572,7 +572,7 @@ Returns the following:
Request GET https://app.datadoghq.com/users was answered with response 200
```

**Note**: `http` is an object and cannot be used in a block (`%{http}` fails), whereas `%{http.method}`, `%{http.status_code}`, or `%{http.url}` returns the corresponding value. Blocks can be used on arrays of values or on a specific attribute within an array.
**Note**: `http` is an object and cannot be used in a block (`%{http}` fails), whereas `%{http.method}`, `%{http.status_code}`, or `%{http.url}` returns the corresponding value. Blocks can be used on arrays of values or on a specific attribute within an array.

* For example, adding the block `%{array_ids}` returns:

Expand Down Expand Up @@ -672,15 +672,15 @@ The lookup processor performs the following actions:
* Looks if the current log contains the source attribute.
* Checks if the source attribute value exists in the mapping table.
* If it does, creates the target attribute with the corresponding value in the table.
* Optionally, if it does not find the value in the mapping table, it creates a target attribute with the default fallback value set in the `fallbackValue` field. You can manually enter a list of `source_key,target_value` pairs or upload a CSV file on the **Manual Mapping** tab.
* Optionally, if it does not find the value in the mapping table, it creates a target attribute with the default fallback value set in the `fallbackValue` field. You can manually enter a list of `source_key,target_value` pairs or upload a CSV file on the **Manual Mapping** tab.

{{< img src="logs/log_configuration/processor/lookup_processor_manual_mapping.png" alt="Lookup processor" style="width:80%;">}}

The size limit for the mapping table is 100Kb. This limit applies across all Lookup Processors on the platform. However, Reference Tables support larger file sizes.

* Optionally, if it does not find the value in the mapping table, it creates a target attribute with the value of the reference table. You can select a value for a [Reference Table][101] on the **Reference Table** tab.
{{< img src="logs/log_configuration/processor/lookup_processor_reference_table.png" alt="Lookup processor"

{{< img src="logs/log_configuration/processor/lookup_processor_reference_table.png" alt="Lookup processor"
style="width:80%;">}}


Expand Down Expand Up @@ -939,6 +939,12 @@ Add an attribute value to the end of a target array attribute in the log.
{{% /tab %}}
{{< /tabs >}}

## Threat intel processor

Add the Threat Intel Process to evaluate logs against the table using a specific Indicator of Compromise (IoC) key, such as an IP address. If a match is found, the log is enriched with relevant Threat Intelligence (TI) attributes from the table, which enhances detection, investigation, and response.

For more information, see [Threat Intelligence][9].

## Further Reading

{{< partial name="whats-next/whats-next.html" >}}
Expand All @@ -954,3 +960,4 @@ Add an attribute value to the end of a target array attribute in the log.
[6]: /logs/search_syntax/
[7]: /integrations/guide/reference-tables/
[8]: /tracing/other_telemetry/connect_logs_and_traces/
[9]: /security/threat_intelligence/
Loading