diff --git a/content/en/observability_pipelines/guide/_index.md b/content/en/observability_pipelines/guide/_index.md index 3d18803abca72..897b08f05edbb 100644 --- a/content/en/observability_pipelines/guide/_index.md +++ b/content/en/observability_pipelines/guide/_index.md @@ -5,5 +5,10 @@ disable_toc: false {{< whatsnext desc="General guides:" >}} {{< nextlink href="observability_pipelines/guide/strategies_for_reducing_log_volume" >}}Strategies for reducing log volume{{< /nextlink >}} +{{< /whatsnext >}} + +{{< whatsnext desc="Processor guides:" >}} {{< nextlink href="observability_pipelines/guide/get_started_with_the_custom_processor" >}}Get started with the Custom Processor{{< /nextlink >}} + {{< nextlink href="observability_pipelines/guide/remap_reserved_attributes" >}}Remap reserved attributes{{< /nextlink >}} + {{< /whatsnext >}} \ No newline at end of file diff --git a/content/en/observability_pipelines/guide/add_field_remap.png b/content/en/observability_pipelines/guide/add_field_remap.png new file mode 100644 index 0000000000000..e5780607d4792 Binary files /dev/null and b/content/en/observability_pipelines/guide/add_field_remap.png differ diff --git a/content/en/observability_pipelines/guide/remap_reserved_attributes.md b/content/en/observability_pipelines/guide/remap_reserved_attributes.md new file mode 100644 index 0000000000000..e37c07c31574a --- /dev/null +++ b/content/en/observability_pipelines/guide/remap_reserved_attributes.md @@ -0,0 +1,78 @@ +--- +title: Remap Reserved Attributes +disable_toc: false +further_reading: +- link: "observability_pipelines/processors/edit_fields/" + tag: "Documentation" + text: "Learn more about the Edit Fields processor" +- link: "observability_pipelines/processors/custom_processor/" + tag: "Documentation" + text: "Learn more about the Custom Processor processor" +--- + +## Overview + +Observability Pipelines' processors enable you to add, edit, and remove log fields. Remapping attributes or rewriting values ensures your logs are processed and standardized properly. For the majority of processing use cases, use the Edit Fields processor to add, remap, or remove fields from your logs. For advanced use cases, use the Custom Processor to conditionally modify fields or rewrite a field's value. + +In Datadog, [reserved attributes][1] are log fields that are set aside for specific processing in the platform. Reserved attributes include ` host`, `source`, `status`, `service`, `trace_id`, and `message`. Reserved attributes are applied when routing logs to the following Observability Pipelines destinations: + +- Datadog Logs +- Amazon S3 (for Log Archives) +- Azure Blob Storage (for Log Archives) +- Google Cloud Storage (for Log Archives) + +There are restrictions in Observability Pipelines on how you can modify reserved attributes. For example, reserved attributes cannot be renamed using the Rename Field processor, but must be remapped instead. This guide walks you through the steps to remap the value of reserved attributes. + +## Remap the value of reserved attributes + +To change or override the value of an existing reserved attribute field, Datadog recommends two approaches using Observability Pipelines. The first uses the Edit Fields processor, and the second uses the Custom Processor. + +### Use an Edit Fields processor for basic field assignments + +1. Use a **Remove field** processor to drop the reserved attribute from the log. +2. Use an **Add field** processor to add the reserved attribute back to the log with your correct field name and value assignment. + +**Note**: In terms of the processor order, the **Add Field** processor should go immediately after the **Remove Field** processor to ensure correct field remapping. + +#### Example +The **Remove field** processor image below removes the improperly named `service` field from the log. + +{{< img src="observability_pipelines/guide/remap_attributes/remove_field_remap.png" alt="A remove field processor that drops the service tag and an add field processor that adds the service field with the value payment-app" style="width:50%;" >}} + +The **Add field** processor image below re-adds the `service` field back with the correct value. + +{{< img src="observability_pipelines/guide/remap_attributes/add_field_remap.png" alt="A remove field processor that drops the service tag and an add field processor that adds the service field with the value payment-app" style="width:50%;" >}} + +### Use the Custom Processor for dynamic or manual assignments + +Use the **Custom Processor** to rewrite the reserved attribute's value. + +#### Dynamically assign the value using template syntax to reference another field's value. + +The following Custom Processor script rewrites the `service` field and dynamically assigns the value of `app_id` to the `service` field's value. + +``` +.service = {{.app_id}} +``` + +In the below example image, the input shows `service` with the value `wrongstatus`. After processing the log with the script, the output shows `service` with the value of `streaming-service`, which is `app_id`'s value. + +{{< img src="observability_pipelines/guide/remap_attributes/custom_processor_dynamically_assign.png" alt="A custom processor showing an input the incorrect status value and the output showing the correct status" style="width:100%;" >}} + +#### Manually rewrite the value of an attribute with a static name + +The following Custom Processor script sets the `status` field to the static value `info`. + +``` +.status = "info" +``` + +In the below example image, the input shows `status` with the value `wrongstatus`. After processing the log with the script, the output shows `status` with `info` as assigned. + +{{< img src="observability_pipelines/guide/remap_attributes/custom_processor_statically_assign.png" alt="A custom processor showing an input the incorrect status value and the output showing the correct status" style="width:100%;" >}} + +## Further reading + +{{< partial name="whats-next/whats-next.html" >}} + +[1]: /logs/log_configuration/attributes_naming_convention/#reserved-attributes \ No newline at end of file diff --git a/content/en/observability_pipelines/guide/remove_fields_remap copy.png b/content/en/observability_pipelines/guide/remove_fields_remap copy.png new file mode 100644 index 0000000000000..9e4b824f0a90a Binary files /dev/null and b/content/en/observability_pipelines/guide/remove_fields_remap copy.png differ diff --git a/content/en/observability_pipelines/processors/custom_processor.md b/content/en/observability_pipelines/processors/custom_processor.md index a68bfc6520311..2fa59027506b3 100644 --- a/content/en/observability_pipelines/processors/custom_processor.md +++ b/content/en/observability_pipelines/processors/custom_processor.md @@ -1,6 +1,10 @@ --- title: Custom Processor disable_toc: false +further_reading: +- link: "/observability_pipelines/guide/remap_reserved_attributes/" + tag: "documentation" + text: "Remap reserved attributes" --- {{< callout src="#" btn_hidden="true" >}} @@ -30,4 +34,8 @@ The Custom Processor is in Preview. Reach out to your account manager to request {{< nextlink href="observability_pipelines/processors/custom_processor/#type" >}}Type{{< /nextlink >}} {{< /whatsnext >}} -{{< vrl-functions >}} \ No newline at end of file +{{< vrl-functions >}} + +## Further reading + +{{< partial name="whats-next/whats-next.html" >}} \ No newline at end of file diff --git a/content/en/observability_pipelines/processors/edit_fields.md b/content/en/observability_pipelines/processors/edit_fields.md index 9be9f3e27045c..a6ba51a69959f 100644 --- a/content/en/observability_pipelines/processors/edit_fields.md +++ b/content/en/observability_pipelines/processors/edit_fields.md @@ -1,8 +1,16 @@ --- title: Edit Fields Processor disable_toc: false +further_reading: +- link: "/observability_pipelines/guide/remap_reserved_attributes/" + tag: "documentation" + text: "Remap reserved attributes" --- {{% observability_pipelines/processors/remap %}} -{{% observability_pipelines/processors/filter_syntax %}} \ No newline at end of file +{{% observability_pipelines/processors/filter_syntax %}} + +## Further reading + +{{< partial name="whats-next/whats-next.html" >}} \ No newline at end of file diff --git a/layouts/shortcodes/observability_pipelines/processors/custom_processor.md b/layouts/shortcodes/observability_pipelines/processors/custom_processor.md index 301e5b0022835..1a72d4925fb95 100644 --- a/layouts/shortcodes/observability_pipelines/processors/custom_processor.md +++ b/layouts/shortcodes/observability_pipelines/processors/custom_processor.md @@ -12,6 +12,8 @@ Use this processor with Vector Remap Language (VRL) to modify and enrich your lo See [Custom functions][10191] for the full list of available functions. +See [Remap Reserved Attributes][10193] on how to use the Custom Processor to manually and dynamically remap attributes. + To set up this processor: - If you have not created any functions yet, click **Add custom processor** and follow the instructions in [Add a function](#add-a-function) to create a function. @@ -27,4 +29,5 @@ To set up this processor: 1. Click **Save**. [10191]: /observability_pipelines/processors/custom_processor#custom-functions -[10192]: /observability_pipelines/guide/get_started_with_the_custom_processor \ No newline at end of file +[10192]: /observability_pipelines/guide/get_started_with_the_custom_processor +[10193]: /observability_pipelines/guide/remap_reserved_attributes \ No newline at end of file diff --git a/layouts/shortcodes/observability_pipelines/processors/remap.en.md b/layouts/shortcodes/observability_pipelines/processors/remap.en.md index cd8a1fcde63dc..9b67897ef54b5 100644 --- a/layouts/shortcodes/observability_pipelines/processors/remap.en.md +++ b/layouts/shortcodes/observability_pipelines/processors/remap.en.md @@ -1,5 +1,7 @@ The remap processor can add, drop, or rename fields within your individual log data. Use this processor to enrich your logs with additional context, remove low-value fields to reduce volume, and standardize naming across important attributes. Select **add field**, **drop field**, or **rename field** in the dropdown menu to get started. +See the [Remap Reserved Attributes][10220] guide on how to use the Edit Fields processor to remap attributes. + ##### Add field Use **add field** to append a new key-value field to your log. @@ -47,3 +49,5 @@ To set up the rename field processor: - Use `outer_key.inner_key` to refer to the key with the value `inner_value`. - Use `outer_key.inner_key.double_inner_key` to refer to the key with the value `double_inner_value`. + +[10220]: /observability_pipelines/guide/remap_reserved_attributes \ No newline at end of file diff --git a/static/images/observability_pipelines/guide/remap_attributes/add_field_remap.png b/static/images/observability_pipelines/guide/remap_attributes/add_field_remap.png new file mode 100644 index 0000000000000..e5780607d4792 Binary files /dev/null and b/static/images/observability_pipelines/guide/remap_attributes/add_field_remap.png differ diff --git a/static/images/observability_pipelines/guide/remap_attributes/custom_processor_dynamically_assign.png b/static/images/observability_pipelines/guide/remap_attributes/custom_processor_dynamically_assign.png new file mode 100644 index 0000000000000..d77f9bc80882a Binary files /dev/null and b/static/images/observability_pipelines/guide/remap_attributes/custom_processor_dynamically_assign.png differ diff --git a/static/images/observability_pipelines/guide/remap_attributes/custom_processor_statically_assign.png b/static/images/observability_pipelines/guide/remap_attributes/custom_processor_statically_assign.png new file mode 100644 index 0000000000000..3b6cd2fb6a44e Binary files /dev/null and b/static/images/observability_pipelines/guide/remap_attributes/custom_processor_statically_assign.png differ diff --git a/static/images/observability_pipelines/guide/remap_attributes/remove_field_remap.png b/static/images/observability_pipelines/guide/remap_attributes/remove_field_remap.png new file mode 100644 index 0000000000000..9e4b824f0a90a Binary files /dev/null and b/static/images/observability_pipelines/guide/remap_attributes/remove_field_remap.png differ