Skip to content

[DOCS-11376] Add Remap Reserved Attributes guide #30322

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Jul 8, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions content/en/observability_pipelines/guide/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,5 +5,10 @@ disable_toc: false

{{< whatsnext desc="General guides:" >}}
{{< nextlink href="observability_pipelines/guide/strategies_for_reducing_log_volume" >}}Strategies for reducing log volume{{< /nextlink >}}
{{< /whatsnext >}}

{{< whatsnext desc="Processor guides:" >}}
{{< nextlink href="observability_pipelines/guide/get_started_with_the_custom_processor" >}}Get started with the Custom Processor{{< /nextlink >}}
{{< nextlink href="observability_pipelines/guide/remap_reserved_attributes" >}}Remap reserved attributes{{< /nextlink >}}

{{< /whatsnext >}}
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
---
title: Remap Reserved Attributes
disable_toc: false
further_reading:
- link: "observability_pipelines/processors/edit_fields/"
tag: "Documentation"
text: "Learn more about the Edit Fields processor"
- link: "observability_pipelines/processors/custom_processor/"
tag: "Documentation"
text: "Learn more about the Custom Processor processor"
---

## Overview

Observability Pipelines' processors enable you to add, edit, and remove log fields. Remapping attributes or rewriting values ensures your logs are processed and standardized properly. For the majority of processing use cases, use the Edit Fields processor to add, remap, or remove fields from your logs. For advanced use cases, use the Custom Processor to conditionally modify fields or rewrite a field's value.

In Datadog, [reserved attributes][1] are log fields that are set aside for specific processing in the platform. Reserved attributes include ` host`, `source`, `status`, `service`, `trace_id`, and `message`. Reserved attributes are applied when routing logs to the following Observability Pipelines destinations:

- Datadog Logs
- Amazon S3 (for Log Archives)
- Azure Blob Storage (for Log Archives)
- Google Cloud Storage (for Log Archives)

There are restrictions in Observability Pipelines on how you can modify reserved attributes. For example, reserved attributes cannot be renamed using the Rename Field processor, but must be remapped instead. This guide walks you through the steps to remap the value of reserved attributes.

## Remap the value of reserved attributes

To change or override the value of an existing reserved attribute field, Datadog recommends two approaches using Observability Pipelines. The first uses the Edit Fields processor, and the second uses the Custom Processor.

### Use an Edit Fields processor for basic field assignments

1. Use a **Remove field** processor to drop the reserved attribute from the log.
2. Use an **Add field** processor to add the reserved attribute back to the log with your correct field name and value assignment.

**Note**: In terms of the processor order, the **Add Field** processor should go immediately after the **Remove Field** processor to ensure correct field remapping.

#### Example
The **Remove field** processor image below removes the improperly named `service` field from the log.

{{< img src="observability_pipelines/guide/remap_attributes/remove_field_remap.png" alt="A remove field processor that drops the service tag and an add field processor that adds the service field with the value payment-app" style="width:50%;" >}}

The **Add field** processor image below re-adds the `service` field back with the correct value.

{{< img src="observability_pipelines/guide/remap_attributes/add_field_remap.png" alt="A remove field processor that drops the service tag and an add field processor that adds the service field with the value payment-app" style="width:50%;" >}}

### Use the Custom Processor for dynamic or manual assignments

Use the **Custom Processor** to rewrite the reserved attribute's value.

#### Dynamically assign the value using template syntax to reference another field's value.

The following Custom Processor script rewrites the `service` field and dynamically assigns the value of `app_id` to the `service` field's value.

```
.service = {{.app_id}}
```

In the below example image, the input shows `service` with the value `wrongstatus`. After processing the log with the script, the output shows `service` with the value of `streaming-service`, which is `app_id`'s value.

{{< img src="observability_pipelines/guide/remap_attributes/custom_processor_dynamically_assign.png" alt="A custom processor showing an input the incorrect status value and the output showing the correct status" style="width:100%;" >}}

#### Manually rewrite the value of an attribute with a static name

The following Custom Processor script sets the `status` field to the static value `info`.

```
.status = "info"
```

In the below example image, the input shows `status` with the value `wrongstatus`. After processing the log with the script, the output shows `status` with `info` as assigned.

{{< img src="observability_pipelines/guide/remap_attributes/custom_processor_statically_assign.png" alt="A custom processor showing an input the incorrect status value and the output showing the correct status" style="width:100%;" >}}

## Further reading

{{< partial name="whats-next/whats-next.html" >}}

[1]: /logs/log_configuration/attributes_naming_convention/#reserved-attributes
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
@@ -1,6 +1,10 @@
---
title: Custom Processor
disable_toc: false
further_reading:
- link: "/observability_pipelines/guide/remap_reserved_attributes/"
tag: "documentation"
text: "Remap reserved attributes"
---

{{< callout src="#" btn_hidden="true" >}}
Expand Down Expand Up @@ -30,4 +34,8 @@ The Custom Processor is in Preview. Reach out to your account manager to request
{{< nextlink href="observability_pipelines/processors/custom_processor/#type" >}}Type{{< /nextlink >}}
{{< /whatsnext >}}

{{< vrl-functions >}}
{{< vrl-functions >}}

## Further reading

{{< partial name="whats-next/whats-next.html" >}}
10 changes: 9 additions & 1 deletion content/en/observability_pipelines/processors/edit_fields.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,16 @@
---
title: Edit Fields Processor
disable_toc: false
further_reading:
- link: "/observability_pipelines/guide/remap_reserved_attributes/"
tag: "documentation"
text: "Remap reserved attributes"
---

{{% observability_pipelines/processors/remap %}}

{{% observability_pipelines/processors/filter_syntax %}}
{{% observability_pipelines/processors/filter_syntax %}}

## Further reading

{{< partial name="whats-next/whats-next.html" >}}
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ Use this processor with Vector Remap Language (VRL) to modify and enrich your lo

See [Custom functions][10191] for the full list of available functions.

See [Remap Reserved Attributes][10193] on how to use the Custom Processor to manually and dynamically remap attributes.

To set up this processor:

- If you have not created any functions yet, click **Add custom processor** and follow the instructions in [Add a function](#add-a-function) to create a function.
Expand All @@ -27,4 +29,5 @@ To set up this processor:
1. Click **Save**.

[10191]: /observability_pipelines/processors/custom_processor#custom-functions
[10192]: /observability_pipelines/guide/get_started_with_the_custom_processor
[10192]: /observability_pipelines/guide/get_started_with_the_custom_processor
[10193]: /observability_pipelines/guide/remap_reserved_attributes
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
The remap processor can add, drop, or rename fields within your individual log data. Use this processor to enrich your logs with additional context, remove low-value fields to reduce volume, and standardize naming across important attributes. Select **add field**, **drop field**, or **rename field** in the dropdown menu to get started.

See the [Remap Reserved Attributes][10220] guide on how to use the Edit Fields processor to remap attributes.

##### Add field
Use **add field** to append a new key-value field to your log.

Expand Down Expand Up @@ -47,3 +49,5 @@ To set up the rename field processor:

- Use `outer_key.inner_key` to refer to the key with the value `inner_value`.
- Use `outer_key.inner_key.double_inner_key` to refer to the key with the value `double_inner_value`.

[10220]: /observability_pipelines/guide/remap_reserved_attributes
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading