Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enhance Datadog Integration Docs: Add serviceMetric Resource #1502

Merged
merged 3 commits into from
Aug 19, 2024
Merged

Enhance Datadog Integration Docs: Add serviceMetric Resource #1502

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
27a00f4
Add service metrics to datadog
phalbert Aug 16, 2024
f5d60cb
Remove unused metadata field
phalbert Aug 19, 2024
d051c53
Merge branch 'main' into PORT-9179-add-service-metrics-to-datadog
Tankilevitch Aug 19, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
279 changes: 270 additions & 9 deletions docs/build-your-software-catalog/sync-data-to-catalog/apm-alerting/datadog.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,11 +11,11 @@ import OceanSaasInstallation from "/docs/build-your-software-catalog/sync-data-t

# Datadog

Our Datadog integration allows you to import `monitors` (also known as alerts), `services`, `slos`, and `sloHistory` from your Datadog account into Port, according to your mapping and definition.
Our Datadog integration allows you to import `monitors` (also known as alerts), `services`, `slos`, `sloHistory`, and `serviceMetrics` from your Datadog account into Port, according to your mapping and definition.

## Common use cases

- Map monitors, services, slos and slo history in your Datadog workspace environment.
- Map monitors, services, slos, slo history, and service metrics in your Datadog workspace environment.
- Watch for object changes (create/update/delete) in real-time, and automatically apply the changes to your entities in Port.
- Create/delete Datadog objects using self-service actions.

Expand Down Expand Up @@ -356,7 +356,6 @@ Make sure to [configure the following GitLab variables](https://docs.gitlab.com/

<br/>


Here is an example for `.gitlab-ci.yml` pipeline file:

```yaml showLineNumbers
Expand All @@ -366,7 +365,7 @@ default:
- docker:24.0.5-dind
before_script:
- docker info

variables:
INTEGRATION_TYPE: datadog
VERSION: latest
Expand Down Expand Up @@ -456,6 +455,7 @@ The following resources can be used to map data from Datadog, it is possible to
- [`Service`](https://docs.datadoghq.com/api/latest/service-definition/#get-all-service-definitions)
- [`SLO`](https://docs.datadoghq.com/api/latest/service-level-objectives/#get-all-slos)
- [`SLO History`](https://docs.datadoghq.com/api/latest/service-level-objectives/#get-an-slos-history)
- [`Service Metric`](https://docs.datadoghq.com/api/latest/metrics/#query-timeseries-points)
:::

- The root key of the integration configuration is the `resources` key:
Expand Down Expand Up @@ -871,9 +871,11 @@ resources:
.monitor_tags + .tags | map(select(startswith("service:"))) |
unique | map(split(":")[1])
```

:::tip Service Relation
Based on the [best practices for tagging infrastructure](https://www.datadoghq.com/blog/tagging-best-practices/), the default mapping connects SLOs to services using tags that starts with the `service` keyword.
:::

</details>

### SLO history
Expand Down Expand Up @@ -965,11 +967,148 @@ resources:
relations:
slo: .slo.id
```

:::tip Service Relation
Based on the [best practices for tagging infrastructure](https://www.datadoghq.com/blog/tagging-best-practices/), the default JQ maps SLOs to services using tags that starts with the `service` keyword
:::

</details>

### Service Metric

<details>
<summary>Service Metric blueprint</summary>

```json showLineNumbers
{
"identifier": "datadogServiceMetric",
"description": "This blueprint represents a Datadog service metric",
"title": "Datadog Service Metric",
"icon": "Datadog",
"schema": {
"properties": {
"query": {
"type": "string",
"title": "Query",
"description": "The Datadog query used to retrieve this metric"
},
"series": {
"type": "array",
"title": "Series",
"description": "Array containing detailed information about the metric series"
},
"res_type": {
"type": "string",
"title": "Response Type",
"description": "The type of response from the Datadog API"
},
"from_date": {
"type": "string",
"format": "date-time",
"title": "From Date",
"description": "Unix timestamp of the start of the queried time period"
},
"to_date": {
"type": "string",
"format": "date-time",
"title": "To Date",
"description": "Unix timestamp of the end of the queried time period"
},
"env": {
"type": "string",
"title": "Environment",
"description": "The environment of the service"
}
},
"required": []
},
"mirrorProperties": {},
"calculationProperties": {},
"relations": {
"service": {
"title": "Service",
"description": "The service associated with this query",
"target": "datadogService",
"required": false,
"many": false
}
}
}
```

</details>

<details>
<summary>Integration configuration</summary>

:::tip Configuration Mapping for Flexible Metric Fetching
The `datadogSelector` section within each `serviceMetric` resource demonstrates how to fetch multiple metrics (e.g., `system.mem.used`, `system.disk.used`) from Datadog with a variety of filters. You can control the:

* **Metric:** Specify the exact metric name (e.g., `avg:system.mem.used`)
* **Environment:** Filter by specific environments (e.g., `prod`, or use `*` for all)
* **Service:** Filter by specific services in your [datadog service catalog](https://docs.datadoghq.com/service_catalog/) (use `*` for all)
* **Timeframe:** Define the time range for data retrieval (in minutes)

This configuration allows you to tailor your data fetching to specific needs and scenarios.
:::

```yaml showLineNumbers
createMissingRelatedEntities: true
deleteDependentEntities: true
resources:
- kind: serviceMetric
selector:
query: "true"
datadogSelector:
metric: "avg:system.mem.used"
env: "*"
service: "*"
timeframe: 10
port:
entity:
mappings:
blueprint: '"datadogServiceMetric"'
identifier: .__query_id
title: .query
properties:
query: .__query
series: .series
res_type: .res_type
from_date: ".from_date / 1000 | todate"
to_date: ".to_date / 1000 | todate"
env: .__env
relations:
service: .__service
- kind: serviceMetric
selector:
query: "true"
datadogSelector:
metric: "avg:system.disk.used"
env: "prod"
service: "*"
timeframe: 5
port:
entity:
mappings:
blueprint: '"datadogServiceMetric"'
identifier: .__query_id
title: .query
properties:
query: .__query
series: .series
res_type: .res_type
from_date: ".from_date / 1000 | todate"
to_date: ".to_date / 1000 | todate"
env: .__env
relations:
service: .__service
```

:::tip Service Relation
Based on the [best practices for tagging infrastructure](https://www.datadoghq.com/blog/tagging-best-practices/), the default JQ maps service metrics to services using tags that starts with the `service` keyword
:::

</details>

## Let's Test It

Expand Down Expand Up @@ -1004,7 +1143,7 @@ Here is an example of the payload structure from Datadog:
"notify_no_data":false,
"timeout_h":0,
"silenced":{

}
},
"multi":true,
Expand All @@ -1027,7 +1166,7 @@ Here is an example of the payload structure from Datadog:
"id":1001199545
},
"matching_downtimes":[

]
}
```
Expand Down Expand Up @@ -1115,13 +1254,13 @@ Here is an example of the payload structure from Datadog:
}
}
}
}
}
}
}
```

</details>


<details>
<summary> SLO response data</summary>

Expand Down Expand Up @@ -1166,6 +1305,7 @@ Here is an example of the payload structure from Datadog:
</details>

<details>

<summary> SLO history response data</summary>

```json showLineNumbers
Expand Down Expand Up @@ -1234,7 +1374,97 @@ Here is an example of the payload structure from Datadog:
"corrections": [],
"state": "breached"
}
}
}
```

</details>

<details>

<summary> Service metric response data</summary>

:::tip Response Enrichment
The Datadog response is enriched with a variety of metadata fields, including:

- `__service`: The name or identifier of the service generating the data.
- `__query_id`: A unique identifier for the query that generated the data.
- `__query`: The original query used to retrieve the data.
- `__env`: The environment associated with the data (e.g., production, staging).

This enrichment significantly enhances the usability of the Datadog response by providing valuable context and facilitating easier analysis and troubleshooting.
:::

```json showLineNumbers
{
"status": "ok",
"res_type": "time_series",
"resp_version": 1,
"query": "avg:system.mem.used{service:inventory-management,env:staging}",
"from_date": 1723796537000,
"to_date": 1723797137000,
"series": [
{
"unit": [
{
"family": "bytes",
"id": 2,
"name": "byte",
"short_name": "B",
"plural": "bytes",
"scale_factor": 1.0
}
],
"query_index": 0,
"aggr": "avg",
"metric": "system.mem.used",
"tag_set": [],
"expression": "avg:system.mem.used{env:staging,service:inventory-management}",
"scope": "env:staging,service:inventory-management",
"interval": 2,
"length": 39,
"start": 1723796546000,
"end": 1723797117000,
"pointlist": [
[1723796546000.0, 528986112.0],
[1723796562000.0, 531886080.0],
[1723796576000.0, 528867328.0],
[1723796592000.0, 522272768.0],
[1723796606000.0, 533704704.0],
[1723796846000.0, 533028864.0],
[1723796862000.0, 527417344.0],
[1723796876000.0, 531513344.0],
[1723796892000.0, 533577728.0],
[1723796906000.0, 533471232.0],
[1723796922000.0, 528125952.0],
[1723796936000.0, 530542592.0],
[1723796952000.0, 530767872.0],
[1723796966000.0, 526966784.0],
[1723796982000.0, 528560128.0],
[1723796996000.0, 530792448.0],
[1723797012000.0, 527384576.0],
[1723797026000.0, 529534976.0],
[1723797042000.0, 521650176.0],
[1723797056000.0, 531001344.0],
[1723797072000.0, 525955072.0],
[1723797086000.0, 529469440.0],
[1723797102000.0, 532279296.0],
[1723797116000.0, 526979072.0]
],
"display_name": "system.mem.used",
"attributes": {}
}
],
"values": [],
"times": [],
"message": "",
"group_by": [],
// highlight-start
"__service": "inventory-management",
"__query_id": "avg:system.mem.used/service:inventory-management/env:staging",
"__query": "avg:system.mem.used",
"__env": "staging"
// highlight-end
}
```

</details>
Expand Down Expand Up @@ -1375,6 +1605,37 @@ The combination of the sample payload and the Ocean configuration generates the
"updatedBy": "<port-client-id>"
}
```

</details>

<details>
<summary>Service metric entity in Port</summary>

```json showLineNumbers
{
"identifier": "avg:system.disk.used/service:inventory-management/env:prod",
"title": "avg:system.disk.used{service:inventory-management,env:prod}",
"icon": null,
"blueprint": "datadogServiceMetric",
"team": [],
"properties": {
"query": "avg:system.disk.used",
"series": [],
"res_type": "time_series",
"from_date": "2024-08-16T07:32:00Z",
"to_date": "2024-08-16T08:02:00Z",
"env": "prod"
},
"relations": {
"service": "inventory-management"
},
"createdAt": "2024-08-15T15:54:36.638Z",
"createdBy": "<port-client-id>",
"updatedAt": "2024-08-16T08:02:02.399Z",
"updatedBy": "<port-client-id>"
}
```

</details>

## Alternative installation via webhook
Expand Down