[APM] ML jobs should support metric documents

[sorenlouv]

Currently ML jobs calculate the average duration like:

aggs: {
  duration: {
    avg: { field: "transaction.duration.us" }
  }
}

For histogram based docs (metrics) this should be calculated using the field transaction.duration.histogram:

aggs: {
  duration: {
    avg: { field: "transaction.duration.histogram" }
  }
}

Metric sample document

Click to view sample

{
  "_index": "apm-7.13.0-metric-live-000001",
  "_type": "_doc",
  "_id": "Pyh98HkBevSlma3Qvbow",
  "_score": 1,
  "fields": {
    "transaction.name.text": [
      "POST /add-cart"
    ],
    "observer.name": [
      "instance-0000000000"
    ],
    "service.node.name": [
      "5e88585c93cd551f19c8aadfe40ae4f7376c53f9b92aea108c9f7f2c9278110c"
    ],
    "transaction.result": [
      "HTTP 2xx"
    ],
    "container.id": [
      "5e88585c93cd551f19c8aadfe40ae4f7376c53f9b92aea108c9f7f2c9278110c"
    ],
    "processor.event": [
      "metric"
    ],
    "kubernetes.pod.name": [
      "frontend-node-685bfb4649-ln2sw"
    ],
    "agent.name": [
      "nodejs"
    ],
    "event.outcome": [
      "success"
    ],
    "service.environment": [
      "prod"
    ],
    "_doc_count": [
      15
    ],
    "service.name": [
      "frontend-node"
    ],
    "processor.name": [
      "metric"
    ],
    "observer.version_major": [
      7
    ],
    "observer.hostname": [
      "17d26171120f"
    ],
    "transaction.duration.histogram": [
      {
        "counts": [
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1
        ],
        "values": [
          1146879,
          1163263,
          1384447,
          1597439,
          1843199,
          2342911,
          3325951,
          3817471,
          6127615,
          7176191,
          8257535,
          9043967,
          10747903,
          11075583,
          12058623
        ]
      }
    ],
    "transaction.type": [
      "request"
    ],
    "metricset.name": [
      "transaction"
    ],
    "observer.id": [
      "219aa009-7229-4210-b76f-39d12a5af7f2"
    ],
    "@timestamp": [
      "2021-06-09T11:17:21.842Z"
    ],
    "service.version": [
      "1.6"
    ],
    "timeseries.instance": [
      "frontend-node:POST /add-cart:5b87bf3db1e822f7"
    ],
    "observer.ephemeral_id": [
      "69d4ce18-e4ac-4a87-a033-b4baf4726217"
    ],
    "observer.version": [
      "7.13.0"
    ],
    "ecs.version": [
      "1.8.0"
    ],
    "observer.type": [
      "apm-server"
    ],
    "transaction.name": [
      "POST /add-cart"
    ]
  }
}

[elasticmachine]

Pinging @elastic/apm-ui (Team:apm)

[elasticmachine]

Pinging @elastic/ml-ui (:ml)

[graphaelli]

The ML team has made a number of recommendations related to this effort that we can choose to investigate - we can determine which are required along the way:

• Sort metric data streams by timestamp - report back on Investigate sorting by service.name, transaction.type, service.environment apm-server#5505
• [APM] Shorten ML Job IDs #111037
• [APM] Set start time of ML jobs to -4 weeks by default #111045
• Use dedicated results index for APM jobs "results_index_name" : "apm"

[graphaelli]

It may also make sense to tackle #92839 while creating this new job.

[dgieselaar]

Re-opening due to the revert in #114961

[peteharverson]

Note that when re-implementing the changes made in #111957, the following additions should also be made:

• Add ignore_unavailable: true to the datafeed config indices_options as done for the previous version of the job in [APM] Ignore unavailable indices for ml jobs #117632
• Add "terms": { "_tier": [ "content", "hot", "warm" ]} to the query in the module manifest.json - see [ML] Recognize modules takes a long time to return results #116696 (comment).