[Task Manager] Workload aggregation caps out at 10k tasks

[gmmorris]

Kibana version: 7.12.0

It seems the aggregation caps out at 10,000 tasks, which means that the aggregation is probably hitting a page and we'er not fetching it.

Task Types are definitely missing in the list when this happens, so we're not providing an accurate image of the workload.

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

[ymao1]

From the example that was provided:

{
 ... other task manager stuff
  "workload": {
    "count": 10000,
    "task_types": {
      "actions:.pagerduty": {
        "count": 91992,
        "status": {
          "failed": 91917,
          "running": 59,
          "idle": 16
        }
      },
     ... other task types
  }
}

It looks like the aggregation is running over all the data but the count that is returned is 10,000 which is the max number returned when track_total_hits is not set to true for a query. reference I believe the aggregation is running over all the data because the number of pagerduty actions is 91,992, which is much greater than 10,000

@gmmorris Should this issue just be to update the count value to reflect the actual number of documents?

[pmuellr]

Oh, my first thought when I saw mention of this was that the problem was the agg was only returning 10 terms (task types), rather than all of them. But I wasn't listening all that closely. I do remember hearing that there were task types we expected to see in the list, that we didn't, so ... somehow we're missing some buckets ...

Ah, just saw the linked PR and can see that is being addressed!

[gmmorris]

I do remember hearing that there were task types we expected to see in the list, that we didn't, so ... somehow we're missing some buckets ...

yeah, indeed.

Ah, just saw the linked PR and can see that is being addressed!

and yay :)