[Fleet] Support multiple ES output in Kibana config file (aka preconfiguration)

[nchaulet]

Why?

Related to #104980

Currently we can only configure one global default output in Fleet. We want to allow users to configure multiple Elasticsearch Output to send integration data or monitoring data.

• to the same Elasticsearch cluster with different settings or different hosts
• to a remote Elasticsearch cluster (Not for 7.16)

Output created using the preconfigure service and the kibana config will not be editable via API or in the UI.

Description

The same way we allow user to configure agent policy, we will allow users to configure outputs in the Kibana config file, and use these outputs in the preconfigured agent policies.

We will not expose outputs management in the UI for 7.16.

We will support output to the same ES cluster as Kibana with different settings or to remote Elasticsearch cluster in this case the user will have to provide a service account token.

Detailed

Data model Saved object

Our data model is already built to support multiple output, we will have to do some tweak:

• allow to configure the data_output_id on a agent policy
• allow to configure the monitoring_output_id on a agent policy
• [] allow to specify a service account token on an output fleet_server_service_token (!We probably want to use encrypted saved object for that).

Agent policy change

• Update the function that create the consolidated policy (.fleet-policy) for agent to use non default outputs, and to pass the service token for remote ES cluster, under fleet_server.service_token
• Update the output permissions creation to generate permissions per output.

Agent Policy example

id: f94867a0-ff5f-11eb-92fb-c5757f79579b
revision: 3
outputs:
  custom_settings:
    type: elasticsearch
    hosts:
      - 'http://test1.fr:9200'
  remote_montoring:
    type: elasticsearch
    hosts:
      - 'http://test2.fr:9200'

output_permissions:
  custom_settings:
    log-1:
      indices:
        - names:
            - logs-generic-default
          privileges:
            - auto_configure
            - create_doc
  remote_montoring:
    _elastic_agent_checks:
      cluster:
        - monitor
      indices:
        - names:
            - logs-elastic_agent-default
            - logs-elastic_agent.elastic_agent-default
            - logs-elastic_agent.apm_server-default
            - logs-elastic_agent.filebeat-default
            - logs-elastic_agent.fleet_server-default
            - logs-elastic_agent.metricbeat-default
            - logs-elastic_agent.osquerybeat-default
            - logs-elastic_agent.packetbeat-default
            - logs-elastic_agent.endpoint_security-default
            - logs-elastic_agent.auditbeat-default
            - logs-elastic_agent.heartbeat-default
            - metrics-elastic_agent-default
            - metrics-elastic_agent.elastic_agent-default
            - metrics-elastic_agent.apm_server-default
            - metrics-elastic_agent.filebeat-default
            - metrics-elastic_agent.fleet_server-default
            - metrics-elastic_agent.metricbeat-default
            - metrics-elastic_agent.osquerybeat-default
            - metrics-elastic_agent.packetbeat-default
            - metrics-elastic_agent.endpoint_security-default
            - metrics-elastic_agent.auditbeat-default
            - metrics-elastic_agent.heartbeat-default
          privileges:
            - auto_configure
            - create_doc
agent:
  monitoring:
    enabled: true
    use_output: remote_montoring
    namespace: default
    logs: true
    metrics: true
inputs:
  - id: f7aca664-3e70-4b7a-bf66-f284a4a7e213
    name: log-1
    revision: 1
    type: logfile
    use_output: custom_settings
    meta:
      package:
        name: log
        version: 0.5.0
    data_stream:
      namespace: default
    streams:
      - id: logfile-log.log-f7aca664-3e70-4b7a-bf66-f284a4a7e213
        data_stream:
          dataset: generic
        paths:
          - /test.log
fleet:
  hosts:
    - 'http://fleet-server:8220'

Config

• We will allow users to configure outputs in the Kibana config file (the same way we allow to configure policy currently)

+ xpack.fleet.outputs:
+  - id: output_custom_host
+     name: Output with custom hosts
+     type:  elasticsearch
+     hosts: ["http://test1.fr", "http://test2.fr"]
+  -  id: remote_output
+     name: Remote Output with custom hosts
+     type:  elasticsearch
+     hosts: ["http://remote1.fr", "http://remote2.fr"]
xpack.fleet.agentPolicies:
    - name: Default policy
+     data_output_id: output_custom_host
+     monitoring_output_id: remote_output

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[nchaulet]

I started to implement the preconfigure API and I think we will have the same issues that preconfigured agent policies has here (#110533) once we have the UI to edit outputs.

I think we should agree on what behavior we should implement:

1. if the output is preconfigured it's not possible to edit through the UI, (I personally like that one it make things really easy and we have only one source of truth)
2. we allow to modify preconfigured output and if the output kibana.yml change we only set the values that has not been modifed by the user.
3. we implement something like the frozen fields for the agent policies
4. Open to other options.

[kpollich]

if the output is preconfigured it's not possible to edit through the UI, (I personally like that one it make things really easy and we have only one source of truth)

On my first read, this seems like the best option to me. Seems like this option leaves the least room for breakage and complexity, and keeps things as strict as possible.

[joshdover]

if the output is preconfigured it's not possible to edit through the UI, (I personally like that one it make things really easy and we have only one source of truth)

+1 to this. Let's keep it simple to make it really easy to understand. Ideally, we'd have a tooltip or similar indicating that this output must be configured via the kibana.yml file.

I also think our first use case for this (the internal Cloud ES URL for the apm-server package) needs this output to not be editable anyways, so I think it makes sense to start with this behavior. I do wonder if we need the ability to 'hide' outputs like this one. Have we spoken with the Cloud team about whether or not this output should be hidden or obfuscated in any way?

[jen-huang]

if the output is preconfigured it's not possible to edit through the UI, (I personally like that one it make things really easy and we have only one source of truth)

+1, it's similar to the is_managed concept in agent policies

I do wonder if we need the ability to 'hide' outputs like this one. Have we spoken with the Cloud team about whether or not this output should be hidden or obfuscated in any way?

That's a fair question when we get to the UI part of this work, but I believe for 7.16 scope this output will definitely be hidden from the user as we're not building a UI.

[mostlyjason]

+1 users should not edit these settings in the UI or API. We should probably hide the service token in the UI too. I added a mockup for this use case in Whimsical and updated the google doc.

[nchaulet]

The scope for 7.16.0 as been updated and only the same cluster is currently supported and this currently implemented in #111002