Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Fleet] Expose TTL option in agent policy settings to support ephemeral agents use case #100617

Closed
6 tasks
jen-huang opened this issue May 25, 2021 · 6 comments · Fixed by #102970
Closed
6 tasks

[Fleet] Expose TTL option in agent policy settings to support ephemeral agents use case #100617

jen-huang opened this issue May 25, 2021 · 6 comments · Fixed by #102970
Assignees
@jfsiii
Labels
enhancement New value added to drive a business result Team:Fleet Team label for Observability Data Collection Fleet team v7.14.0

Comments

@jen-huang
Copy link
Contributor

jen-huang commented May 25, 2021 •
edited by blakerouse
Loading

Some agents will be enrolled as ephermal agents, see elastic/beats#25030 for details. To support this use case, we want to allow users to specify a TTL setting in their agent policies, that would get sent down to the Fleet Server to handle unenrollment and revoking of API keys:

  • Work with Agent/Fleet Server team to decide on TTL field name in agent policy: Unenroll ephemeral Agents fleet-server#446
  • Add mapping/type/schema definition for the new field in agent policy saved object
  • Show input field labelled Unenrollment timeout in agent policy settings that reads/writes to the new field
    • In Create agent flyout, this would live under Advanced options
  • Ensure that this field can be populated using preconfigured agent policies defined in kibana.yml
    • Ensure that this field can be populated if the user has a preconfigured policy that does not have this field initially, but then updates their kibana.yml later to include it
@jen-huang jen-huang added enhancement New value added to drive a business result Team:Fleet Team label for Observability Data Collection Fleet team v7.14.0 labels May 25, 2021
@elasticmachine
Copy link
Contributor

Pinging @elastic/fleet (Team:Fleet)

@jfsiii jfsiii mentioned this issue May 26, 2021
Closed
4 tasks
@mostlyjason
Copy link
Contributor

Lets put this on old until we finalize definition for ephemeral agents. The current proposal does not define a type of agent. Instead, any agent have a TTL defined by the agent policy.

@jen-huang jen-huang assigned jen-huang and unassigned jfsiii Jun 9, 2021
@jen-huang
Copy link
Contributor Author

jen-huang commented Jun 9, 2021 •
edited
Loading

Assigning back to me for now to define.

@jen-huang jen-huang changed the title [Fleet] Surface ephemeral agent type in UI [Fleet] Expose TTL option in agent policy settings to support ephemeral agents use case Jun 14, 2021
@jen-huang jen-huang assigned jfsiii and unassigned jen-huang Jun 14, 2021
@jen-huang
Copy link
Contributor Author

Hi @jfsiii, I updated the AC here.

@blakerouse
Copy link

I have created a pull request to elasticsearch to add the required fields to the indexes. elastic/elasticsearch#74180

I went with ephemeral boolean on the .fleet-agents and ephemeral_ttl integer (in seconds) on the .fleet-policies.

@blakerouse
Copy link

elastic/elasticsearch#74180 has been merged into elasticsearch and the change for Fleet Server is in review elastic/fleet-server#472

The fields where updated because I was confused on the feature.

unenroll_timeout is set on the policy to un-enroll all Elastic Agents that have not checked in for that amount of time and unenroll_reason was added to mark the reason for the unenrollment as timeout.

@jfsiii jfsiii mentioned this issue Jun 22, 2021
Merged
6 tasks
jfsiii pushed a commit that referenced this issue Jun 23, 2021
[Fleet] Add UI and mappings for agent policy unenroll_timeout (#102970)
bb7bff5 
## Summary

closes #100617

UI and mappings related to ephemeral agents

- [x] Adds mapping/type/schema definition for the new field in agent policy saved object
- [x] Shows input field labelled `Unenrollment timeout` in agent policy settings that reads/writes to the new field
- [x] Same input in `Advanced options` section of create agent flyout
- [x] `unenroll_timeout` can be set using preconfigured agent policies defined in `kibana.yml`
- [x] `unenroll_timeout` can be populated if the user has a preconfigured policy that _does not_ have this field initially, but then updates their `kibana.yml` later to include it

<details><summary>Screenshot - editing an existing agent policy</summary>

<img width="1251" alt="Screen Shot 2021-06-22 at 1 42 50 PM" src="https://user-images.githubusercontent.com/57655/122974924-d479b800-d360-11eb-8d37-306a6860f140.png">

</details>


<details><summary>Screenshots - adding a new agent policy</summary>
<img width="1258" alt="Screen Shot 2021-06-22 at 1 45 01 PM" src="https://user-images.githubusercontent.com/57655/122975141-0be86480-d361-11eb-9ccb-dde0cc44e438.png">
<img width="1257" alt="Screen Shot 2021-06-22 at 1 45 35 PM" src="https://user-images.githubusercontent.com/57655/122975139-0be86480-d361-11eb-90ae-3541dbbd68e6.png">
<img width="1258" alt="Screen Shot 2021-06-22 at 1 45 44 PM" src="https://user-images.githubusercontent.com/57655/122975138-0b4fce00-d361-11eb-8347-d6f0cb43ab91.png">
<img width="1254" alt="Screen Shot 2021-06-22 at 1 45 56 PM" src="https://user-images.githubusercontent.com/57655/122975136-0b4fce00-d361-11eb-9c51-ec7a9e52ea2b.png">
</details>

<details><summary>Using <code>kibana.dev.yml</code></summary>

<h3>No <code>unenroll_timeout</code></h3>

```yml
xpack.fleet.agentPolicies:
  - name: Preconfigured Policy From Config
    description: From kibana.dev.yml (no timeout given)
    id: 1
    namespace: test
    package_policies:
      - package:
          name: system
        name: System Integration
        inputs:
          - type: system/metrics
            enabled: true
            vars:
              - name: system.hostfs
                value: home/test
            streams:
              - data_stream:
                  dataset: system.core
                enabled: true
                vars:
                  - name: period
                    value: 20s
          - type: winlog
            enabled: false
```

<h4>UI (saved object)</h4>
<img width="1072" alt="Screen Shot 2021-06-23 at 10 28 03 AM" src="https://user-images.githubusercontent.com/57655/123117442-bf0f9700-d40f-11eb-8e8f-685fee3ccb4b.png">

<h4><code>fleet-policies</code>index</h4>
<img width="1690" alt="Screen Shot 2021-06-23 at 10 52 39 AM" src="https://user-images.githubusercontent.com/57655/123119430-883a8080-d411-11eb-990f-4953845a76b5.png">


<h4>Updated <code>kibana.dev.yml</code> to include <code>unenroll_timeout</code></h4>

```yml
xpack.fleet.agentPolicies:
  - name: Preconfigured Policy From Config
    description: From kibana.dev.yml (updated with timeout)
    id: 1
    namespace: test
    unenroll_timeout: 234
    package_policies:
      - package:
          name: system
        name: System Integration
        inputs:
          - type: system/metrics
            enabled: true
            vars:
              - name: system.hostfs
                value: home/test
            streams:
              - data_stream:
                  dataset: system.core
                enabled: true
                vars:
                  - name: period
                    value: 20s
          - type: winlog
            enabled: false
```

<h4>UI (saved object)</h4>
<img width="1073" alt="Screen Shot 2021-06-23 at 10 35 17 AM" src="https://user-images.githubusercontent.com/57655/123117444-bfa82d80-d40f-11eb-94a9-99d4ca4bed2a.png">

<h4><code>fleet-policies</code>index</h4>
<img width="1690" alt="Screen Shot 2021-06-23 at 10 35 41 AM" src="https://user-images.githubusercontent.com/57655/123117446-bfa82d80-d40f-11eb-9563-1add80f43f0e.png">

</details>

### Checklist
- [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
kibanamachine pushed a commit to kibanamachine/kibana that referenced this issue Jun 23, 2021
@kibanamachine
[Fleet] Add UI and mappings for agent policy unenroll_timeout (elasti…
6b9ac2f 
…c#102970)

## Summary

closes elastic#100617

UI and mappings related to ephemeral agents

- [x] Adds mapping/type/schema definition for the new field in agent policy saved object
- [x] Shows input field labelled `Unenrollment timeout` in agent policy settings that reads/writes to the new field
- [x] Same input in `Advanced options` section of create agent flyout
- [x] `unenroll_timeout` can be set using preconfigured agent policies defined in `kibana.yml`
- [x] `unenroll_timeout` can be populated if the user has a preconfigured policy that _does not_ have this field initially, but then updates their `kibana.yml` later to include it

<details><summary>Screenshot - editing an existing agent policy</summary>

<img width="1251" alt="Screen Shot 2021-06-22 at 1 42 50 PM" src="https://user-images.githubusercontent.com/57655/122974924-d479b800-d360-11eb-8d37-306a6860f140.png">

</details>


<details><summary>Screenshots - adding a new agent policy</summary>
<img width="1258" alt="Screen Shot 2021-06-22 at 1 45 01 PM" src="https://user-images.githubusercontent.com/57655/122975141-0be86480-d361-11eb-9ccb-dde0cc44e438.png">
<img width="1257" alt="Screen Shot 2021-06-22 at 1 45 35 PM" src="https://user-images.githubusercontent.com/57655/122975139-0be86480-d361-11eb-90ae-3541dbbd68e6.png">
<img width="1258" alt="Screen Shot 2021-06-22 at 1 45 44 PM" src="https://user-images.githubusercontent.com/57655/122975138-0b4fce00-d361-11eb-8347-d6f0cb43ab91.png">
<img width="1254" alt="Screen Shot 2021-06-22 at 1 45 56 PM" src="https://user-images.githubusercontent.com/57655/122975136-0b4fce00-d361-11eb-9c51-ec7a9e52ea2b.png">
</details>

<details><summary>Using <code>kibana.dev.yml</code></summary>

<h3>No <code>unenroll_timeout</code></h3>

```yml
xpack.fleet.agentPolicies:
  - name: Preconfigured Policy From Config
    description: From kibana.dev.yml (no timeout given)
    id: 1
    namespace: test
    package_policies:
      - package:
          name: system
        name: System Integration
        inputs:
          - type: system/metrics
            enabled: true
            vars:
              - name: system.hostfs
                value: home/test
            streams:
              - data_stream:
                  dataset: system.core
                enabled: true
                vars:
                  - name: period
                    value: 20s
          - type: winlog
            enabled: false
```

<h4>UI (saved object)</h4>
<img width="1072" alt="Screen Shot 2021-06-23 at 10 28 03 AM" src="https://user-images.githubusercontent.com/57655/123117442-bf0f9700-d40f-11eb-8e8f-685fee3ccb4b.png">

<h4><code>fleet-policies</code>index</h4>
<img width="1690" alt="Screen Shot 2021-06-23 at 10 52 39 AM" src="https://user-images.githubusercontent.com/57655/123119430-883a8080-d411-11eb-990f-4953845a76b5.png">


<h4>Updated <code>kibana.dev.yml</code> to include <code>unenroll_timeout</code></h4>

```yml
xpack.fleet.agentPolicies:
  - name: Preconfigured Policy From Config
    description: From kibana.dev.yml (updated with timeout)
    id: 1
    namespace: test
    unenroll_timeout: 234
    package_policies:
      - package:
          name: system
        name: System Integration
        inputs:
          - type: system/metrics
            enabled: true
            vars:
              - name: system.hostfs
                value: home/test
            streams:
              - data_stream:
                  dataset: system.core
                enabled: true
                vars:
                  - name: period
                    value: 20s
          - type: winlog
            enabled: false
```

<h4>UI (saved object)</h4>
<img width="1073" alt="Screen Shot 2021-06-23 at 10 35 17 AM" src="https://user-images.githubusercontent.com/57655/123117444-bfa82d80-d40f-11eb-94a9-99d4ca4bed2a.png">

<h4><code>fleet-policies</code>index</h4>
<img width="1690" alt="Screen Shot 2021-06-23 at 10 35 41 AM" src="https://user-images.githubusercontent.com/57655/123117446-bfa82d80-d40f-11eb-9563-1add80f43f0e.png">

</details>

### Checklist
- [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
kibanamachine added a commit that referenced this issue Jun 23, 2021
@kibanamachine
[Fleet] Add UI and mappings for agent policy unenroll_timeout (#102970)…
dbfb32b 
… (#103161)

## Summary

closes #100617

UI and mappings related to ephemeral agents

- [x] Adds mapping/type/schema definition for the new field in agent policy saved object
- [x] Shows input field labelled `Unenrollment timeout` in agent policy settings that reads/writes to the new field
- [x] Same input in `Advanced options` section of create agent flyout
- [x] `unenroll_timeout` can be set using preconfigured agent policies defined in `kibana.yml`
- [x] `unenroll_timeout` can be populated if the user has a preconfigured policy that _does not_ have this field initially, but then updates their `kibana.yml` later to include it

<details><summary>Screenshot - editing an existing agent policy</summary>

<img width="1251" alt="Screen Shot 2021-06-22 at 1 42 50 PM" src="https://user-images.githubusercontent.com/57655/122974924-d479b800-d360-11eb-8d37-306a6860f140.png">

</details>


<details><summary>Screenshots - adding a new agent policy</summary>
<img width="1258" alt="Screen Shot 2021-06-22 at 1 45 01 PM" src="https://user-images.githubusercontent.com/57655/122975141-0be86480-d361-11eb-9ccb-dde0cc44e438.png">
<img width="1257" alt="Screen Shot 2021-06-22 at 1 45 35 PM" src="https://user-images.githubusercontent.com/57655/122975139-0be86480-d361-11eb-90ae-3541dbbd68e6.png">
<img width="1258" alt="Screen Shot 2021-06-22 at 1 45 44 PM" src="https://user-images.githubusercontent.com/57655/122975138-0b4fce00-d361-11eb-8347-d6f0cb43ab91.png">
<img width="1254" alt="Screen Shot 2021-06-22 at 1 45 56 PM" src="https://user-images.githubusercontent.com/57655/122975136-0b4fce00-d361-11eb-9c51-ec7a9e52ea2b.png">
</details>

<details><summary>Using <code>kibana.dev.yml</code></summary>

<h3>No <code>unenroll_timeout</code></h3>

```yml
xpack.fleet.agentPolicies:
  - name: Preconfigured Policy From Config
    description: From kibana.dev.yml (no timeout given)
    id: 1
    namespace: test
    package_policies:
      - package:
          name: system
        name: System Integration
        inputs:
          - type: system/metrics
            enabled: true
            vars:
              - name: system.hostfs
                value: home/test
            streams:
              - data_stream:
                  dataset: system.core
                enabled: true
                vars:
                  - name: period
                    value: 20s
          - type: winlog
            enabled: false
```

<h4>UI (saved object)</h4>
<img width="1072" alt="Screen Shot 2021-06-23 at 10 28 03 AM" src="https://user-images.githubusercontent.com/57655/123117442-bf0f9700-d40f-11eb-8e8f-685fee3ccb4b.png">

<h4><code>fleet-policies</code>index</h4>
<img width="1690" alt="Screen Shot 2021-06-23 at 10 52 39 AM" src="https://user-images.githubusercontent.com/57655/123119430-883a8080-d411-11eb-990f-4953845a76b5.png">


<h4>Updated <code>kibana.dev.yml</code> to include <code>unenroll_timeout</code></h4>

```yml
xpack.fleet.agentPolicies:
  - name: Preconfigured Policy From Config
    description: From kibana.dev.yml (updated with timeout)
    id: 1
    namespace: test
    unenroll_timeout: 234
    package_policies:
      - package:
          name: system
        name: System Integration
        inputs:
          - type: system/metrics
            enabled: true
            vars:
              - name: system.hostfs
                value: home/test
            streams:
              - data_stream:
                  dataset: system.core
                enabled: true
                vars:
                  - name: period
                    value: 20s
          - type: winlog
            enabled: false
```

<h4>UI (saved object)</h4>
<img width="1073" alt="Screen Shot 2021-06-23 at 10 35 17 AM" src="https://user-images.githubusercontent.com/57655/123117444-bfa82d80-d40f-11eb-94a9-99d4ca4bed2a.png">

<h4><code>fleet-policies</code>index</h4>
<img width="1690" alt="Screen Shot 2021-06-23 at 10 35 41 AM" src="https://user-images.githubusercontent.com/57655/123117446-bfa82d80-d40f-11eb-9563-1add80f43f0e.png">

</details>

### Checklist
- [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios

Co-authored-by: John Schulz <john.schulz@elastic.co>
@jen-huang jen-huang mentioned this issue Jun 23, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jfsiii jfsiii

Labels
enhancement New value added to drive a business result Team:Fleet Team label for Observability Data Collection Fleet team v7.14.0
Projects
None yet
Milestone
No milestone
5 participants
@jfsiii @jen-huang @blakerouse @mostlyjason @elasticmachine