[chore][otel] Enable persistence in otel.yml #5549

VihasMakwana · 2024-09-17T11:53:49Z

Enhancement
Docs

What does this PR do?

This PR adds the filestorage extension in otel.yaml and updates the documentation.
Updates otel dependencies to 0.110.0

Why is it important?

By default, the OpenTelemetry Collector operates in a stateless mode, meaning it does not save file offsets to disk. To resolve this, we enable persistence in the default configuration files.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in ./changelog/fragments using the changelog tool
I have added an integration test or an E2E test

How to test this PR locally

Build from this branch
Keep the config default and run elastic-agent otel --config otel.yaml
Ingest some data and stop the process.
Restart the process and ingest some more data.
You shouldn't see any data loss/duplication.

Related issues

Closes https://github.com/elastic/opentelemetry-dev/issues/281

mergify · 2024-09-17T11:54:29Z

This pull request does not have a backport label. Could you fix it @VihasMakwana? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-./d./d is the label to automatically backport to the 8./d branch. /d is the digit

mergify · 2024-09-17T11:54:30Z

backport-v8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.

elasticmachine · 2024-09-18T10:19:53Z

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

VihasMakwana · 2024-09-18T17:17:36Z

@elastic/elastic-agent-control-plane

I propose two options for default directory:

Set it as per current sample configurations and modify them during the onboarding process. Currently, the sample configs use STORAGE_DIR env as the default directory option
Follow Filebeat's directory layout and set the default to a path under path.data.

For the latter option, we need to implement a config converter which maps path.data to /opt/Elastic/Agent/data (in case of Linux).

Let me know how do you feel about this proposal.

cc: @ycombinator @pierrehilbert @andrzej-stencel

VihasMakwana · 2024-09-18T17:19:53Z

Introducing a new config converter would be straightforward and could be easily integrated into our distribution.

cmacknz · 2024-09-19T13:32:15Z

As discussed let's leave this as an environment variable. How the collector is run and installed (if it is installed as a service at all) is currently completely up to the user and this goes along with that.

The only place we likely want a default is in our reference Kubernetes configurations, which can set the environment variable or set a default in the configuration it includes directly.

In general we do not want to use any of the paths that are also used by Beats or Agent, because a standalone OTel collector must be able to run alongside them without issue.

In our OTel distribution I think we solve this by having an install command that puts this in whatever base install directory we pick (/opt/Elastic, C:\Program Files\Elastic, or /Library/Elastic) but we need an installer that runs as a privileged user to do that. You can't do it with config alone.

mergify · 2024-09-27T11:05:00Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b enabl-file-storage-update-docs upstream/enabl-file-storage-update-docs
git merge upstream/main
git push upstream enabl-file-storage-update-docs

otel.yml

internal/pkg/otel/README.md

changelog/fragments/1726572104-enable-persistence-by-default.yaml

go.mod

andrzej-stencel · 2024-09-27T12:46:44Z

otel.yml

+  file_storage/filelogreceiver:
+    create_directory: true
+    directory: ${env:STORAGE_DIR}


The addition of the File Storage extension may cause problems for users using this configuration with read-only filesystems, like in Docker, in Kubernetes, unless they configure a writable storage layer in their containers/pods. Should we be concerned about this?

The addition of the File Storage extension may cause problems for users using this configuration with read-only filesystems, like in Docker, in Kubernetes, unless they configure a writable storage layer in their containers/pods. Should we be concerned about this?

I believe we will have a separate recommendation in our k8s configs based on @cmacknz's comment.

This change breaks running Elatic Agent as OTel Collector in Docker. I'd rather have this change applied in a way that doesn't break running the EDOT Collector in Docker. If we want to enable persistence by default, it should "just work", like it "just works" with the regular Elastic Agent.

$ docker run --rm -ti --entrypoint="elastic-agent" docker.elastic.co/beats/elastic-agent:9.0.0-SNAPSHOT otel Starting in otel mode 2024-09-30T09:06:39.260Z info service@v0.110.0/service.go:137 Setting up own telemetry... 2024-09-30T09:06:39.261Z info service@v0.110.0/service.go:186 Skipped telemetry setup. 2024-09-30T09:06:39.261Z info builders/builders.go:26 Development component. May change in the future. {"kind": "exporter", "data_type": "logs", "name": "debug"} 2024-09-30T09:06:39.262Z info builders/extension.go:50 Development component. May change in the future. {"kind": "extension", "name": "memory_limiter"} 2024-09-30T09:06:39.262Z info memorylimiter/memorylimiter.go:75 Memory limiter configured {"kind": "extension", "name": "memory_limiter", "limit_mib": 700, "spike_limit_mib": 180, "check_interval": 1} 2024-09-30T09:06:39.252Z warn envprovider@v1.16.0/provider.go:59 Configuration references unset environment variable {"name": "STORAGE_DIR"} failed to build extensions: failed to create extension "file_storage/filelogreceiver": mkdir /var/lib/otelcol: permission denied

@andrzej-stencel I see what you mean. The error is due to an empty env:STORAGE_DIR and hence, the filestorage extension defaults to /var/lib/otelcol. Since the default user in the Docker container is elastic-agent, this causes a permission denied error.

If I use any of following commands, it works.

docker run --rm -ti --entrypoint="elastic-agent" --mount type=bind,source=$(pwd)/data,target=/usr/share/otel -e STORAGE_DIR=/usr/share/otel/ docker.elastic.co/beats/elastic-agent:9.0.0-SNAPSHOT otel

or

docker run --user root --rm -ti --entrypoint="elastic-agent" docker.elastic.co/beats/elastic-agent:9.0.0-SNAPSHOT otel

I wonder how should we approach this. Maybe add documentation? or update EDOT according to @cmacknz's comment. Or do both.

In our OTel distribution I think we solve this by having an install command that puts this in whatever base install directory we pick (/opt/Elastic, C:\Program Files\Elastic, or /Library/Elastic) but we need an installer that runs as a privileged user to do that. You can't do it with config alone.

@cmacknz @andrzej-stencel I'd appreciate your opinions over this!

I agree with you that is should "just work".

In the container we control the paths, and can pick a default that is writable by our user in the container. Elastic agent uses the STATE_PATH environment variable and puts it under /usr/share/elastic-agent.

elastic-agent/internal/pkg/agent/cmd/container.go

Lines 48 to 49 in 95021e1

agentBaseDirectory = "/usr/share/elastic-agent" // directory that holds all elastic-agent related files

defaultStateDirectory = agentBaseDirectory + "/state" // directory that will hold the state data

We could add an /otelcol sub-directory to /usr/share/elastic-agent for one possibility.

Possibly we can reuse the STATE_PATH variable and change which sub-directory we use based on whether the entrypoint is the otel command or the elastic-agent container command.

Makes sense.

Co-authored-by: Andrzej Stencel <andrzej.stencel@elastic.co>

mergify · 2024-10-01T03:05:27Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b enabl-file-storage-update-docs upstream/enabl-file-storage-update-docs
git merge upstream/main
git push upstream enabl-file-storage-update-docs

VihasMakwana · 2024-10-04T12:18:33Z

@andrzej-stencel @cmacknz PTAL!

VihasMakwana · 2024-10-04T12:20:12Z

internal/pkg/agent/cmd/otel.go

+		// The filestorage extension will handle directory creation since create_directory: true is set by default.
+		// If the user hasn’t specified the env:STORAGE_DIR in filestorage, they may have opted for a custom path, and the extension will create the directory accordingly.
+		// In this case, setting env:STORAGE_DIR will have no effect.
+		if err := os.Setenv("STORAGE_DIR", filepath.Join(paths.Top(), "otel_registry")); err != nil {


We do this to avoid introducing a breaking change, as we expect the collector to function smoothly without issues.

mergify · 2024-10-07T07:04:42Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b enabl-file-storage-update-docs upstream/enabl-file-storage-update-docs
git merge upstream/main
git push upstream enabl-file-storage-update-docs

VihasMakwana · 2024-10-08T15:20:21Z

@andrzej-stencel @cmacknz PTAL 😅

elastic-sonarqube · 2024-10-08T16:01:04Z

Quality Gate failed

Failed conditions
0.0% Coverage on New Code (required ≥ 40%)

See analysis details on SonarQube

VihasMakwana added 2 commits September 17, 2024 17:12

chore: update config, docs

2d56cba

Merge branch 'main' into enabl-file-storage-update-docs

038027b

VihasMakwana added documentation Improvements or additions to documentation enhancement New feature or request Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team opentelemetry Related to the Elastic Distribution of the OpenTelemetry Collector labels Sep 17, 2024

VihasMakwana self-assigned this Sep 17, 2024

mergify bot added the backport-8.x Automated backport to the 8.x branch with mergify label Sep 17, 2024

VihasMakwana and others added 2 commits September 17, 2024 18:12

fix: spelling mistake

a82d734

Merge branch 'main' into enabl-file-storage-update-docs

c28889f

VihasMakwana marked this pull request as ready for review September 18, 2024 10:19

VihasMakwana requested a review from a team as a code owner September 18, 2024 10:19

VihasMakwana requested review from blakerouse and pchila September 18, 2024 10:19

VihasMakwana requested a review from andrzej-stencel September 18, 2024 10:20

pchila approved these changes Sep 24, 2024

View reviewed changes

VihasMakwana added 2 commits September 27, 2024 16:34

fix: go tidy

539931e

remove comments

564a619

VihasMakwana added 2 commits September 27, 2024 16:37

Merge branch 'main' into enabl-file-storage-update-docs

36238a3

fix: go tidy

a5cbe5f

andrzej-stencel reviewed Sep 27, 2024

View reviewed changes

VihasMakwana and others added 2 commits September 29, 2024 16:38

fix: add filestorage to extension

899607a

Update changelog/fragments/1726572104-enable-persistence-by-default.yaml

eafd9cd

Co-authored-by: Andrzej Stencel <andrzej.stencel@elastic.co>

VihasMakwana requested a review from andrzej-stencel September 30, 2024 07:24

VihasMakwana added 2 commits September 30, 2024 15:42

chore: update readme

6986051

Merge branch 'main' into enabl-file-storage-update-docs

9be9819

VihasMakwana added 6 commits October 4, 2024 17:14

chore: update readme

b465b1c

Merge branch 'main' into enabl-file-storage-update-docs

44370be

go.mod

5c2aaa1

chore: go.mod

afaac31

comments

d9b667a

comments

f66e01f

VihasMakwana requested review from pchila and cmacknz October 4, 2024 12:18

VihasMakwana commented Oct 4, 2024

View reviewed changes

VihasMakwana and others added 2 commits October 4, 2024 18:29

chore: update notice

1f0ecc9

Merge branch 'main' into enabl-file-storage-update-docs

b17e355

VihasMakwana added 3 commits October 8, 2024 20:45

Merge branch 'main' into enabl-file-storage-update-docs

b203fb4

chore: go.sum and notice

eac340c

chore: go.sum and notice

5baa9ac

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[chore][otel] Enable persistence in otel.yml #5549

[chore][otel] Enable persistence in otel.yml #5549

VihasMakwana commented Sep 17, 2024 •

edited

Loading

mergify bot commented Sep 17, 2024

mergify bot commented Sep 17, 2024

elasticmachine commented Sep 18, 2024

VihasMakwana commented Sep 18, 2024 •

edited

Loading

VihasMakwana commented Sep 18, 2024

cmacknz commented Sep 19, 2024

mergify bot commented Sep 27, 2024

andrzej-stencel Sep 27, 2024

VihasMakwana Sep 29, 2024

andrzej-stencel Sep 30, 2024 •

edited

Loading

VihasMakwana Sep 30, 2024

VihasMakwana Sep 30, 2024

cmacknz Oct 2, 2024 •

edited

Loading

cmacknz Oct 2, 2024

VihasMakwana Oct 3, 2024

mergify bot commented Oct 1, 2024

VihasMakwana commented Oct 4, 2024

VihasMakwana Oct 4, 2024

mergify bot commented Oct 7, 2024

VihasMakwana commented Oct 8, 2024

elastic-sonarqube bot commented Oct 8, 2024

	agentBaseDirectory = "/usr/share/elastic-agent" // directory that holds all elastic-agent related files
	defaultStateDirectory = agentBaseDirectory + "/state" // directory that will hold the state data

[chore][otel] Enable persistence in otel.yml #5549

Are you sure you want to change the base?

[chore][otel] Enable persistence in otel.yml #5549

Conversation

VihasMakwana commented Sep 17, 2024 • edited Loading

What does this PR do?

Why is it important?

Checklist

How to test this PR locally

Related issues

mergify bot commented Sep 17, 2024

mergify bot commented Sep 17, 2024

elasticmachine commented Sep 18, 2024

VihasMakwana commented Sep 18, 2024 • edited Loading

VihasMakwana commented Sep 18, 2024

cmacknz commented Sep 19, 2024

mergify bot commented Sep 27, 2024

andrzej-stencel Sep 27, 2024

Choose a reason for hiding this comment

VihasMakwana Sep 29, 2024

Choose a reason for hiding this comment

andrzej-stencel Sep 30, 2024 • edited Loading

Choose a reason for hiding this comment

VihasMakwana Sep 30, 2024

Choose a reason for hiding this comment

VihasMakwana Sep 30, 2024

Choose a reason for hiding this comment

cmacknz Oct 2, 2024 • edited Loading

Choose a reason for hiding this comment

cmacknz Oct 2, 2024

Choose a reason for hiding this comment

VihasMakwana Oct 3, 2024

Choose a reason for hiding this comment

mergify bot commented Oct 1, 2024

VihasMakwana commented Oct 4, 2024

VihasMakwana Oct 4, 2024

Choose a reason for hiding this comment

mergify bot commented Oct 7, 2024

VihasMakwana commented Oct 8, 2024

elastic-sonarqube bot commented Oct 8, 2024

Quality Gate failed

VihasMakwana commented Sep 17, 2024 •

edited

Loading

VihasMakwana commented Sep 18, 2024 •

edited

Loading

andrzej-stencel Sep 30, 2024 •

edited

Loading

cmacknz Oct 2, 2024 •

edited

Loading