[receiver/k8sclusterreceiver] Fatal error on startup

[atoulme]

Component(s)

No response

What happened?

Description

A user reports the collector fails to start on a Kubernetes deployment.

See original issue with complete log file here: signalfx/splunk-otel-collector-chart#418

Steps to Reproduce

Not sure.

Expected Result

Actual Result

Collector version

0.42.0

Environment information

No response

OpenTelemetry Collector configuration

No response

Log output

E0408 01:52:25.564670       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: Failed to watch *v1.CronJob: failed to list *v1.CronJob: the server could not find the requested resource
2022-04-08T01:52:44.588Z	error	k8sclusterreceiver@v0.47.0/receiver.go:73	Timed out waiting for initial cache sync.	{"kind": "receiver", "name": "k8s_cluster"}
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/k8sclusterreceiver.(*kubernetesReceiver).Start.func1
	/builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/github.com/open-telemetry/opentelemetry-collector-contrib/receiver/k8sclusterreceiver@v0.47.0/receiver.go:73
2022-04-08T01:52:44.588Z	error	components/host_wrapper.go:40	Component fatal error	{"kind": "receiver", "name": "k8s_cluster", "error": "failed to start receiver: k8s_cluster"}
go.opentelemetry.io/collector/service/internal/components.(*hostWrapper).ReportFatalError
	/builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/go.opentelemetry.io/collector@v0.47.0/service/internal/components/host_wrapper.go:40
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/k8sclusterreceiver.(*kubernetesReceiver).Start.func1
	/builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/github.com/open-telemetry/opentelemetry-collector-contrib/receiver/k8sclusterreceiver@v0.47.0/receiver.go:74
2022-04-08T01:52:44.588Z	error	service/collector.go:169	Asynchronous error received, terminating process	{"error": "failed to start receiver: k8s_cluster"}
go.opentelemetry.io/collector/service.(*Collector).runAndWaitForShutdownEvent
	/builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/go.opentelemetry.io/collector@v0.47.0/service/collector.go:169
go.opentelemetry.io/collector/service.(*Collector).Run
	/builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/go.opentelemetry.io/collector@v0.47.0/service/collector.go:257
go.opentelemetry.io/collector/service.NewCommand.func1
	/builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/go.opentelemetry.io/collector@v0.47.0/service/command.go:39
github.com/spf13/cobra.(*Command).execute
	/builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:856
github.com/spf13/cobra.(*Command).ExecuteC
	/builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:974
github.com/spf13/cobra.(*Command).Execute
	/builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:902
main.runInteractive
	/builds/o11y-gdi/splunk-otel-collector-releaser/cmd/otelcol/main.go:408
main.run
	/builds/o11y-gdi/splunk-otel-collector-releaser/cmd/otelcol/main_others.go:24
main.main
	/builds/o11y-gdi/splunk-otel-collector-releaser/cmd/otelcol/main.go:129
runtime.main
	/usr/local/go/src/runtime/proc.go:250
2022-04-08T01:52:44.588Z	info	service/collector.go:267	Starting shutdown...
2022-04-08T01:52:44.588Z	info	healthcheck/handler.go:129	Health Check state change	{"kind": "extension", "name": "health_check", "status": "unavailable"}
2022-04-08T01:52:44.588Z	info	service/service.go:117	Stopping receivers...
2022-04-08T01:52:44.589Z	info	service/service.go:122	Stopping processors...
2022-04-08T01:52:44.589Z	info	builder/pipelines_builder.go:73	Pipeline is shutting down...	{"kind": "pipeline", "name": "metrics"}
2022-04-08T01:52:44.589Z	info	builder/pipelines_builder.go:77	Pipeline is shutdown.	{"kind": "pipeline", "name": "metrics"}
2022-04-08T01:52:44.589Z	info	builder/pipelines_builder.go:73	Pipeline is shutting down...	{"kind": "pipeline", "name": "metrics/collector"}
2022-04-08T01:52:44.589Z	info	builder/pipelines_builder.go:77	Pipeline is shutdown.	{"kind": "pipeline", "name": "metrics/collector"}
2022-04-08T01:52:44.589Z	info	service/service.go:127	Stopping exporters...
2022-04-08T01:52:44.589Z	info	service/service.go:132	Stopping extensions...
2022-04-08T01:52:44.589Z	info	service/collector.go:281	Shutdown complete.

### Additional context

_No response_

[github-actions]

Pinging code owners:

• receiver/k8scluster: @dmitryax

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[atoulme]

I believe it would be good to provide actionable feedback and context to the user when this error occurs, so they know the collector might not be to blame here, and that the API server is unable to serve content.

[github-actions]

Pinging code owners for receiver/k8scluster: @dmitryax. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• receiver/k8scluster: @dmitryax

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• receiver/k8scluster: @dmitryax

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[dmitryax]

This should be fixed by #9523 which is released in 0.51.0. Please reopen if it's not the case