Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

quay-operator crashes when adding rados registryBackends to CR #133

Closed
gorantornqvist opened this issue Jan 17, 2020 · 7 comments
Closed

quay-operator crashes when adding rados registryBackends to CR #133

gorantornqvist opened this issue Jan 17, 2020 · 7 comments

Comments

@gorantornqvist
Copy link

gorantornqvist commented Jan 17, 2020 •
edited
Loading

When I try to add a rados registry backend to my CR:

apiVersion: redhatcop.redhat.io/v1alpha1
kind: QuayEcosystem
metadata:
  name: my-quayecosystem
spec:
  quay:
    registryBackends:
       - name: rados
         rados:
           hostname: rook-ceph-rgw-object-store.rook-ceph
           port: 8080
           secure: false
           accessKey: ********
           secretKey: ********
           bucketName: quay-bucket-6a744ef2-a516-4d09-b3b8-31f56db86146

The quay-operator crashes with error:

{"level":"info","ts":1579245717.8176355,"logger":"controller_quayecosystem","msg":"Reconciling QuayEcosystem","Request.Namespace":"quay-enterprise","Request.Name":"my-quayecosystem"}
E0117 07:21:57.918246       1 runtime.go:73] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 1612 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x13dcda0, 0x219e050)
	/home/travis/gopath/pkg/mod/k8s.io/apimachinery@v0.0.0-20190817020851-f2f3a405f61d/pkg/util/runtime/runtime.go:69 +0x7b
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/k8s.io/apimachinery@v0.0.0-20190817020851-f2f3a405f61d/pkg/util/runtime/runtime.go:51 +0x82
panic(0x13dcda0, 0x219e050)
	/home/travis/.gimme/versions/go1.13.linux.amd64/src/runtime/panic.go:679 +0x1b2
github.com/redhat-cop/quay-operator/pkg/controller/quayecosystem/validation.Validate(0x17b4600, 0xc000351fb0, 0xc000460a80, 0x0, 0xc000304638, 0xc000bf9060)
	/home/travis/gopath/src/github.com/redhat-cop/quay-operator/pkg/controller/quayecosystem/validation/validate.go:358 +0x19d4
github.com/redhat-cop/quay-operator/pkg/controller/quayecosystem.(*ReconcileQuayEcosystem).Reconcile(0xc0009b3770, 0xc000d2e290, 0xf, 0xc000bf9060, 0x12, 0xc00097dcd8, 0xc00054da70, 0xc000c0b0e8, 0x17753c0)
	/home/travis/gopath/src/github.com/redhat-cop/quay-operator/pkg/controller/quayecosystem/quayecosystem_controller.go:126 +0x5b1
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc00012d540, 0x1431420, 0xc0009eccc0, 0xc000d32200)
	/home/travis/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.3.0/pkg/internal/controller/controller.go:216 +0x162
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc00012d540, 0x0)
	/home/travis/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.3.0/pkg/internal/controller/controller.go:192 +0xcb
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker(0xc00012d540)
	/home/travis/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.3.0/pkg/internal/controller/controller.go:171 +0x2b
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc000d23490)
	/home/travis/gopath/pkg/mod/k8s.io/apimachinery@v0.0.0-20190817020851-f2f3a405f61d/pkg/util/wait/wait.go:152 +0x5e
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000d23490, 0x3b9aca00, 0x0, 0x1, 0xc0000ae300)
	/home/travis/gopath/pkg/mod/k8s.io/apimachinery@v0.0.0-20190817020851-f2f3a405f61d/pkg/util/wait/wait.go:153 +0xf8
k8s.io/apimachinery/pkg/util/wait.Until(0xc000d23490, 0x3b9aca00, 0xc0000ae300)
	/home/travis/gopath/pkg/mod/k8s.io/apimachinery@v0.0.0-20190817020851-f2f3a405f61d/pkg/util/wait/wait.go:88 +0x4d
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start
	/home/travis/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.3.0/pkg/internal/controller/controller.go:157 +0x32e
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x11aa7f4]

goroutine 1612 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/k8s.io/apimachinery@v0.0.0-20190817020851-f2f3a405f61d/pkg/util/runtime/runtime.go:58 +0x105
panic(0x13dcda0, 0x219e050)
	/home/travis/.gimme/versions/go1.13.linux.amd64/src/runtime/panic.go:679 +0x1b2
github.com/redhat-cop/quay-operator/pkg/controller/quayecosystem/validation.Validate(0x17b4600, 0xc000351fb0, 0xc000460a80, 0x0, 0xc000304638, 0xc000bf9060)
	/home/travis/gopath/src/github.com/redhat-cop/quay-operator/pkg/controller/quayecosystem/validation/validate.go:358 +0x19d4
github.com/redhat-cop/quay-operator/pkg/controller/quayecosystem.(*ReconcileQuayEcosystem).Reconcile(0xc0009b3770, 0xc000d2e290, 0xf, 0xc000bf9060, 0x12, 0xc00097dcd8, 0xc00054da70, 0xc000c0b0e8, 0x17753c0)
	/home/travis/gopath/src/github.com/redhat-cop/quay-operator/pkg/controller/quayecosystem/quayecosystem_controller.go:126 +0x5b1
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc00012d540, 0x1431420, 0xc0009eccc0, 0xc000d32200)
	/home/travis/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.3.0/pkg/internal/controller/controller.go:216 +0x162
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc00012d540, 0x0)
	/home/travis/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.3.0/pkg/internal/controller/controller.go:192 +0xcb
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker(0xc00012d540)
	/home/travis/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.3.0/pkg/internal/controller/controller.go:171 +0x2b
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc000d23490)
	/home/travis/gopath/pkg/mod/k8s.io/apimachinery@v0.0.0-20190817020851-f2f3a405f61d/pkg/util/wait/wait.go:152 +0x5e
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000d23490, 0x3b9aca00, 0x0, 0x1, 0xc0000ae300)
	/home/travis/gopath/pkg/mod/k8s.io/apimachinery@v0.0.0-20190817020851-f2f3a405f61d/pkg/util/wait/wait.go:153 +0xf8
k8s.io/apimachinery/pkg/util/wait.Until(0xc000d23490, 0x3b9aca00, 0xc0000ae300)
	/home/travis/gopath/pkg/mod/k8s.io/apimachinery@v0.0.0-20190817020851-f2f3a405f61d/pkg/util/wait/wait.go:88 +0x4d
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start
	/home/travis/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.3.0/pkg/internal/controller/controller.go:157 +0x32e

Using quay-operator:v1.0.1
@sabre1041 sabre1041 mentioned this issue Jan 22, 2020
Merged
@gorantornqvist
Copy link
Author

Not really sure if this should work but tried changing deployment quay-operator image: 'quay.io/redhat-cop/quay-operator:latest'
But just get: exec: "quay-operator": executable file not found in $PATH

(File deploy/operator.yaml doesnt appear to have changed since 1.0.1 ...)

@sabre1041 sabre1041 mentioned this issue Jan 30, 2020
Merged
@sabre1041
Copy link
Contributor

@gorantornqvist we recently migrated CI from Travis to GitHub Actions. You discovered a bug which was corrected in #139. Can you try to use the new latest image?

@gorantornqvist
Copy link
Author

latest image works now, great

@gorantornqvist
Copy link
Author

The operator doesnt crash anymore.
However I get another error now:

quayconfig [31/Jan/2020:08:25:01 +0000] "POST /api/v1/superuser/config/validate/registry-storage HTTP/1.1" 200 79 "-" "Go-http-client/1.1" (0.003 2624 0.003)
2020-01-31 08:25:01,887 [65] [ERROR] [util.config.validator] Validation exception
Traceback (most recent call last):
  File "/quay-registry/util/config/validator.py", line 79, in validate_service_for_config
    VALIDATORS[service](validator_context)
  File "/quay-registry/util/config/validators/validate_storage.py", line 18, in validate
    providers = _get_storage_providers(config, ip_resolver, config_provider).items()
  File "/quay-registry/util/config/validators/validate_storage.py", line 55, in _get_storage_providers
    raise ConfigValidationException("Missing required parameter(s) for storage %s" % name)
ConfigValidationException: Missing required parameter(s) for storage rados

My config is:

spec:
  registryBackends:
    - name: rados
         rados:
           hostname: rook-ceph-rgw-ocs-storagecluster-cephobjectstore.openshift-storage
           port: 80
           secure: false
           accessKey: **************
           secretKey: **************
           bucketName: quay-bucket-2649ee59-66cd-4ebd-92cf-223533aa3b78

@sabre1041 sabre1041 mentioned this issue Jan 31, 2020
Merged
@sabre1041
Copy link
Contributor

@gorantornqvist #140 hopefully resolves the issues that you are experiencing. Can you verify?

@scouturier
Copy link
Contributor

scouturier commented Mar 30, 2020 •
edited
Loading

@sabre1041 FYI I am getting the same results even with #140

Manual configuration looks like this:

DISTRIBUTED_STORAGE_CONFIG:
  default:
  - RadosGWStorage
  - access_key: <accesskey>
    bucket_name: <bucket>
    hostname: <hostname>
    is_secure: <true/false>
    port: <port>
    secret_key: <secretkey>
    storage_path: <storagepath>

@ricardomaraschini
Copy link
Contributor

Closing this, if necessary please feel free to reopen.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@scouturier @sabre1041 @ricardomaraschini @gorantornqvist