Skip to content
This repository has been archived by the owner on Jan 18, 2023. It is now read-only.

cmk cluster-init failed to re-run with new added node #231

Open
jackiehjm opened this issue Apr 15, 2019 · 1 comment · May be fixed by #232 or #234
Open

cmk cluster-init failed to re-run with new added node #231

jackiehjm opened this issue Apr 15, 2019 · 1 comment · May be fixed by #232 or #234

Comments

@jackiehjm
Copy link

Description

cmk cluster-init failed with new added node

ERROR:root:Exception when creating secret: (409)
Reason: Conflict
HTTP response headers: HTTPHeaderDict({'Date': 'Thu, 11 Apr 2019 05:03:41 GMT', 'Content-Length': '218', 'Content-Type': 'application/json'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"secrets \"cmk-webhook-certs\" already exists","reason":"AlreadyExists","details":{"name":"cmk-webhook-certs","kind":"secrets"},"code":409}


ERROR:root:Aborting webhook deployment ...

steps to reproduce

1. create cmk-cluster-init-pod with the following args:

  - args:
      # Change this value to pass different options to cluster-init.
      - "/cmk/cmk.py cluster-init --host-list=node-0 --saname=cmk-serviceaccount"

2. check that the cmk cluster-init is successful

$ kubectl get pods -o wide
NAME                                             READY   STATUS      RESTARTS   AGE    IP             NODE           NOMINATED NODE
cmk-cluster-init-pod                             0/1     Completed   0          128m   172.16.0.8     node-0         <none>
cmk-init-install-discover-pod-node-0             0/2     Completed   0          128m   172.16.0.9     node-0         <none>
cmk-reconcile-nodereport-ds-node-0-g5ww7         2/2     Running     0          127m   172.16.0.10    node-0         <none>
cmk-webhook-deployment-5b7895df7f-zzwgc          1/1     Running     0          127m   172.16.0.11    node-0         <none>

3. add a new node node-1 and create cmk-cluster-init-pod-node-1 with the following args:

  - args:
      # Change this value to pass different options to cluster-init.
      - "/cmk/cmk.py cluster-init --host-list=node-1 --saname=cmk-serviceaccount"

cmk-cluster-init-pod-node-1 failed with errors:

$ kubectl get pods cmk-cluster-init-pod-node-1 -o wide
NAME                                             READY   STATUS      RESTARTS   AGE    IP             NODE           NOMINATED NODE
cmk-cluster-init-pod-node-1                      0/1     Error       0          18m    172.16.1.98    node-1         <none>

$ kubectl describe logs cmk-cluster-init-pod-node-1
error: the server doesn't have a resource type "logs"
$ kubectl logs cmk-cluster-init-pod-node-1
INFO:root:Used ServiceAccount: cmk-serviceaccount
INFO:root:Creating cmk pod for ['init', 'install', 'discover'] commands ...
INFO:root:Waiting for cmk pod running ['init', 'install', 'discover'] cmds to enter Succeeded state.
INFO:root:Creating cmk pod for ['reconcile', 'nodereport'] commands ...
INFO:root:Waiting for cmk pod running ['reconcile', 'nodereport'] cmds to enter Running state.
ERROR:root:Exception when creating secret: (409)
Reason: Conflict
HTTP response headers: HTTPHeaderDict({'Date': 'Thu, 11 Apr 2019 05:03:41 GMT', 'Content-Length': '218', 'Content-Type': 'application/json'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"secrets \"cmk-webhook-certs\" already exists","reason":"AlreadyExists","details":{"name":"cmk-webhook-certs","kind":"secrets"},"code":409}


ERROR:root:Aborting webhook deployment ...
jackiehjm added a commit to jackiehjm/CPU-Manager-for-Kubernetes that referenced this issue Apr 15, 2019
@jackiehjm
deploy_webhook: ignore err if the webhook has been deployed
e1d3650 
This ignores the AlreadyExists error so that the user can successfully
re-run cmk cluster-init when there is new node added into to cluster.

Fixes intel#231.

Signed-off-by: Jackie Huang <jackie.huang@windriver.com>
@jackiehjm jackiehjm linked a pull request Apr 15, 2019 that will close this issue
Open
jackiehjm added a commit to jackiehjm/CPU-Manager-for-Kubernetes that referenced this issue Apr 16, 2019
@jackiehjm
deploy_webhook: ignore err if the webhook has been deployed
b8f9d9a 
This ignores the AlreadyExists error so that the user can successfully
re-run cmk cluster-init when there is new node added into to cluster.

Fixes intel#231.

Signed-off-by: Jackie Huang <jackie.huang@windriver.com>
@jackiehjm jackiehjm linked a pull request Apr 16, 2019 that will close this issue
Open
jackiehjm added a commit to jackiehjm/CPU-Manager-for-Kubernetes that referenced this issue Apr 16, 2019
@jackiehjm
deploy_webhook: ignore err if the webhook has been deployed
25c4f0d 
This ignores the AlreadyExists error so that the user can successfully
re-run cmk cluster-init when there is new node added into to cluster.

Fixes intel#231.

Signed-off-by: Jackie Huang <jackie.huang@windriver.com>
@lmdaly
Copy link
Contributor

lmdaly commented May 16, 2019

@jackiehjm Are you removing CMK before re-running with a new node added? Or running cluster-init again on an already running CMK Cluster?

cbf123 pushed a commit to cbf123/CPU-Manager-for-Kubernetes that referenced this issue May 23, 2019
@jackiehjm @cbf123
deploy_webhook: ignore err if the webhook has been deployed
300664d 
This ignores the AlreadyExists error so that the user can successfully
re-run cmk cluster-init when there is new node added into to cluster.

Fixes intel#231.

Signed-off-by: Jackie Huang <jackie.huang@windriver.com>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
2 participants
@jackiehjm @lmdaly