Issues with private Docker Cloud repos and the 1.9.0 CLI

[pluttrell]

/kind bug

We're having trouble retrieving images from a private registry on DockerCloud. For nearly a year the following has worked flawlessly. Last Friday I pulled down the 1.9.0 CLI (via brew) and since then have not been able to access any image in a private registry on DockerCloud. I've tried this on 10+ brand new clusters created with kops v1.8.0. The Kubernetes cluster version have been 1.8.0, 1.8.4, 1.8.5 and 1.9.0 (which is not supported by kops yet) and all exhibit the same problem.

We create the secret as such:

kubectl create secret docker-registry dockerclouduser \
    --docker-server=https://index.docker.io/v1/ \
    --docker-username={my-username} \
    --docker-password={my-password} \
    --docker-email={my-email}

And verify it using:

kubectl get secret dockerclouduser --output=json | jq --raw-output '.data | .".dockercfg"' | base64 -D | python -m json.tool

Which yields:

{
    "auths": {
        "https://index.docker.io/v1/": {
            "auth": "{what-appears-to-be-a-token}",
            "email": "{my-email}",
            "password": "{my-password}",
            "username": "{my-username}"
        }
    }
}

And create a test service with kubectl create -f with these contents:

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: test-service
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: test-service
    spec:
      containers:
      - name: test-service
        image: {my-org}/{my-repo-name}:{my-tag}
        imagePullPolicy: Always
      imagePullSecrets:
        - name: dockerclouduser

The pod fails immediatly with:

Failed to pull image "{my-org}/{mmy-repo-name}:{my-tag}": rpc error: code = Unknown desc = Error response from daemon: repository {my-org}/{mmy-repo-name} not found: does not exist or no pull access

I've reviewed the 1.9.0 release notes and nothing jumps out as the root cause.

For reference I've tried putting quotes around the {my-password} value when creating the secret to no avail.

I've also tried prefixing the spec.template.spec.containers.[0].image value with index.docker.io, so the full value would read: image: index.docker.io/{my-org}/{my-repo-name}:{my-tag} also to no avail.

Environment:
Kubernetes version:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.0", GitCommit:"925c127ec6b946659ad0fd596fa959be43f0cc05", GitTreeState:"clean", BuildDate:"2017-12-16T03:16:50Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.0", GitCommit:"925c127ec6b946659ad0fd596fa959be43f0cc05", GitTreeState:"clean", BuildDate:"2017-12-15T20:55:30Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}

Cloud provider and installation info:

• AWS cluster built with Kops 1.8.0, using (to isolate this issue) nearly the default settings for everything.

[pluttrell]

@kubernetes/sig-cli-bugs
@kubernetes/sig-cli-maintainers

[k8s-ci-robot]

@pluttrell: Reiterating the mentions to trigger a notification:
@kubernetes/sig-cli-bugs

In response to this:

@kubernetes/sig-cli-bugs
@kubernetes/sig-cli-maintainers

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[drazul]

I have same problem, and I "solved" it creating a docker-registry secret in an old kubernetes version and pasting it on new kubernetes

[ITler]

I'm recently (I know it worked perfectly about last month) facing issues with kubectl create secret docker-registry ... as well. For me it ended in a similar manner saying: Failed to pull image "{my-org}/{my-repo-name}:{my-tag}": rpc error: code = Unknown desc = unauthorized: authentication required

I was able to confirm, that this is a kubernetes CLI related issue and not end user. Based on this guide and this article I was able to create properly formatted auth information looking like this: { "https://my.repo/v1/": { "auth": "SOMEnAUTHnTOKENnGENERATEDnBYnDOCKERnLOGIN" } }. When encoding this one-liner using base64 (MacOS workstation) and apply like kubectl apply -f /tmp/manually_encoded_secret.yaml this works fine and images can be pulled from private repository again.
However this breaks all automation and can't be considered as solution

Confirmed working with kubectl 1.8.5 and server 1.8.4 (guess 1.7.9 as well)

[dims]

looks like there's an extra "auths" element when adding the secret using 1.9 kubectl

[dims@bigbox 11:34] ~/go/src/k8s.io/kubernetes ⟩ kubectl get secret dockerclouduser --output=json | jq --raw-output '.data | .".dockercfg"' | base64 -d | python -m json.tool
{
    "auths": {
        "https://index.docker.io/v1/": {
            "auth": "ZGltczpPcGVuU3RhY2s=",
            "email": "dims@yahoo.com",
            "password": "OpenStack",
            "username": "dims"
        }
    }
}
[dims@bigbox 11:34] ~/go/src/k8s.io/kubernetes ⟩ kubectl get secret dockerclouduser-old --output=json | jq --raw-output '.data | .".dockercfg"' | base64 -d | python -m json.tool
{
    "https://index.docker.io/v1/": {
        "auth": "ZGltczpPcGVuU3RhY2s=",
        "email": "dims@yahoo.com",
        "password": "OpenStack",
        "username": "dims"
    }
}

[liggitt]

Added in #53916

/assign @juanvallejo

[pluttrell]

@dims Many thanks for the quick fix on this.

Any idea when this might get rolled out?

Also, it might be a good idea if this was added to the known issue list for v1.9.0. It would have saved us a bunch of time trying to track down the root cause. Just trying to make sure no one else needs to do the same.

[dims]

linking to the "v1.9.0 known issues" list #57159

[dims]

@pluttrell ack, asking Mehdy since he is the patch manager for v1.9 branch.

@mbohlool Mehdy, any idea when we can ship 1.9.1 with this fix?

[liggitt]

workarounds for folks until 1.9.1 is released:

if you have a .dockerconfigjson for your private registry already, you can manually specify the type and data key (e.g. kubectl create secret generic my-secret-name --type=kubernetes.io/dockerconfigjson --from-file .dockerconfigjson=/path/to/.dockerconfigjson)

if you don't have a .dockerconfigjson file already, you can fix up the secret produced by kubectl create secret docker-registry manually:

1. add --dry-run -o yaml > secret.yaml
2. change the type from kubernetes.io/dockercfg to kubernetes.io/dockerconfigjson
3. change the data key from .dockercfg to .dockerconfigjson
4. create the modified secret with kubectl create -f secret.yaml

[mbohlool]

1.9.1 will be released tomorrow

[aknuds1]

I can confirm @liggitt's solution works, thanks so much!

[WillPlatnick]

I am still having issues in 1.9.1. This worked in the 1.8.x branches:

  docker-registry gitlab-registry \
  --docker-server="$CI_REGISTRY" \
  --docker-username="$GL_USERNAME" \
  --docker-password="$GL_PASSWORD" \
  --docker-email="$GITLAB_USER_EMAIL" \
  -o yaml --dry-run | kubectl apply -n $KUBE_NAMESPACE -f -```

results in the following error:
```The Secret "gitlab-registry" is invalid: type: Invalid value: "kubernetes.io/dockerconfigjson": field is immutable```

[liggitt]

you have to remove the gitlab-registry secret before you can apply an update... type is immutable, and the fix in 1.9.1 required changing the type of the secret