Private git repos with SSH does not seem to work

[johnqa]

Hi,

What happened:
Since my litmus-portal deployment is an air gapped environment, I created a git repository in Azure DevOps Server and downloaded the https://github.com/litmuschaos/chaos-charts repo and put it in that internal one.

I am trying to add this internal repo as a private chaos hub using SSH authentication but I a, getting an error when submitting it:

Error: empty git-upload-pack given

If I look in the portal-server logs I see:

2021/03/19 19:31:45 Private Key generated

Fri, Mar 19 2021 8:31:45 pm
2021/03/19 19:31:45 Public key generated

Fri, Mar 19 2021 8:31:50 pm
time="2021-03-19T19:31:50Z" level=info msg="Running GitOps DB Sync..."

Fri, Mar 19 2021 8:31:50 pm
time="2021-03-19T19:31:50Z" level=info msg="Updating : []"

Fri, Mar 19 2021 8:31:50 pm
time="2021-03-19T19:31:50Z" level=info msg="GitOps DB Sync Complete"

Fri, Mar 19 2021 8:32:40 pm
File reading error open /tmp/version/3db98aa5-7562-4d71-a2b3-8f3ff57a3a95/<hubname>/charts/: no such file or directory

Fri, Mar 19 2021 8:32:42 pm
time="2021-03-19T19:32:42Z" level=info msg="Executed isRepositoryExists()... " repositoryExists=false

Fri, Mar 19 2021 8:32:42 pm
time="2021-03-19T19:32:42Z" level=error msg="empty git-upload-pack given"

Fri, Mar 19 2021 8:32:42 pm
Error in cloningFile reading error open /tmp/version/3db98aa5-7562-4d71-a2b3-8f3ff57a3a95/<hubname>/charts/: no such file or directory

Fri, Mar 19 2021 8:32:45 pm
File reading error open /tmp/version/3db98aa5-7562-4d71-a2b3-8f3ff57a3a95/<hubname>/charts/: no such file or directory

What you expected to happen:
I would expect to work and be able to create hubs this way.

Anything else we need to know?:
If I manually create /charts/ folder in /tmp/version/3db98aa5-7562-4d71-a2b3-8f3ff57a3a95/ and retry, the hub says it's connected with error but it's empty.

Thank you,
John

[imrajdas]

Hi @johnqa, As per the logs, it is saying cloned repo is empty. Can you check your cloned repo?

[johnqa]

Hi,

If I clone my repo manually on a linux box, it is not empty.

By the way, my installation is on a RKE cluster.

[imrajdas]

@johnqa If you create anything here /tmp/version/3db98aa5-7562-4d71-a2b3-8f3ff57a3a95/, it will not work because it has to be updated in the DB first.

I am tagging the other team members @gdsoumya and @amityt for it. Meantime, if you can try it again from the portal and send me the logs.

[johnqa]

which logs, the ones from litmus-portal-server?

[imrajdas]

which logs, the ones from litmus-portal-server?
Yes @johnqa

[johnqa]

There is no difference:

2021/03/19 19:59:09 Private Key generated

Fri, Mar 19 2021 8:59:09 pm
2021/03/19 19:59:09 Public key generated

Fri, Mar 19 2021 8:59:38 pm
Error in cloningError in cloningFile reading error open /tmp/version/fba1dd4d-1537-44ea-a4da-d1b9ede07907/tfs/charts/: no such file or directory

Fri, Mar 19 2021 8:59:50 pm
time="2021-03-19T19:59:50Z" level=info msg="Running GitOps DB Sync..."

Fri, Mar 19 2021 8:59:50 pm
time="2021-03-19T19:59:50Z" level=info msg="Updating : []"

Fri, Mar 19 2021 8:59:50 pm
time="2021-03-19T19:59:50Z" level=info msg="GitOps DB Sync Complete"

Fri, Mar 19 2021 9:01:00 pm
time="2021-03-19T20:01:00Z" level=info msg="Executed isRepositoryExists()... " repositoryExists=false

Fri, Mar 19 2021 9:01:00 pm
time="2021-03-19T20:01:00Z" level=error msg="empty git-upload-pack given"

Fri, Mar 19 2021 9:01:01 pm
Error in cloningFile reading error open /tmp/version/fba1dd4d-1537-44ea-a4da-d1b9ede07907/tfs/charts/: no such file or directory

When you say from the portal you mean from the UI. Is there maybe a cli way of doing things?

[imrajdas]

When you say from the portal you mean from the UI. Is there maybe a cli way of doing things?

Yes, I meant from the UI. Currently, we don't support private hub setup from CLI.

[johnqa]

I have the same error if I want to use the same repository for GitOPS.

By the way, now I am using 2.00-Beta1, but I had the same issue with the stable version.

[amityt]

Hi @johnqa , are you providing the SSH link of the repo. It should be something like this git@github.com:<org>/<repo>.git

[imrajdas]

Also, For debugging purpose. Can you exec into the litmus-portal-server(graphl-server) container and try to do a git clone of your repo? I am suspecting some network/proxy issue.

[johnqa]

Yes I tried that but I am getting git: not found

[imrajdas]

It's a alpine image, so you have to do apk add git

[johnqa]

no luck

apk add git
ERROR: Unable to lock database: Permission denied
ERROR: Failed to open apk database: Permission denied

[imrajdas]

Can you check with wget and curl?

[imrajdas]

@johnqa Currently, litmusportal-server(graphql-server) container is a non-root container, that's why it is showing permission error. I just built a root image. Can you replace the graphql-server container image with imrajdas/litmusportal-graphql-server:ci?

After that, Can you exec into the new container and do wget of your git repo?

[johnqa]

Hi,

I have done that, but wget from alpine doesn't allow --username and --password flags so I get 401 Unauthorized

[johnqa]

Hi @johnqa , are you providing the SSH link of the repo. It should be something like this git@github.com:<org>/<repo>.git

My git repo is not in github but in Azure DevOps Server and it looks like
ssh://<fqdn>:22/<org>/<repo>

[imrajdas]

Then, Can you do the following steps ?

  apk add git
  git clone git@url:<org>/<repo>.git

Before that, make sure you follow this thread #2578 (comment)

[johnqa]

hi @rajdas98 I cannot follow the steps as I am in air gapped environment and apk add git tries to connect to some repos that are outside this.

If I try in the portal to use git@... I have another error saying: Error: exec: "git": executable file not found in $PATH

[ishangupta-ds]

@johnqa if the container does not have sufficient permissions, then it may not able to read / write from / to azure repositories, since you have already added chaos charts, you may try creating a custom scope map giving read / write permissions to the chaos charts internal repo and a private ACR then upload litmus portal images to it and update their scope with the custom scope map from configurations, then the ACR may be added to RKE as a private registry, the privileged images should now be able to access the chaos charts repo on azure depending on scope map permissions.

[johnqa]

hi, could it be related to this? argoproj/argo-workflows#5235

[imrajdas]

Hi @johnqa, Looks like the same issue. But we are using the go-git library to do the git operations. I found this issue go-git/go-git#64 in their repo.