[crmsh-4.6] Dev: bootstrap: implement ssh-agent support (jsc#PED-5774)

[nicholasyang2022]

Use Cases

In a typical cloud-based deployment, a server may have password-based
authentication disabled for ssh, and an adminstrator's ssh public key
is added to authorized_keys during initialization.

For this case, it is impossible for crmsh to log into cluster node with
interactive authentication and create new key pairs for further
operations. Instead, crmsh can make use of the administrator's key, by
authenticating ssh session with ssh-agent forwarded form the
adminstrator's PC.

Usage Example

alice@alice-pc: ~> ssh -A root@node1
root@node1:~ # crm cluster init --use-ssh-agent -y
root@node1:~ # exit
alice@alice-pc: ~> ssh -A root@node2
root@node2:~ # crm cluster join --use-ssh-agent -c node1 -y

[codecov]

Codecov Report

Attention: 99 lines in your changes are missing coverage. Please review.

Comparison is base (373ddb9) 52.52% compared to head (6a49af9) 53.06%.

Files	Patch %	Lines
crmsh/bootstrap.py	78.99%	46 Missing ⚠️
crmsh/ssh_key.py	68.90%	37 Missing ⚠️
crmsh/sh.py	83.60%	10 Missing ⚠️
crmsh/report/collect.py	25.00%	3 Missing ⚠️
crmsh/report/utillib.py	0.00%	1 Missing ⚠️
crmsh/ui_cluster.py	83.33%	1 Missing ⚠️
crmsh/utils.py	91.66%	1 Missing ⚠️

Additional details and impacted files

@@              Coverage Diff              @@
##           crmsh-4.6    #1261      +/-   ##
=============================================
+ Coverage      52.52%   53.06%   +0.53%     
=============================================
  Files             83       83              
  Lines          25354    25617     +263     
=============================================
+ Hits           13317    13593     +276     
+ Misses         12037    12024      -13     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[liangxin1300]

After using ssh-agent to setup a cluster, some commands will fail without adding SSH_AUTH_SOCK:

• crm cluster run "cmd"
• crm report
• crm cluster stop --all
  Can crmsh be aware this cluster had been set up via ssh-agent? So that crmsh can give a warning to hint use how to do?

How about check config.core.no_generating_ssh_key?
@nicholasyang2022

[nicholasyang2022]

Can crmsh be aware this cluster had been set up via ssh-agent? So that crmsh can give a warning to hint use how to do?

@liangxin1300 ClusterShell has the ability to catch an ssh call with 255 exit code and raise an AuthorizationError exception. It is easy to add diagnose messages for that exception.

However, the old get_stdout_or_raise_error method does not handle 255 specially. To avoid breaking things, I made ClusterShell to raise AuthorizationError only when requested explicitly.

I will try enabling that exception by default.

[liangxin1300]

Compare with current crmsh-4.6 code:

# crm cluster init -y

INFO: SSH key for root does not exist, hence generate it now
INFO: SSH key for hacluster does not exist, hence generate it now

Above 2 lines are missing while not using ssh-agent;
And

INFO: SSH key for hacluster does not exist, hence generate it now

Above line is missing while using ssh-agent (but hacluster's key still be generated locally? I mean don't we consider hacluster also using ssh-agent? @zzhou1 )

[liangxin1300]

SSH_AUTH_SOCK=/root/ssh-auth-sock crm report will hang, please check

And, it's better to add crm report case under ssh-agent scenario

Works now, thanks!

[zzhou1]

Compare with current crmsh-4.6 code:

# crm cluster init -y

...

INFO: SSH key for hacluster does not exist, hence generate it now

Above line is missing while using ssh-agent (but hacluster's key still be generated locally? I mean don't we consider hacluster also using ssh-agent? @zzhou1 )

Agree, "hacluster" ssh-key should be inside the cluster nodes since hawk might call for it.

[zzhou1]

Suggested change

	buf.write(' Please make sure environment variables are preserved across sudo calls.')
	buf.write(' Probably could consider, eg. sudo --preserve-env=SSH_AUTH_SOCK .')

[nicholasyang2022]

Lacking --preserve-env=SSH_AUTH_SOCK is only one of the possible causes of SSH_AUTH_SOCK not found in environment variables. How about Please check whether an ssh-agent is available and consider using "sudo --preserve-env=SSH_AUTH_SOCK"

[liangxin1300]

Please check whether ssh-agent is available and consider using "sudo --preserve-env=SSH_AUTH_SOCK"