SSHExecutor respects MaxSessions

[maxfischer2781]

This PR changes SSHExecutor to use at most MaxSessions at once.

• Detect remote sshd MaxSessions via probing
• Limit concurrent sessions

This is a rather straightforward implementation of step 1 in #217. After opening a new connection, it is probed until ChannelOpenError to deduce the remote MaxSessions. This limit is then enforced using a Semaphore on accessing the connection, queueing additional requests.
The large diff of run_command is because the Semaphore logic works best with a context manager.

[codecov-commenter]

Codecov Report

Merging #218 (2d31c69) into master (2afcbd4) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master     #218   +/-   ##
=======================================
  Coverage   99.55%   99.55%           
=======================================
  Files          41       41           
  Lines        1789     1810   +21     
=======================================
+ Hits         1781     1802   +21     
  Misses          8        8           

Impacted Files	Coverage Δ
tardis/utilities/executors/sshexecutor.py	100.00% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2afcbd4...2d31c69. Read the comment docs.

[mschnepf]

Looks good to me 👍
Thanks for the improvement

[RHofsaess]

I don't really like the probing but after a little research I also did not find another way to get the max sessions.
So I think the changes are fine!
Thanks.

[giffels]

Thanks a lot for improving the ssh connection handling. I have a few questions concerning the implementation and a request to further improve unittesting before the approval. Could you take this into account, please?

[giffels]

Thanks a lot for your contribution and applying the suggested changes. Everything looks good now.

[mschnepf]

Looks good to me. 👍
You also get bonus points from me

[RHofsaess]

What is idea behind this test?
Isn't it automatically successful?
Or am I misunderstanding something here?

[maxfischer2781]

It's a sanity test that the Mock/probe behaves as expected. Other tests rely on the default as well, so if they fail this test reveals whether the issue is with the default or the other tests.

[giffels]

It tests the functionality of the probe_max_session function defined in

tardis/tardis/utilities/executors/sshexecutor.py

Lines 15 to 31 in 2d31c69

	async def probe_max_session(connection: asyncssh.SSHClientConnection):
	"""
	Probe the sshd `MaxSessions`, i.e. the multiplexing limit per connection
	"""
	sessions = 0
	# It does not actually matter what kind of session we open here, but:
	# - it should stay open without a separate task to manage it
	# - it should reliably and promptly clean up when done probing
	# `create_process` is a bit heavy but does all that.
	async with AsyncExitStack() as aes:
	try:
	while True:
	await aes.enter_context(await connection.create_process())
	sessions += 1
	except asyncssh.ChannelOpenError:
	pass
	return sessions

. In case someone changes the code so, that the functionality is not obtained anymore. The test will fail. For example someone removes

tardis/tardis/utilities/executors/sshexecutor.py

Line 28 in 2d31c69

sessions += 1

.

[RHofsaess]

Okay thanks. That makes perfectly sense!

[RHofsaess]

👍