Failure to connect to storage backends should fail readiness probe

[concaf]

Feature request

one of the basic functionalities of chains is to sign and attest workloads and push these somewhere which is defined by the storage backends like tekton, oci, gcs, docdb, grafeas.

when chains cannot connect to these storage backends to push signatures, attestations, etc, that violates one of the fundamental guarantees that chains provides and the readiness probe (or liveness???) should fail in such a case.

Use case

2 users have reported that they use mongo DB as a storage backend (via artifacts.taskrun.storage: docdb) but when the creds to connect to mongo DB were rotated, they expected chains to fail a probe and start restarting the chains controller pod so they could do remediation - instead chains continued throwing errors that it was not able to connect to mongo but did not start failing while not pushing any signatures, etc to mongo DB.

[lcarva]

+1 this would be a great feature. I believe this was already done for KMS: #936