fix(repo-server): excess git requests, cache lock on revisions

[nromriell]

Fixes argoproj#14725

Third part of a split of the argoproj#16309 and follow-up to argoproj#16410

Another piece of the fixes for argoproj#14725. This resolves the excess ls-remote requests that still occur in high application count environments due to the fact that all active processes get their cache invalidated at the same time. When this happens they all try to re-fetch the remotes at the same time and all miss the cache.

The solution is to add a soft cache level lock on the key, the first process to try and access that key on a miss will obtain the lock and attempt to refresh the cache.

This changed required adding the ability to use "set only if not exists" on the util/cache (redis NX or memcached add) interface. Without this option all of the processes could still end up in a position where they think they own the lock, so there are some minor changes to the util/cache interface:

• Adds a DisableOverwrite option to represent the NX/add during set option
• The Item struct now also contains options for the cache set actions. This allows passing Delete and DisableOverwrite in a shared options value and replaces the position of the current delete option in the SetItem call

This is the final major logic portion of the changes to remove all the excess git calls, but we can get some good performance improvements by also adding the two layer cache to the caching layer for revisions. So I'll be following up with another PR to add that. Most of the abstraction to prepare for that was handled as part of this PR.

Baseline v2.9.0 with 200 multi-source applications with a ref only source:

With all the changes up to and including this PR under the same conditions:

Checklist:

• Either (a) I've created an enhancement proposal and discussed it with the community, (b) this is a bug fix, or (c) this does not need to be in the release notes.
• The title of the PR states what changed and the related issues number (used for the release note).
• The title of the PR conforms to the Toolchain Guide
• I've included "Closes [ISSUE #]" or "Fixes [ISSUE #]" in the description to automatically close the associated issue.
• I've updated both the CLI and UI to expose my feature, or I plan to submit a second PR with them.
• Does this PR require documentation updates?
• I've updated documentation as required by this PR.
• Optional. My organization is added to USERS.md.
• I have signed off all my commits as required by DCO
• I have written unit and/or e2e tests for my change. PRs without these are unlikely to be merged.
• My build is green (troubleshooting builds).
• My new feature complies with the feature status guidelines.
• I have added a brief description of why this PR is necessary and/or what this PR solves.