[v23.3.x] archival: keep locks until outcome is known

[vbotbuildovich]

Backport #21348 to branch v23.3.x. Note that this issue was created as a placeholder, since the original PR's commit(s) could not be automatically cherry-picked.
Here are the commands to execute:

git checkout v23.3.x
git checkout -b vbotbuildovich/backport-21348-v23.3.x-430
git cherry-pick -x a01c673311623fb5499eed73d9b7e7743da67365 69a5f70359e60f2a4a0239e53fef42c38d8460ba

git push origin vbotbuildovich/backport-21348-v23.3.x-430
gh pr create \
  --title "[v23.3.x] archival: keep locks until outcome is known" \
  --base "v23.3.x" \
  --label "kind/backport" \
  --head "vbotbuildovich/backport-21348-v23.3.x-430" \
  --draft \
  --repo "redpanda-data/redpanda" \
  --milestone "v23.3.x-next" \
  --body "Backport of PR https://github.com/redpanda-data/redpanda/pull/21348 "

JIRA Link: [CORE-5705](https://redpandadata.atlassian.net/browse/CORE-5705)