"stale file handle" or "error in file IO" sporadically when copying from server to HPC

[Aariq]

Not sure if this is an rsync bug or something else, but I'm looking for help debugging.

I'm working on a server via SSH and need to copy a directory from the server to an HPC with rsync:

rsync -a -q "sourcedir" "username@hpc:~/destdir/"

In this example, sourcedir is on the server and contains a sub-directory which contains a small .csv file. destdir doesn't exist on the HPC, but gets created by rsync (when it works).

It works roughly 3/4th of the time, but sometimes fails with one of two errors:

1. a 'stale file handle':

   rsync: recv_generator: failed to stat "/home/u27/username/destdir/path/to/file.csv": Stale file handle (116)
   rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1196) [sender=3.1.2]

OR

2. 'error in file IO' error

   rsync: mkdir "/home/u27/username/destdir" failed: File exists (17)
   rsync error: error in file IO (code 11) at main.c(656) [Receiver=3.1.2]

The stale file handle error (1) happens more often than the error in file IO (2).

Environment:

• Host is a Virtual Machine running Ubuntu 18.04.6 on OpenStack. Data are stored in a mounted volume. Both the VM and data are hosted on OpenStack.
• The HPC is running CentOS Linux 7 and is accessed via SSH through a bastion host.

I suspect that the problem is happening on the host, not the HPC, because I can't reproduce this error on my laptop running macOS. I can reproduce this problem using both the sftp set up for transferring files to the HPC as well as via ssh.

Any ideas what could be causing this, or what further steps I could do to debug or eliminate this error?

Cross-posted here: https://superuser.com/questions/1755612/rsync-errors-with-stale-file-handle-or-error-in-file-io-sporadically-when-co