Controlling Caching

[atzannes]

Let me start by saying that I am a big fan of fsspec and universal_pathlib. I had written a version of fsspec which I am trying to migrate off of in favor of fsspec. Especially when combined with UPath, it is very powerful.

I do keep running up against some issues with the caching, however, and I am wondering if I am missing something. Whenever I use UPath/fsspec in a process that writes data, things work well, but whenever I try to use it to read files that are being updated I run into issues with the listings cache: the default behavior is to enable the listings cache with no timeout. UPath reuses fsspec instances, so even if I create a new UPath instance to the same URI, it will find and reuse the instance of the filesystem (which holds the listings cache). My (not so elegant solution) is to pass use_listings_cache=False to UPath, but I am coming to the realization that implementations of AbstractFileSystem don't have a uniform API for forwarding kwargs to super(). For example, FTPFileSystem takes kwargs that it forwards to super(), but SFTPFileSystem takes ssh_kwargs which it both forwards to its super() and uses to create the client, which chokes on use_listings_cache=False. Perhaps, this is simply a bug and I'm happy to report it, but I am wondering if I'm not just using UPath incorrectly because I imagine that using it for reading externally changing filesystems is a common pattern, and there may be a different preferred way of doing that.

[martindurant]

Perhaps, this is simply a bug

Definitely, such kwargs should be passed up and not result in instantiating. If you can't use use_listings_cache=False, it's a bug and should be fixed. This fix might belong in fsspec.spec._Cached .

You should also note that not all filesystems use a directory listings cache at all, usually depending on how big the latency on a list call is expected to be.