huggingface-cli download: FileNotFoundError No such file or directory

[noahgorstein]

Describe the bug

Hey 👋 ! When attempting to download a model into a local directory using the huggingface-cli, I am seeing this issue occur non-deterministically where a .gitignore.lock file is not found.

It looked like there was a similar issue reported in the past however it looks like a commit was made to try and fix it. I believe I am on the latest version of the cli which should have that commit.

#2276

Thanks in advance for your help 😄 !

Reproduction

This happens with different models for me. This is just one example. Link to it on Hugging Face: https://huggingface.co/protectai/deberta-v3-base-prompt-injection-v2

Here is my huggingface-cli command:

huggingface-cli download --local-dir "app/models/protectai/deberta-v3-base-prompt-injection-v2" "protectai/deberta-v3-base-prompt-injection-v2"

Logs

0.103 Downloading models...
Fetching 19 files:   0%|          | 0/19 [00:00<?, ?it/s]Downloading 'README.md' to '/app/models/protectai/deberta-v3-base-prompt-injection-v2/.huggingface/download/README.md.cb68f2938b4d43cc207677cd87d3583764087640.incomplete'
0.665 Downloading 'onnx/added_tokens.json' to '/app/models/protectai/deberta-v3-base-prompt-injection-v2/.huggingface/download/onnx/added_tokens.json.8ee2b3623dc526b123cde0aaa401755b82299af2.incomplete'
0.668 Downloading 'onnx/config.json' to '/app/models/protectai/deberta-v3-base-prompt-injection-v2/.huggingface/download/onnx/config.json.3366690a064655366dd7dd57f9b3cf1e8ba8e78f.incomplete'
0.681 Downloading 'model.safetensors' to '/app/models/protectai/deberta-v3-base-prompt-injection-v2/.huggingface/download/model.safetensors.6521cb8d0ac08148c81464899c424e6148fcc62befa371089fa4061d8b6e0424.incomplete'
0.684 Downloading '.gitattributes' to '/app/models/protectai/deberta-v3-base-prompt-injection-v2/.huggingface/download/.gitattributes.a6344aac8c09253b3b630fb776ae94478aa0275b.incomplete'
0.691 Download complete. Moving file to /app/models/protectai/deberta-v3-base-prompt-injection-v2/README.md
0.698 Download complete. Moving file to /app/models/protectai/deberta-v3-base-prompt-injection-v2/onnx/config.json
0.701 Download complete. Moving file to /app/models/protectai/deberta-v3-base-prompt-injection-v2/onnx/added_tokens.json
0.708 Downloading 'config.json' to '/app/models/protectai/deberta-v3-base-prompt-injection-v2/.huggingface/download/config.json.75347aec096be871790c81deeb4175ec9a2be98a.incomplete'
0.715 Download complete. Moving file to /app/models/protectai/deberta-v3-base-prompt-injection-v2/.gitattributes
Fetching 19 files:   5%|▌         | 1/19 [00:00<00:02,  6.02it/s]Downloading 'onnx/spm.model' to '/app/models/protectai/deberta-v3-base-prompt-injection-v2/.huggingface/download/onnx/spm.model.c679fbf93643d19aab7ee10c0b99e460bdbc02fedf34b92b05af343b4af586fd.incomplete'
0.729 Downloading 'onnx/special_tokens_map.json' to '/app/models/protectai/deberta-v3-base-prompt-injection-v2/.huggingface/download/onnx/special_tokens_map.json.83fb22de184abab1b193b1f6001e3015e865e50b.incomplete'
0.733 Downloading 'onnx/model.onnx' to '/app/models/protectai/deberta-v3-base-prompt-injection-v2/.huggingface/download/onnx/model.onnx.f0ea7f239f765aedbde7c9e163a7cb38a79c5b8853d3f76db5152172047b228c.incomplete'
0.736 Download complete. Moving file to /app/models/protectai/deberta-v3-base-prompt-injection-v2/config.json
0.747 Downloading 'onnx/tokenizer.json' to '/app/models/protectai/deberta-v3-base-prompt-injection-v2/.huggingface/download/onnx/tokenizer.json.00c50710453478105265c838bfb3bad69b410891.incomplete'
0.761 Download complete. Moving file to /app/models/protectai/deberta-v3-base-prompt-injection-v2/onnx/special_tokens_map.json
0.763 Downloading 'LICENSE' to '/app/models/protectai/deberta-v3-base-prompt-injection-v2/.huggingface/download/LICENSE.2bb9ad240fa04c8cf706a4901c4807878e90c2dc.incomplete'
0.774 Downloading 'onnx/tokenizer_config.json' to '/app/models/protectai/deberta-v3-base-prompt-injection-v2/.huggingface/download/onnx/tokenizer_config.json.7a76cf2e0d2a72e3baf09096c657dc5d977b04ec.incomplete'
0.786 Downloading 'added_tokens.json' to '/app/models/protectai/deberta-v3-base-prompt-injection-v2/.huggingface/download/added_tokens.json.8ee2b3623dc526b123cde0aaa401755b82299af2.incomplete'
0.789 Downloading 'special_tokens_map.json' to '/app/models/protectai/deberta-v3-base-prompt-injection-v2/.huggingface/download/special_tokens_map.json.2c9cb07c8fdeeb5ac3ceafb170592e990b204dcd.incomplete'
0.809 Download complete. Moving file to /app/models/protectai/deberta-v3-base-prompt-injection-v2/LICENSE
0.812 Download complete. Moving file to /app/models/protectai/deberta-v3-base-prompt-injection-v2/onnx/tokenizer_config.json
0.818 Download complete. Moving file to /app/models/protectai/deberta-v3-base-prompt-injection-v2/added_tokens.json
Fetching 19 files:  26%|██▋       | 5/19 [00:00<00:00, 18.60it/s]
0.821 Download complete. Moving file to /app/models/protectai/deberta-v3-base-prompt-injection-v2/special_tokens_map.json
0.851 Downloading 'spm.model' to '/app/models/protectai/deberta-v3-base-prompt-injection-v2/.huggingface/download/spm.model.c679fbf93643d19aab7ee10c0b99e460bdbc02fedf34b92b05af343b4af586fd.incomplete'
0.867 Downloading 'tokenizer.json' to '/app/models/protectai/deberta-v3-base-prompt-injection-v2/.huggingface/download/tokenizer.json.e68aa4603df5ac5423c3a2ecae182bd1a80f509a.incomplete'
0.870 Downloading 'tokenizer_config.json' to '/app/models/protectai/deberta-v3-base-prompt-injection-v2/.huggingface/download/tokenizer_config.json.13d6406af565bd9e59dd649afd4c2b48bddb6a4c.incomplete'
0.943 Download complete. Moving file to /app/models/protectai/deberta-v3-base-prompt-injection-v2/tokenizer_config.json
2.674 Download complete. Moving file to /app/models/protectai/deberta-v3-base-prompt-injection-v2/spm.model
4.778 Download complete. Moving file to /app/models/protectai/deberta-v3-base-prompt-injection-v2/onnx/spm.model
5.795 Download complete. Moving file to /app/models/protectai/deberta-v3-base-prompt-injection-v2/onnx/tokenizer.json
7.193 Download complete. Moving file to /app/models/protectai/deberta-v3-base-prompt-injection-v2/tokenizer.json
86.40 Download complete. Moving file to /app/models/protectai/deberta-v3-base-prompt-injection-v2/model.safetensors
129.4 Download complete. Moving file to /app/models/protectai/deberta-v3-base-prompt-injection-v2/onnx/model.onnx
129.4 Traceback (most recent call last):
129.4   File "/usr/local/bin/huggingface-cli", line 8, in <module>
129.4     sys.exit(main())
129.4              ^^^^^^
129.4   File "/usr/local/lib/python3.11/site-packages/huggingface_hub/commands/huggingface_cli.py", line 51, in main
129.4     service.run()
129.4   File "/usr/local/lib/python3.11/site-packages/huggingface_hub/commands/download.py", line 146, in run
129.4     print(self._download())  # Print path to downloaded files
129.4           ^^^^^^^^^^^^^^^^
129.4   File "/usr/local/lib/python3.11/site-packages/huggingface_hub/commands/download.py", line 180, in _download
129.4     return snapshot_download(
129.4            ^^^^^^^^^^^^^^^^^^
129.4   File "/usr/local/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
129.4     return fn(*args, **kwargs)
129.4            ^^^^^^^^^^^^^^^^^^^
129.4   File "/usr/local/lib/python3.11/site-packages/huggingface_hub/_snapshot_download.py", line 294, in snapshot_download
129.4     thread_map(
129.4   File "/usr/local/lib/python3.11/site-packages/tqdm/contrib/concurrent.py", line 69, in thread_map
129.4     return _executor_map(ThreadPoolExecutor, fn, *iterables, **tqdm_kwargs)
129.4            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
129.4   File "/usr/local/lib/python3.11/site-packages/tqdm/contrib/concurrent.py", line 51, in _executor_map
129.4     return list(tqdm_class(ex.map(fn, *iterables, chunksize=chunksize), **kwargs))
129.4            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
129.4   File "/usr/local/lib/python3.11/site-packages/tqdm/std.py", line 1181, in __iter__
129.5     for obj in iterable:
129.5   File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 619, in result_iterator
129.5     yield _result_or_cancel(fs.pop())
129.5           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
129.5   File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 317, in _result_or_cancel
129.5     return fut.result(timeout)
129.5            ^^^^^^^^^^^^^^^^^^^
129.5   File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 449, in result
129.5     return self.__get_result()
129.5            ^^^^^^^^^^^^^^^^^^^
129.5   File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
129.5     raise self._exception
129.5   File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
129.5     result = self.fn(*self.args, **self.kwargs)
129.5              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
129.5   File "/usr/local/lib/python3.11/site-packages/huggingface_hub/_snapshot_download.py", line 268, in _inner_hf_hub_download
129.5     return hf_hub_download(
129.5            ^^^^^^^^^^^^^^^^
129.5   File "/usr/local/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
129.5     return fn(*args, **kwargs)
129.5            ^^^^^^^^^^^^^^^^^^^
129.5   File "/usr/local/lib/python3.11/site-packages/huggingface_hub/file_download.py", line 1202, in hf_hub_download
129.5     return _hf_hub_download_to_local_dir(
129.5            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
129.5   File "/usr/local/lib/python3.11/site-packages/huggingface_hub/file_download.py", line 1406, in _hf_hub_download_to_local_dir
129.5     paths = get_local_download_paths(local_dir=local_dir, filename=filename)
129.5             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
129.5   File "/usr/local/lib/python3.11/site-packages/huggingface_hub/_local_folder.py", line 138, in get_local_download_paths
129.5     metadata_path = _huggingface_dir(local_dir) / "download" / f"{sanitized_filename}.metadata"
129.5                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^
129.5   File "/usr/local/lib/python3.11/site-packages/huggingface_hub/_local_folder.py", line 223, in _huggingface_dir
129.5     with WeakFileLock(gitignore_lock):
129.5   File "/usr/local/lib/python3.11/contextlib.py", line 137, in __enter__
129.5     return next(self.gen)
129.5            ^^^^^^^^^^^^^^
129.5   File "/usr/local/lib/python3.11/site-packages/huggingface_hub/utils/_fixes.py", line 84, in WeakFileLock
129.5     lock.acquire()
129.5   File "/usr/local/lib/python3.11/site-packages/filelock/_api.py", line 332, in acquire
129.5     self._acquire()
129.5   File "/usr/local/lib/python3.11/site-packages/filelock/_unix.py", line 42, in _acquire
129.5     fd = os.open(self.lock_file, open_flags, self._context.mode)
129.5          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
129.5 FileNotFoundError: [Errno 2] No such file or directory: '/app/models/protectai/deberta-v3-base-prompt-injection-v2/.huggingface/.gitignore.lock'

System info

huggingface_hub version: 0.23.4
Platform: Linux-6.8.0-35-generic-x86_64-with-glibc2.31
Python version: 3.11.9
Running in iPython ?: No
Running in notebook ?: No
Running in Google Colab ?: No
Token path ?: /root/.cache/huggingface/token
Has saved token ?: False
Configured git credential helpers:
FastAI: N/A
Tensorflow: N/A
Torch: 2.3.1
Jinja2: 3.1.4
Graphviz: N/A
keras: N/A
Pydot: N/A
Pillow: N/A
hf_transfer: N/A
gradio: N/A
tensorboard: N/A
numpy: 1.26.4
pydantic: 2.7.4
aiohttp: 3.9.5
ENDPOINT: https://huggingface.co
HF_HUB_CACHE: /root/.cache/huggingface/hub
HF_ASSETS_CACHE: /root/.cache/huggingface/assets
HF_TOKEN_PATH: /root/.cache/huggingface/token
HF_HUB_OFFLINE: False
HF_HUB_DISABLE_TELEMETRY: False
HF_HUB_DISABLE_PROGRESS_BARS: None
HF_HUB_DISABLE_SYMLINKS_WARNING: False
HF_HUB_DISABLE_EXPERIMENTAL_WARNING: False
HF_HUB_DISABLE_IMPLICIT_TOKEN: False
HF_HUB_ENABLE_HF_TRANSFER: False
HF_HUB_ETAG_TIMEOUT: 10
HF_HUB_DOWNLOAD_TIMEOUT: 10

[Wauplin]

Hi @noahgorstein, sorry about the inconvenience. This fix you mentioned (#2288) has not been officially released yet. We will ship this very soon now. In the meantime, you can already benefit from the fix by installing from source.

[j-adamczyk]

@Wauplin is the fixed deployed now? It frequently, but randomly, breaks our tests in scikit-fingerprints

[Wauplin]

@j-adamczyk sorry for the inconvenience. The fix has been shipped yesterday as part of the 0.24.0 release :)

[j-adamczyk]

Great news, thanks!