Serialization: support saving torch state dict to disk

[Wauplin]

Implement save_torch_state_dict to save a torch state dictionary to disk (first part of #2065). It uses split_torch_state_dict_into_shards under the hood (#1938).

State dict is saved either to a single file (if less than 5GB) or to shards with the corresponding index.json. By default, shards are saved as safetensors but safe_serialization=False can be passed to save as pickle. A warning is logged when saving as pickle and hopefully we should be able to dropped support for it when transformers/diffusers/accelerate/... completely phase out from .bin saving. cc @LysandreJik I'd like to get your opinion on this. I'm fine with not adding support for .bin files at all but worry it would slow down adoption in our libraries.

For the implementation, I took inspiration from huggingface/diffusers#7830 + accelerate/transformers. What it does:

1. Split state dict into shard (logic already exists)
2. Clean existing directory (remove previous shard/index files)
3. Write shards to disk
4. Write index to disk (optional)

Example:

>>> from huggingface_hub import save_torch_state_dict
>>> model = ... # A PyTorch model

# Save state dict to "path/to/folder"
# The model is split into shards of 5GB each and saved as safetensors.
>>> state_dict = model_to_save.state_dict()
>>> save_torch_state_dict(state_dict, "path/to/folder")

cc @amyeroberts / @ArthurZucker for transformers, @sayakpaul for diffusers, @SunMarc @muellerzr for accelerate
Happy to get feedback on this type of critical part. The goal is to standardize things to be consistent across libraries so please let me know if you want to add/remove something!

(documentation has also been updated)

---

note: I also removed split_numpy_state_dict_into_shards which is a breaking change but I don't expect anything to break in the wild. Better to just remove it to avoid future maintenance (I shouldn't have added it in the first place).

(failing CI is unrelated)

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[sayakpaul]

Very nicely done.

[SunMarc]

Thanks for adding this ! LGTM ! Just a small nit

[SunMarc]

filename_pattern is modified even when the user passes a filename_pattern != None.

[Wauplin]

Good catch! Fixed it in 5c4cac7.

[amyeroberts]

Very nice ❤️

[amyeroberts]

Nice :)

[Wauplin]

Thanks everyone for the reviews! ❤️