Support multiple tokens locally

[hanouticelina]

Closes #2446.

This PR adds a multi-token support allowing users to easily manage and switch between multiple tokens.

List of changes

• Added a token-name argument logout helpers and the corresponding commands.
• Refactored all modules related to authentification into huggingface_hub/src/utils/_auth.py.
• Added new commands huggingface-cli auth switch and huggingface-cli auth list to switch between tokens and list saved tokens respectively. New helpers huggingface_hub._login.auth_switch and huggingface_hub._login.auth_list are also added.
• Multiple helper functions are added : get_stored_tokens(), _save_stored_tokens(), _get_token_by_name(), _save_token().
• Unit tests are added in tests/test_auth.py.
• Documentation: A new section added to quick-start.md, guides/cli.md and login package reference are also updated.

Usage

Login

huggingface-cli login [--token TOKEN]

• Logs in with the provided token or prompts for one if not provided.
• The token is saved locally under the name provided by the server.
• If --token is not provided, prompts the user for token interactively.

List locally stored tokens

huggingface-cli auth list

• Lists all saved tokens.

Example output:

name                     | token          
-------------------------|---------------
  dummy_token            | hf_****XXXX    
  * all_perm             | hf_****YYYY    


Note: Environment variable `HF_TOKEN` is set and is the current active token.

Logout

huggingface-cli logout [--token-name TOKEN_NAME]

• Removes the specified token.
• If --token-name is not specified, logs out from all tokens.

Switching the current active token

huggingface-cli auth switch [--token-name TOKEN_NAME]

• Sets the specified token as the active token for subsequent commands.
• If --token-name is not provided, prompts the user to select from locally saved tokens.
• This command will override the token written in ~/.cache/huggingface/token (defined by HF_TOKEN_PATH env variable).

How it Works

Token storage

Tokens will be stored in ~/.cache/huggingface/stored_tokens (or HF_STORED_TOKENS_PATH env variable if set), which is INI format file:

[token_name_1] 
hf_token = hf_XXXXXXX 
[token_name_2] 
hf_token = hf_YYYYYYY
[token_name_3] 
hf_token = hf_ZZZZZZZ

Token Selection Priority

We will keep the same token retrieval priority order

1. HF_TOKEN env variable.
2. Token file ~/.cache/huggingface/token. The token stored in this file will be updated when switching between tokens.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[Wauplin]

Thanks for working on it @hanouticelina !

I've not properly tested it locally yet but started to make a pass on the implementation. I think we can regroup some things to avoid multiple utils modules and decouple the low-level logic (e.g. how the config file is structured) from the high-level logic (e.g. how to present stuff to the users).

[Wauplin]

Hey there! I finally took some time to extensively test things locally. I think you are on the right track to get this feature shipped. I found a few corner cases that I think should be tackled differently. Other than that, it's mostly cosmetic changes in the API design

[Wauplin]

TODO in a future PR: remove all print statement from this module and replace them with proper logs. And then in the CLI set verbosity level to INFO.

[Wauplin]

(btw, you can set the PR to "ready for review" when it's the case :) )

[Wauplin]

Looking good! As discussed offline, let's use the "token name" returned by the server as profile names :) Once updated we should be good to merge. Thanks for the extensive tests!

[Wauplin]

Thanks for the change! Last round of review I think :) I tested everything locally and the UX is really smooth now 🤗

Feel free to merge once the comments below are addressed and CI is green.

[hanouticelina]

thanks a lot @Wauplin and sorry for the lengthy back and forth reviews 🙈 with your detailed feedback on this PR, I'm sure the next ones will be much easier to review 😄
failing tests seem unrelated

[Wauplin]

All good!

[hanouticelina]

I think we are good with this one, let's merge :)

[Pierrci]

nice!! 🎉🎉🎉