load peft-model error (gradio_web_server)

[jackaihfia2334]

According to the correction in (#1933), loading the peft model through python3 -m fastchat.serve.cli --model-path /data2/project/peft-model successfully

However, when I use python3 -m fastchat.serve.gradio_web_server, the same error still occurs.

I checked the code and found that it is because in gradio_web_server.py, the incoming model_path is model_name, in my case it is "peft-model", instead of "data2/projcet/peft-model", which caused the adapter_model.config to not be found locally (#2262 (comment)), and then searched in huggingface, resulting in the same error.
So I hope you can fix this bug so that gradio_web_server can support peft_model loading.

Error following

---

root@docker-desktop:/# python3 -m fastchat.serve.gradio_web_server
2023-08-18 16:24:00 | INFO | gradio_web_server | args: Namespace(host='0.0.0.0', port=None, share=False, controller_url='http://localhost:21001', concurrency_count=10, model_list_mode='once', moderate=False, add_chatgpt=False, add_claude=False, add_palm=False, gradio_auth_path=None)
2023-08-18 16:24:00 | INFO | gradio_web_server | Models: ['peft-model']
2023-08-18 16:24:00 | INFO | stdout | Running on local URL: http://0.0.0.0:7860
2023-08-18 16:24:00 | INFO | stdout |
2023-08-18 16:24:00 | INFO | stdout | To create a public link, set share=True in launch().
2023-08-18 16:24:05 | INFO | gradio_web_server | load_demo. ip: 127.0.0.1. params: {}
2023-08-18 16:24:05 | INFO | httpx | HTTP Request: POST http://localhost:7860/api/predict "HTTP/1.1 200 OK"
2023-08-18 16:24:05 | INFO | httpx | HTTP Request: POST http://localhost:7860/reset "HTTP/1.1 200 OK"
2023-08-18 16:24:10 | INFO | gradio_web_server | add_text. ip: 127.0.0.1. len: 2
'(ReadTimeoutError("HTTPSConnectionPool(host='huggingface.co', port=443): Read timed out. (read timeout=10)"), '(Request ID: 6b2753fa-c523-43c3-80bf-70ebedbb1769)')' thrown while requesting HEAD https://huggingface.co/peft-model/resolve/main/adapter_config.json
2023-08-18 16:24:20 | WARNING | huggingface_hub.utils._http | '(ReadTimeoutError("HTTPSConnectionPool(host='huggingface.co', port=443): Read timed out. (read timeout=10)"), '(Request ID: 6b2753fa-c523-43c3-80bf-70ebedbb1769)')' thrown while requesting HEAD https://huggingface.co/peft-model/resolve/main/adapter_config.json
2023-08-18 16:24:20 | ERROR | stderr | Traceback (most recent call last):
2023-08-18 16:24:20 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/peft/utils/config.py", line 119, in from_pretrained
2023-08-18 16:24:20 | ERROR | stderr | config_file = hf_hub_download(
2023-08-18 16:24:20 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn
2023-08-18 16:24:20 | ERROR | stderr | return fn(*args, **kwargs)
2023-08-18 16:24:20 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/huggingface_hub/file_download.py", line 1291, in hf_hub_download
2023-08-18 16:24:20 | ERROR | stderr | raise LocalEntryNotFoundError(
2023-08-18 16:24:20 | ERROR | stderr | huggingface_hub.utils._errors.LocalEntryNotFoundError: Connection error, and we cannot find the requested files in the disk cache. Please try again or make sure your Internet connection is on.
2023-08-18 16:24:20 | ERROR | stderr |
2023-08-18 16:24:20 | ERROR | stderr | During handling of the above exception, another exception occurred:
2023-08-18 16:24:20 | ERROR | stderr |
2023-08-18 16:24:20 | ERROR | stderr | Traceback (most recent call last):
2023-08-18 16:24:20 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/gradio/routes.py", line 442, in run_predict
2023-08-18 16:24:20 | ERROR | stderr | output = await app.get_blocks().process_api(
2023-08-18 16:24:20 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/gradio/blocks.py", line 1392, in process_api
2023-08-18 16:24:20 | ERROR | stderr | result = await self.call_function(
2023-08-18 16:24:20 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/gradio/blocks.py", line 1097, in call_function
2023-08-18 16:24:20 | ERROR | stderr | prediction = await anyio.to_thread.run_sync(
2023-08-18 16:24:20 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/anyio/to_thread.py", line 33, in run_sync
2023-08-18 16:24:20 | ERROR | stderr | return await get_asynclib().run_sync_in_worker_thread(
2023-08-18 16:24:20 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
2023-08-18 16:24:20 | ERROR | stderr | return await future
2023-08-18 16:24:20 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/anyio/_backends/_asyncio.py", line 807, in run
2023-08-18 16:24:20 | ERROR | stderr | result = context.run(func, *args)
2023-08-18 16:24:20 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/gradio/utils.py", line 703, in wrapper
2023-08-18 16:24:20 | ERROR | stderr | response = f(*args, **kwargs)
2023-08-18 16:24:20 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/fastchat/serve/gradio_web_server.py", line 210, in add_text
2023-08-18 16:24:20 | ERROR | stderr | state = State(model_selector)
2023-08-18 16:24:20 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/fastchat/serve/gradio_web_server.py", line 68, in init
2023-08-18 16:24:20 | ERROR | stderr | self.conv = get_conversation_template(model_name)
2023-08-18 16:24:20 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/fastchat/model/model_adapter.py", line 291, in get_conversation_template
2023-08-18 16:24:20 | ERROR | stderr | return adapter.get_default_conv_template(model_path)
2023-08-18 16:24:20 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/fastchat/model/model_adapter.py", line 498, in get_default_conv_template
2023-08-18 16:24:20 | ERROR | stderr | config = PeftConfig.from_pretrained(model_path)
2023-08-18 16:24:20 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/peft/utils/config.py", line 123, in from_pretrained
2023-08-18 16:24:20 | ERROR | stderr | raise ValueError(f"Can't find '{CONFIG_NAME}' at '{pretrained_model_name_or_path}'")
2023-08-18 16:24:20 | ERROR | stderr | ValueError: Can't find 'adapter_config.json' at 'peft-model'
2023-08-18 16:24:20 | INFO | httpx | HTTP Request: POST http://localhost:7860/api/predict "HTTP/1.1 500 Internal Server Error"
2023-08-18 16:24:20 | INFO | httpx | HTTP Request: POST http://localhost:7860/reset "HTTP/1.1 200 OK"
2023-08-18 16:24:20 | INFO | gradio_web_server | bot_response. ip: 127.0.0.1
2023-08-18 16:24:20 | ERROR | stderr | Traceback (most recent call last):
2023-08-18 16:24:20 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/gradio/routes.py", line 442, in run_predict
2023-08-18 16:24:20 | ERROR | stderr | output = await app.get_blocks().process_api(
2023-08-18 16:24:20 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/gradio/blocks.py", line 1392, in process_api
2023-08-18 16:24:20 | ERROR | stderr | result = await self.call_function(
2023-08-18 16:24:20 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/gradio/blocks.py", line 1111, in call_function
2023-08-18 16:24:20 | ERROR | stderr | prediction = await utils.async_iteration(iterator)
2023-08-18 16:24:20 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/gradio/utils.py", line 346, in async_iteration
2023-08-18 16:24:20 | ERROR | stderr | return await iterator.anext()
2023-08-18 16:24:20 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/gradio/utils.py", line 339, in anext
2023-08-18 16:24:20 | ERROR | stderr | return await anyio.to_thread.run_sync(
2023-08-18 16:24:20 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/anyio/to_thread.py", line 33, in run_sync
2023-08-18 16:24:20 | ERROR | stderr | return await get_asynclib().run_sync_in_worker_thread(
2023-08-18 16:24:20 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
2023-08-18 16:24:20 | ERROR | stderr | return await future
2023-08-18 16:24:20 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/anyio/_backends/_asyncio.py", line 807, in run
2023-08-18 16:24:20 | ERROR | stderr | result = context.run(func, *args)
2023-08-18 16:24:20 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/gradio/utils.py", line 322, in run_sync_iterator_async
2023-08-18 16:24:20 | ERROR | stderr | return next(iterator)
2023-08-18 16:24:20 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/gradio/utils.py", line 691, in gen_wrapper
2023-08-18 16:24:20 | ERROR | stderr | yield from f(*args, **kwargs)
2023-08-18 16:24:20 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/fastchat/serve/gradio_web_server.py", line 300, in bot_response
2023-08-18 16:24:20 | ERROR | stderr | if state.skip_next:
2023-08-18 16:24:20 | ERROR | stderr | AttributeError: 'NoneType' object has no attribute 'skip_next'
2023-08-18 16:24:20 | INFO | httpx | HTTP Request: POST http://localhost:7860/api/predict "HTTP/1.1 500 Internal Server Error"
2023-08-18 16:24:20 | INFO | httpx | HTTP Request: POST http://localhost:7860/reset "HTTP/1.1 200 OK"

---

[dijkstra-mose]

I encountered the same problem. I found there is a bug in gradio_web_server.py. It doesn't read the conv template correctly while loading Peft model. There are two approaches to solve the problem.

Option I: To add the parameter "--conv-template" manually.

Option II: To fix the code bug in gradio_web_server.py.

class State:
    def __init__(self, model_name):
        self.conv = get_conversation_template(model_name)   

The State init calls get_conversation_template() here. But since in model_adapter.py, PeftModelAdapter dynamically reads conv_template, the result is incorrect here.The correct logic should be to call the model_worker's API dynamically to read the conv:

class State:
    def __init__(self, model_name):
        ret = requests.post(
            controller_url + "/get_worker_address", json={"model": model_name}
        )
        worker_addr = ret.json()["address"]
        ret = requests.post(worker_addr + "/worker_get_conv_template")
        conv = ret.json()["conv"]
        self.conv = Conversation(
            name=conv["name"],
            system_template=conv["system_template"],
            system_message=conv["system_message"],
            roles=conv["roles"],
            messages=conv["messages"],
            offset=conv["offset"],
            sep_style=conv["sep_style"],
            sep=conv["sep"],
            sep2=conv["sep2"],
            stop_str=conv["stop_str"],
            stop_token_ids=conv["stop_token_ids"],
        )
        logger.info(f"model_name: {model_name}, worker_addr: {worker_addr}, worker_get_conv_template")

Thus, the issue of deploying Lora checkpoint should have been resolved.

[LOMOGO]

有一个骚操作，就是你在使用 python3 -m fastchat.serve.model_worker 时，把 --model-names 改成你微调后的模型的路径就好了（设置值和 --model-path 的值一样）