Is there any way to reduce the inference time for this part

[BugO0]

Is there any way to reduce the inference time for this part
The following is the output information of one inference, which seems to take much longer than pure llama.cpp inference, and mainly because this part takes time

[gjmulder]

Use a faster GPU or a smaller model. e.g. 7B params with a 3080TI:

llama_print_timings: prompt eval time = 695.29 ms / 150 tokens ( 4.64 ms per token)

Use the cache:

llama_cpp.llama.Llama.set_cache

[Dampfinchen]

The new matmul kernels by Johannes (upstream) are merged now and can reduce VRAM a lot as well as increasing prompt processing speed for non k-quants (k-quants will be slower for now).

Once its merged in the python bindings, you can enable it using the --mul-mat-q command.

[BugO0]

Use a faster GPU or a smaller model. e.g. 7B params with a 3080TI:

llama_print_timings: prompt eval time = 695.29 ms / 150 tokens ( 4.64 ms per token)

Use the cache:

llama_cpp.llama.Llama.set_cache

I have found the reason for the slow inference speed. I was using version 0.1.77, but the speed was much faster with version 0.1.68, the latest version 0.1.78 also has normal speed.

[BugO0]

I have found the reason for the slow inference speed. I was using version 0.1.77, but the speed was much faster with version 0.1.68, the latest version 0.1.78 also has normal speed.

I have found the reason for the slow inference speed. I was using version 0.1.77, but the speed was much faster with version 0.1.68, the latest version 0.1.78 also has normal speed.