Should llamafile works with mistral nemo? #543
-
|
clear I downloaded the model and tried the commando below: llamafile -m Mistral-Nemo-Instruct-2407-f16.gguf I have 64 gigabytes of DDR5 ram and a Rayzen 7 7700X Processor.... As a result of the command I got the output below: C:\Users\CPU.cache\lm-studio\models\second-state\Mistral-Nemo-Instruct-2407-GGUF>llamafile -m Mistral-Nemo-Instruct-2407-f16.gguf C:\Users\CPU.cache\lm-studio\models\second-state\Mistral-Nemo-Instruct-2407-GGUF> |
Beta Was this translation helpful? Give feedback.
Replies: 4 comments 11 replies
-
|
Try passing |
Beta Was this translation helpful? Give feedback.
-
|
I am not sure if I am doing right, but I tried your sugestion -c 2048 and I also tried -c 1024 ... The error message that calls my attention from the terminal output is "error loading model hyperparameters: invalid n_row 128, expected 160" ... This message is the same without -c paremeter and with both -c 2048 or -c 1024.... Below are the outputs printed in my screen: C:\Users\CPU.cache\lm-studio\models\second-state\Mistral-Nemo-Instruct-2407-GGUF>llamafile.exe -c 2048 -m Mistral-Nemo-Instruct-2407-f16.gguf C:\Users\CPU.cache\lm-studio\models\second-state\Mistral-Nemo-Instruct-2407-GGUF>llamafile.exe -c 1024 -m Mistral-N C:\Users\CPU.cache\lm-studio\models\second-state\Mistral-Nemo-Instruct-2407-GGUF> |
Beta Was this translation helpful? Give feedback.
-
|
Chances are you're using an old version of the llamafile software. You need to use 0.8.13. To make things easy for you, I've started uploading llamafiles for Mistral Nemo Instruct here: I can confirm that these are working. Enjoy! |
Beta Was this translation helpful? Give feedback.
-
|
I am very glad to communicate that your instructions made it work! I had two issues: an earlier version (0.8.11) and needing to input -c 2048 as a parameter. Many thanks for your help! =D |
Beta Was this translation helpful? Give feedback.
-
|
on CPU, v0.8.13: ... ./llamafile-0.8.13/bin/llamafile-bench -p "32,64,128,256,512" -n "16" -m "Mistral-Nemo-Instruct-2407-Q6_K.gguf"
64Go RAM. (but Q6 not F16... )
|
Beta Was this translation helpful? Give feedback.
-
For the case of Mistral-Nemo, https://mistral.ai/fr/news/mistral-nemo/:
We do not have FP8 with llama.cpp or llamafile, but it mean that in that case FP16 and BF16 is more than enough, no need for FP32. For other Mistral model there are natively distribut in BF16 format to. So as jar say , some model "support" hight quantisation, and some other not, which makes the question even more "interesting" 😉 and need some testing. A problem/limite with the big models is their execution time, for exemple, Mistral-Large-Instruct-2407-Q2_K.gguf (~45Go) don't give stupid answer, but is really slow on CPU. |
Beta Was this translation helpful? Give feedback.
-
|
llamafile and llama.cpp style quantization formats are probably much more dynamic and adaptable than FP8, which I assume has a specific defined range of floating point numbers. With our quants, there's a certain degree of tolerance for floating point numbers floating around the spectrum, since an independent scaling factor variable is stored for each block, among other tricks. With FP8 I imagine you have to train your model to be friendly to it. But with our quants you just have to not go too wild on the numbers. At least that's how I imagine it works based on my reading of some code. You'd need to ask someone like Kawrakow to get a more authoritative answer. |
Beta Was this translation helpful? Give feedback.
-
|
Also I would really like to have FP8 now that you mention it. Nothing would make me happier than to be able to index individual 8-bit samples in my GEMM algorithm. |
Beta Was this translation helpful? Give feedback.
-
Yes I think so... but with GPU like MI300/and last datacenter NVIDIA GPU that support native compute with FP8, and mistral-nemo that have been train to work with it...
With E4M3 we may have better result that with Q4 with nothing special to do. |
Beta Was this translation helpful? Give feedback.
-
|
Could you help me find the best C reference implementation for encoding / decoding FP8 numbers of that specific variety used by NVIDIA GPUs? For example, here's what it looks like for BF16. static inline float ggml_compute_bf16_to_fp32(ggml_bf16_t h) {
union {
float f;
uint32_t i;
} u;
u.i = (uint32_t)h.bits << 16;
return u.f;
}
static inline ggml_bf16_t ggml_compute_fp32_to_bf16(float s) {
ggml_bf16_t h;
union {
float f;
uint32_t i;
} u;
u.f = s;
if ((u.i & 0x7fffffff) > 0x7f800000) { /* nan */
h.bits = (u.i >> 16) | 64; /* force to quiet */
return h;
}
h.bits = (u.i + (0x7fff + ((u.i >> 16) & 1))) >> 16;
return h;
}If you can find me the canonical functions for doing the above with an FP8 design that's compatible with Nemo, then I can implement it into llamafile potentially in a few hours. |
Beta Was this translation helpful? Give feedback.
Chances are you're using an old version of the llamafile software. You need to use 0.8.13.
To make things easy for you, I've started uploading llamafiles for Mistral Nemo Instruct here:
I can confirm that these are working. Enjoy!