spike: small model presets experiments

[jalling97]

Model presets

LeapfrogAI currently has two primary models that are used on the backend, but more should be added/tested. By implementing certain small models and evaluating their efficacy from a human perspective, we can make better decisions as to what models to use and evaluate against.

Goal

To determine a list of models and model configs that work well in LFAI from a human-in-the-loop perspective (no automated evals).

Methodology

• Determine a short list (up to 5) models to test in LFAI (from HuggingFace is the simplest way to do this)
• Run LFAI in a local dev context, replacing the model backend with one of these choices
• Change the config parameters to gauge performance
• Record config

Limitations

• Model licenses are very permissive (e.g MIT, Apache-2.0)
• Must be compatible with vllm or llama-cpp-python (the two frameworks currently supported by LFAI)
• Limited vRAM requirements (12-16Gb including model weights and context)

Delivery

• A list of models, different config options, and a respective set of subjective scores gauging their performance within LFAI.
• A report of the methodology used to evaluate these models (so the experiment can be replicated)
• Potentially a repository that contains code used to run these evaluations (and what was evaluated on)

[jxtngx]

I can take this.

Here's an example from AWS on collecting HIL feedback for LM evals:

https://github.com/aws-samples/human-in-the-loop-llm-eval-blog

Are the questions in that example appropriate for the intent of this experiment?

[jxtngx]

Here's a list of additional models for consideration.

1. https://huggingface.co/meta-llama/Meta-Llama-3.1-8B
2. https://huggingface.co/nvidia/Llama-3.1-Minitron-4B-Width-Base # distilled Llama 3.1 model
3. https://huggingface.co/nvidia/Llama-3.1-Minitron-4B-Depth-Base # distilled Llama 3.1 model
4. https://huggingface.co/nvidia/Mistral-NeMo-Minitron-8B-Base
5. https://huggingface.co/nvidia/Minitron-8B-Base
6. https://huggingface.co/nvidia/Minitron-4B-Base
7. https://huggingface.co/RWKV/v6-Finch-7B-HF
8. https://huggingface.co/RWKV/v6-Finch-3B-HF

The above models would be used as is – without any quantization; unless it is preferred that I also quantize the models and provide feedback on 8-bit and 4-bit versions.

I can also provide feedback on the models listed in the docs.

1. https://huggingface.co/TheBloke/SynthIA-7B-v2.0-GGUF # CPU
2. https://huggingface.co/TheBloke/SynthIA-7B-v2.0-GPTQ # GPU
3. https://huggingface.co/defenseunicorns/Hermes-2-Pro-Mistral-7B-4bit-32g-GPTQ
4. https://huggingface.co/NousResearch/Nous-Hermes-2-Mixtral-8x7B-DPO-GGUF

[jxtngx]

I realize that not all of the models may be supported by llama-cpp-python or vLLM, and that I may need to add support for a custom model; especially for CPU only on macOS.

Here are two GGUF conversion examples for reference:

1. https://github.com/ggerganov/llama.cpp/discussions/ 2948
2. https://brev.dev/blog/convert-to-llamacpp

[jxtngx]

As for the models that have proprietary licenses – if anything, it will be beneficial to run these subjective evals and provide feedback for the community.

From a user perspective – I'd probably opt for these models on my own, foregoing the default selections – though I understand the need for the LFAI team to ship with a default model that is very permissive in terms of licensing.

[jalling97]

For reference I've updated the issue description a little bit to help clarify a few things.

For the AWS HIL example, that framework looks like it makes sense for what we're asking for, so if you would like to use it as a basis, go for it!

I added these to the description, but we have a few limitations I didn't outline originally:

• Model licenses are very permissive (e.g MIT, Apache-2.0)
• Models must be compatible with vllm or llama-cpp-python (the two frameworks currently supported by LFAI)
• Limited vRAM requirements (12-16Gb including model weights and context)

If you're on MacOS, we won't ask you to work outside of the deployment context available to you, so anything that can be run on llama-cpp-python is great. For simplicity, let's stick to model licenses that are as permissive as Apache-2.0 or greater.

The vRAM requirements are ideally less than 12Gb, but anything that would fit under 16Gb is worth checking for our purposes (i.e single-GPU laptop deployment scenarios). That likely means quantizations, so if you can find quantizations for models you want to test, great! We're also open to managing our own quantized models, so feel free to experiment with your own quantizations if you want to, but it's certainly not required.

It would be greatly helpful to compare any models you test with the current defaults from the docs (as you already listed). That would act as a fantastic point of comparison.

[jxtngx]

Important

RWKV is a recurrent model architecture (paper)

RWKV is a different architecture than transformers based models. The model arch is not available in llama-cpp, but can be made available for use with llama-cpp-python by using the gguf-my-repo HF Space.

With regard to the HF Space, a user must understand the quantization key provided below:

https://huggingface.co/docs/hub/en/gguf#quantization-types.

Note

quantized, llama-cpp-python compatible models will be made available in this HF collection.