A comprehensive benchmark designed to evaluate the performance and resource consumptions of LLMs & LMMs for on-device use cases.
To install MobileBench, follow these steps:
- Clone the Repository:
git clone --recurse-submodules https://github.com/SalesforceAIResearch/MobileAIBench.git
- Create a Conda Environment:
conda create -n mobile_bench python=3.10 conda activate mobile_bench
- Run the Makefile:
make
- Add OpenAI API Key:
export OPENAI_API_KEY=<OPENAI_API_KEY>
Here are some usage examples for running MobileAIBench:
-
Dataset: hotpot_qa & databricks-15k
-
Model: xgen2-3b.gguf
-
Run on GPU:
python ./src/mobile_bench.py --task question_answering --model_lib llama_cpp_python --model_name xgen2-3b.gguf --use_gpu
-
Run on CPU:
python ./src/mobile_bench.py --task question_answering --model_lib llama_cpp_python --model_name xgen2-3b.gguf
-
Model: xgen2-3b.gguf
-
Run on GPU:
python ./src/mobile_bench.py --task all --model_lib llama_cpp_python --model_name xgen2-3b.gguf --use_gpu
-
Run on CPU:
python ./src/mobile_bench.py --task all --model_lib llama_cpp_python --model_name xgen2-3b.gguf
- To run ios mobile app, refer to ./ios-app/README.md
- Here's a screenshot taken from the ios-app