Run Local LLM Offline

Run Local LLM Offline

LetMeDoIt AI support open source Hugging Face language models, with running local LLM server 'ollama'.

An Ollama chatbot is designed for users to chat with various open source offline models.

We also created some plugins for users to seamlessly integrates the offline chatbot with LetMeDoIt AI prompt.

Requirement

LLM server: Install 'Ollama' first. Read https://ollama.com

Memory: You should have at least 8 GB of RAM available to run the 7B models, 16 GB to run the 13B models, and 32 GB to run the 33B models.

Ollama Chat

Launch Ollama Chat and select a model.

Find model information at https://ollama.com/library

During the initial execution of a newly chosen model, LetMeDoIt AI will automatically download it for you.

After the initial download, you can enter your query in the Ollama chat prompt

Plugins

Related plugins are: ask ollama, ask gemma, ask mistral, ask llama2, ask llava

The ask ollama is the wildcard option, as you can choose any language models supported by ollama.   Enable these plugins allows you to ask Ollam / Gemma / Mistral / Llama2 / Llava directly from LetMeDoIt AI prompt, e.g.:

ask mistral the same question

Launch from Tray Menu

You can launch ollama chat models from tray menu.

Edit Tray menu

You can modify extended items displayed on LetMeDoIt AI tray menu:

1. Run '.editconfigs' in LetMeDoIt AI prompt

2. Locate and edit item 'customTrayCommands' # default: ['mistral', 'llama2', 'gemma7b', 'lava']

3. Press ctrl+s & ctrl+q to save and return to LetMeDoIt AI prompt

Launch from CLI commands

Available CLI commands: 'ollamachat', 'mistral', 'llama2', 'llama213b', 'llama270b', 'gemma2b', 'gemma7b', 'llava', 'phi', 'vicuna'

All the commands support default argument as default entry.

Only 'ollamachat' accepts -m to specify a model.