Added mistral instruct chat format as "mistral-instruct"

[Rafaelblsilva]

@abetlen Thanks for you work!

Here is my small contribution.

[alex4321]

Did you checked - does it (llama.cpp nowadays tokenizer) even tokenize text correctly?

Because from what I found at https://huggingface.co/TheBloke/Mistral-7B-Instruct-v0.1-GGUF - prompt format is:

<s>[INST] {prompt} [/INST]

And inside Llama::create_chat_completion prompt converted to a string before passing to the tokenizer into Llama:_create_completion:

        prompt_tokens: List[int] = (
            self.tokenize(prompt.encode("utf-8"))
            if prompt != ""
            else [self.token_bos()]
        )

While I tried the following code:

for token in model.tokenize("<s>[INST] Test prompt [/INST]".encode(), add_bos=False):
    print(model.detokenize([token]))

And the output was:

b' <'
b's'
b'>'
b'['
b'INST'
b']'
b' Test'
b' prompt'
b' ['
b'/'
b'INST'
b']'

I guess <s>, [INST] and [/INST] should be a special tokens?
However, I did not dive into the issue much yet and when the tokenization will work it will be useful.

So I am asking just in case I missed something and this stuff should work already.

@Rafaelblsilva

[Rafaelblsilva]

@alex4321

I have not checked that deep.
If i understood correctly your observation suggest there might be underlying issues even with llama-2 tokenization?

Is this model dependent, or something that would need to be implemented on llama.cpp?

I've quickly looked on open issues there and found these:

ggerganov/llama.cpp#3475
ggerganov/llama.cpp#2820

But at least on mistral model with my changes it works a bit better than just using the llama-2 prompt.

[fakerybakery]

Hi can we merge this?

[zhangp365]

Based on this document, I believe this pull request is accurate for a single round. However, according to the document, for multiple rounds of history, each round should conclude with '< /s >'.

[fakerybakery]

Hi @abetlen might it be possible to merge this, now that Mistral Instruct v0.2 has been released?

[handshape]

I'm not certain that the models are consistently emitting </s> as a single token. I'm hand-cranking the sampling/detokenization loop, and seeing that about 80% of the time, the eos token is coming out as a single token, and the other 20% of the time it's coming out as a sequence of four single-character tokens. I'm having to catch all permutations.

[abetlen]

Hey @Rafaelblsilva sorry for missing this, for some reason I thought it overlapped with another format that was already supported. Thank you for the contribution!

Note: I renamed the chat format to mistral-instruct so it's consistent with the finetuned model name.