Update core.py to have 1 extra token

[mosheber]

• added 1 token from fix history and all correct from the target

[keyboardAnt]

Please see the attached comments.

[keyboardAnt]

Since total_tokens is not used after the following line, please consider this minor suggested change:

Suggested change

	total_tokens += correct + 1
	sim_shared_dict["total_tokens"] = total_tokens
	sim_shared_dict["total_tokens"] = total_tokens + correct + 1

[mosheber]

done

[keyboardAnt]

Does it mean we generate max_tokens+1 new tokens? Why not >=?

Suggested change

	if res_dict["total_tokens"] > args.max_tokens:
	if res_dict["total_tokens"] >= args.max_tokens:

[mosheber]

done

[keyboardAnt]

We call this line if the target accepts all the 'lookahead' draft tokens, right? Does setting res_dict["total_tokens"] += res_dict["correct"] + 1 mean we accept an additional token? I'm asking because we shouldn't accept an additional token unless it is the last token. We accept an additional token only if it is the last token (e.g., the 50th token where config.S == 50) or the target rejects at least one draft token (in the current iteration). Instead, we should terminate the iteration that speculates this additional token with probability 1 - acceptance_rate.

To conclude, there are two changes to the online simulation to boost the speedup of DSI:

1. Accept an extra token if the target rejects a draft or if the extra token is the last.
2. Simulate an immediate validation of the extra token by terminating the corresponding speculating iteration with probability 1 - acceptance_rate.

We can separate them into two PRs.

[mosheber]

done

[keyboardAnt]

@mosheber, please see my previous comments and let me know when the tests pass so I can do another iteration.

[keyboardAnt]

Please merge or rebase main and remove the skip marker in test_duration and test_num_of_fix_history (tests/integration/online/test_simul.py). To run these two tests serially: python ./scripts/test.py online -- -vvv