Logits warper for batch generation

[bnurbekov]

🚀 Feature request

Is there a way to modify logits_warper, so that it will apply different parameters (top_p, top_k, temperature) for each item in a batch instead of applying the same parameters for all items in the batch:

transformers/src/transformers/generation_utils.py

Line 1562 in f25a933

next_token_scores = logits_warper(input_ids, next_token_scores)

Motivation

It will very useful for inference because it will increase the throughput. Currently, you cannot really use #7552 for inference because the same parameters apply to the entire batch.

Your contribution

Can help with implementing this feature and reviewing the PR.

[patrickvonplaten]

Hey @bnurbekov,

Sorry for replying so late. We recently allowed to pass customized logits wrapper to generate(). Could you maybe try to build a custom wrapper for your purpose this way?

#14779

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.