How to compute scores after USMLE evaluation?

[anonymoususerr1]

Hi, can you share the script to compute the scores after evaluating each model on USMLE data via eval_usmle.py?

[kbressem]

I used this notebook. JSON files were created with eval_usmle.py.

eval_usmle.ipynb.zip

[anonymoususerr1]

Using the notebook you shared, I always get nan as the scores for every model including your pretrained models such as medalpaca-lora-7b-8bit. Do you have any idea why this happens?

[kbressem]

Does the model fail on some scores and produces nan? I believe if one nan is in the list, numpy will return nan for all.