Benchmark Runner Error

[carina-ding]

Hi, Thank you for developing this amazing benchmark!

I created a virtual environment using anaconda and successfully installed crfm-helm (I followed the instructions from the official document https://crfm-helm.readthedocs.io/en/latest/installation/.

When I tried to run benchmarking with the example given in the document https://crfm-helm.readthedocs.io/en/latest/quick_start/, I followed exactly the instructions given. However, when I run "helm-run --conf-paths run_entries.conf --suite v1 --max-eval-instances 10", I got "NotADirectoryError" and "helm.benchmark.runner.RunnerError". I attached some screenshots of traceback below.

Could anyone help me with this issue?

Thank you very much!

[yifanmai]

Hi, it looks like you're trying to run helm-run on Windows, which is not currently supported. We're planning to add support in the next couple of months. Alternatively, you can run helm-run on Linux instead using WSL2.

[carina-ding]

Hi @yifanmai , thank you very much for your reply! I will try on Linux using WSL2.
Thank you!