Update schema for instruction-following #2292
Conversation
| @@ -0,0 +1,238 @@ | |||
| --- | |||
| ############################################################ | |||
| models: | |||
There was a problem hiding this comment.
I thought we are reading information out of the metadata yaml now...do we still need this?
| @@ -2232,6 +2232,107 @@ run_groups: | |||
| when: n/a | |||
| language: synthetic | |||
|
|
|||
| ## Instruction following scenarios | |||
There was a problem hiding this comment.
Why are these not in schema_instruction_following.yaml?
There was a problem hiding this comment.
Reverted schema_classic.yaml.
|
@percyliang This is ready for review again. |
| @@ -0,0 +1,210 @@ | |||
| --- | |||
| ############################################################ | |||
| adapter: | |||
There was a problem hiding this comment.
We should at some point figure out how to avoid this duplication.
There was a problem hiding this comment.
Acknowledged. I have some idea of how to do this.
| # Ideally, the name should be "best_chatgpt_prompts". | ||
| # But unfortunately the group name in the results is "grammar", | ||
| # so the schema has to match the same group name. | ||
| # TODO: Change the group name in the "grammar" run spec, and then change this group name. |
There was a problem hiding this comment.
Would changing this require regenerating the benchmark output files? If so, do we want to make this change before deploying?
There was a problem hiding this comment.
The changes are non-trivial so I would probably have to spend a day if we want to change "grammar" to "best_chatgpt_prompts". We can't re-run helm-run easily, so we would have to edit the JSON files post-hoc.
In any case, I would prefer to do it in a different pull request.
No description provided.