Add MultipanelVQA and POPE vision-language scenarios

[ImKeTT]

Hello, this PR is to add two vision-language scenarios to VHELM --- MultipanelVQA from https://arxiv.org/abs/2401.15847 and the POPE benchmark from https://aclanthology.org/2023.emnlp-main.20/.

There are two subjects (synthetic or real-world) and two question_type (multiple-choice or open) for MultipanelVQA, I use get_short_answer_generation_adapter_spec for open-ended generation and get_multiple_choice_joint_adapter_spec for multiple-choice type questions. For both scenarios, I use get_exact_match_metric_specs for evaluation.

Here's a screenshot after running ./pre-commit.sh

Here're several screenshots and the scenario_state.json of toy runs on two scenarios (Qwen-VL-Chat on 25 instances):
POPE

pope_scenario_state.json

MultipanelVQA-real-world

mpvqa-real-open-scenario_state.json
mpvqa-real-mc-scenario_state.json

MultipanelVQA-synthetic

mpvqa-syn-open-scenario_state.json
mpvqa-syn-mc-scenario_state.json

Please let me know how I can improve it.
Thanks!

[teetone]

@ImKeTT Thanks for adding these! I had a few minor comments. Could you also add the conf file you used to run in the PR description?

[ImKeTT]

Thanks for reviewing @teetone ! I've re-framed POPE to the MCQA task and added more detailed descriptions for these two scenarios.

Here are the configuration files I used for this PR.
For MultipanelVQA

entries: [
    {description: "multipanelvqa:subject=synthetic,question_type=multiple-choice,model=qwen/qwen-vl-chat", priority: 1}
    {description: "multipanelvqa:subject=synthetic,question_type=open,model=qwen/qwen-vl-chat", priority: 1}
    {description: "multipanelvqa:subject=real-world,question_type=multiple-choice,model=qwen/qwen-vl-chat", priority: 1}
    {description: "multipanelvqa:subject=real-world,question_type=open,model=qwen/qwen-vl-chat", priority: 1}
    ]

For POPE

entries: [
    {description: "pope:model=qwen/qwen-vl-chat", priority: 1}
    ]

[teetone]

Thanks for reviewing @teetone ! I've re-framed POPE to the MCQA task and added more detailed descriptions for these two scenarios.

Here are the configuration files I used for this PR. For MultipanelVQA

entries: [
    {description: "multipanelvqa:subject=synthetic,question_type=multiple-choice,model=qwen/qwen-vl-chat", priority: 1}
    {description: "multipanelvqa:subject=synthetic,question_type=open,model=qwen/qwen-vl-chat", priority: 1}
    {description: "multipanelvqa:subject=real-world,question_type=multiple-choice,model=qwen/qwen-vl-chat", priority: 1}
    {description: "multipanelvqa:subject=real-world,question_type=open,model=qwen/qwen-vl-chat", priority: 1}
    ]

For POPE

entries: [
    {description: "pope:model=qwen/qwen-vl-chat", priority: 1}
    ]

Thanks @ImKeTT! could you address one last comment in schema_vlm.yaml?

[ImKeTT]

Thanks @ImKeTT! could you address one last comment in schema_vlm.yaml?

Sure, I think it's ready to go now, thanks @teetone !