Ranking Output Format

This page describes a XML format to save the intermediate answer choice ranking results. It will be used as the submission format for the NTCIR-11 QA-Lab Phase 1 Center Exam Combination Runs.

Tag	Description
TOPIC_SET	root element that contains a meta data and a list of topics
METADATA	include meta information on run id and system description
TOPIC	each TOPIC is associated with one question in the center exam, and must specify the question's anscolumn_ID as its ID
CHOICE	each CHOICE is associated with one choice of current question, and must specify the choice number as its ID
RANK	current answer choice's ranking number
SCORE	the confidence score of selecting current answer choice

DTD

<!ELEMENT TOPIC_SET (METADATA,TOPIC*)>
<!ELEMENT METADATA (RUNID?,DESCRIPTION?)>
<!ELEMENT RUNID (#PCDATA)>
<!ELEMENT DESCRIPTION (#PCDATA)>
<!ELEMENT TOPIC (CHOICE)*>
<!ATTLIST TOPIC
        ID CDATA #REQUIRED>
<!ELEMENT CHOICE (#PCDATA)>
<!ATTLIST CHOICE
        ID CDATA #REQUIRED
        RANK CDATA #REQUIRED
        SCORE CDATA #REQUIRED>

Example XML

<TOPIC_SET>
    <METADATA>
        <RUNID>Center-2009--Main-WorldHistoryB_TEAM1_EN_QA_01</RUNID>
        <DESCRIPTION>
            Our team uses Wikipedia retrieval
            scores to rank answer choice
        </DESCRIPTION>
    </METADATA>

    <TOPIC ID="A3">
        <CHOICE ID="C1" SCORE="0.95" RANK="1" />
        <CHOICE ID="C2" SCORE="0.65" RANK="3" />
        <CHOICE ID="C3" SCORE="0.85" RANK="2" />
        <CHOICE ID="C4" SCORE="0.15" RANK="4" />
    </TOPIC>
</TOPIC_SET>