# Submissions

version 1.0 (August 27th, 2015)

## Run Types

In the Query Understanding subtask, we accept two types of runs.

• Q-Run: Runs for the regular Query Understanding subtask; systems are required to identify both subtopics and relevant verticals for given topics.
• S-Run: Optional runs designed for those who wants to focus on the Subtopic mining. Systems are required to identify subtopics, but not verticals.

## Run Names

Run files for the Query Understanding subtask should be named as follows. Make your [GroupID] is exactly what you registered in NTCIR-12.

[GroupID]-Q-[JCE]-[priority][QS].tsv

e.g., KYOTO-Q-E-1Q.tsv
(E means English – use J for Japanese, C for Chinese. 1 means this run has the highest priority to be included in the pool. Due to limited resources, we may not include all submitted runs in the result pool. Q means Q-run – use S for S-run.)

For example, run files for the Japanese Query Understanding subtasks should like this:

KYOTO-Q-J-1Q.tsv
KYOTO-Q-J-2Q.tsv
KYOTO-Q-J-3Q.tsv
KYOTO-Q-J-4Q.tsv


## Number of Runs

For each language, a participating team can submit up to five runs.
For example, if a team is participating both English and Chinese QU subtasks, they can submit up to 10 runs (five runs for English and 5 runs for Chinese.)

## Run Submission Format

We accept a tab-delimited-values (TSV) file as a run,
where the first line must be a short description of your system.
The rest of the file should contain lines of the form:

[QueryID][TAB][Subtopic][TAB][Vertical][TAB][Score][TAB][RunName]\n


For example, an English Query Understanding run should look like this:

This is a sample English Query Understanding run.
IMINE2-E-001 cvs stores Web 0.9 KYOTO-Q-E-1Q
IMINE2-E-001 concurrent versions system Encyclopedia 0.7 KYOTO-Q-E-1Q
IMINE2-E-001 cvs coupons Web 0.681 KYOTO-Q-E-1Q
IMINE2-E-002 Bumblebee Pictures Image 0.9 KYOTO-Q-E-1Q
...


As for the available verticals for each language, please check the task guideline page.

As for S-runs, [Vertical] should be empty; thus the line should look like this:

IMINE2-E-001[TAB]cvs stores[TAB][TAB]0.9[TAB]KYOTO-Q-E-1S
...


Return no more than 10 subtopics per topic. The run file should be saved as a UTF-8 encoded file. It’s okay if your ranked lists are empty for some topics. Note that we do not use [Score] values for our evaluation and use only the order of subtopics in the evaluation; the ranks of the subtopics are determined just by their appearance orders in the submission file.

## Run Types

In the Vertical Incorporating subtask, we accept two types of runs.

• M-Run: Mandatory runs which use the document corpus provided by IMine-2 (IMine-2 Web Corpus).
• O-Run: Optional runs which use SogoutT (for Chinese runs) or ClueWeb12-B13 (for English runs)

Please note that those who participate in the Vertical Incorporating subtask is required to submit at least one M-run.

## Run Names

Run files for the Vertical Incorporating subtask should be named as follows. Make your [GroupID] is exactly what you registered.

[GroupID]-V-[CE]-[priority][MO].tsv

e.g., KYOTO-V-E-1M.tsv (E means English – use C for Chinese; 1 means this run has the highest priority to be included in the pool; M means M-run – use O for O-run.)

For example, run files for English Vertical Incorporating subtasks should like this:

KYOTO-V-E-1M.tsv
KYOTO-V-E-2M.tsv
KYOTO-V-E-3O.tsv
KYOTO-V-E-4O.tsv


## Number of Runs

For each language, a participating team can submit up to five runs, which include at least one M-run.

## Run Submission Format

Similar to the Query Understanding subtask, we accept a tab-delimited-values (TSV) file as a Vertical Incorporating run,
where the first line must be a short description of your system.
The rest of the file should contain lines of the form:

[QueryID][TAB][DocumentID][TAB][Rank][TAB][Score][TAB][RunName]\n


For example, an English Vertical Incorporating M-run should look like this:

This is a sample English VI run.
IMINE2-E-001 IMINE2-E-001-021.html 1 0.84 KYOTO-V-E-1M
IMINE2-E-001 Vertical-Image 2 0.80 KYOTO-V-E-1M
IMINE2-E-001 Vertical-QA 3 0.70 KYOTO-V-E-1M
IMINE2-E-002 IMINE2-E-002-103.html 1 0.90 KYOTO-V-E-1M
...


As for the special virtual documents for each language, please check the task guideline page.
Return no more than 100 documents per topic. The run file should be saved as a UTF-8 encoded file. It’s okay if your ranked lists are empty for some topics. Note that we do not use [Score] nor [Rank] values for our evaluation and use only the order of documents in the evaluation; the ranks of the documents are determined just by their appearance orders in the submission file.

1. Please zip all your runs into a file named “[GroupID].zip” (e.g., KYOTO.zip).