Submissions

version 1.0 (August 27th, 2015)

Query Understanding Subtask

Run Types

In the Query Understanding subtask, we accept two types of runs.

  • Q-Run: Runs for the regular Query Understanding subtask; systems are required to identify both subtopics and relevant verticals for given topics.
  • S-Run: Optional runs designed for those who wants to focus on the Subtopic mining. Systems are required to identify subtopics, but not verticals.

Run Names

Run files for the Query Understanding subtask should be named as follows. Make your [GroupID] is exactly what you registered in NTCIR-12.

[GroupID]-Q-[JCE]-[priority][QS].tsv

e.g., KYOTO-Q-E-1Q.tsv
(E means English – use J for Japanese, C for Chinese. 1 means this run has the highest priority to be included in the pool. Due to limited resources, we may not include all submitted runs in the result pool. Q means Q-run – use S for S-run.)

For example, run files for the Japanese Query Understanding subtasks should like this:

KYOTO-Q-J-1Q.tsv
KYOTO-Q-J-2Q.tsv
KYOTO-Q-J-3Q.tsv
KYOTO-Q-J-4Q.tsv

Number of Runs

For each language, a participating team can submit up to five runs.
For example, if a team is participating both English and Chinese QU subtasks, they can submit up to 10 runs (five runs for English and 5 runs for Chinese.)

Run Submission Format

We accept a tab-delimited-values (TSV) file as a run,
where the first line must be a short description of your system.
The rest of the file should contain lines of the form:

[QueryID][TAB][Subtopic][TAB][Vertical][TAB][Score][TAB][RunName]\n

For example, an English Query Understanding run should look like this:

This is a sample English Query Understanding run.
IMINE2-E-001 cvs stores Web 0.9 KYOTO-Q-E-1Q
IMINE2-E-001 concurrent versions system Encyclopedia 0.7 KYOTO-Q-E-1Q
IMINE2-E-001 cvs coupons Web 0.681 KYOTO-Q-E-1Q
IMINE2-E-002 Bumblebee Pictures Image 0.9 KYOTO-Q-E-1Q
...

As for the available verticals for each language, please check the task guideline page.

As for S-runs, [Vertical] should be empty; thus the line should look like this:

IMINE2-E-001[TAB]cvs stores[TAB][TAB]0.9[TAB]KYOTO-Q-E-1S
...

Return no more than 10 subtopics per topic. The run file should be saved as a UTF-8 encoded file. It’s okay if your ranked lists are empty for some topics. Note that we do not use [Score] values for our evaluation and use only the order of subtopics in the evaluation; the ranks of the subtopics are determined just by their appearance orders in the submission file.

Vertical Incorporating Subtask

Run Types

In the Vertical Incorporating subtask, we accept two types of runs.

  • M-Run: Mandatory runs which use the document corpus provided by IMine-2 (IMine-2 Web Corpus).
  • O-Run: Optional runs which use SogoutT (for Chinese runs) or ClueWeb12-B13 (for English runs)

Please note that those who participate in the Vertical Incorporating subtask is required to submit at least one M-run.

Run Names

Run files for the Vertical Incorporating subtask should be named as follows. Make your [GroupID] is exactly what you registered.

[GroupID]-V-[CE]-[priority][MO].tsv

e.g., KYOTO-V-E-1M.tsv (E means English – use C for Chinese; 1 means this run has the highest priority to be included in the pool; M means M-run – use O for O-run.)

For example, run files for English Vertical Incorporating subtasks should like this:

KYOTO-V-E-1M.tsv
KYOTO-V-E-2M.tsv
KYOTO-V-E-3O.tsv
KYOTO-V-E-4O.tsv

Number of Runs

For each language, a participating team can submit up to five runs, which include at least one M-run.

Run Submission Format

Similar to the Query Understanding subtask, we accept a tab-delimited-values (TSV) file as a Vertical Incorporating run,
where the first line must be a short description of your system.
The rest of the file should contain lines of the form:

[QueryID][TAB][DocumentID][TAB][Rank][TAB][Score][TAB][RunName]\n

For example, an English Vertical Incorporating M-run should look like this:

This is a sample English VI run.
IMINE2-E-001 IMINE2-E-001-021.html 1 0.84 KYOTO-V-E-1M
IMINE2-E-001 Vertical-Image 2 0.80 KYOTO-V-E-1M
IMINE2-E-001 Vertical-QA 3 0.70 KYOTO-V-E-1M
IMINE2-E-002 IMINE2-E-002-103.html 1 0.90 KYOTO-V-E-1M
...

As for the special virtual documents for each language, please check the task guideline page.
Return no more than 100 documents per topic. The run file should be saved as a UTF-8 encoded file. It’s okay if your ranked lists are empty for some topics. Note that we do not use [Score] nor [Rank] values for our evaluation and use only the order of documents in the evaluation; the ranks of the documents are determined just by their appearance orders in the submission file.

Uploading Your Runs

1. Please zip all your runs into a file named “[GroupID].zip” (e.g., KYOTO.zip).

2. Please visit here and upload the zipped archive.
If you are not able to access Dropbox, please send the zipped archive to tyamamot@dl.kuis.kyoto-u.ac.jp.

3. Once we receive your submission file, we will send the confirmation mail to the contact address of your group.