University of Washington: Linguistics: Ling 573: Spring 2011: Deliverable #3

Ling 573 - Natural Language Processing Systems and Applications
Spring 2011
Deliverable #3: Passage Retrieval: Due May 10, 2011: 09:00

Goals

In this deliverable, you will continue development of your question-answering system. You will

Develop the intermediate stage of a question-answering system by performing passage retrieval.
Implement a passage retrieval component, based on techniques discussed in class and in the readings.
Perform question processing, such as query expansion to aid passage retrieval and passage ranking.
Explore passage ranking to improve retrieval results.
Identify the resources - software and corpus - that can support this task.

Passage Retrieval

For this deliverable, you will need to implement a passage retrieval system. You may build on techniques presented in class, described in the reading list, and proposed in other research articles.

You system must include indexing and retrieval based on a standard IR engine, such as those described in the resource list. In addition, you system may exploit:

Query expansion, either semantically-based or corpus-based
Passage selection, such as sentence-based retrieval, fixed window-based retrieval, or some other segmentation approach
Passage reranking, either heuristic, rule-based, or machine-learning based

Data

Document Collection

The Aquaint Corpus was employed as the document collection for the question-answering task for a number of years, and will form the basis of retrieval for this deliverable . The collection can be found on patas in /corpora/LDC/LDC02T31/.

Training Data and Development Test Data

You may use any of the pre-2005 TREC question collections for training retrieval. For 2003, 2004 there are prepared gold standard documents and answer patterns to allow you to train and tune your passage retrieval system.

All pattern files appear in /dropbox/10-11/573/Data/patterns.

All question files appear in /dropbox/10-11/573/Data/Questions.

Test Data

You should evaluate on the TREC-2005 questions and their corresponding documents and answer string patterns. You are only required to test on the factoid questions.
NOTE:Please do NOT tune on these questions.

Evaluation

You will compute two sets of evaluation measures.

Document Ranking

Standard information retrieval measures, including mean average precision, using /dropbox/10-11/573/code/trec_eval. The standard command is:
trec_eval -q -c -M1000 official_qrels IR.outputs,
where the official_qrels files are the judgment files corresponding to the questions (in /dropbox/10-11/573/Data/patterns/qrels*.txt) and IR.outputs your ranked results file. The -q switch prints per-question results; omitting it produces only overall averaged results.
These scores should be placed in a file called IR.results in the results directory.
NOTE: Some questions (e.g 2004: 1.1, 1.2) do not have ANY relevant documents. If that is the case, they may not appear in the corresponding qrels file and will not affect the score produced by trec_eval.
A detailed explanation of trec_eval appears here

Passage Retrieval

MRR (Mean Reciprocal Rank) of results from your basic system. These scores should be placed a file called PR.results in the results directory.

python2.6 compute_mrr.py pattern_file PR.outputs {type}

pattern_file is the factoid Litowski pattern file,
PR.outputs is your passage retrieval output file, and
type is "strict" or "lenient". If you omit the type, it will default to strict

Outputs

Create two output files in the outputs directory, based on running your passage retrieval system on the test data file. You should do this in two phases:

Document Retrieval and Ranking

You should produce a ranked list of documents for each factoid question in the standard information retrieval style. The required format for the document ranking evaluation appears here. The file should be named IR.outputs and should appear in the outputs directory.

Passage Retrieval and Ranking

You should return the top 20 passages, where a passage is no longer than 1000 characters in length. The required format for the passage retrieval phase appears here. The file should be named PR.outputs and should appear in the outputs directory.

Extending the project report

This extended version should include all the sections from the original report (with many still as stubs) and additionally include the following new material:

Approach

Passage Retrieval: this subsection describes your passage retrieval approach so far.

Evaluation

Component Evaluation: this subsection will describe your passage retrieval evaluation.

Please name your report D3.pdf.

Presentation

Your presentation may be prepared in any computer-projectable format, including HTML, PDF, PPT, and Word. Your presentation should take about 10 minutes to cover your main content, including:

Passage Retrieval
Issues and successes
Related reading which influenced your approach

Your presentation should be deposited in your doc directory, but it is not due until the actual presentation time. You may continue working on it after the main deliverable is due.

Summary

Finish coding and document all code.
Verify that all code runs effectively on patas using Condor.
Add any specific execution or other notes to a README.
Create your D3.pdf and add it to the doc directory.
Verify that all components have been added and any changes checked in.

Ling 573 - Natural Language Processing Systems and Applications Spring 2011 Deliverable #3: Passage Retrieval: Due May 10, 2011: 09:00