Book Reviews

TREC: Experiment and Evaluation in Information Retrieval

TREC: Experiment and Evaluation in Information Retrieval. Ellen M. Voorhees and Donna K. Harman, editors. Cambridge, Massachusetts: MIT Press, 2005. ISBN 0-262-22073-3. $45.00

The Text Retrieval Conference (TREC) is an annual workshop aimed at building the infrastructure required for extracting relevant information from large volumes of electronic documents. The first TREC conference was held in 1992 and research into improved text-retrieval methodologies has continued ever since.

TREC: Experiment and Evaluation in Information Retrieval chronicles the outcomes of the past TREC conferences, summarizing the research results and best practices developed. The book also presents some of the lessons learned and offers suggestions for research still needed. While neither a "best of" sampling of past conference proceedings nor a guide to the literature of text-retrieval research, the book chronicles the evolution of text-retrieval systems and text-retrieval research over the duration of the TREC conferences.

There are three parts to TREC: Experiment and Evaluation in Information Retrieval. Part one describes TREC's origins and history, its mission and purpose, the major products of TREC -- i.e., the test collections -- and the retrieval evaluation methodology. The chapters of part two summarize seven of TREC's "tracks" or subtopics such as retrieval of streaming text, cross-language text retrieval, web searching and question answering. Each chapter gives a brief history of the track, the state of the research prior to the first TREC conference, what was learned at each TREC conference and conclusions based on the research findings to date. The third part summarizes research from seven participating research groups -- four of which have been participating since the original TREC conference, and three who have used TREC results to validate their own approaches to text retrieval. The participating groups include universities such as the University of Massachusetts at Amherst, Cornell University and the University of Waterloo, as well as companies such as IBM. Finally, an epilogue reflects on TREC's overall contributions to fields such as information retrieval itself, its evaluation and web searching. An index is provided along with numerous references, many of which are to past TREC proceedings, as would be expected.

Both editors, Ellen M. Voorhees and Donna K. Harman, are affiliated with the National Institute of Standards and Technology (NIST), the host of the TREC conference. Both have written extensively on the subject of text retrieval, and they have been involved with TREC for several years.

The book was not written as a replacement for the TREC conference proceedings themselves. As such, it does not delve into the level of detail that would be found in the actual proceedings. The book is useful, however, as an overview and a starting point for finding TREC conference literature.

The book is written in such a way that it lends itself to many uses: as an overview of past TREC research, a "state of the technology" report, a history of text-retrieval development and a retrospective look at the work of TREC and its impact. As such, it would be a useful addition in academic or research libraries engaging in the development of text-retrieval systems.