This package contains the information retrieval test collection from Task 3 (User-centred health information retrieval) of the CLEFeHealth Lab 2013. Task 3 aims at facilitating the evaluation of information retrieval techniques for the medical domain, specifically information retrieval to address questions patients may have when reading clinical reports.

The CLEF Initiative

The CLEF Initiative (Conference and Labs of the Evaluation Forum, formerly known as Cross-Language Evaluation Forum) is a self-organized body whose main mission is to promote research, innovation, and development of information access systems with an emphasis on multilingual and multimodal information with various levels of structure.

Through the organisation of system evaluation campaigns, the CLEF Coordination has created a community of researchers and developers studying the same problems and facilitated collaborative initiatives between groups with similar interests. Since 2000, CLEF has established strong links, exchanging ideas and sharing results, with similar cross-language evaluation initiatives in the US and Asia, working on other sets of languages.

The CLEF Initiative is structured in two main parts:

  1. a series of Evaluation Labs, i.e. laboratories to conduct evaluation of information access systems and workshops to discuss and pilot innovative evaluation activities;

  2. a peer-reviewed Conference on a broad range of issues, including:

    • investigation continuing the activities of the Evaluation Labs;

    • experiments using multilingual and multimodal data; in particular, but not only, data resulting from CLEF activities; research in evaluation methodologies and challenges.

Since 2010, CLEF has taken the form of an independent event, constituted by a peer-reviewed conference jointly organised with a set of evaluation labs.

Evaluation Approach

It is a standard TREC-style information retrieval (IR) task using:

(a) a 2012 crawl of approximately one million medical documents made available by the EU-FP7 Khresmoi project in plain text form,

(b) general public queries that individuals may realistically pose based on the content of their discharge summaries. Queries were generated from discharge summaries used in Tasks 1 and 2 of the CLEF eHealth Lab 2013. The goal of Task 3 is to retrieve the relevant documents for the user queries.

Content of this Test Suite

In this package you will find:

How to use this Package

Complete guidelines on how to perform your own system evaluation are supplied.

Required acknowledgment

Any paper describing experiments using the CLEFeHealth Task 3 Evaluation Package should cite the Overview paper of Task 3 (reference in BibTex format):

ShARe/CLEF eHealth Evaluation Lab 2013, Task 3: Information Retrieval to Address Patients' Questions when Reading Clinical Reports
Lorraine Goeuriot, Gareth J.F. Jones, Liadh Kelly, Johannes Leveling, Allan Hanbury, Henning Müller, Sanna Salanterä, Hanna Suominen and Guido Zuccon. CLEF online working notes, (2013).


In the Package documentation we provide html links to external Internet resources, like websites. Please note that these links may change over time.