Presentation
This package contains the information retrieval test collection from Task 3 (User-centred health information retrieval) of the CLEFeHealth Lab 2013. Task 3 aims at facilitating the evaluation of information retrieval techniques for the medical domain, specifically information retrieval to address questions patients may have when reading clinical reports.
The CLEF Initiative
The CLEF Initiative (Conference and Labs of the Evaluation Forum, formerly known as Cross-Language Evaluation Forum) is a self-organized body whose main mission is to promote research, innovation, and development of information access systems with an emphasis on multilingual and multimodal information with various levels of structure.
Through the organisation of system evaluation campaigns, the
CLEF Coordination has created a community of researchers
and developers studying the same problems and facilitated collaborative initiatives between groups with similar
interests. Since 2000, CLEF has established strong links, exchanging ideas and sharing results, with similar
cross-language evaluation initiatives in the US and Asia, working on other sets of languages.
The CLEF Initiative is structured in two main parts:
-
a series of Evaluation Labs, i.e. laboratories to conduct evaluation of information access systems and workshops to discuss and pilot innovative evaluation activities;
-
a peer-reviewed Conference on a broad range of issues, including:
-
investigation continuing the activities of the Evaluation Labs;
-
experiments using multilingual and multimodal data; in particular, but not only, data resulting from CLEF activities; research in evaluation methodologies and challenges.
-
Since 2010, CLEF has taken the form of an independent event, constituted by a peer-reviewed conference jointly organised with a set of evaluation labs.
Evaluation Approach
It is a standard TREC-style information retrieval (IR) task using:
(a) a 2012 crawl of approximately one million medical documents made available by the EU-FP7 Khresmoi project in plain text form,
(b) general public queries that individuals may realistically pose based on the content of their discharge summaries. Queries were generated from discharge summaries used in Tasks 1 and 2 of the CLEF eHealth Lab 2013. The goal of Task 3 is to retrieve the relevant documents for the user queries.
Content of this Test Suite
In this package you will find:
- All datasets used for the Task
- The queries used for these evaluations
- The corresponding relevance assessments
- The guidelines provided to the participants
- The official results obtained by the participants
- The working notes of the CLEF campaigns
How to use this Package
Complete guidelines on how to perform your own system evaluation are supplied.
- Learn about the CLEF eHealth Lab and the original program : tasks, languages, topics…
- Read about the document data collections
- Read how to perform your own evaluation experiments
Required acknowledgment
Any paper describing experiments using the CLEFeHealth Task 3 Evaluation Package should cite the Overview paper of Task 3 (reference in BibTex format):
ShARe/CLEF eHealth Evaluation Lab 2013, Task 3: Information Retrieval to Address Patients'
Questions when Reading Clinical Reports
Lorraine Goeuriot, Gareth J.F. Jones, Liadh Kelly, Johannes Leveling, Allan Hanbury, Henning Müller, Sanna
Salanterä, Hanna Suominen and Guido Zuccon.
CLEF online working notes, (2013).
Disclaimer
In the Package documentation we provide html links to external Internet resources, like websites. Please note that these links may change over time.