Welcome to CLEF 2005 - Consiglio Nazionale delle Ricerche

Welcome to CLEF 2007
Carol Peters
ISTI-CNR Pisa, Italy
CLEF Objectives
 Stimulate the development of multilingual IR systems
for European languages
 To create a CLIR/MLIA community
 Construct publicly available test-suites
 Conducting annual evaluation campaigns
 Designing tracks/tasks to meet emerging needs and
to stimulate research in the”right” direction
CLEF 2007 Workshop, Budapest, Hungary
19-21 September 2007
CLEF Coordination
CLEF is coordinated by the Istituto di Scienza e Tecnologie dell'Informazione, Consiglio Nazionale delle Ricerche, Pisa
The following Institutions are contributing to the organisation of the different tracks of the CLEF 2007campaign:















Inst. For Information technology, Hyderabad, India
Centre for the Evaluation of Human Language and
Multimodal Communication Technologies (CELCT),

Inst. of Formal and Applied Linguistics, Charles
Trento, Italy
University, Czech Rep
College of Information Studies and Institute for

LSI-UNED, Madrid, Spain
Advanced Computer Studies, U. Maryland, USA

Linguateca, Sintef, Oslo, Norway
Dept. of Computer Science, U. Indonesia

Linguistic Modelling Lab., Bulgarian Acad Sci
Depts. of Computer Science & Medical Informatics,

Microsoft Research Asia
RWTH Aachen U., Germany

NIST, USA
Dept. of Computer Science and Information Systems,

Biomedial Informatics, Oregon Health and Science
U. Limerick, Ireland
University, USA
Dept. of Computer Science and Information

Research Computing Center of Moscow State U.
Engineering, National U. Taiwan

Research Institute for Linguistics, Hungarian
Dept. of Information Engineering, U. Padua, Italy
Academy of Sciences
Dept. of Information Sci, U. Hildesheim, Germany

School of Computer Science and Mathematics,
Dept. of Information Studies, U. Sheffield, UK
Victoria U., Australia
Evaluations and Language Resources Distribution

School of Computing, DCU, Ireland
Agency Sarl, Paris, France

UC Data Archive and School of Information
Fondazione Bruno Kessler FBK-irst, Trento, Italy
Management and Systems, UC Berkeley, USA
German Research Centre for Artificial Intelligence,

University "Alexandru Ioan Cuza", IASI, Romania
DFKI, Saarbrücken, Germany

U. Hospitals and U.of Geneva, Switzerland
Information and Language Processing Systems, U.

Vienna University of Technology, Austria
Amsterdam, Netherlands
IZ Bonn, Germany
CLEF 2007 Workshop, Budapest, Hungary
19-21 September 2007
CLEF
Steering Committee




















Maristella Agosti, U.Padove, Italy
Martin Braschler, Zurich, Switzerland
Amedeo Cappelli, ISTI-CNR & CELCT, Italy
Hsin-Hsi Chen, National Taiwan U., Taipei, Taiwan
Khalid Choukri, ELRA/ELDA, Paris, France
Paul Clough, University of Sheffield, UK
Thomas Deselaers, RWTH Aachen University, Germany
Giorgio Di Nunzio, U. Padova, Italy
David A. Evans, Clairvoyance Corporation, USA
Nicola Ferro, U. Padova, Italy
Christian Fluhr, CEA-LIST, Fontenay-aux-Roses, France
Norbert Fuhr, University of Duisburg, Germany
Frederic C. Gey, U.C. Berkeley, USA
Julio Gonzalo, LSI-UNED, Madrid, Spain
Donna Harman, NIST, USA
Gareth Jones, Dublin City University, Ireland
Franciska de Jong, University of Twente, Netherlands
Noriko Kando, NII, Tokyo, Japan
Jussi Karlgren, SICS, Sweden
Michael Kluck, German Institute for International and
Security Affairs, Berlin, Germany


















Natalia Loukachevitch, Moscow State University, Russia
Bernardo Magnini, ITC-irst, Trento, Italy
Thomas Mandl, U. Hildesheim, Germany
Paul McNamee, Johns Hopkins University, USA
Henning Müller, University & University Hospitals of
Geneva, Switzerland
Douglas W. Oard, University of Maryland, USA
Anselmo Peňas, LSI-UNED, Madrid, Spain
Maarten de Rijke, University of Amsterdam, Netherlands
Diana Santos, Linguateca, Sintef, Oslo, Norway
Jacques Savoy, University of Neuchatel, Switzerland
Peter Schäuble, Eurospider Information Technologies,
Switzerland
Richard Sutcliffe, University of Limerick, Ireland
Max Stempfhuber, Informationszentrum
Sozialwissenschaften Bonn, Germany
Hans Uszkoreit, German Research Center for Artificial
Intelligence (DFKI), Germany
Felisa Verdejo, LSI-UNED, Madrid, Spain
José Luis Vicedo, University of Alicante, Spain
Ellen Voorhees, NIST, USA
Christa Womser-Hacker, University of Hildesheim, Germany
CLEF 2007 Workshop, Budapest, Hungary
19-21 September 2007
CLEF 2007:
Track Coordinators
 Ad Hoc: Giorgio Di Nunzio, Nicola Ferro and Thomas Mandl
 Domain-Specific: Vivien Petras, Stefan Baerisch, Maximillian
Stempfhuber
 QA@CLEF: Danilo Giampiccolo, Bernardo Magnini, Anselmo
Peñas, Christelle Ayache, Petya Osenova,, Maarten de Rijke,
Bogdan Sacaleanu, Diana Santos and Richard Sutcliffe
 ImageCLEF: Allan Hanbury, Paul Clough, Henning Müller,
Thomas Deselaers , Michael Grubinger, Jayashree Kalpathy–
Cramer, and William Hersh
 CL-SR: Douglas W. Oard, Gareth J. F. Jones, and Pavel Pecina
 Web-CLEF: Valentin Jijkoun and Maarten de Rijke
 GeoCLEF: Thomas Mandl, Fredric Gey, Giorgio Di Nunzio, Nicola
Ferro, Ray Larson, Mark Sanderson, Diana Santos, Christa
Womser-Hacker, Xing Xie
CLEF 2007 Workshop, Budapest, Hungary
19-21 September 2007
CLEF 2007:
Participating Groups





















Brown U., USA
California State U. SanMarcos, USA**
Charles U., Prague, Czech Rep.
Daedalus & Madrid Univs, Spain ****
Ching Yun Univ., Taiwan
DFKI-Artificial Intelligence, DE****
Dokuz Eylul U.,Turkey*
Dublin City U. - Comp.Sci., Ireland ***
Fondazione Bruno Kessler********
Helsinki U. of Technology
Hungarian Acad. Sci.
IDIAP Research Inst., CH
Imperial College, London, UK**
Ist.Nac.Astrofisica, Optica, Electronica,
Mexico**
Indian Statistical Inst., India*
Indian Inst. Technology (IIT-Bombay)
Indian Inst. Technology (IITKharagpur)
Inst.Infocomm Research, Singapore **
Inst. Superior Técníco (DEI-IST)
IPAL-CNRS (IR2), Singapore ****
IRIT / SIG - Toulouse *****
Jadavpur University, Kolkata, India
Johns Hopkins U., USA *******
Language Computer Corp., USA*

LIMSI-CNRS, France ****



























Linguateca-Sintef, Norway ***
Linguit Ltd, UK
Microsoft Asia*
Microsoft India
MRIM Group – LIG, Grenoble*
Nat. Inst.Informatics, Japan ***
Nat.Taiwan U. - Comp-Sci, *****
Open Text Corp.(ex Hummingbird)
Oregon Health & Sci. U., USA **
Priberam Informatica, Portugal *
Research Inst. for AI of Romaian
Academy*
RWTH Aaachen-CS., Germany ***
RWTH Aachen - Med.Inf., DE***
SUNY Buffalo – Informat, USA ****
SYNAPSE Développement, France**
Tech U. Chemnitz, Germany*
Tokyo Inst. Technology, Japan*
U.Alicante, Spain (2) ******
U.AI.I Cuza Iasi, Romania*
U.Amsterdam - Informatics, N ******
U. Basil, Seitzerland
U. Chicago, USA **
U.Concordia - CINDI, Canada**
U.Concordia - CLAK, Canada
U.Coruna & U.Sunderland, ES/UK*





Univ. Evora, Portugal **
U.Freiburg – Pattern Recog., Germany U. & Hospitals
Geneva, CH ***
U.Groningen - Inf.Sci, The Netherlands** (2)
U.Hagen – IICS, Germany ****
U.Hildesheim - Inf.Sci, Germany *** *





















U.Indonesia - Comp.Sci, Indonesia **
U.Jaen - Intell.Systems, Spain ******
U.Liege - Elect.Eng.&CS, Belgium**
U.Lisbon – Informatics, Portugal ***
Univ. Macquarie, Australia
Univ. Nacional Colombia
U.Neuchatel – Informatique, Switzerland ******
Univ. Nottingham, UK
U.Ottawa - IT & Eng, Canada*
U.Politecnica Catalunya – TALP, Spain**
U.Politecnica Valencia - Comp.Sci, Spain**
U. Porto, Portugal*
U.Salamanca – REINA, Spain *****
U.Stockholm, NLP, Sweden ***
U.Tampere, Fiinland ****
U.Wolverhampton, UK *
UC Berkeley - IM&S-1, USA *******
UNED-LSI, Spain ******
Univ. West Bohemia, Czech rRp.*
Vienna Univ. Technology, Austria
Xerox XRCE, France *
CLEF 2007 Workshop, Budapest, Hungary
19-21 September 2007
CLEF: Trend in
Participation
CLEF 2000-2007 Participation
100
90
80
70
60
50
40
30
20
10
0
2000
Oceania
South America
North America
Asia
Europe
2001
2002
2003
2004
2005
2006
2007
Europe = 51(59.5); N. America = 14(4.5); Asia = 14(10), S. America = 1(4), Oceania = 1(2)
CLEF 2007 Workshop, Budapest, Hungary
19-21 September 2007
CLEF 2007 Tracks
 multilingual textual document retrieval on news
collections (Ad Hoc)
 mono- and cross-language information on structured
scientific data (Domain-Specific)
 multiple language question answering (QA@CLEF)
 cross-language retrieval in image collections
(ImageCLEF)
 cross-language spoken document retrieval (CL-SR)
 multilingual retrieval of Web documents (WebCLEF)
 cross-language geographical retrieval (GeoCLEF)
Plus: CLEF@SemEval and CLEF@MorphoChallenge
CLEF 2007 Workshop, Budapest, Hungary
19-21 September 2007
No. of Participants
per Track




Ad Hoc: 22(25)
Domain-Spec: 5(4)
iCLEF: 0(3)
QA@CLEF: 28(37)




ImageCLEF: 35(25)
CL-SR: 8(6)
WebCLEF: 4(8)
GeoCLEF: 13(17)
CLEF 2007 Workshop, Budapest, Hungary
19-21 September 2007
CLEF 2000 – 2007
Tracks
CLEF 2000-2007 Tracks
AdHoc
40
Participating Groups
35
DomSpec
30
iCLEF
25
CL-SR
20
15
QA@CLEF
10
ImageCLEF
5
WebClef
0
2000
2001
2002
2003
2004
2005
2006
Years
CLEF 2007 Workshop, Budapest, Hungary
19-21 September 2007
2007
GeoClef
CLEF 2007:
Test Collections
2000
 News documents in 4 languages
 GIRT German Social science database
2007
 CLEF multilingual comparable corpus of more than 3M news docs in 13
languages: CZ,DE,EN,ES,FI,FR,IT,NL,RU,SV,PT,BG and HU
 GIRT-4 social science database in EN and DE, Russian ISISS collection;
Cambridge Sociological Abstracts
 Malach collection of conversational speech derived from the Shoah
archives EN & CZ
 EuroGOV, a multilingual collection of approx 3M webpages crawled from
European governmental sites
 IAPR TC-12 photo database; PASCAL VOC 2006 training data
 ImageCLEFmed radiological database consisting of 6 distinct datasets;
 IRMA collection in EN and DE for automatic medical image
annotation:10,000 images
CLEF 2007 Workshop, Budapest, Hungary
19-21 September 2007
CLEF 2007: Highlights
 Slight fall in participation 81 groups in 2007 (90 in 2006); workshop
>115 Participants (130 in 2006)
 Expansion of test-suites
 Ad Hoc – mixed results – but good success of the non-European topic
languages task
 Domain-specific holds its own!
 Enormous success of ImageCLEF
 Confirmation of interest in QA@CLEF, GeoCLEF and CL-SR
 iCLEF -<didn’t happen
 WebCLEF – what happened???
 CLEF 2006 Proceedings ???
CLEF 2007 Workshop, Budapest, Hungary
19-21 September 2007
CLEF 2007: Highlights
 Slight fall in participation 81 groups in 2007 (90 in 2006); workshop
>115 Participants (130 in 2006)
 Expansion of test-suites
 Ad Hoc – mixed results – but good success of the non-European topic
languages task
 Domain-specific holds its own!
 Enormous success of ImageCLEF
 Confirmation of interest in QA@CLEF, GeoCLEF and CL-SR
 iCLEF -<didn’t happen
 WebCLEF – what happened???
 CLEF 2006 Proceedings – DID HAPPEN – A Miracle?
CLEF 2007 Workshop, Budapest, Hungary
19-21 September 2007
CLEF 2006 Proceedings
Evaluation of Multilingual and Multi-modal
Information Retrieval
7th Workshop of the Cross-Language Evaluation
Forum, CLEF 2006, Alicante, Spain, September,
2006, Revised Selected Papers
Lecture Notes in Computer Science, Vol. 4730
Peters, C.; Clough, P., Gey, F.C.; Karlgren, J.;
Magnini, B.; Oard, D.W.; de Rijke, M.: Stempfhuber
(Eds.) 2006
CLEF 2007 Workshop, Budapest, Hungary
19-21 September 2007
2006: Points for Discussion
 What new tasks/evaluation methodologies are
needed to address more advanced information
requirements?
 How can we best reduce the gap between research
and application communities?
 Who are the users?
Does CLEF have a future?
The challenge represented by i2010
CLEF 2007 Workshop, Budapest, Hungary
19-21 September 2007
Treble-CLEF
The CLEF research results have led to development of a
new generation of multilingual retrieval system prototypes
BUT lack of technology transfer
Treble-CLEF will extend the CLEF activity by:




continuing to promote MLIA R&D via evaluation campaigns;
providing a consistent training activity: tutorials, workshops,
summer school;
producing best practice guidelines for system implementation;
providing resources to encourage the multilingual system
development.
Treble-CLEF will begin activity with a brainstorming
workshop in January 2008
CLEF 2007 Workshop, Budapest, Hungary
19-21 September 2007