Collaborative Information Retrieval: Concepts, Models and Evaluation
Traditional conceptualizations of an IR task generally rely on an individual user's perspective. Accordingly, a great amount of research in the IR domain mostly dealt with both the design of enhanced document ranking models and a deep user's behavior understanding with the aim of improving an individual search effectiveness. However, in practice, collaboration among a community of users is increasingly acknowledged as an effective mean for gathering the complementary skills and/or knowledge of individual users in order to solve complex shared search tasks, such as fact-finding tasks (e.g., travel planning) or exploratory search tasks. Collaboration allows the group achieving a result that is more effective than the simple aggregation of the individual results. This class of complex search settings frequently occurs within a wide-range of domain-applications, such as the medical domain, the legal domain or the librarian domain to cite but a few.
These last years and particularly since 2005, collaborative information retrieval has became an emerging topic, addressed in several IR and Information Seeking conferences including SIGIR, CIKM and ECIR. While a large amount of work have investigated collaborative search according to the behavioral or computer-supported approaches, the first generation of collaborative models focused on the search interactivity to adapt to document ranking to users' preferences or roles following two main collaboration paradigms (division of labor or sharing of knowledge). However, they are still remaining challenges in IR dealing with the session asynchronicity, multi-level collaborative search or the consideration of users-users interactions as well as crowd-search perspectives for collaborative work.
Collaborative Information Retrieval (CIR) results in collaborative information behavior processes, such as information sharing, evaluation, synthesis and sense-making. Two fundamental research challenges are faced by the design of CIR systems: 1) allowing effective communication and coordination among the collaborators and 2) achieving high synergic effectiveness of the search results. This tutorial will focus on the second challenge and pay a great deal of attention to how collaboration could be integrated in IR models and effectiveness evaluation processes. Our goal is to provide concepts and motivation to researchers so that participants could investigate this emerging IR domain as well as giving them some clues on how to experiment their models. More specifically, the tutorial aims to:
- Give an overview of the concepts underlying collaborative information behavior and retrieval;
- Present state-of-the art techniques and models that tackle the search effectiveness challenge;
- Synthesize the metrics used for the evaluation of the effectiveness of CIR systems.
With respect with the above-mentioned objectives, the main body of the tutorial is composed of five parts. In the first part of the tutorial, we will introduce the notion of collaboration in a search context, and more particularly concepts underlying CIR. In the second part of the tutorial, we will shift our focus to collaborative ranking models proposed in the literature. We will present the two main categories of models (based on symmetric or asymmetric roles). In the third part of the tutorial we address the evaluation issue. In particular, we will pay attention to the evaluation frameworks for CIR models as well as the related metrics. The fourth part will be devoted to present some perspectives in CIR. Finally, we will conclude by a discussion phase.
Part 1: Collaborative Information Retrieval Fundamental Notions
- Notion of collaboration in information seeking and retrieval
- Dimensions of Collaboration
- Collaboration Paradigms
- Behavior Processes
Part 2: Models and Techniques for Collaborative Document Seeking and Retrieval
- Algorithmic-Driven Division of Labor-based Document Ranking models (symmetric roles)
- Role-Based Document Ranking Models
Part 3: Effectiveness Evaluation of Collaborative Document Seeking and Retrieval
- Taxonomy of evaluation methodologies
- Evaluation metrics
Part 4: Perspectives
- User-driven CIR models
- Community-based and Social-Media-Based Collaborative Information Information Retrieval Systems
- Standardization and CIR evaluation campaigns
Part 5: Interaction, Questions and Discussion with the Instructors
The tutorial should attract PhD students and early career scientists interested in designing CIR/CIS frameworks where collaboration is held either by small working teams or social-media-based communities. The tutorial should also attract researchers and experts aiming at tackling the core issue of evaluation in CIR/CIS.