SAUCE 2016

1st International Workshop on System And User Centered Evaluation Approaches in Interactive Information Retrieval

co-located with 1st ACM SIGIR Conference on Human Information Interaction and Retrieval (CHIIR 2006)
17 March 2016, Chapel Hill, North Carolina, USA



Nowadays, IIR continues to increase in complexity: user tasks and needs are demanding; data and information systems are rapidly evolving and greatly heterogeneous; and the interaction between users and IR systems is much more articulated. For example, consider what Web search is today: highly diversified results are returned from Web pages, news, social media, image and video search, products and more, and all are merged through adaptive strategies driven by current and previous user-systems interactions. As a result, experimental evaluation needs to appropriately model these evolving tasks, needs, data sources and user interactions. An additional challenge pertains to the anticipated outcome of IIR research and application. It is no longer sufficient to focus solely on precision, recall and satisfaction: successful IIR systems must engage, inform, and relate to users, taking into account single session and more long-term use and re-use.

To progress current evaluation methodologies and ensure they are able to effectively support the development of next generation IIR systems, one of the most compelling prospects is to bridge system-oriented and user-oriented evaluation methods. Both methodological approaches have their advantages and drawbacks: while system-centered methods ensure greater internal validity, they may fail to take into account user and contextual factors that influence IIR; user-oriented methods may better approximate actual user behavior, affect and cognition, but provide less experimental control of independent variables.

The goal of this workshop is to unite system- and user-centered IIR researchers for the purposes of:

  • Sharing different user-centered and system-centred research methods, measures, and tools in order to foster knowledge exchange;
  • Exploring the addition of user-centered evaluation strategies to system-oriented studies, and vice versa; and
  • Initiating collaborations between user- and system-oriented researchers to further IIR research.

Important Dates

Submission deadline: January 22, 2016 (extended)

Notification of acceptance: February 12, 2016

Camera ready: February 26, 2016

Workshop day: March 17, 2016

Conference days: March 13-17, 2016

Call for Position Papers

General areas of interests include, but are not limited to, the following topics:

  • User-centred approaches that incorporate elements of system-centred evaluation
  • System-centred approaches that incorporate elements of user-centred evaluation
  • Phenomenon of interest or problems in interactive information retrieval that could be best approached using a combination of user- and system-centred evaluation
  • Descriptions of research or system design that are incorporate system- and user-centred methods
  • Correlation between system-centred and user-centred evaluation
  • Prediction of user behaviour, satisfaction, engagement, ... from system-centred evaluation

Papers should be formatted according to the ACM SIG Proceedings Template.

Papers should be two-four pages (maximum) in length.

Papers will be peer-reviewed by members of the program committee through double-blind peer review, i.e. authors must be anonymized. Selection will be based on originality, clarity, and technical quality. Papers should be submitted in PDF format to the following address:

Accepted papers will be published online as a volume of the CEUR-WS proceeding series.


Heather L. O'Brien, University of British Columbia, Canada,

Nicola Ferro, University of Padua, Italy

Hideo Joho, University of Tsukuba, Japan

Dirk Lewandowski, Hamburg University of Applied Sciences, Germany

Paul Thomas, CSIRO, Australia,

Keith van Rijsbergen, University of Glasgow, UK,


Keynote Talks

Eric Wiebe
Department of STEM Education at North Carolina State University, USA

Eric Wiebe


Dr. Wiebe is a Professor in the Department of STEM Education at NC State University and Senior Research Fellow at the Friday Institute for Educational Innovation.
Dr. Wiebe works on many different facets of STEM Education, including the design and evaluation of innovative uses of computing technologies in STEM instructional settings, the use of multimedia tools for teaching and learning, and student engagement and persistence in STEM career pathways.
Specific research programs include the use of intelligent agents to support science learning in classrooms and basic research in the how instructional technologies (including game-based learning environments) shape student engagement and learning.
Research contexts for this work range from individual student studies in the lab, to classroom studies, to large scale remote data collection activities with MOOCs and other cloud-based learning environments.

Goals and Engagement: Learners in Information Environments

This talk will at the learner in increasingly complex educational environments. I will focus on psychological models of affect and cognition to look at how researchers in education, psychology, and computer science are analyzing behaviors of learners, including information seeking and retrieval, in digital environments.
Examples will be drawn from my project work over the last 15 years in building and studying cyberlearning tools in grades 4-16 science classrooms and in the laboratory. A particular focus will be on how students engage in goal-directed behavior as they construct knowledge, and what psychological models have been particularly useful for studying this. An array of methodologies will also be discussed as well as implications of emerging adaptive systems supporting learners.

Ioannis Arapakis
Eurecat, Barcelona, Spain

Eric Wiebe


Dr. Arapakis received his M.Sc. degree in Information Technology from the Royal Institute of Technology (KTH), Sweden. He then pursued a Ph.D. degree at the SoCS of the University of Glasgow, in the area of Information Retrieval, under the supervision of Prof. Joemon M. Jose, and Prof. C. J. Keith van Rijsbergen. In the past four years, he worked as a full-time researcher at Yahoo Labs, Barcelona, where he conducted research in the areas of Data Mining, HCI, and IR. During this period he was an active member of the User Engagement, Web Retrieval, and Ad Processing and Retrieval groups and has been involved in several key internal projects. His groups have achieved many high quality publications in top-tier venues and journals, and have addressed research problems related to pattern recognition, predictive modelling, multimedia mining and search, social media analysis, distributed systems, and machine learning. Currently, he is a Senior Data Scientist at Eurecat (Barcelona) and member of the Data Mining group.

User Behaviour Modelling - Online and Offline Methods, Metrics, and Challenges

Network flows, social networking, smart devices, the Internet-of-Things. These innovations carry no deep value in themselves. Value invariably comes from understanding, obtaining accurate measurements, predicting, and controlling. This vision, however, rests on the application of machine intelligence and data mining techniques that can tackle large and diverse data collections, but also on our capacity to operationalise an interdisciplinary research in the intersection of many domains, such as statistics, signal processing, neuroscience, privacy and security, to name a few. In this talk, through the narration of my involvement in past and recent projects, I will share my experience in the domain of user behaviour analysis and predictive modelling. I will discuss offline and online experimental methods (and how they can be brought together), present current practices in measuring human behaviour in the online world, and highlight research challenges and opportunities that I have encountered.


09:00-09:15 - Opening and Welcome
09:15-10:30 - Keynote with Dr. Eric Wiebe, North Carolina State University
10:30-11:00 - Coffee Break
11:00-12:30 - Keynote with Dr. Ioannis Arapakis and World Cafe Session, Part 1
12:30-13:30 - Lunch
13:30-15:00 - Position Papers and World Cafe, Part 2

Understanding user adjustment to slow search   

Ryan Burton, School of Information, University of Michigan, USA
Kevyn Collins-Thompson, School of Information, University of Michigan, USA

Challenges for Measuring Usefulness of Interactive IR Systems with Log-based Approaches   

Daniel Hienert, GESIS - Leibniz Institute for the Social Sciences, Germany
Peter Mutschke, GESIS - Leibniz Institute for the Social Sciences, Germany

Process-Learning as a factor in evaluation   

Catherine Smith, Kent State University, USA
15:00-15:30 - Coffee Break
15:30-16:30 - Expert Panel
Jacek Gwizdka, University of Texas at Austin, USA
Noriko Kando, National Institute of Informatics, Japan
Diane Kelly, University of North Carolina Chapel Hill, USA
16:30-17:00 - Wrap Up and Closing