Search

Abstract: So far, predictions of user quality judgments in response to spoken dialog systems have been achieved on the basis of interaction parameters describing the dialog, e.g. in the PARADISE framework. These parameters do not take into account the temporal position of events happening in the dialog. It seems promising to apply sequence classification algorithms to the raw annotations of the data, instead of interaction parameters describing the overall dialog. As dialogs can be of very different length, Hidden Markov Models (HMM) and Markov Chains (MC) are handy, because they describe the likelihood of traversing to a state given only the previous state and the transition probability, thus they can be trained and applied to sequences of different lengths. This paper analyzes the feasibility of predicting user judgments with HMMs and MCs. In order to test the models, we acquire data with different types of users, forcing users to do as similar interactions as possible, and asking for user judgments after each turn. This allows comparing predicted distributions of judgments to the distributions measured empirically. We also apply the models to less rich corpora and compare them with results from Linear Regression models as used in the PARADISE framework. [Copyright &y& Elsevier]

Abstract: The evaluation of spoken dialog systems still relies on subjective interaction experiments for quantifying interaction behavior and user-perceived quality. In this paper, we present a simulation approach replacing subjective tests in early system design and evaluation phases. The simulation is based on a model of the system, and a probabilistic model of user behavior. Probabilities for the next user action vary in dependence of system features and user characteristics, as defined by rules. This way, simulations can be conducted before data have been acquired. In order to evaluate the simulation approach, characteristics of simulated interactions are compared to interaction corpora obtained in subjective experiments. As was previously proposed in the literature, we compare interaction parameters for both corpora and calculate recall and precision of user utterances. The results are compared to those from a comparison of real user corpora. While the real corpora are not equal, they are more similar than the simulation is to the real data. However, the simulations can predict differences between system versions and user groups quite well on a relative level. In order to derive further requirements for the model, we conclude with a detailed analysis of utterances missing in the simulated corpus and consider the believability of entire dialogs. [Copyright &y& Elsevier]

Evaluation of spoken dialogue systems has been traditionally carried out in terms of instrumentally or expert-derived measures (usually called “objective” evaluation) and quality judgments of users who have previously interacted with the system (also called “subjective” evaluation). Different research efforts have been made to extract relationships between these evaluation criteria. In this paper we report empirical results obtained from statistical studies, which were carried out on interactions of real users with our spoken dialogue system. These studies have rarely been exploited in the literature. Our results show that they can indicate important relationships between criteria, which can be used as guidelines for refinement of the systems under evaluation, as well as contributing to the state-of-the-art knowledge about how quantitative aspects of the systems affect the user’s perceptions about them. [Copyright &y& Elsevier]

Borrowing from the negotiation literature, we tested 2 factors that might improve stakeholder dialogue in program and policy evaluation. Undergraduate stakeholders (61 pairs) engaged in dialogue about their universities' alcohol policies. Pairs were randomly assigned to levels of accountability audience and dialogue structure. The audience for the videotaped dialogue was described as holding either (a) views about the policy similar to the participant's, consistent across audience members (homogeneous), or (b) mixed views, on both sides of the issue (heterogeneous). Pairs approached the dialogue with either (a) problem-solving goals or (b) no particular strategy. Dyads accountable to a heterogeneous audience and given problem-solving instructions exhibited the most effective dialogue. Accountability to a heterogeneous audience facilitated satisfaction with and optimism about dialogue. Accountability to homogeneous audiences and adopting no particular strategy yielded the least positive perceptions of dialogue. Implications for stakeholder dialogue, and for the role of social psychology in evaluation are discussed. [ABSTRACT FROM AUTHOR]

Spoken dialogue systems (SDSs) can be used to operate devices, e.g. in the automotive environment. People using these systems usually have different levels of experience. However, most systems do not take this into account. In this paper, we present a method to build a dialogue system in an automotive environment that automatically adapts to the user’s experience with the system. We implemented the adaptation in a prototype and carried out exhaustive tests. Our usability tests show that adaptation increases both user performance and user satisfaction. We describe the tests that were performed, and the methods used to assess the test results. One of these methods is a modification of PARADISE, a framework for evaluating the performance of SDSs [Walker MA, Litman DJ, Kamm CA, Abella A (Comput Speech Lang 12(3):317–347, 1998)]. We discuss its drawbacks for the evaluation of SDSs like ours, the modifications we have carried out, and the test results. [ABSTRACT FROM AUTHOR]

Comments on the research "Relational Rationality and Children's Interview Responses," by Karin Aronsson and Karsten Hundeide, which proposed that the idea of the appropriateness of children's answers being judged in terms of scientific rationality was inappropriate. Rationality of cooperation; Genetic approach to development; Need for some degree of interpersonal attunement for a meaningful conversations.

Dialogue in evaluation is presented as a fundamental value commitment to engagement, particularly engagement with the relational, moral and political dimensions of our contexts and our craft. Because it is at root a relational and communicative activity, effective dialogue in evaluation is inclusive of all legitimate stakeholder interests and involves interactions that are respectful, reciprocal and equitable. Effective dialogue, in turn, can enhance the democratizing potential of evaluation. [ABSTRACT FROM AUTHOR]