Hokkaido University Collection of Scholarly and Academic Papers >
Graduate School of Information Science and Technology / Faculty of Information Science and Technology >
Peer-reviewed Journal Articles, etc >
An Automatic Evaluation Method for Conversational Agents Based on Affect-as-Information Theory
Title: | An Automatic Evaluation Method for Conversational Agents Based on Affect-as-Information Theory |
Authors: | Ptaszynski, Michal Browse this author | Dybala, Pawel Browse this author | Rzepka, Rafal Browse this author | Araki, Kenji Browse this author →KAKEN DB |
Keywords: | Affect-as-Information Theory | Conversational Agents | Evaluation Methods |
Issue Date: | Feb-2010 |
Publisher: | Japan Society for Fuzzy Theory and Intelligent Informatics |
Journal Title: | Journal of Japan Society for Fuzzy Theory and Intelligent Informatics |
Journal Title(alt): | 知能と情報 |
Volume: | 22 |
Issue: | 1 |
Start Page: | 73 |
End Page: | 89 |
Publisher DOI: | 10.3156/jsoft.22.73 |
Abstract: | This paper presents a method for automatic evaluation of conversational agents. The method consists of several steps. First, an affect analysis system is used to detect users' general emotional engagement in the conversation and classify their specific emotional states. Next, we interpret this data with the use of reasoning based on Affect-as-Information Theory to obtain information about users' general attitudes to the conversational agent and its performance. The affect analysis system was also enhanced with a procedure for analysis of Contextual Valence Shifters to help determine the semantic orientation of emotive expressions. The method is used as a background procedure during users' conversations with two Japanese-speaking conversational agents. To verify the usability of the method, the users' attitudes to the conversational agents determined automatically during the conversations were compared to the results of a questionnaire taken after the conversations. The results provided by the system revealed similar tendencies to the questionnaire. Therefore we can say that the method is applicable as a means of evaluation for Japanese-speaking conversational agents. |
Type: | article |
URI: | http://hdl.handle.net/2115/46974 |
Appears in Collections: | 情報科学院・情報科学研究院 (Graduate School of Information Science and Technology / Faculty of Information Science and Technology) > 雑誌発表論文等 (Peer-reviewed Journal Articles, etc)
|
Submitter: 荒木 健治
|