A Taxonomy of Social Cues for Conversational Agents

When using the taxonomy, please cite as Feine, J., Gnewuch U., Morana S. and Maedche A. (2019): “A Taxonomy of Social Cues for Conversational Agents” International Journal of Human-Computer Studies. To read the paper, please click here.

Social cue: Eye movement
Communication system: Visual
Cue category: Kinesics
Cue Description
The CA moves its eyes to intentionally or unintentionally fixate or track objects.
Cue example
CA is looking in the direction of the other person’s face.
Cue impact
Eye blinking creates the impression that the agent is being alive (Kraemer et al. 2016; Becker et al. 2005), impacts believability (Becker et al. 2004), human-likeness (McBreen et al. 2001) and social reactions of the user towards the agent (Appel et al. 2012). However, it does not enhance the perceived social presence like reactive nonverbal feedback (Puetten et al. 2010; Von der Pütten et al. 2009). Furthermore, too unrealistic movements do not positively impact the agent evaluation (Bonito et al 1999). Eye contact can increase trustworthiness (Cowell, Stanney 2005), builds rapport (Krämer et al. 2016), impacts the likeability (Cafaro et al. 2016), and influences the perceived dominance of the agent (Gebhard et al. 2014). Furthermore, moving eye gaze makes the agent more natural (Mersiol et al. 2002), human-like (McBreen et al. 2001) and indicates turn-taking attempts (Cassel et al. 1999). It is further useful as part of relational behavior strategies to ensure a long-term working alliance (Bickmore et al. 2005). Gaze behavior can further be used for flirtation which impacts the user’s enjoyment, increased their interest to continue the interaction or even to engage in a conversation (Bee et al. 2009).The perception of being monitored by an animated character, who is making eye contact, has the same effects on anxiety and performance as being monitored by a human (Rickenberg, Reeves 2000).
Reference List
1. Appel, J., Pütten, A. von der, Krämer, N. C., & Gratch, J. (2012). Does Humanity Matter?: Analyzing the Importance of Social Cues and Perceived Agency of a Computer System for the Emergence of Social Reactions during Human-Computer Interaction. Advances in Human-Computer Interaction (2012:2), pp. 1-10.
2. Becker, C., Kopp, S., & Wachsmuth, L. (2004). Simulating the emotion dynamics of a multimodal conversational agent. In E. Andre, L. Dybkjaer, W. Minker, & P. Heisterkamp (Eds.): Lecture Notes in Computer Science, AFFECTIVE DIALOGUE SYSTEMS, PROCEEDINGS (pp. 154-165).
3. Becker, C., Prendinger, H., Ishizuka, M., & Wachsmuth, I. (Eds.). 2005. Evaluating affective feedback of the 3D agent max in a competitive cards game: Springer.
4. Bee, N., André, E., & Tober, S. (2009). Breaking the Ice in Human-Agent Communication: Eye-Gaze Based Initiation of Contact with an Embodied Conversational Agent. In Z. Ruttkay, M. Kipp, A. Nijholt, & H. H. Vilhjálmsson (Eds.), Intelligent Virtual Agents (pp. 229–242). Berlin, Heidelberg: Springer Berlin Heidelberg
5. Bickmore, T. W., & Picard, R. W. (2005). Establishing and Maintaining Long-term Human-computer Relationships. ACM TRANSACTIONS ON COMPUTER-HUMAN INTERACTION (12:2), pp. 293-327, from http://doi.acm.org/10.1145/1067860.1067867.
6. Bonito, J. A., Burgoon, J. K., & Bengtsson, B. (1999). The Role of Expectations in Human-computer Interaction. In : GROUP ’99, Proceedings of the International ACM SIGGROUP Conference on Supporting Group Work (pp. 229-238). New York, NY, USA: ACM.
7. Cafaro, A., Vilhjalmsson, H. H., & Bickmore, T. (2016). First Impressions in Human-Agent Virtual Encounters. ACM TRANSACTIONS ON COMPUTER-HUMAN INTERACTION (23:4).
8. Cassell, J., Bickmore, T., Billinghurst, M., Campbell, L., Chang, K., Vilhjálmsson, H., & Yan, H. (1999). Embodiment in Conversational Interfaces: Rea. In : CHI ’99, Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 520-527). New York, NY, USA: ACM.
9. Cowell, A. J., & Stanney, K. M. (2005). Manipulation of non-verbal interaction style and demographic embodiment to increase anthropomorphic computer character credibility. INTERNATIONAL JOURNAL OF HUMAN-COMPUTER STUDIES (62:2), pp. 281-306.
10. Gebhard, P., Baur, T., Damian, I., Mehlmann, G., Wagner, J., & André, E. (2014). Exploring Interaction Strategies for Virtual Characters to Induce Stress in Simulated Job Interviews. In : AAMAS ’14, Proceedings of the 2014 International Conference on Autonomous Agents and Multi-agent Systems (pp. 661-668). Richland, SC: International Foundation for Autonomous Agents and Multiagent Systems.
11. Kraemer, N. C., Karacora, B., Lucas, G., Dehghani, M., Ruether, G., & Gratch, J. (2016). Closing the gender gap in STEM with friendly male instructors? On the effects of rapport behavior and gender of a virtual agent in an instructional interaction. COMPUTERS & EDUCATION (99, pp. 1-13.
12. McBreen, H. M., & Jack, M. A. (2001). Evaluating humanoid synthetic agents in e-retail applications. IEEE TRANSACTIONS ON SYSTEMS MAN AND CYBERNETICS PART A-SYSTEMS AND HUMANS (31:5), pp. 394-405.
13. Mersiol, M., Chateau, N., & Maffiolo, V. (Eds.). 2002. Talking heads: Which matching between faces and synthetic voices? Proceedings. Fourth IEEE International Conference on Multimodal Interfaces.
14. Puetten, A. M. von der, Kraemer, N. C., Gratch, J., & Kang, S.-H. (2010). “It doesn’t matter what you are!” Explaining social effects of agents and avatars. Computers in Human Behavior (26:6), pp. 1641-1650.
15. Rickenberg, R., & Reeves, B. (2000). The Effects of Animated Characters on Anxiety, Task Performance, and Evaluations of User Interfaces. In : CHI ’00, Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 49-56). New York, NY, USA: ACM.
16. von der Pütten, Astrid Marieke, Krämer, N., & Gratch, J. (2009). Who s there? Can a Virtual Agent Really Elicit Social Presence? The 12th Annual International Workshop on Presence.
17. Cassell, J. (2000). Embodied conversational interface agents? Communications of the ACM 43 (4), 70?78.
18. Cassell, J. and K. R. Thorisson (1999). ?The power of a nod and a glance: Envelope vs. emotional feedback in animated conversational agents? Applied Artificial intelligence 13 (4-5), 519?538.
19. Rosis, F. de, C. Pelachaud, I. Poggi, V. Carofiglio and B. de Carolis (2003). ?From Greta's mind to her face: modelling the dynamics of affective states in a conversational embodied agent? International Journal of Human-Computer Studies 59 (1-2), 81?118.
20. Bickmore, T. and J. Cassell (2005). Social Dialogue with Embodied Conversational Agents. In J. C. J. Kuppevelt, N. O. Bernsen and L. Dybkjær (eds.) Advances in Natural Multimodal Dialogue Systems, pp. 23?54. Dordrecht: Springer.
21. Thiebaux, M., S. Marsella, A. N. Marshall and M. Kallmann (2008). Smartbody: Behavior realization for embodied conversational agents. In International Foundation for Autonomous Agents and Multiagent SystemsInternational Foundation for Autonomous Agents and Multiagent Systems.
22. Cassell, J. (2001). Embodied Conversational Agents. Representation and Intelligence in User Interfaces AI MAGAZINE 22 (4), 67?83.
23. Ryokai, K., C. Vaucelle and J. Cassell (2003). Virtual peers as partners in storytelling and literacy learning Journal of Computer Assisted Learning 19 (2), 195?208.
24. Carolis, B. de, C. Pelachaud, I. Poggi and M. Steedman (2004). APML, a Markup Language for Believable Behavior Generation. In H. Prendinger and M. Ishizuka (eds.) Life-Like Characters: Tools, Affective Functions, and Applications, pp. 65?85. Berlin, Heidelberg: Springer Berlin Heidelberg.
25. Bailenson, J. N. and N. Yee (2005). Digital chameleons. Automatic assimilation of nonverbal gestures in immersive virtual environments Psychological science 16 (10), 814?819.
26. Pelachaud, C. (2005). Multimodal Expressive Embodied Conversational Agents. In: Proceedings of the 13th Annual ACM International Conference on Multimedia. New York, NY, USA: ACM, pp. 683?689.
27. Bevacqua, E., S. Pammi, S. J. Hyniewska, M. Schröder and C. Pelachaud (2010). Multimodal Backchannels for Embodied Conversational Agents. In: Intelligent Virtual Agents. Ed. by J. Allbeck, N. Badler, T. Bickmore, C. Pelachaud, A. Safonova. Berlin, Heidelberg: Springer Berlin Heidelberg, pp. 194?200.