A Taxonomy of Social Cues for Conversational Agents

When using the taxonomy, please cite as Feine, J., Gnewuch U., Morana S. and Maedche A. (2019): “A Taxonomy of Social Cues for Conversational Agents” International Journal of Human-Computer Studies. To read the paper, please click here.

Social cue: Arm and hand gesture
Communication system: Visual
Cue category: Kinesics
Cue Description
The CA moves its arm or hand.
Cue example
Point a finger at something, raise an arm to express an idea or feeling.
Cue impact
Hand gestures can be useful to indicate turn-taking attempts (Bickmore, Cassel 2001; Cassel, Bickmore 2000), complement speech content (Cassel et al. 1999), can improve relationship building, and is useful as part of relational behavior strategies to ensure a long-term working alliance (Bickmore et al. 2005). Space-taking and freely conducted gestures also impact the perceived dominance (Gebhard et al. 2014; McBreen et al. 2001) and extroversion of the agent (Isbister, Nass 2000). Agent that conducts note-taking gestures during a task creates fear and anxiety like in real world examination situation (Rickenberg, Reeves 2000). Furthermore, gestures that are not content related can also distract the user and decrease the user involvement (Hess et al. 2005).
Reference List
1. Bickmore, T. W., & Picard, R. W. (2005). Establishing and Maintaining Long-term Human-computer Relationships. ACM TRANSACTIONS ON COMPUTER-HUMAN INTERACTION (12:2), pp. 293-327, from http://doi.acm.org/10.1145/1067860.1067867.
2. Bickmore, T., & Cassell, J. (2001). Relational agents: A Model and Implementation of Building User Trust. In J. Jacko (Ed.), Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 396-403). New York, NY: ACM.
3. Cassell, J., & Bickmore, T. (2000). EXTERNAL MANIFESTATIONS OF TRUST WORTHINESS IN THE INTERFACE. Communications of the ACM (43:12), pp. 50-56.
4. Cassell, J., Bickmore, T., Billinghurst, M., Campbell, L., Chang, K., Vilhjálmsson, H., & Yan, H. (1999). Embodiment in Conversational Interfaces: Rea. In : CHI ’99, Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 520-527). New York, NY, USA: ACM.
5. Gebhard, P., Baur, T., Damian, I., Mehlmann, G., Wagner, J., & André, E. (2014). Exploring Interaction Strategies for Virtual Characters to Induce Stress in Simulated Job Interviews. In : AAMAS ’14, Proceedings of the 2014 International Conference on Autonomous Agents and Multi-agent Systems (pp. 661-668). Richland, SC: International Foundation for Autonomous Agents and Multiagent Systems.
6. HESS, T. J., FULLER, M. A., & MATHEW, J. (2005). Involvement and Decision-Making Performance with a Decision Aid: The Influence of Social Multimedia, Gender, and Playfulness. JOURNAL OF MANAGEMENT INFORMATION SYSTEMS (22:3), pp. 15-54.
7. Isbister, K., & Nass, C. (2000). Consistency of personality in interactive characters: Verbal cues, non-verbal cues, and user characteristics. International Journal of Human-Computer Studies (53:2), pp. 251-267.
8. McBreen, H. M., & Jack, M. A. (2001). Evaluating humanoid synthetic agents in e-retail applications. IEEE TRANSACTIONS ON SYSTEMS MAN AND CYBERNETICS PART A-SYSTEMS AND HUMANS (31:5), pp. 394-405.
9. Rickenberg, R., & Reeves, B. (2000). The Effects of Animated Characters on Anxiety, Task Performance, and Evaluations of User Interfaces. In : CHI ’00, Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 49-56). New York, NY, USA: ACM.
10. Cassell, J. (2000). Embodied conversational interface agents? Communications of the ACM 43 (4), 70?78.
11. Bickmore, T. and J. Cassell (2005). Social Dialogue with Embodied Conversational Agents. In J. C. J. Kuppevelt, N. O. Bernsen and L. Dybkjær (eds.) Advances in Natural Multimodal Dialogue Systems, pp. 23?54. Dordrecht: Springer.
12. Kopp, S. and I. Wachsmuth (2004). Synthesizing multimodal utterances for conversational agents Computer animation and virtual worlds 15 (1), 39?52.
13. Cassell, J. (2001). Embodied Conversational Agents. Representation and Intelligence in User Interfaces AI MAGAZINE 22 (4), 67?83.
14. Pelachaud, C. (2005). Multimodal Expressive Embodied Conversational Agents. In: Proceedings of the 13th Annual ACM International Conference on Multimedia. New York, NY, USA: ACM, pp. 683?689.