Multimodal feedback i social interaktion

Authors

  • Patrizia Paggio Københavns Universitet
  • Costanza Navarretta

DOI:

https://doi.org/10.7146/nys.v1i56.111943

Keywords:

multimodal feedback; nonverbal expressions; interaction corpora; head movements

Abstract

This paper presents an analysis of spoken and nonverbal feedback in two multimodal corpora reflecting quite different communicative situations. The first consists in eight video recordings of map task dialogues, in which one of the participants explains to the other an itinerary on a map. The two participants sit in different rooms without being able to see each other and communicate through headphones. The other corpus is a collection of twelve video recordings of free face-to-face conversations in which the participants, who met for the first time, were recorded in a studio while standing in front of each other talking. The language used in both corpora is Danish. We describe how spoken and nonverbal feedback have been annotated, and give an account of the distribution of different types of feedback expressions in the two corpora. The analysis uncovers interesting differences both in amount and type (e.g. unimodal vs. multimodal) of feedback, differences which can be explained in terms of the different physical settings as well as the nature of the interaction, which is functional to the map task in the one corpus, and only serves the purpose of talking for the sake of talking in the other.

 

References

Allwood, J., J. Nivre & E. Ahlsén. 1992. On the semantics and pragmatics of linguistic feedback. Journal of Semantics, 9 (1), 1–26. https://doi.org/10.1093/jos/9.1.1
Allwood, J. 2000. Structure of dialog. M. Taylor, D. Bouwhuis, & F. Neel (red.) The structure of multimodal dialogue II, 3–24. Amsterdam: Benjamins.
Allwood, J. , L. Cerrato, K. Jokinen, C. Navarretta & P. Paggio. 2007. The MUMIN coding scheme for the annotation of feedback, turn management and sequencing. Multimodal Corpora for Modelling Human Multimodal Behaviour, Special issue of the International Journal of Language Resources and Evaluation, vol. 41, 273–287, Springer.
Allwood, J., S. Kopp, K. Grammer, E. Ahlsén, E. Oberzaucher & M. Koppensteiner. 2007. The analysis of embodied communicative feedback in multimodal corpora: A prerequisite for behavior simulation. Language Resources and Evaluation, 41(3–4), 255–272. https://doi.org/10.1007/s10579-007-9056-2
Caris-Verhallen, W.M., A. Kerkstra & J.M. Bensing. 1999. Non-verbal behaviour in nurse–elderly patient communication. Journal of advanced nursing, 29 (4), 808–818. https://doi.org/10.1046/j.1365-2648.1999.00965.x
Cerrato, L. 2007. Investigating communicative feedback phenomena across languages and modalities. Ph.D. thesis, School of Speech and Music Communication, Stockholm, KTH.
Clark, H. H. & E.F. Schaefer. 1989. Contributing to Discourse. Cognitive Science, 13: 259–294. https://doi.org/10.1207/s15516709cog1302_7
Grønnum, N. 2006. DanPASS – a Danish phonetically annotated spontaneous speech corpus. Proceedings of the 5th LREC, 1578–1583, Genoa, May.
Grønnum, N. 2009. A Danish phonetically annotated spontaneous speech corpus (DanPASS). Speech Communication, 51 (7), 594–603. https://doi.org/10.1016/j.specom.2008.11.002
Heylen, D. 2006. Head Gestures, Gaze And The Principles Of Conversational Structure International Journal of Humanoid Robotics 2006 03:03, 241–267.
Kraut, R.E., S.H. Lewis & L.W. Swezey. 1982. Listener responsiveness and the coordination of conversation. Journal of personality and social psychology, 43(4), 718–731. https://doi.org/10.1037/0022-3514.43.4.718
McClave, E. Z. 2000. Linguistic functions of head movements in the context of speech. Journal of Pragmatics, 32(7), 855–878. https://doi.org/10.1016/S0378-2166(99)00079-X
Mol, L., E. Krahmer, A. Maes & M. Swerts. 2011. Seeing and Being Seen: The Effects on Gesture Production, Journal of Computer-Mediated Communication, 17 (1), 77–100, https://doi.org/10.1111/j.1083-6101.2011.01558.x
Morency, L-P., I. A. de Kok & J. Gratch. 2008. Predicting Listener Backchannels: A Probabilistic Multimodal Approach. H. Prendinger, J. Lester, & M. Ishizuka (red.), Proceedings of the Eight International Conference on Intelligent Virtual Agents 2008, 176–190. https://doi.org/10.1007/978-3-540-85483-8_18
Mottet, T.P., 2000. Interactive television instructors' perceptions of students' nonverbal responsiveness and their influence on distance teaching. Communication Education, 49(2), 146–164. https://doi.org/10.1080/03634520009379202
Navarretta, C. & P. Paggio. 2010. Classification of Feedback Expressions in Multimodal Data. Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics (ACL 2010), Uppsala, Sweden, Juli 11-16, 318–324.
Paggio, P., J. Allwood, E. Ahlsén, K. Jokinen & C. Navarretta. 2010. The NOMCO Multimodal Nordic Resource - Goals and Characteristics. Calzolari et al (red.) Proceedings of the Seventh conference on International Language Resources and Evaluation (LREC'10), 2968–2974,Valletta, Malta, May.
Paggio P. & C. Navarretta. 2013. Head movements, facial expressions and feedback in conversations - Empirical evidence from Danish multimodal data. Journal on Multimodal User Interfaces - Special Issue on Multimodal Corpora, 7 (1–2), 29–37, Springer Verlag.
Paggio,P. & C. Navarretta. 2016. The Danish NOMCO corpus: Multimodal interaction in first acquaintance conversations. Language Resources and Evaluation, 51 (2), 463–494. https://doi.org/10.1007/s10579-016-9371-6
Poppe R., K.P. Truong & D. Heylen. 2011. Backchannels: Quantity, Type and Timing Matters. Vilhjálmsson H.H., Kopp S., Marsella S., Thórisson K.R. (red.) Intelligent Virtual Agents. IVA 2011. Lecture Notes in Computer Science, vol 6895. Springer.
Tickle-Degnen, L. & R. Rosenthal. 1990. The nature of rapport and its nonverbal correlates. Psychological inquiry, 1(4), 285–293. https://doi.org/10.1207/s15327965pli0104_1
Thompson, H. S., A. Anderson, E. Gurman Bard, G. Doherty-Sneddon, A. Newlands & C. Sotillo. 1993. The HCRC map task corpus: natural dialogue for speech recognition. Proceedings of the 1993 Workshop on Human Language Technology, 25–30, Princeton, New Jersey. https://doi.org/10.3115/1075671.1075677
Yngve, V. 1970. On getting a word in edgewise. Chicago Linguistic Society 6, 567–578.

Downloads

Published

2019-05-27

How to Cite

Paggio, P., & Navarretta, C. (2019). Multimodal feedback i social interaktion. NyS, Nydanske Sprogstudier, 1(56), 77–101. https://doi.org/10.7146/nys.v1i56.111943