Multimodal feedback i social interaktion

  • Patrizia Paggio Københavns Universitet
  • Costanza Navarretta
Nøgleord: multimodal feedback; nonverbale udtryk; interaktionskorpora; hovedbevægelser

Resumé

I denne artikel analyserer vi sproglig og nonverbal feedback i to multimodale korpora som afspejler ret forskellige kommunikationssituationer. Det ene består af otte videooptagede såkaldte ’map task’-samtaler, hvor en af samtalepartnerne forklarer den anden ruten på et kort. De to deltagere sidder i forskellige rum uden at kunne se hinanden og kommunikerer igennem høretelefoner. Det andet korpus er en samling af tolv videooptagelser af frie ansigt-til-ansigt-samtaler hvor deltagerne, som ikke kendte hinanden på forhånd, blev optaget mens de stod over for hinanden i et studie og talte sammen. Sproget er dansk i begge korpora. Vi beskriver hvordan sproglig og nonverbal feedback er opmærket og gennemgår distributionen af forskellige typer feedbackudtryk i de to korpora. Vores analyse viser interessante forskelle i både mængden og typen (mono- kontra multimodal) af feedback, som dels skyldes de forskellige fysiske opsætninger, dels interaktionens natur, som er funktionel i ’map task’-dialogerne, og som bare er samtale for samtalens skyld i de frie ansigt-til-ansigt-dialoger.

Referencer

Allwood, J., J. Nivre & E. Ahlsén. 1992. On the semantics and pragmatics of linguistic feedback. Journal of Semantics, 9 (1), 1–26. https://doi.org/10.1093/jos/9.1.1
Allwood, J. 2000. Structure of dialog. M. Taylor, D. Bouwhuis, & F. Neel (red.) The structure of multimodal dialogue II, 3–24. Amsterdam: Benjamins.
Allwood, J. , L. Cerrato, K. Jokinen, C. Navarretta & P. Paggio. 2007. The MUMIN coding scheme for the annotation of feedback, turn management and sequencing. Multimodal Corpora for Modelling Human Multimodal Behaviour, Special issue of the International Journal of Language Resources and Evaluation, vol. 41, 273–287, Springer.
Allwood, J., S. Kopp, K. Grammer, E. Ahlsén, E. Oberzaucher & M. Koppensteiner. 2007. The analysis of embodied communicative feedback in multimodal corpora: A prerequisite for behavior simulation. Language Resources and Evaluation, 41(3–4), 255–272. https://doi.org/10.1007/s10579-007-9056-2
Caris-Verhallen, W.M., A. Kerkstra & J.M. Bensing. 1999. Non-verbal behaviour in nurse–elderly patient communication. Journal of advanced nursing, 29 (4), 808–818. https://doi.org/10.1046/j.1365-2648.1999.00965.x
Cerrato, L. 2007. Investigating communicative feedback phenomena across languages and modalities. Ph.D. thesis, School of Speech and Music Communication, Stockholm, KTH.
Clark, H. H. & E.F. Schaefer. 1989. Contributing to Discourse. Cognitive Science, 13: 259–294. https://doi.org/10.1207/s15516709cog1302_7
Grønnum, N. 2006. DanPASS – a Danish phonetically annotated spontaneous speech corpus. Proceedings of the 5th LREC, 1578–1583, Genoa, May.
Grønnum, N. 2009. A Danish phonetically annotated spontaneous speech corpus (DanPASS). Speech Communication, 51 (7), 594–603. https://doi.org/10.1016/j.specom.2008.11.002
Heylen, D. 2006. Head Gestures, Gaze And The Principles Of Conversational Structure International Journal of Humanoid Robotics 2006 03:03, 241–267.
Kraut, R.E., S.H. Lewis & L.W. Swezey. 1982. Listener responsiveness and the coordination of conversation. Journal of personality and social psychology, 43(4), 718–731. https://doi.org/10.1037/0022-3514.43.4.718
McClave, E. Z. 2000. Linguistic functions of head movements in the context of speech. Journal of Pragmatics, 32(7), 855–878. https://doi.org/10.1016/S0378-2166(99)00079-X
Mol, L., E. Krahmer, A. Maes & M. Swerts. 2011. Seeing and Being Seen: The Effects on Gesture Production, Journal of Computer-Mediated Communication, 17 (1), 77–100, https://doi.org/10.1111/j.1083-6101.2011.01558.x
Morency, L-P., I. A. de Kok & J. Gratch. 2008. Predicting Listener Backchannels: A Probabilistic Multimodal Approach. H. Prendinger, J. Lester, & M. Ishizuka (red.), Proceedings of the Eight International Conference on Intelligent Virtual Agents 2008, 176–190. https://doi.org/10.1007/978-3-540-85483-8_18
Mottet, T.P., 2000. Interactive television instructors' perceptions of students' nonverbal responsiveness and their influence on distance teaching. Communication Education, 49(2), 146–164. https://doi.org/10.1080/03634520009379202
Navarretta, C. & P. Paggio. 2010. Classification of Feedback Expressions in Multimodal Data. Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics (ACL 2010), Uppsala, Sweden, Juli 11-16, 318–324.
Paggio, P., J. Allwood, E. Ahlsén, K. Jokinen & C. Navarretta. 2010. The NOMCO Multimodal Nordic Resource - Goals and Characteristics. Calzolari et al (red.) Proceedings of the Seventh conference on International Language Resources and Evaluation (LREC'10), 2968–2974,Valletta, Malta, May.
Paggio P. & C. Navarretta. 2013. Head movements, facial expressions and feedback in conversations - Empirical evidence from Danish multimodal data. Journal on Multimodal User Interfaces - Special Issue on Multimodal Corpora, 7 (1–2), 29–37, Springer Verlag.
Paggio,P. & C. Navarretta. 2016. The Danish NOMCO corpus: Multimodal interaction in first acquaintance conversations. Language Resources and Evaluation, 51 (2), 463–494. https://doi.org/10.1007/s10579-016-9371-6
Poppe R., K.P. Truong & D. Heylen. 2011. Backchannels: Quantity, Type and Timing Matters. Vilhjálmsson H.H., Kopp S., Marsella S., Thórisson K.R. (red.) Intelligent Virtual Agents. IVA 2011. Lecture Notes in Computer Science, vol 6895. Springer.
Tickle-Degnen, L. & R. Rosenthal. 1990. The nature of rapport and its nonverbal correlates. Psychological inquiry, 1(4), 285–293. https://doi.org/10.1207/s15327965pli0104_1
Thompson, H. S., A. Anderson, E. Gurman Bard, G. Doherty-Sneddon, A. Newlands & C. Sotillo. 1993. The HCRC map task corpus: natural dialogue for speech recognition. Proceedings of the 1993 Workshop on Human Language Technology, 25–30, Princeton, New Jersey. https://doi.org/10.3115/1075671.1075677
Yngve, V. 1970. On getting a word in edgewise. Chicago Linguistic Society 6, 567–578.
Publiceret
2019-05-27
Citation/Eksport
Paggio, P., & Navarretta, C. (2019). Multimodal feedback i social interaktion. NyS, Nydanske Sprogstudier, 1(56), 77-101. https://doi.org/10.7146/nys.v1i56.111943