Multimodal feedback i social interaktion
DOI:
https://doi.org/10.7146/nys.v1i56.111943Keywords:
multimodal feedback; nonverbal expressions; interaction corpora; head movementsAbstract
This paper presents an analysis of spoken and nonverbal feedback in two multimodal corpora reflecting quite different communicative situations. The first consists in eight video recordings of map task dialogues, in which one of the participants explains to the other an itinerary on a map. The two participants sit in different rooms without being able to see each other and communicate through headphones. The other corpus is a collection of twelve video recordings of free face-to-face conversations in which the participants, who met for the first time, were recorded in a studio while standing in front of each other talking. The language used in both corpora is Danish. We describe how spoken and nonverbal feedback have been annotated, and give an account of the distribution of different types of feedback expressions in the two corpora. The analysis uncovers interesting differences both in amount and type (e.g. unimodal vs. multimodal) of feedback, differences which can be explained in terms of the different physical settings as well as the nature of the interaction, which is functional to the map task in the one corpus, and only serves the purpose of talking for the sake of talking in the other.
References
Allwood, J. 2000. Structure of dialog. M. Taylor, D. Bouwhuis, & F. Neel (red.) The structure of multimodal dialogue II, 3–24. Amsterdam: Benjamins.
Allwood, J. , L. Cerrato, K. Jokinen, C. Navarretta & P. Paggio. 2007. The MUMIN coding scheme for the annotation of feedback, turn management and sequencing. Multimodal Corpora for Modelling Human Multimodal Behaviour, Special issue of the International Journal of Language Resources and Evaluation, vol. 41, 273–287, Springer.
Allwood, J., S. Kopp, K. Grammer, E. Ahlsén, E. Oberzaucher & M. Koppensteiner. 2007. The analysis of embodied communicative feedback in multimodal corpora: A prerequisite for behavior simulation. Language Resources and Evaluation, 41(3–4), 255–272. https://doi.org/10.1007/s10579-007-9056-2
Caris-Verhallen, W.M., A. Kerkstra & J.M. Bensing. 1999. Non-verbal behaviour in nurse–elderly patient communication. Journal of advanced nursing, 29 (4), 808–818. https://doi.org/10.1046/j.1365-2648.1999.00965.x
Cerrato, L. 2007. Investigating communicative feedback phenomena across languages and modalities. Ph.D. thesis, School of Speech and Music Communication, Stockholm, KTH.
Clark, H. H. & E.F. Schaefer. 1989. Contributing to Discourse. Cognitive Science, 13: 259–294. https://doi.org/10.1207/s15516709cog1302_7
Grønnum, N. 2006. DanPASS – a Danish phonetically annotated spontaneous speech corpus. Proceedings of the 5th LREC, 1578–1583, Genoa, May.
Grønnum, N. 2009. A Danish phonetically annotated spontaneous speech corpus (DanPASS). Speech Communication, 51 (7), 594–603. https://doi.org/10.1016/j.specom.2008.11.002
Heylen, D. 2006. Head Gestures, Gaze And The Principles Of Conversational Structure International Journal of Humanoid Robotics 2006 03:03, 241–267.
Kraut, R.E., S.H. Lewis & L.W. Swezey. 1982. Listener responsiveness and the coordination of conversation. Journal of personality and social psychology, 43(4), 718–731. https://doi.org/10.1037/0022-3514.43.4.718
McClave, E. Z. 2000. Linguistic functions of head movements in the context of speech. Journal of Pragmatics, 32(7), 855–878. https://doi.org/10.1016/S0378-2166(99)00079-X
Mol, L., E. Krahmer, A. Maes & M. Swerts. 2011. Seeing and Being Seen: The Effects on Gesture Production, Journal of Computer-Mediated Communication, 17 (1), 77–100, https://doi.org/10.1111/j.1083-6101.2011.01558.x
Morency, L-P., I. A. de Kok & J. Gratch. 2008. Predicting Listener Backchannels: A Probabilistic Multimodal Approach. H. Prendinger, J. Lester, & M. Ishizuka (red.), Proceedings of the Eight International Conference on Intelligent Virtual Agents 2008, 176–190. https://doi.org/10.1007/978-3-540-85483-8_18
Mottet, T.P., 2000. Interactive television instructors' perceptions of students' nonverbal responsiveness and their influence on distance teaching. Communication Education, 49(2), 146–164. https://doi.org/10.1080/03634520009379202
Navarretta, C. & P. Paggio. 2010. Classification of Feedback Expressions in Multimodal Data. Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics (ACL 2010), Uppsala, Sweden, Juli 11-16, 318–324.
Paggio, P., J. Allwood, E. Ahlsén, K. Jokinen & C. Navarretta. 2010. The NOMCO Multimodal Nordic Resource - Goals and Characteristics. Calzolari et al (red.) Proceedings of the Seventh conference on International Language Resources and Evaluation (LREC'10), 2968–2974,Valletta, Malta, May.
Paggio P. & C. Navarretta. 2013. Head movements, facial expressions and feedback in conversations - Empirical evidence from Danish multimodal data. Journal on Multimodal User Interfaces - Special Issue on Multimodal Corpora, 7 (1–2), 29–37, Springer Verlag.
Paggio,P. & C. Navarretta. 2016. The Danish NOMCO corpus: Multimodal interaction in first acquaintance conversations. Language Resources and Evaluation, 51 (2), 463–494. https://doi.org/10.1007/s10579-016-9371-6
Poppe R., K.P. Truong & D. Heylen. 2011. Backchannels: Quantity, Type and Timing Matters. Vilhjálmsson H.H., Kopp S., Marsella S., Thórisson K.R. (red.) Intelligent Virtual Agents. IVA 2011. Lecture Notes in Computer Science, vol 6895. Springer.
Tickle-Degnen, L. & R. Rosenthal. 1990. The nature of rapport and its nonverbal correlates. Psychological inquiry, 1(4), 285–293. https://doi.org/10.1207/s15327965pli0104_1
Thompson, H. S., A. Anderson, E. Gurman Bard, G. Doherty-Sneddon, A. Newlands & C. Sotillo. 1993. The HCRC map task corpus: natural dialogue for speech recognition. Proceedings of the 1993 Workshop on Human Language Technology, 25–30, Princeton, New Jersey. https://doi.org/10.3115/1075671.1075677
Yngve, V. 1970. On getting a word in edgewise. Chicago Linguistic Society 6, 567–578.
Downloads
Published
How to Cite
Issue
Section
License
Forfatteren/forfatterne og NyS har ophavsret til de artikler og anmeldelser der bringes i tidsskriftet. NyS har ophavsretten til den udgivne version af tidsskriftet. Forfatteren har ophavsretten til sin egen tekst. Forfattere kan arkivere den publicerede artikel på deres institutions forskningsarkiv (Institutional Repository) eller en privat hjemmeside, når forfatteren samtidig linker til artiklen med den officielle DOI.
For artikler publiceret i NyS tillades at læsere kan downloade, kopiere, udskrive, søge eller linke til og citere fra artikler til ethvert lovligt formål. Artikler kan frit deles og linkes til på forsknings- og undervisningsnetværk (så som Blackboard, Moodle, Canvas o.a.). Link foretrækkes fordi det giver oplysning om brug af tidsskriftets artikler, og fordi det anerkender tidsskriftets redaktionelle arbejde. NyS tillader ikke at læsere bruger artikler eller dele af dem i egne artikler uden at citere, eller at læsere på anden vis anvender dem til kommercielle formål.
