Skip to Main content Skip to Navigation

Etude de la coordination gestes manuels/parole dans le cadre de la désignation

Abstract : The work synthesized in this thesis aims at studying the coordination between manual gestures and speech during multimodal utterances production. More precisely, the temporal relationship between the two modalities is considered. The coordination is studied in a designation framework since designating is possible both manually (pointing gesture) and using speech (one can "show with the voice" using focus and/or demonstratives for example). All the studies presented in this work are done in a lab setting thus allowing to get precise and reproducible measurements while minimizing potential external sources of variation (either between or within participants). Participants' productions were then compared to each other focusing on factors of interest while keeping other sources of variation as low as possible. A part of the work consisted in designing rather natural experimental protocols so as to ensure productions were not too artificial. The first two experiments studied to co-production of manual gestures and speech containing a focused part. Different types of gestures were compared (pointing gesture, beat, button-push) in a designation task. It has been shown that producing focus did temporally attract manual gesture whichever its type but that this attraction was finer and less variable for pointing gesture. Another interesting finding was that the apex of pointing gesture seems to be cooccurring with articulatory targets rather than acoustic ones. The second study manipulates the designation link between manual gestures and speech. By showing that participants can be split up into two groups using different multimodal coordination strategies, it put forward the complexity of underlying mechanisms of this coordination. The last experiment focuses on the coordination in a more natural interactive and collaborative task. In this task, participants used pointing gestures as a natural way to show where a card should be placed on a playing board. They also produced accompanying sentences containing demonstratives. Results show a co-ocurrence of the part of the gesture that shows and with the complementary information in speech (ie. the name of the object to be placed at the spot pointed at by the manual gesture) rather than with the part of speech that shows (ie. demonstrative). The influence of impairing interaction by broadcasting a surrounding noise is also an issue which is addressed. However speech production shows a classical Lombard effect, the production of manual gesture undergoes only slight changes: mainly, the part of the manual gesture that shows lasts longer and this lengthening is related to the lengthening observed in speech. The work presented in this manuscript moreover put forward a systematic way of labeling semi-constrained interactive tasks which can be generalized. The conclusion puts in perspective the results so as to improve some manual gestures/speech co-production models and indicates paths for reflection about embodied conversational agents and early detection of pathological cases.
Document type :
Complete list of metadatas

Cited literature [47 references]  Display  Hide  Download
Contributor : Abes Star :  Contact
Submitted on : Friday, September 20, 2013 - 10:26:19 AM
Last modification on : Wednesday, October 14, 2020 - 3:51:27 AM
Long-term archiving on: : Friday, April 7, 2017 - 12:23:12 AM


Version validated by the jury (STAR)


  • HAL Id : tel-00759199, version 2


Benjamin Roustan. Etude de la coordination gestes manuels/parole dans le cadre de la désignation. Médecine humaine et pathologie. Université de Grenoble, 2012. Français. ⟨NNT : 2012GRENS015⟩. ⟨tel-00759199v2⟩



Record views


Files downloads