Skip to Main content Skip to Navigation

The Emergence of Multimodal Concepts : From Perceptual Motion Primitives to Grounded Acoustic Words

Olivier Mangin 1
1 Flowers - Flowing Epigenetic Robots and Systems
Inria Bordeaux - Sud-Ouest, U2IS - Unité d'Informatique et d'Ingénierie des Systèmes
Abstract : This thesis focuses on learning recurring patterns in multimodal perception. For that purpose it develops cognitive systems that model the mechanisms providing such capabilities to infants; a methodology that fits into thefield of developmental robotics.More precisely, this thesis revolves around two main topics that are, on the one hand the ability of infants or robots to imitate and understand human behaviors, and on the other the acquisition of language. At the crossing of these topics, we study the question of the how a developmental cognitive agent can discover a dictionary of primitive patterns from its multimodal perceptual flow. We specify this problem and formulate its links with Quine's indetermination of translation and blind source separation, as studied in acoustics.We sequentially study four sub-problems and provide an experimental formulation of each of them. We then describe and test computational models of agents solving these problems. They are particularly based on bag-of-words techniques, matrix factorization algorithms, and inverse reinforcement learning approaches. We first go in depth into the three separate problems of learning primitive sounds, such as phonemes or words, learning primitive dance motions, and learning primitive objective that compose complex tasks. Finally we study the problem of learning multimodal primitive patterns, which corresponds to solve simultaneously several of the aforementioned problems. We also details how the last problems models acoustic words grounding.
Document type :
Complete list of metadatas

Cited literature [198 references]  Display  Hide  Download
Contributor : Abes Star :  Contact
Submitted on : Tuesday, May 5, 2015 - 4:57:06 PM
Last modification on : Saturday, October 5, 2019 - 3:40:06 AM
Long-term archiving on: : Wednesday, April 19, 2017 - 5:27:07 PM


Version validated by the jury (STAR)


  • HAL Id : tel-01148936, version 1



Olivier Mangin. The Emergence of Multimodal Concepts : From Perceptual Motion Primitives to Grounded Acoustic Words. Other [cs.OH]. Université de Bordeaux, 2014. English. ⟨NNT : 2014BORD0002⟩. ⟨tel-01148936⟩



Record views


Files downloads